Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

nix-processmgmt: An experimental Nix-based process manager-agnostic framework

337 views

Published on

NixCon 2020 talk about an experimental framework that integrates the Nix package manager with all kinds of process managers, such as : sysvinit, systemd, launchd, and even Docker

Published in: Software
  • Be the first to comment

  • Be the first to like this

nix-processmgmt: An experimental Nix-based process manager-agnostic framework

  1. 1. nix-processmgmt: An experimental Nix-based process manager-agnostic framework Sander van der Burg October 17, 2020 Sander van der Burg nix-processmgmt
  2. 2. Nix package manager: a powerful solution Conveniently construct packages from source code and all its required build-time dependencies Build determinism. Same hash code → (nearly) bit-identical build Transparent binary deployments (by using substitutes) Store multiple versions and variants safely next to each other Thanks to the hash prefixes and the Nix store Unprivileged user deployments Multiple operating systems: Linux, macOS, FreeBSD, others Sander van der Burg nix-processmgmt
  3. 3. Nix: development environments We can conveniently install and use all kinds of packages without interfering with the host system’s packages. nix-shell example $ python --version Python 2.7.15 $ node --version node: command not found $ nix-shell -p python3 nodejs $ python --version Python 3.8.5 $ which python /nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/bin/python $ node --version v12.18.4 $ which node /nix/store/2w6ilfh7zmbz9zqvphgxinmbn3wdqa1b-nodejs-12.18.4/bin/node Sander van der Burg nix-processmgmt
  4. 4. Nix package manager: deploying services? Sander van der Burg nix-processmgmt
  5. 5. Nix package manager: not a service manager Nix does not manage application services/processes’ life-cycles. Sander van der Burg nix-processmgmt
  6. 6. Nix: service deployment integrations There are sister projects that complement Nix with process manage- ment: NixOS. Generates systemd unit files to manage services. Requires you to switch to a fully Nix-managed Linux distribution. nix-darwin. Generates launchd daemon configuration files. Only works on macOS. Sander van der Burg nix-processmgmt
  7. 7. Nix: service deployment integrations Nix can also be used to augment other process management solu- tions: Docker. Docker uses a package manager in Dockerfiles for the construction of images. Nix can be used as a replacement for conventional package managers. Nix can be used to fully build Docker images. Not always not a compelling use case for Nix beginners → they typically download prebuilt images from Docker Hub. Docker is built around Linux technologies (e.g. namespaces) and deploys Linux software Sander van der Burg nix-processmgmt
  8. 8. nix-processmgmt: A general solution complementing Nix with process management Sander van der Burg nix-processmgmt
  9. 9. nix-processmgmt: A general solution complementing Nix with process management Driven by Nix and the Nix expression language Based on simple conventions: function definitions and an attribute set with function invocations Similar to how packages are organized in Nixpkgs Works with process dependencies as well: the framework arranges the ordering, if needed Process-manager agnostic: Integrates with sysvinit scripts, supervisord, systemd, launchd, bsdrc scripts, cygrunsrv Even with systems that are not qualified as process managers: disnix, docker Sander van der Burg nix-processmgmt
  10. 10. nix-processmgmt: A general solution complementing Nix with process management Operating system agnostic: Works on NixOS, but it is not a requirement Conventional Linux distros, macOS, FreeBSD, Cygwin Unprivileged user installations A switch to disable creation of users, and changing user permissions No advanced concepts required, such as namespaces and cgroups The solution relies on conflict avoidance, rather than isolation Sander van der Burg nix-processmgmt
  11. 11. Example: a simple web application system Sander van der Burg nix-processmgmt
  12. 12. Packaging the webapp process (sysvinit, verbose) {createSystemVInitScript, webapp, port ? 5000}: createSystemVInitScript { name = "webapp"; description = "Example web application"; environment.PORT = port; activities = { start = ’’ log_info_msg "Starting Example web application..." loadproc ${webapp}/bin/webapp -D evaluate_retval ’’; stop = ’’ log_info_msg "Stopping Example web application..." killproc ${webapp}/bin/webapp evaluate_retval ’’; restart = "$0 stop; sleep 1; $0 start"; status = "statusproc ${webapp}/bin/webapp"; }; runlevels = [ 3 4 5 ]; } Sander van der Burg nix-processmgmt
  13. 13. Packaging the webapp process (sysvinit, declarative) {createSystemVInitScript, webapp, port ? 5000}: createSystemVInitScript { name = "webapp"; process = "${webapp}/bin/webapp"; args = [ "-D" ]; runlevels = [ 3 4 5 ]; environment.PORT = port; } We can also specify the daemon that we want to manage, instead of the activity implementations. Most sysvinit activities (start, stop, restart, reload, status) can be inferred. Sander van der Burg nix-processmgmt
  14. 14. Packaging the webapp process (systemd) The following function composes a systemd unit instead of a sysvinit script: {createSystemdService, webapp, port ? 5000}: createSystemdService { name = "webapp"; Unit = { Description = "Example web application"; }; Service = { ExecStart = "${webapp}/bin/webapp"; Environment.PORT = port; Type = "simple"; }; } The framework contains many other process manager- specific abstraction functions: createSupervisordProgram, createLaunchdDaemon, createBSDRCScript etc. Sander van der Burg nix-processmgmt
  15. 15. Packaging the webapp process (agnostic) Process manager-agnostic abstraction of the webapp service: {createManagedProcess, webapp, port ? 5000}: createManagedProcess { name = "webapp"; description = "Example web application"; process = "${webapp}/bin/webapp"; daemonArgs = [ "-D" ]; # For process managers that prefer daemons environment.PORT = port; overrides = { sysvinit.runlevels = [ 3 4 5 ]; }; } Invokes the required target-specific abstraction function, e.g. createSystemVInitScript, createSystemdService overrides override/augment process manager-specific parameters You can treat foreground processes and daemons separately, for optimal user experience Sander van der Burg nix-processmgmt
  16. 16. Instantiatable webapp processes {createManagedProcess, webapp}: {instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}", port ? 5000}: createManagedProcess { name = instanceName; inherit instanceName; # To ensure a unique PID file name description = "Example web application"; process = "${webapp}/bin/webapp"; daemonArgs = [ "-D" ]; # For process managers that prefer daemons environment.PORT = port; overrides = { sysvinit.runlevels = [ 3 4 5 ]; }; } instanceName: ensures that the daemon command generates unique PID file Outer-function header: parameters that apply to all instances. Inner-function header: instance parameters. A unique combination ensures that multiple instances can co-exist. Sander van der Burg nix-processmgmt
  17. 17. Composing process instances { pkgs ? import <nixpkgs> {} , stateDir ? "/var" , forceDisableUserChange ? false, processManager}: let constructors = import ./constructors.nix { inherit pkgs stateDir forceDisableUserChange processManager; }; in rec { webapp1 = rec { # First webapp instance port = 5000; dnsName = "webapp1.local"; pkg = constructors.webapp { inherit port; instanceSuffix = "1"; }; }; webapp2 = rec { # Second webapp instance port = 5001; dnsName = "webapp2.local"; pkg = constructors.webapp { inherit port; instanceSuffix = "2"; }; }; ... } Sander van der Burg nix-processmgmt
  18. 18. Composing process instances { pkgs ? import <nixpkgs> {} , stateDir ? "/var" , forceDisableUserChange ? false, processManager}: let constructors = import ./constructors.nix { inherit pkgs stateDir forceDisableUserChange processManager; }; in rec { ... # Nginx with a config that redirects users to the appropriate webapp # instance based on the virtual host header nginx = { pkg = constructors.nginxReverseProxyHostBased { # Process dependencies used to set up redirections in nginx.conf webapps = [ webapp1 webapp2 ]; port = 8080; } {}; }; } Sander van der Burg nix-processmgmt
  19. 19. Demo: deploying the system and using it Deploy the process instances as sysvinit scripts: $ nixproc-sysvinit-switch processes.nix Open the first webapp instance (via the Nginx reverse proxy): $ curl -H ’Host: webapp1.local’ http://localhost:8080 Open the second webapp instance (via the Nginx reverse proxy): $ curl -H ’Host: webapp2.local’ http://localhost:8080 Sander van der Burg nix-processmgmt
  20. 20. Demo: all kinds of process manager integrations Deploy as systemd units (in a user session): $ nixproc-systemd-switch --user processes.nix Deploy as supervisord programs (stateless): $ nixproc-supervisord-deploy-stateless processes.nix Deploy on FreeBSD as BSD rc scripts: $ nixproc-bsdrc-switch processes.nix Deploy as Docker containers per service (with shared Nix store and host networking): $ nixproc-docker-switch processes.nix Sander van der Burg nix-processmgmt
  21. 21. Demo: screenshots Sander van der Burg nix-processmgmt
  22. 22. Other features of nix-processmgmt Automatic creation of users and groups (createCredentials function) nixproc-id-assign: Automated assignment of unique IDs for TCP/UDP ports, UIDs, GIDs to process instances Using Disnix as a process orchestrator (works on all platforms where Nix/Disnix is supported) Sander van der Burg nix-processmgmt
  23. 23. Combining nix-processmgmt with Disnix We can use any process manager, do distributed deployment and combine processes with non-process based services (e.g. Java web applications in an Apache Tomcat container): Sander van der Burg nix-processmgmt
  24. 24. Example services packaged for nix-processmgmt HTTP/application servers: Apache HTTP server Nginx Apache Tomcat Database: PostgreSQL MariaDB/MySQL MongoDB InfluxDB Misc: Docker Supervisord svnserve Sander van der Burg nix-processmgmt
  25. 25. Future work Deploy containers with multiple processes (WIP) Mutable service containers (WIP) s6 / s6-rc backend Work on a test strategy for services Maybe write an RFC? Sander van der Burg nix-processmgmt
  26. 26. Availability https://github.com/svanderburg/nix-processmgmt The implementation is still a work in progress and relies on devel- opment versions of dependencies! Sander van der Burg nix-processmgmt
  27. 27. References Blog posts: A Nix-based functional organization for managing processes, https://sandervanderburg.blogspot.com/2019/11/ a-nix-based-functional-organization-for.html A declarative process manager-agnostic deployment framework based on Nix tooling, https://sandervanderburg.blogspot.com/2020/02/ a-declarative-process-manager-agnostic.html Deploying container and application services with Disnix, https://sandervanderburg.blogspot.com/2020/04/ deploying-container-and-application.html Sander van der Burg nix-processmgmt
  28. 28. References Blog posts: Using Disnix as a simple and minimalistic dependency-based process manager, https://sandervanderburg.blogspot.com/2020/06/ using-disnix-as-simple-and-minimalistic.html Experimenting with Nix and the service management properties of Docker, https://sandervanderburg.blogspot.com/2020/08/ experimenting-with-nix-and-service.html Assigning unique IDs to services in Disnix deployment models, https://sandervanderburg.blogspot.com/2020/09/ assigning-unique-ids-to-services-in.html Sander van der Burg nix-processmgmt
  29. 29. References E. Dolstra, The Purely Functional Software Deployment Model, PhD thesis, Chapter 9, https://edolstra.github.io/pubs/phd-thesis.pdf Sander van der Burg nix-processmgmt
  30. 30. Questions Sander van der Burg nix-processmgmt

×