%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
The NixOS project and deploying systems declaratively
1. The NixOS project and deploying systems
declaratively
Sander van der Burg
March 12, 2016
Sander van der Burg The NixOS project and deploying systems declaratively
2. On being declarative
A declarative sentence makes a statement. It is punctuated by a
period:
The dog in the neighbor’s yard is barking.
(Source: http://www.slideshare.net/luigi a97/parts-of-a-sentence-8862361)
Sander van der Burg The NixOS project and deploying systems declaratively
3. On being imperative
An imperative sentence is a command or polite request:
(Source: https://xkcd.com/149/)
Sander van der Burg The NixOS project and deploying systems declaratively
4. On being declarative in programming
A style of building the structure and elements of computer
programs – that expresses the logic of a computation without
describing its control flow
(Source: https://en.wikipedia.org/wiki/Declarative programming)
Sander van der Burg The NixOS project and deploying systems declaratively
5. On being declarative in programming
Declarative: describing ”what” is to be computed rather than
”how” to compute the result/behavior
Imperative: a description of a computation that involves
implicit effects, usually mutable state and input/output.
(Source:
http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html)
Sander van der Burg The NixOS project and deploying systems declaratively
6. On being declarative in programming
Declarative: describing ”what” is to be computed rather than
”how” to compute the result/behavior
Imperative: a description of a computation that involves
implicit effects, usually mutable state and input/output.
(Source:
http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html)
Sander van der Burg The NixOS project and deploying systems declaratively
Declarative
“declarative” is a spectrum – hard to draw a hard line
between “what” and “how”.
Imperative is not necessarily the opposite of
declarative.
7. Example: HTML and CSS
<!DOCTYPE html>
<html>
<head>
<title>Test</title>
<link rel=”stylesheet” href=”style.css” type=”text/css”>
</head>
<body>
<div id=”outer”>
<div id=”inner”>
<p>HTML and CSS are declarative and so cool!</p>
</div>
</div>
</body>
</html>
#outer {
margin−left: auto;
margin−right: auto;
width: 20%;
border−style: solid;
}
#inner {
width: 500px;
}
Sander van der Burg The NixOS project and deploying systems declaratively
8. Example: HTML and CSS
Sander van der Burg The NixOS project and deploying systems declaratively
9. Deployment: What do we want?
Sander van der Burg The NixOS project and deploying systems declaratively
11. Deployment complexity
Diverse technology imposes many kinds of deployment procedures:
Different operating systems, different dependencies, many
variants
Sander van der Burg The NixOS project and deploying systems declaratively
12. Deployment complexity
Deployment may need to be done on a large scale:
Sander van der Burg The NixOS project and deploying systems declaratively
13. Deployment complexity
How to update the deployment frequently?
How not to break the system while upgrading?
How to minimize downtimes?
How to roll back in case of a failure?
Sander van der Burg The NixOS project and deploying systems declaratively
14. Deployment automation
To deal with deployment complexities automation is needed!
Sander van der Burg The NixOS project and deploying systems declaratively
15. Deployment automation
To deal with deployment complexities automation is needed!
Many automated deployment solutions available
Automation is typically driven by a specification
Some solutions have been developed for specific kinds of
technology:
Apache Felix (for OSGi components)
Some solutions are general:
Chef
Puppet
CFEngine
Nix
Some solutions use declarative deployment specifications
Sander van der Burg The NixOS project and deploying systems declaratively
16. On being declarative in deployment
Declare what system you want to run in the consumer environment,
not the activities that need to be executed to accomplish it!
Sander van der Burg The NixOS project and deploying systems declaratively
17. Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
remote_file wordpress_latest do
source "http://wordpress.org/latest.tar.gz"
mode "0644"
end
directory node["phpapp"]["path"] do
owner "root"
group "root"
mode "0755"
action :create
recursive true
end
execute "untar-wordpress" do
cwd node[’phpapp’][’path’]
command "tar --strip-components 1 -xzf " + wordpress_latest
creates node[’phpapp’][’path’] + "/wp-settings.php"
end
(Source: http://gettingstartedwithchef.com/first-steps-with-chef.html)
Sander van der Burg The NixOS project and deploying systems declaratively
18. Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
remote_file wordpress_latest do
source "http://wordpress.org/latest.tar.gz"
mode "0644"
end
directory node["phpapp"]["path"] do
owner "root"
group "root"
mode "0755"
action :create
recursive true
end
execute "untar-wordpress" do
cwd node[’phpapp’][’path’]
command "tar --strip-components 1 -xzf " + wordpress_latest
creates node[’phpapp’][’path’] + "/wp-settings.php"
end
(Source: http://gettingstartedwithchef.com/first-steps-with-chef.html)
Sander van der Burg The NixOS project and deploying systems declaratively
Declarative
The specification captures the outcome of a set of
changes as a fixpoint. Chef converges to the outcome.
Specification applies to set of machines – but does not
guarantee that an entire machine’s configuration can
be reproduced elsewhere
How to roll back to a previous configuration?
How to mimimize downtime?
19. NixOS
NixOS: A GNU/Linux distribution using the Nix package manager
Sander van der Burg The NixOS project and deploying systems declaratively
21. NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages and generates all configuration files, e.g.
OpenSSH configuration
Upgrades are (almost) atomic
Components are stored safely next to each other, due to hashes
No files are automatically removed or overwritten
Users can switch to older generations of system configurations
not garbage collected yet
Sander van der Burg The NixOS project and deploying systems declaratively
23. Nix store
Main idea: store all packages
in isolation from each other:
/nix/store/rpdqxnilb0cg...
-firefox-3.5.4
Paths contain a 160-bit
cryptographic hash of all
inputs used to build the
package:
Sources
Libraries
Compilers
Build scripts
. . .
/nix/store
l9w6773m1msy...-openssh-4.6p1
bin
ssh
sbin
sshd
smkabrbibqv7...-openssl-0.9.8e
lib
libssl.so.0.9.8
c6jbqm2mc0a7...-zlib-1.2.3
lib
libz.so.1.2.3
im276akmsrhv...-glibc-2.5
lib
libc.so.6
Sander van der Burg The NixOS project and deploying systems declaratively
24. Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Sander van der Burg The NixOS project and deploying systems declaratively
25. Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
openssl = import ../development/libraries/openssl {
inherit fetchurl stdenv perl;
};
stdenv = ...;
openssl = ...;
zlib = ...;
perl = ...;
nix-env -f all-packages.nix -iA openssh
Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1
package in the Nix store.
Sander van der Burg The NixOS project and deploying systems declaratively
26. User environments
Users can have
different sets of
installed applications.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
Sander van der Burg The NixOS project and deploying systems declaratively
27. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
28. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
29. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
30. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
31. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env --remove-generations old)
Sander van der Burg The NixOS project and deploying systems declaratively
32. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-collect-garbage)
Sander van der Burg The NixOS project and deploying systems declaratively
33. NixOS
In NixOS, all packages including the Linux kernel and
configuration files are managed by Nix.
NixOS does not have directories such as: /lib and /usr
NixOS has a minimal /bin and /etc
Sander van der Burg The NixOS project and deploying systems declaratively
34. Distributed deployment
NixOS has good properties for deployment of a single system
Can we extend these properties to distributed systems?
Sander van der Burg The NixOS project and deploying systems declaratively
36. Motivating example: Trac
Trac can be deployed in a distributed environment:
Subversion server
Database server
Web server
Sander van der Burg The NixOS project and deploying systems declaratively
38. Distributed deployment
$ nixops create network.nix -d production
$ nixops deploy -d production
Build system configurations by the Nix package manager
Transfer complete system and all dependencies to target
machines in the network
Efficient: only missing store paths must be transferred
Safe: Existing configuration is not affected, because no files
are overwritten or removed
Activate new system configuration
In case of a failure, roll back all configurations
Relatively cheap operation, because old configuration is stored
next to new configuration
Sander van der Burg The NixOS project and deploying systems declaratively
39. The Nix project
Tools part of the Nix-project: http://nixos.org:
Nix. A purely functional package manager
NixOS. Nix based GNU/Linux distribution
Hydra. Nix based continuous build and integration server
Disnix. Nix based distributed service deployment
NixOps. NixOS-based multi-cloud deployment tool
Sander van der Burg The NixOS project and deploying systems declaratively
40. The Nix project
Automated deployment using declarative specifications with the
following properties:
Generic. Can be used with many programming languages,
component technologies, and operating systems.
Reproducible. (Almost) no impurities – if inputs are the same,
result should be the same regardless of its location
Reliable. Dependency completeness, (almost) atomic
upgrades and rollbacks.
Efficient. Only the required deployment activities are
executed.
Sander van der Burg The NixOS project and deploying systems declaratively
41. Nix-related tools: how declarative are they?
Nix-related tools solve problems in a technical domain:
e.g. deployment of packages, machines, services, ...
What about your domain?
Sander van der Burg The NixOS project and deploying systems declaratively
42. A real world example: Conference Compass
Conference Compass provides a service to improve the way
people experience events
Most visible part of the service: apps for conference attendees
Each customer basically gets “their own” app.
Sander van der Burg The NixOS project and deploying systems declaratively
43. A real world example: Conference Compass
We have a product-line using a Nix-based build infrastructure,
including Hydra, driven by simple app specific configurations:
{
name = "wroclove.rb 2016";
homepage = "http://www.wrocloverb.com";
iconSet = ./icons;
backgroundImage" = ./background.png;
...
}
Sander van der Burg The NixOS project and deploying systems declaratively
44. A real world example: Conference Compass
The app’s contents is customizable with a configurator service
allowing organizers to create and update their content
Apps connect to a configurator to retrieve the data to be
displayed and other configuration settings
Integration with third party information systems is also
possible
Sander van der Burg The NixOS project and deploying systems declaratively
45. A real world example: Conference Compass
{
wrocloverb = {
eventName = "wroclove.rb 2016";
domain = "http://www.wrocloverb.com";
channels = [ "wrocloverb" ];
};
otherevent = ...;
yetanotherevent = ...;
...
}
We have developed a formalism to concisely model such
configurations and to automatically deploy them
Tool figures out which machines to configure, what services to
deploy etc.
If underlying implementation and technology evolves,
specifications (probably) remains the same.
Sander van der Burg The NixOS project and deploying systems declaratively
46. Conclusions
I have illustated a declarative deployment vision
I have demonstrated NixOS and the Nix package manager
I have explained that domain specific deployment tools can be
built on top of tools from the Nix project
Sander van der Burg The NixOS project and deploying systems declaratively
47. References
NixOS project homepage: http://nixos.org
Software available under free and open-source licenses
(LGPL/X11)
Nix package manager can be used on any Linux system, Mac
OS X, and (in some extent) Cygwin and FreeBSD.
Sander van der Burg The NixOS project and deploying systems declaratively