Deploying NPM packages with the Nix package manager
1. Deploying NPM packages with the Nix package
manager
Sander van der Burg
Feb 4, 2017
Sander van der Burg Deploying NPM packages with the Nix package manager
2. The Nix project
Family of declarative software deployment tools:
Nix. A purely functional package manager
NixOS. Nix based GNU/Linux distribution
Hydra. Nix based continuous build and integration server
Disnix. Nix based distributed service deployment
NixOps. NixOS-based multi-cloud deployment tool
Sander van der Burg Deploying NPM packages with the Nix package manager
3. The Nix project
Non-functional properties:
Generic. Can be used with many programming languages,
component technologies, and operating systems.
Reproducible. (Almost) no impurities – if inputs are the same,
result should be the same.
Reliable. Dependency completeness, (almost) atomic
upgrades and rollbacks.
Efficient. Only the required deployment activities are
executed.
Sander van der Burg Deploying NPM packages with the Nix package manager
5. NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages and generates all configuration files, e.g.
OpenSSH configuration
Upgrades are (almost) atomic
Components are stored safely next to each other, due to hashes
No files are automatically removed or overwritten
Users can switch to older generations of system configurations
not garbage collected yet
Sander van der Burg Deploying NPM packages with the Nix package manager
7. The Nix project
Basis of all tools: The Nix package manager
Sander van der Burg Deploying NPM packages with the Nix package manager
8. Nix store
Main idea: store all packages
in isolation from each other:
/nix/store/rpdqxnilb0cg...
-firefox-3.5.4
Paths contain a 160-bit
cryptographic hash of all
inputs used to build the
package:
Sources
Libraries
Compilers
Build scripts
. . .
/nix/store
l9w6773m1msy...-openssh-4.6p1
bin
ssh
sbin
sshd
smkabrbibqv7...-openssl-0.9.8e
lib
libssl.so.0.9.8
c6jbqm2mc0a7...-zlib-1.2.3
lib
libz.so.1.2.3
im276akmsrhv...-glibc-2.5
lib
libc.so.6
Sander van der Burg Deploying NPM packages with the Nix package manager
9. Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
10. Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
openssl = import ../development/libraries/openssl {
inherit fetchurl stdenv perl;
};
stdenv = ...;
openssl = ...;
zlib = ...;
perl = ...;
nix-env -f all-packages.nix -iA openssh
Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1
package in the Nix store.
Sander van der Burg Deploying NPM packages with the Nix package manager
11. User environments
Users can have
different sets of
installed applications.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
Sander van der Burg Deploying NPM packages with the Nix package manager
12. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
13. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
14. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
15. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env -u openssh)
Sander van der Burg Deploying NPM packages with the Nix package manager
16. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
l9w6773m1msy...-openssh-4.6p1
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-env --remove-generations old)
Sander van der Burg Deploying NPM packages with the Nix package manager
17. User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox
(nix-collect-garbage)
Sander van der Burg Deploying NPM packages with the Nix package manager
18. Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Nix complements existing build tools with a pure environment
You can also run other build tools in an environment, e.g.
cmake, ant, scons
Sander van der Burg Deploying NPM packages with the Nix package manager
19. Pure environments
Precautions in service of making build results bit-identical
(regardless on what machine the package has been built):
Clearing (most) environment variables or setting them to
dummy values
Modifying search environment variables to contain Nix store
paths to only specified dependencies (e.g. PATH, PYTHONPATH,
CLASSPATH, PERL5LIB)
Using designated temp folders and output folders
Making packages immutable by making their files read-only
Resetting timestamps to 1
Chroot environments/namespaces
Restricting network access
Sander van der Burg Deploying NPM packages with the Nix package manager
20. What about Node.js/NPM projects?
NPM is Node.js’ ubiquitous deployment tool.
NPM also does dependency management in addition to build
management
NPM’s dependency management conflicts with Nix’s
dependency management
Sander van der Burg Deploying NPM packages with the Nix package manager
21. package.json configuration
{
"name" :"nijs",
"version" :"0.0.23",
"description" :"An internal DSL for the Nix package manager in JavaScript",
"repository" :{
"type" :"git",
"url" :"https://github.com/svanderburg/nijs.git"
},
"author" :"Sander van der Burg",
"license" :"MIT",
"main" :"./lib/nijs",
"dependencies" :{
"optparse" :">= 1.0.3",
"slasp": "0.0.4"
}
}
$ npm install
$ ls node modules/
optparse slasp
Sander van der Burg Deploying NPM packages with the Nix package manager
22. Dealing with a conflicting dependency manager
Solution: substitute the conflicting dependency manager by
providing the dependencies with Nix first!
Sander van der Burg Deploying NPM packages with the Nix package manager
23. Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
stdenv.mkDerivation {
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};
buildInputs = [ nodejs ];
buildCommand = ’’
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
24. Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
stdenv.mkDerivation {
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};
buildInputs = [ nodejs ];
buildCommand = ’’
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
Substituting dependencies
We may be able to automatically generate a Nix expression
from a package.json file since they capture similar proper-
ties!
25. Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
stdenv.mkDerivation {
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};;
buildInputs = [ nodejs ];
buildCommand = ’’
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}
Sander van der Burg Deploying NPM packages with the Nix package manager
Substituting dependencies
Appears to be straight forward! Is it really that straight
forward?
26. Generating Nix expressions
It is actually much more complicated!
Sander van der Burg Deploying NPM packages with the Nix package manager
27. NPM dependency classes
Dependencies:
Run-time dependencies of a package
Development dependencies:
Build-time dependencies, e.g. transpilers
Should only be installed if they have been requested
Peer dependencies:
Check whether a shared dependency does not conflict.
In old versions of NPM, missing peer dependencies were also
installed.
Bundled dependencies:
Statically bundled with a package. No need to install.
Optional dependencies:
Dependencies that are allowed to break.
Typically used to bundle platform-specific code.
Permitting errors (especially non-deterministic ones)
incompatible with Nix deployment model
Sander van der Burg Deploying NPM packages with the Nix package manager
28. NPM dependency classes
Dependencies:
Run-time dependencies of a package
Development dependencies:
Build-time dependencies, e.g. transpilers
Should only be installed if they have been requested
Peer dependencies:
Check whether a shared dependency does not conflict.
In old versions of NPM, missing peer dependencies were also
installed.
Bundled dependencies:
Statically bundled with a package. No need to install.
Optional dependencies:
Dependencies that are allowed to break.
Typically used to bundle platform-specific code.
Permitting errors (especially non-deterministic ones)
incompatible with Nix deployment model
Sander van der Burg Deploying NPM packages with the Nix package manager
Interpreting dependency classes
We only need to consider dependencies and development de-
pendencies (if they are requested) and substract the bundled
dependencies
29. Version specifiers: NPM registry
NPM version specifiers are nominal and semantically versioned:
Precise version numbers: e.g. 1.0.0, 2.1.1
Version ranges and wildcards: e.g. 1.0.x, *, >= 1.0.1
Tags: latest, beta
The above version specifiers refer to packages obtained from the
NPM registry.
Sander van der Burg Deploying NPM packages with the Nix package manager
30. Version specifiers: external packages
Local filesystem paths, e.g. /home/sander/nijs
External URLs, e.g.
http://example.com/packages/nijs-0.0.23.tgz
Git URLs, e.g.
https://github.com/svanderburg/nijs.git
GitHub, BitBucket, GitLab, e.g. github:svanderburg/nijs
Above specifiers obtain packages from other sources than the
registry. They indirectly resolve to a package with a name and
version number.
Sander van der Burg Deploying NPM packages with the Nix package manager
31. Version specifiers: translation to Nix
Nix version specifiers are exact (any point of variation is reflected
in the hash prefix):
/nix/store/rpdqxnilb0cg...-firefox-3.5.4
Sander van der Burg Deploying NPM packages with the Nix package manager
32. Version specifiers: translation to Nix
Nix version specifiers are exact (any point of variation is reflected
in the hash prefix):
/nix/store/rpdqxnilb0cg...-firefox-3.5.4
To make a translation from NPM version specifiers to exact
version specifiers:
We must snapshot a version range and pinpoint the resolved
version (version ranges are unsupported in Nix)
We must supply the output hash to make it deterministic:
src = fetchurl {
url = "https://registry.npmjs.org/nijs/-/nijs-0.0.23.tgz";
sha1 = "dbf8f4a0acafbe3b8d9b71c24cbd1d851de6c31a";
};
Sander van der Burg Deploying NPM packages with the Nix package manager
33. Private, shared and cyclic dependencies
When including a package as a CommonJS module:
./node2nix/node modules/nijs/lib/execute/index.js
var slasp = require(’slasp’);
The module loader searches recursively upwards in node modules/
sub folders for the corresponding package:
./node2nix/nijs/node_modules/slasp
./node2nix/node_modules/slasp
./node_modules/slasp
Sander van der Burg Deploying NPM packages with the Nix package manager
34. Private, shared and cyclic dependencies
Private dependency: a package residing in a package’s
node modules/ sub folder.
Shared dependency: a package residing in a parent directory’s
node modules/ sub folder.
Sander van der Burg Deploying NPM packages with the Nix package manager
35. Private, shared and cyclic dependencies
When installing dependencies with NPM, NPM will not install a
dependency privately if a conforming shared dependency exists.
Some undesired implications:
A version range does not always yield the latest version that
fits in a version range.
Cyclic dependencies are permitted. Bad practice, as packages
are supposed to be units of reuse.
Sander van der Burg Deploying NPM packages with the Nix package manager
36. Flat-module installations
npm < 3.x installs dependencies privately by default (unless a
conforming dependency exists in any parent node modules/
folder):
webapp/
...
package.json
node_modules/
express/
...
package.json
node_modules/
accepts/
array-flatten/
content-disposition/
...
ejs/
...
package.json
Sander van der Burg Deploying NPM packages with the Nix package manager
37. Flat-module installations
npm ≥ 3.x implements flat-module installations (moving packages
as high as possible in the node modules/ hierarchy):
webapp/
...
package.json
node_modules/
accepts/
array-flatten/
content-disposition/
express/
...
package.json
ejs/
...
package.json
Because no packages conflict, they all appear in the top level
node modules/ folder. Implications:
Shorter path names, some degree of deduplication
The order in which packages are installed matters
Sander van der Burg Deploying NPM packages with the Nix package manager
38. Simulating flat-module installations
We cannot build each NPM dependency as a Nix package, with a
separate store path:
CommonJS module resolves its own symlink location. Some
relative paths may not work:
var slasp = require(’../slasp’);
Flattening the module hierarchy is imperative.
Nix packages are made immutable after they have been built.
Solution: compose the entire dependency tree statically ahead of
time in one Nix package
Sander van der Burg Deploying NPM packages with the Nix package manager
39. Replacing impure version specifiers
Some version specifiers (e.g. tags and Git URLs), trigger NPM to
always check the remote locations for changes in each npm
install run:
Replace these version specifiers by: ’*’. Downside: it
sometimes confuses NPM, in particular flat module
installations.
Better solution: create fake processes (e.g git) giving a
deterministic result. This is still an open issue.
Sander van der Burg Deploying NPM packages with the Nix package manager
40. The solution1
: node2nix
1
Sort of, but not entirely :-)
Sander van der Burg Deploying NPM packages with the Nix package manager
41. Example: using node2nix on project-level
Generating Nix expressions from a package.json file:
$ node2nix
Building the project as a Nix package:
$ nix-build -A package
./result/bin/node2nix --help
Opening a development shell:
$ nix-shell -A shell
$ node bin/node2nix.js --help
Sander van der Burg Deploying NPM packages with the Nix package manager
42. Example: using node2nix on a package set
custom-packages.json
[
"node2nix"
,"bower"
,{"nijs": "https://github.com/svanderburg/nijs.git" }
,{"grunt-cli": "1.2.0" }
]
Generating Nix expressions:
$ node2nix -i custom-packages.json
Installing an NPM package with Nix:
$ nix-env -f default.nix -iA node2nix
$ node2nix --help
Sander van der Burg Deploying NPM packages with the Nix package manager
43. Example: simulating global dependencies
Global packages do not exist in Nix build environments. This is
problematic for certain kinds of projects, such as Grunt projects,
requiring the grunt-cli to be installed globally:
package.json
{
"name": "grunt-test",
"version": "0.0.1",
"private": "true",
"devDependencies": {
"grunt": "*",
"grunt-contrib-jshint": "*",
"grunt-contrib-watch": "*"
}
}
$ npm install
$ grunt build
Sander van der Burg Deploying NPM packages with the Nix package manager
44. Example: simulating global dependencies
We can supply global packages as extra packages:
supplement.json
[
"grunt-cli"
]
And generate the expressions as follows:
$ node2nix -d -i package.json
--supplement-input supplement.json
Sander van der Burg Deploying NPM packages with the Nix package manager
45. Example: simulating global dependencies
override.nix
{ pkgs ? import <nixpkgs> {}
, system ? builtins.currentSystem
}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
package = nodePackages.package.override {
postInstall = "grunt";
};
}
$ nix-build override.nix
Sander van der Burg Deploying NPM packages with the Nix package manager
46. Conclusion
I have explained Nix and NPM’s deployment concepts
I have described node2nix generating Nix expressions from
package.json configurations, making it possible to deploy
NPM packages with the Nix package manager.
Besides the package manager, it also becomes possible to
deploy entire systems using Node.js with NixOps and Disnix.
Sander van der Burg Deploying NPM packages with the Nix package manager
47. Lessons
Naming packages: a name and version number typically does
not suffice!
Nix uses hash codes derived from all build inputs
.NET global assembly cache: strong names
Organization: isolate dependencies
Cyclic dependencies: disallow them – packages are supposed
to be units of reuse
Sander van der Burg Deploying NPM packages with the Nix package manager
48. Related work: other generators
npm2nix: Original generation attempt done by Shea Levy.
Composes Nix store paths for each dependency and symlinks
them.
Faster, but less accurate when dealing when shared
dependencies
Does not support flat module installations or cyclic
dependencies
nixfromnpm: Implementation of npm2nix’s Nix concepts in
Haskell. Can partially regenerate expressions.
Sander van der Burg Deploying NPM packages with the Nix package manager
49. Related work: NPM alternatives
yarn: parallelization for speed improvements, version locking
ied: parallelization, content-addressable store, atomicity,
maximal sharing of dependencies (some of its concepts are
heavily inspired by Nix according to its author :-) )
Sander van der Burg Deploying NPM packages with the Nix package manager
50. References
Nix project: http://nixos.org.
Nix package manager: http://nixos.org/nix.
node2nix: https://github.com/svanderburg/node2nix.
Sander van der Burg Deploying NPM packages with the Nix package manager