Functional programming finds its roots in mathematics - the pursuit of purity and completeness. We functional programmers look to formalize system behaviors in an algebraic and total manner. Despite this, when it comes time to deploy ones beautiful monadic ivory towers to production, most organizations cast caution to the wind and use a myriad of bash scripts and sticky tape to get the job done. In this talk, the speaker will introduce you to Nelson, an open-source project from Verizon that looks to provide rigor to your large distributed system, whilst offering best-in-class security, runtime traffic shifting and a fully immutable approach to application lifecycle. Nelson itself is entirely composed of free algebras and coproducts, and the speaker will show not only how this has enabled development, but also how it provided a frame with which to reason about solutions to fundamental operational problems.

1. Nelson
automated, multi-region container deployment
timperrett

2. Hello.
!
verizon.github.io/nelson/

3. Rig·or·ous.
Extremely thorough, exhaustive, or accurate.

4. Rigorous Systems.

5. – Dr Tara Smith
“Pragmatism is the opposition to principal, on principle.”

6. Rigorous Systems.
• Theory is the foundation.
• Category Theory, Combinatorics, Set Theory, Group Theory…
• Principles of formal verification becoming more widespread.
• Type systems act as proofs about your program.
• Amazon S3 (among others) have TLA+ models.
• Never fear practical application of rigorous systems.
• Invest now to save your sanity tomorrow (i.e. 2am PagerDuty alert).

7. Rigorous Systems.
Time
Pain
Principled Pragmatic

8. Rigorous Systems.
Time
Pain
Principled Pragmatic
"

9. Rigorous Systems.
Time
Pain
Principled Pragmatic
#

10. Rigorous Systems.
Time
Pain
Principled Pragmatic
$

11. Deployment.
sticky tape, bandages and bash scripts.
antithesis
of rigor

12. Problem
• Provisioning applications is still too slow (bare metal or cloud).
• Runtime traffic control systems are medieval at best.
• Coupling CI and CD creates monolithic operational systems.
• These systems do everything. This is a distinct problem.
• Current market solutions limited or hard to adopt.
• Most teams have brittle, painful automation nobody wants to use.
• Many teams attempt CD ignorant of the side-effects.

13. Lessons
• Automate every part of the system.
• Testing a distributed system locally is a fable.
• Emergent properties. Scaling issues etc.
• Uniformity is highly desirable and wildly advantageous.
• Beautiful, unique snowflakes are however, inevitable.
• Automated lifecycle management is required.

14. Goals
• Use the minimally powerful components.
• System elements should be awesome at just one thing.
• Reduce overall platform complexity.
• Increase responsibility of engineering teams. Break it, you bought it.
• Decentralize process gatekeepers.
• No build team. No ticket filing for deployments. No configuration
management.

15. Goals
• All application specifications are checked in.
• Build. Deployment. Alerting etc.
• Reduce deployment time to 2 minutes or less.
• Support multi-DC topologies from the get-go.
• Automatic credential management and secure-introduction
• Transparent, strong encryption for application I/O on the wire.

16. Nelson.

17. – Vice Admiral Horatio Nelson, 1758-1805
“Desperate affairs require desperate remedies.”

18. – Vice Admiral Horatio Nelson, 1758-1805
“Desperate affairs require desperate remedies.”
#opslife

19. Overview
• Provides “best practices” for your organization; focus on shipping.
• Github driven developer workflow (.com or enterprise).
• Choose whatever build / CI system you want.
• State of the art runtime routing via Envoy.
• Secure introduction for safe distribution of credentials from Vault.
• Target any datacenter running a scheduler (Nomad, Mesos, etc).

21. Overview.
pie of
responsibility!

22. Overview.
users

23. Overview.
operators

24. Features.
• Declarative, simplistic YAML manifest per-repository.
• Active lifecycle management.
• Management of edge ingress (e.g. public load balancers).
• Optional routing control plane.
• Comprehensive auditing.
• Extensible with arbitrary scheduling, routing or artifact systems.

25. Foundations.
graphs and coproducts.

26. Mathematics.
Combinatorics.
Graph Theory.
https://github.com/Verizon/quiver

27. Vertex.

28. Vertex.
vertex[N,L]

29. A
Edge.
B

30. A
Directed Edge.
B

31. Directed Edge.
LEdge[N,A]
Edge[N]
A B

32. Graph.

33. Monoids.
A B
CD
A
F
G
H
(A, A) => A
A => A

34. Monoids.
B
C
D
A
F
G
H

35. A
D
C
B
E
depends ondepends on
depends on
depends on

36. depends ondepends on
depends on
depends on

37. depends ondepends on
depends on
depends on
foo@1.2 bar@4.5
qux@2.1
buzz@1.1
klang@1.0

38. Graphs.
• Every application declares their dependencies a-priori.
• No routing to disconnected vertices.
• No Vault policy attributes for disconnected vertices.
• Edge ingress points are the “root” of a graph.
• Intersecting graphs with shared or discrete roots.
• Provides anchor-point for lifecycle management.
• Entire world view falls out of a union of all discrete graphs.

39. Lifecycle.
deployment is the easy part.

40. Lifecycle.
Amazon
in 2008
https://twitter.com/Werner/status/741673514567143424

42. based on
consul

43. typical state

44. pending
GC

45. user
activated

46. pluggable

47. borrowed
time

48. garbage
collection

49. Graph
Pruning

50. X
X
Upgraded!

51. last two
major revsXX X

52. last two
featuresXX X

53. Security.
that thing you wish you did better.
graphs help
here too!

54. Security.
• Derivation of security policy, as opposed to proclamation of policy.
• Excellent audibility
• All dependencies are known - internal and external.
• Every single application instance has unique credentials.
• Dynamic certificates, dynamic DB access etc.
• Compromised containers have the minimal attack surface.
• Joined with a service mesh, transparent mTLS for everyone is a breeze.

55. Namespaces.

56. machines

57. scheduler

58. namespaces

59. namespaces
entirely
virtual!

61. root
namespace

62. qa/unstable

63. qa/staging/tim

65. Discovery & Routing.

66. Discovery.
• Discovery protocol written to Consul KV for every stack
• We call this Lighthouse protocol
• Application dependencies are declared a-priori.
• You cannot route to that which you do not tell Nelson about.
• Makes for awesome auditing and security.
• Language implementations need only consume the protocol.

67. Routing.
• Non-prescriptive approach to routing tier implementation.
• Provides a control plane protocol describe routing actions.
• Typically implemented with Envoy, but you can choose.
• Minor application changes required.
• Incentivized these with tracing and context propagation.
• Models traffic shifting as a time vs traffic policy curve.

69. Workflow.
how the sausage is made.

70. Workflow.
HTTPQueue
Workflow X
Workflow Y
End
End
Executor Pool
GitHub

71. Workflow.
and thenand then
start
replicate
containers
create
vault policies
???
and then

72. Workflow.
Looks like a graph.
Walks like a graph.
Must be a… coproduct!

73. Workflow.
Coproduct[F[_], G[_], A](run: F[A] / G[A])

74. Workflow.
type Op0[A] = Coproduct[DockerOp, ConsulOp, A]
…
type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A]
type WorkflowF[A] = Free.FreeC[WorkflowOp, A]

75. Workflow.
for {
_ <- status(id, Pending, "workflow about to start")
i <- dockerOps(id, unit, dc.docker.registry)
_ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul")
_ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
///////////////////////////////
// remove for space limitations
///////////////////////////////
l <- launch(i, dc, ns.name, vunit, p, hash)
_ <- debug(s"response from scheduler $l”)
_ <- status(id, getStatus(unit, p), "======> workflow completed <======")
} yield ()
Free monad

76. Challenges
• Non-trivial level of investment and execution.
• Tight integration with Hashistack is both pro or con.
• Community adding k8s + Istio support %
• Containerizing legacy applications can be “interesting”.
• Migration can be a challenge if not collocated with “the new world”.
• Small organizations better served by existing solutions.

77. Summary
• Provide rigor to your application Death Star.
• Fully automated application lifecycle: no manual housekeeping.
• Choose whatever CI setup best fits your team.
• Secure your deployments.
• Transparent mTLS and rotating credentials.
• Automatic Vault policy management.

78. EOF
timperrett
verizon.github.io/nelson/

79. Related Links.
• Nelson documentation site:
verizon.github.io/nelson/
• Nelson CLI:
https://github.com/Verizon/nelson-cli
• Quiver: Scala port of Haskell FGL
https://github.com/Verizon/quiver
• Slipway: integrate any CI system with Nelson
https://github.com/Verizon/slipway