In this talk, using the Erlang hacker's semi-official motto "Let it Crash!" as a lens, I'll speak to how radical simplicity of implementation, straight-forward runtime characteristics and discoverability of the running system lead to computer systems which have great success in networked, always-on deployments. I will argue that while Erlang natively implements many features which aid the construction of such systems--functional programming language semantics, lack of global mutable state, first-class networking, for instance--these characteristics can be replicated in any computer system, as a part of initial design of new systems or the gradual evolution of an existing project. I'll discuss common design patterns and anti-patterns, using my own work at AdRoll and project experience reports from a variety of fields--both successes and failures--to advance my argument. This will be a very practical talk and will be accessible to engineers of all backgrounds.

1. Let it Crash!
The Erlang Approach
to Building Reliable
Services

2. Hi, folks.

3. This talk is not about convincing
Facebook to pay you 19$ billion.

4. This is a talk about doing good,
careful work…

5. …on systems that might kill
people or ruin companies,
which is totally rad.

6. many
processors

7. in a network

8. inside an even
bigger network

9. and none of it’s
reliable

10. and also it’s
spread across
the globe

11. Sometimes we don’t care
to make the illusion of
whole-system reliability.

12. Sometimes we do.

13. In this talk, we do.

14. Erlang

15. A handful simple pieces.

16. An old-fashioned, explicit
functional language.

17. All data is immutable,
without exception.

18. Processes are the smallest
unit of control flow.

19. Processes are sequential
internally.

20. Processes are concurrent
to one another.

21. Processes can
communicate only
through message
passing.

22. Messages are copied
between processes.

23. Processes can “link” and
receive messages about
linked pair deaths.

24. This is called
“trapping exits”.

25. OR

26. Processes can “link” and
die when linked pairs die.

27. This is the
default
behaviour.

28. That’s basically it.

29. The implications are
really fun.

30. Supervision

31. Well known process traps exits,
restart other processes that fail.

32. The failed process is restarted
from a static specification.

33. The failed process state
is not preserved.

34. The failed process state
is not preserved.
(intentionally)

35. Registered
Names

36. These are decoupled
from the underlying
process.

37. You can change processes
without consumers getting
wise to it.

38. Protocols

39. You can’t manipulate another
process’ state.

40. You have to convince
it to change itself.

41. Network
Transparency

42. Network
Transparency
(kinda)

43. Messages may be addressed
to explicit pid, process name.
Node may be specified, too.

44. Erlang hides the network
details, leaving just the
abstraction.

45. Hot Code
Reloading

46. Code updates can be
inserted at runtime.

47. New functions and new
processes get inserted
under old names.

48. Tricky without VM support.

49. This is all driving to one end:

50. Fault
Tolerance

51. Everything in Erlang (and
everything not) is in the
service of building fault-tolerant
systems.

52. The short-hand for this is:

53. Let it crash!

54. Faulty subsystems are
difficult to correct.

55. Don’t bother.
Just restart.

56. What’s unique
to Erlang?

57. VM support for
swapping function
implementations.

58. Cheap processes.

59. Cheap processes.
(309 words of memory)

60. Strict process isolation.

61. It’s all small
stuff really.

62. What can you
do now?

63. Short of using Erlang, that is…

64. Immutable
Data

65. This means
embracing
copying.

66. Rub some
Queue on it.

67. Queue
+ Async Execution
= Erlang Process

68. Structure your
system as
loosely coupled
sub-systems.

69. Sub-systems communicate
through a well-known
protocol.

70. They may be co-resident
in memory or across
machine boundaries.

71. Decoupling brings
many secondary
advantages.

72. Isolate.

73. Don’t manipulate
memory that
isn’t yours.

74. Critical components get
their own machines.

75. Plan for
failure.

76. If sub-system A depends
on B and B fails, what
does A need to do?

77. “Hey, who knew
that system C
relied on B too?”

78. Supervise.

79. Negotiate the restarts
of failed sub-systems
into well-known states.

80. Erlang implements all of
this for you, but there’s
nothing special about it.

81. Anything else?

82. Understand the
Network.

83. 0. The network is reliable.
1. Latency is zero.
2. Bandwidth is infinite.
3. The network is secure.
4. Topology doesn’t change.
5. There is one administrator.
6. Transport cost is zero.
7. The network is homogenous.

84. P.S.

85. This is how the
Internet works.

86. This is how hard
real-time systems
work.

87. This is how
multi-processors
work.

88. There’s decades
of material to
learn from.

89. Thanks!
<3
@bltroutwine