Presentation by Oleksiy Kurnenkov, OnApp. During 7th #pivorak Video from the MeetUp is available here https://www.youtube.com/watch?v=ENxqIvLEIck

1. Building
Distributed Systems
decision guide

2. Oleksiy Kurnenkov
Core Infrastructure
Domain lead
OnApp

3. Building
Distributed Systems
decision guide

4. Introduction
Part I

5. Distributed system
is a set of interconnected functional units with separate
runtime context

6. MONOLITH : DISTRIBUTED
1 Runtime context
Coroutines
Threads
Process
N Runtime contexts
Nodes
Components
Clusters
Grids

7. SCALE
Small ->
Middle ->
Global ->
OS, LAN
LAN
WAN

8. TYPES
CDN
Grids
Clouds
Clusters
Enterprise

9. CDN

10. Grid

11. Globus Toolkit 6
BOINC
x-Torrent

12. Cloud

13. Cluster

14. Enterprise

15. TASKS
MMOG
Analysis
Sensors
Real Time Enterprise
Media
Storage
Computation
Transactions

16. Challenges
Availability
Scalability
Security
Openness
Heterogeneity
Concurrency
Transparency
Feasibility

17. It IS there
99.9999999%
Availability

18. It grows and shrinks
Scalability

19. U ain’ gonna crack it
>:|
Security

20. It is open for development
and integration
Opennes

21. Everything changes
Heterogeneity

22. Parallel workloads
Concurrency

23. Less know - less depend
Transparency

24. Simplify as possible
Feasibility

25. Fallacies
The network is reliable
Latency is zero
Bandwidth is infinite
The network is secure
Topology doesn't change
There is one administrator
Transport cost is zero
The network is homogeneous

26. Mechanics
Part II

27. Mechanics
I
Membership
Concurrency / Parallelism
Synchronization
Coordination
Consensus
Clustering

28. Who’s there?
Membership

29. Membership
Heartbeats
Up-2-date list of nodes

30. We are Legion
Concurrency / Parallelism

31. Concurrency / Parallelism
Actors
Process Calculi
Shared memory

32. MY precious!
Synchronization

33. Synchronisation
Mutex | Semaphore
Distributed Lock

34. Roger that!
Coordination

35. Coordination
Leader election
Orchestration
the ‘Two Generals Problem’

36. Deal
Consensus

37. Consensus
Quorum
Consistency Availability Partition tolerance

38. Altogether
Clustering

39. Clustering
Load Balancing
Redundancy
Availability
Replication
Partition tolerance

40. Mechanics
II
Scaling
Fault Detection
Failover
Recover
Fail back

41. Scaling
Vertical
Horisontal
Automatic

42. Fault Detection
Monitoring
Supervision
Activity Analysis

43. Failover
Data | State Accessibility
Resource allocation
Spare Capacity
Business Continuity

44. Recovery
State Reconstruction
Data backup
State dump

45. Failback
Data consistency
Conflict resolution

46. Apache YARN
Apache Zookeeper
Apache Mesos
Apache Drill
Google Chubby
CoreOS
Google Kubernetes
Google Dremel

47. Architecture
Part III

48. Client - Server

49. N-tier

50. Service oriented Architecture

51. Event Driven Architecture

52. P2P

53. Context

55. System Allocation

56. Architectural
entities
decomposition
Subsystem
Service
Component
Process
Object
Middleware

57. Fractal

59. Subsystem -> Component -> Process -> Object

60. Subsystem -> Component -> Process -> Object

61. Subsystem -> Component -> Process -> Object

62. Communication
Infrastructure
aka Middleware
IPC
-----------------------------
Remote Invocation
Group Communication
Messaging

63. IPC
Share data
Send data

64. Remote Invocation
Request - Reply
RPC
RMI

65. Group Communication
{ Uni | Multi | Broad } cast
Publish | Subscribe

66. Messaging
Message Queue
Message Broker
Pipe | Filter | Translator

67. Implementation
Part IV

68. Microcosm ->
Macrocosm
Middleware
[ * ] Process
Host
Network

69. Within a process
Threads
Shared memory
Mutex | Monitor

70. require 'thread'
class RaceCondition
def initialize(resource, concurrency_level)
@resource = resource
@concurrency_level = concurrency_level
end
def perform
@concurrency_level.times do
threads << Thread.new { action }
end
threads.each(&:join)
end
private
def threads
@threads ||= []
end
def action
@resource += 1
puts @resource
end
end
RaceCondition.new(0, 10).perform
# 12
# 3
# 5
# 6
# 7
# 8
# 910
# 4

71. require 'thread'
class Sycnhronizer
def initialize(resource, concurrency_level)
@resource = resource
@concurrency_level = concurrency_level
end
def perform
@concurrency_level.times do
threads << Thread.new { action }
end
threads.each(&:join)
end
private
def threads
@threads ||= []
end
def action
lock.synchronize do
@resource += 1
puts @resource
end
end
def lock
@lock ||= Mutex.new
end
end
Sycnhronizer.new(0, 10).perform
# 1
# 2
# 3
# 4
# 5
# 6
# 7
# 8
# 9
# 10

72. Microcosm ->
Macrocosm
Process
[ * ] Host
NetworkMiddleware

73. Within a host
Processes
MQ | Shared memory | Pipes | UNIX Socket
Semaphore

74. # http://bogomips.org/ruby_posix_mq/
require 'posix_mq'
class Producer
attr_reader :mq
def initialize(mq_name)
@mq = POSIX_MQ.new("/foo", :rw)
end
def send(message, prio = 0)
puts "Send: #{message}. Priority #{prio}"
mq.send("#{message} #{prio}", prio)
end
end
p = Producer.new("/test_ipc")
p.send("Hello from #{Process.pid}", 10)
p.send("Hello from #{Process.pid}", 2)
p.send("Hello from #{Process.pid}", 0)
p.send("Hello from #{Process.pid}", 1)
p.send("Hello from #{Process.pid}", 20)
# ruby posix_mq/producer.rb
#
# Send: Hello from 12635. Priority 10
# Send: Hello from 12635. Priority 2
# Send: Hello from 12635. Priority 0
# Send: Hello from 12635. Priority 1
# Send: Hello from 12635. Priority 20

75. # http://bogomips.org/ruby_posix_mq/
require 'posix_mq'
class Consumer
attr_reader :mq
def initialize(mq_name)
@mq = POSIX_MQ.new("/foo", :rw)
end
def receive
mq.receive.first
end
def receive_non_block
mq.nonblock = true
begin
receive
rescue Errno::EAGAIN
mq.nonblock = false
puts "Nothing"
end
end
def shift
mq.tryshift
end
end
c = Consumer.new("/test_ipc")
while m = c.shift
puts "got: #{m}"
end
# ruby posix_mq/consumer.rb
# got: Hello from 12635 10
# got: Hello from 12635 20
# got: Hello from 12635 2
# got: Hello from 12635 1
# got: Hello from 12635 0

76. # https://github.com/pmahoney/process_shared
require 'process_shared'
mutex = ProcessShared::Mutex.new
mem = ProcessShared::SharedMemory.new(:int)
mem.put_int(0, 0)
pid1 = fork do
puts "in process 1 (#{Process.pid})"
10.times do
sleep 0.01
mutex.synchronize do
value = mem.get_int(0)
sleep 0.01
puts "process 1 (#{Process.pid}) incrementing"
mem.put_int(0, value + 1)
end
end
end
pid2 = fork do
puts "in process 2 (#{Process.pid})"
10.times do
sleep 0.01
mutex.synchronize do
value = mem.get_int(0)
sleep 0.01
puts "process 2 (#{Process.pid}) decrementing"
mem.put_int(0, value - 1)
end
end
end
Process.wait(pid1)
Process.wait(pid2)
# ruby shm.rb
#
# in process 1 (8038)
# in process 2 (8041)
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# process 1 (8038) incrementing
# process 2 (8041) decrementing
# value should be zero: 0

77. rd_child, wr_parent = IO.pipe
rd_parent, wr_child = IO.pipe
pid = fork do
rd_parent.close
wr_parent.close
wr_child.puts "sent from child process"
puts rd_child.gets
end
rd_child.close
wr_child.close
wr_parent.write "sent from parent process"
puts rd_parent.gets
# ruby pipes/pipes.rb
#
# sent from child process
# sent from parent process

78. require 'eventmachine'
module UnixServer
def receive_data(data)
puts data
EM.stop if data.chomp == "exit"
send_data("Server #{Process.pid}: Got #{data.chomp} from you")
close_connection_after_writing
end
end
EM.run do
puts "Started UNIX socket server on /tmp/sock"
EM::start_unix_domain_server("/tmp/sock", UnixServer)
end
# ruby server.rb
#
# Started UNIX socket server on /tmp/sock
#
# HELLO! My pid is 13847

79. require 'socket'
UNIXSocket.open("/tmp/sock") do |c|
c.write("HELLO! My pid is #{Process.pid}")
puts c.read
end
# ruby client.rb
#
# Server 13843: Got HELLO! My pid is 13847 from you

80. Microcosm ->
Macrocosm
Process
Host
[ * ] NetworkMiddleware

81. Over the Network
{ TCP | UDP } Socket
HTTP Client - Server
Async messaging

82. require "bunny"
conn = Bunny.new
conn.start
ch = conn.create_channel
q = ch.queue("hello")
puts " [*] Waiting for messages in #{q.name}. To exit press CTRL+C"
q.subscribe(:block => true) do |delivery_info, properties, body|
puts " [x] Received #{body}"
# cancel the consumer to exit
delivery_info.consumer.cancel
end
require "bunny"
conn = Bunny.new(:hostname => "rabbit.local")
conn.start
ch = conn.create_channel
q = ch.queue("hello")
ch.default_exchange.publish("Hello World!", :routing_key => q.name)
puts " [x] Sent 'Hello World!'"
conn.close

83. require "bunny"
conn = Bunny.new
conn.start
ch = conn.create_channel
x = ch.fanout("logs")
msg = ARGV.empty? ? "Hello World!" : ARGV.join(" ")
x.publish(msg)
puts " [x] Sent #{msg}"
conn.close
require "bunny"
conn = Bunny.new
conn.start
ch = conn.create_channel
x = ch.fanout("logs")
q = ch.queue("", :exclusive => true)
q.bind(x)
puts " [*] Waiting for logs. To exit press CTRL+C"
begin
q.subscribe(:block => true) do |delivery_info, properties, body|
puts " [x] #{body}"
end
rescue Interrupt => _
ch.close
conn.close
end

85. Apache Kafka
RabbitMQ

86. Next time
To be continued:
Algorithms, patterns,
code!

87. Thank you!
Q&A