Some simple examples of how to pipeline data through beanstalkd or 0MQ, using Dominic memory encoding as an example.

1. IN THE BACK OF YOUR MIND
beanstalkd and 0MQ through the Dominic System

3. THE DOMINIC SYSTEM

4. LEARN HOW
TO IMPROVE
YOUR MIND
A book I am currently reading

5. MEMORY TECHNIQUES

6. MEMORY TECHNIQUES
• The book includes various techniques for improving memory

7. MEMORY TECHNIQUES
• The book includes various techniques for improving memory
• Ithelps to translate abstract data (the brain is weak with) into
vivid images (a strength of the brain)

8. MEMORY TECHNIQUES
• The book includes various techniques for improving memory
• Ithelps to translate abstract data (the brain is weak with) into
vivid images (a strength of the brain)
• The Dominic System is used to translate numbers into
characters and actions
• It can help you memorize large numbers

9. SINGLE DIGITS

10. SINGLE DIGITS
• Just
count through the
alphabet
• But replace F with S
• And skip to N when you
hit 9

11. SINGLE DIGITS
1 A
2 B
• Just
count through the 3 C
alphabet 4 D
5 E
• But replace F with S
6 S
• And skip to N when you 7 G
hit 9 8 H
9 N
0 O

12. DOUBLE DIGITS

13. DOUBLE DIGITS
• Convert digit pairs into letters
• 15 = AE, 23 = BC, 27 = BG, 39 = CN, 80 = HO, 80 = HA

14. DOUBLE DIGITS
• Convert digit pairs into letters
• 15 = AE, 23 = BC, 27 = BG, 39 = CN, 80 = HO, 80 = HA
• Use the letters as initials for a character
• AE = Albert Einstein, BG = Bill Gates, CN = Chuck Norris

15. DOUBLE DIGITS
• Convert digit pairs into letters
• 15 = AE, 23 = BC, 27 = BG, 39 = CN, 80 = HO, 80 = HA
• Use the letters as initials for a character
• AE = Albert Einstein, BG = Bill Gates, CN = Chuck Norris
• Or use letters to indicate characters in other ways
• BC = Jesus Christ, HO = Santa Claus, HA = Julia Sweeney

16. IMAGERY IS KEY

17. IMAGERY IS KEY
• Each digit pair should create an image in your mind

18. IMAGERY IS KEY
• Each digit pair should create an image in your mind
• The image need to be a character doing their signature action
• Chuck Norris performing a roundhouse kick
• Santa Claus delivering presents
• Julia Sweeney dressing asexually

19. FOUR DIGIT NUMBERS

20. FOUR DIGIT NUMBERS
• Tohandle four digit numbers, use the first two for the
character and the second two as the action
• 2339 = BCCN = Jesus Christ performing a roundhouse kick
• 8081 = HOHA = Santa Claus dressing asexually

21. FOUR DIGIT NUMBERS
• Tohandle four digit numbers, use the first two for the
character and the second two as the action
• 2339 = BCCN = Jesus Christ performing a roundhouse kick
• 8081 = HOHA = Santa Claus dressing asexually
• You’re up to a 10,000 number memory at this point!

22. LARGER NUMBERS

23. LARGER NUMBERS
• “Chunk” up bigger numbers into two and four digit groups

24. LARGER NUMBERS
• “Chunk” up bigger numbers into two and four digit groups
• Build wacky stories using those characters and actions
• 27808039 = 2780 8039 = BGHO HOCN = Bill Gates
started delivering presents, but this angered Santa Claus
who gave him a roundhouse kick.

25. A CAST OF CHARACTERS

26. A CAST OF CHARACTERS
http://ron.ludism.org/mnemonics_public.txt
…
40 = DO = Fred Flintstone, breaking rocks in the quarry (Yabba dabba
DOo!)
41 = DA = Douglas Adams, putting a Babelfish into his ear (_The Hitch
Hiker's Guide to the Galaxy_)
42 = DB = PERSONAL
43 = DC = George Washington, chopping down cherry tree (Washington DC)
44 = DD = Danny Dunn (fictional character), descending in a
bathysphere
45 = DE = PERSONAL
46 = DS = Ivan Stang (a.k.a. Doug Smith), ranting on a soapbox
47 = DG = PERSONAL
48 = DH = Douglas Hofstadter, recursively holding a picture of himself
49 = DN = Dan (_Roseanne_ TV show), wearing sunglasses to switch on
Christmas lights
…

27. A CAST OF CHARACTERS
http://ron.ludism.org/mnemonics_public.txt
…
40 = DO = Fred Flintstone, breaking rocks in the quarry (Yabba dabba
DOo!)
41 = DA = Douglas Adams, putting a Babelfish into his ear (_The Hitch
Hiker's Guide to the Galaxy_)
42 = DB = PERSONAL
43 = DC = George Washington, chopping down cherry tree (Washington DC)
44 = DD = Danny Dunn (fictional character), descending in a
bathysphere
45 = DE = PERSONAL
46 = DS = Ivan Stang (a.k.a. Doug Smith), ranting on a soapbox
47 = DG = PERSONAL
48 = DH = Douglas Hofstadter, recursively holding a picture of himself
49 = DN = Dan (_Roseanne_ TV show), wearing sunglasses to switch on
Christmas lights
…

29. BEANSTALKD (VIA STALKER)

30. BEANSTALKD

31. BEANSTALKD
•A simple but highly effective message queue

32. BEANSTALKD
•A simple but highly effective message queue
• I’ve used it many times, including to do moves in Go vs Go

33. BEANSTALKD
•A simple but highly effective message queue
• I’ve used it many times, including to do moves in Go vs Go
• It’s definitely one of my favorite tools for background jobs

34. CHARACTER LOOKUPS

35. CHARACTER LOOKUPS
require "open-uri"
def lookup_table
return @lookup_table if defined? @lookup_table
@lookup_table = { }
open("http://ron.ludism.org/mnemonics_public.txt") do |page|
page.each do |line|
if line =~ /A(d{2})s*=s*[A-Z]{2}s*=s*(.+)/
@lookup_table[$1] = $2
elsif line =~ /As+(.+)/
@lookup_table.values.last << " #{$1}"
end
end
end
@lookup_table
end

36. CHARACTER LOOKUPS
require "open-uri"
def lookup_table
return @lookup_table if defined? @lookup_table
@lookup_table = { }
open("http://ron.ludism.org/mnemonics_public.txt") do |page|
page.each do |line|
if line =~ /A(d{2})s*=s*[A-Z]{2}s*=s*(.+)/
@lookup_table[$1] = $2
elsif line =~ /As+(.+)/
@lookup_table.values.last << " #{$1}"
end
end
end
@lookup_table
end

37. CHARACTER LOOKUPS
require "open-uri"
def lookup_table
return @lookup_table if defined? @lookup_table
@lookup_table = { }
open("http://ron.ludism.org/mnemonics_public.txt") do |page|
page.each do |line|
if line =~ /A(d{2})s*=s*[A-Z]{2}s*=s*(.+)/
@lookup_table[$1] = $2
elsif line =~ /As+(.+)/
@lookup_table.values.last << " #{$1}"
end
end
end
@lookup_table
end

38. ENCODING CHUNKS

39. ENCODING CHUNKS
def encode(n, as = :both)
if n.size == 4 and as == :both
"#{encode(n[0..1], :character)} #{encode(n[2..3], :action)}."
elsif as == :both
"#{encode(n, :character)} #{encode(n, :action)}."
elsif as == :character
lookup_table[n][/As*([^,]+(?:([^)]+))?)/, 1]
elsif as == :action
lookup_table[n][/As*[^,]+(?:([^)]+))?,s*(.+?)s*z/, 1]
end
end

40. ENCODING CHUNKS
def encode(n, as = :both)
if n.size == 4 and as == :both
"#{encode(n[0..1], :character)} #{encode(n[2..3], :action)}."
elsif as == :both
"#{encode(n, :character)} #{encode(n, :action)}."
elsif as == :character
lookup_table[n][/As*([^,]+(?:([^)]+))?)/, 1]
elsif as == :action
lookup_table[n][/As*[^,]+(?:([^)]+))?,s*(.+?)s*z/, 1]
end
end

41. ENCODING CHUNKS
def encode(n, as = :both)
if n.size == 4 and as == :both
"#{encode(n[0..1], :character)} #{encode(n[2..3], :action)}."
elsif as == :both
"#{encode(n, :character)} #{encode(n, :action)}."
elsif as == :character
lookup_table[n][/As*([^,]+(?:([^)]+))?)/, 1]
elsif as == :action
lookup_table[n][/As*[^,]+(?:([^)]+))?,s*(.+?)s*z/, 1]
end
end

42. ENCODING NUMBERS

43. ENCODING NUMBERS
job "number.encode" do |args|
number = args["number"].to_s.delete("^0-9")
number = "0#{number}" if (number.size % 2).nonzero?
chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2)
encodings = [ ]
chunks.permutation.to_a.uniq.each do |grouping|
encodings << number.match(/A#{grouping.map { |g| "(.{#{g}})" }.join}z/)
.captures
.map { |n| encode(n) }
.join(" ")
.tr("_", '"')
end
Stalker.enqueue( "number.display", number: args["number"],
encodings: encodings )
end

44. ENCODING NUMBERS
job "number.encode" do |args|
number = args["number"].to_s.delete("^0-9")
number = "0#{number}" if (number.size % 2).nonzero?
chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2)
encodings = [ ]
chunks.permutation.to_a.uniq.each do |grouping|
encodings << number.match(/A#{grouping.map { |g| "(.{#{g}})" }.join}z/)
.captures
.map { |n| encode(n) }
.join(" ")
.tr("_", '"')
end
Stalker.enqueue( "number.display", number: args["number"],
encodings: encodings )
end

45. ENCODING NUMBERS
job "number.encode" do |args|
number = args["number"].to_s.delete("^0-9")
number = "0#{number}" if (number.size % 2).nonzero?
chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2)
encodings = [ ]
chunks.permutation.to_a.uniq.each do |grouping|
encodings << number.match(/A#{grouping.map { |g| "(.{#{g}})" }.join}z/)
.captures
.map { |n| encode(n) }
.join(" ")
.tr("_", '"')
end
Stalker.enqueue( "number.display", number: args["number"],
encodings: encodings )
end

46. ENCODING NUMBERS
job "number.encode" do |args|
number = args["number"].to_s.delete("^0-9")
number = "0#{number}" if (number.size % 2).nonzero?
chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2)
encodings = [ ]
chunks.permutation.to_a.uniq.each do |grouping|
encodings << number.match(/A#{grouping.map { |g| "(.{#{g}})" }.join}z/)
.captures
.map { |n| encode(n) }
.join(" ")
.tr("_", '"')
end
Stalker.enqueue( "number.display", number: args["number"],
encodings: encodings )
end

47. DISPLAYING ENCODINGS

48. DISPLAYING ENCODINGS
job "number.display" do |args|
Array(args["encodings"]).each do |encoded|
puts "#{args['number']}:"
puts encoded
puts
end
end

49. QUEUING NUMBERS

50. QUEUING NUMBERS
abort "USAGE: #{$PROGRAM_NAME} NUMBER" if ARGV.empty?
require "stalker"
Stalker.enqueue("number.encode", number: ARGV.shift)

51. RUNNING JOBS

52. RUNNING JOBS
$ beanstalkd

53. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536

54. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms

55. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms

56. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms

57. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms

58. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms

59. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms

60. RUNNING JOBS
$ beanstalkd
$ ruby encode_number.rb 405-285-0536
$ rvm 1.9.2
$ RUBYLIB=. stalk jobs.rb number.encode
[2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ]
[2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536)
[2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
$ RUBYLIB=. stalk jobs.rb number.display
…
405-285-0536:
Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!).
Stuart Little (tiny fictional mouse wearing street clothes) releasing
helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into
the Wardrobe.
…

61. FOR MORE INFORMATION
See Railscasts #243

62. FOR MORE INFORMATION
See Railscasts #243

63. BEANSTALKD PROS

64. BEANSTALKD PROS
• Extremely lightweight (especially if you drop the Rails stack)

65. BEANSTALKD PROS
• Extremely lightweight (especially if you drop the Rails stack)
• Ridiculously fast

66. BEANSTALKD PROS
• Extremely lightweight (especially if you drop the Rails stack)
• Ridiculously fast
• Designed to be a queue

67. BEANSTALKD PROS
• Extremely lightweight (especially if you drop the Rails stack)
• Ridiculously fast
• Designed to be a queue
• “Tubes” make it trivial to divide work or pipeline jobs

68. BEANSTALKD PROS
• Extremely lightweight (especially if you drop the Rails stack)
• Ridiculously fast
• Designed to be a queue
• “Tubes” make it trivial to divide work or pipeline jobs
• Really easy to spread across ruby versions and environments

69. BEANSTALKD CONS

70. BEANSTALKD CONS
• Not ideal for durable queuing needs (though it does have a
binlog)

71. BEANSTALKD CONS
• Not ideal for durable queuing needs (though it does have a
binlog)
• Missing
niceties like logging (though Stalker adds logging above
beanstalkd)

72. BEANSTALKD CONS
• Not ideal for durable queuing needs (though it does have a
binlog)
• Missing
niceties like logging (though Stalker adds logging above
beanstalkd)
• Needs job introspection

74. 0MQ
(AKA ZEROMQ OR ZMQ)

75. 0MQ

76. 0MQ
•A high-level socket API over an asynchronous message queue
• Request and reply, publish subscribe, and pipeline models

77. 0MQ
•A high-level socket API over an asynchronous message queue
• Request and reply, publish subscribe, and pipeline models
• This was my first time using it (I was curious about it)
•I don’t think it’s well suited to this task

78. THE END OF THE PIPELINE

79. THE END OF THE PIPELINE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
up.bind("tcp://127.0.0.1:5000")
while message = up.recv
puts message
end

80. THE END OF THE PIPELINE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
up.bind("tcp://127.0.0.1:5000")
while message = up.recv
puts message
end

81. THE END OF THE PIPELINE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
up.bind("tcp://127.0.0.1:5000")
while message = up.recv
puts message
end

82. THE END OF THE PIPELINE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
up.bind("tcp://127.0.0.1:5000")
while message = up.recv
puts message
end

83. NUMBER LOOKUPS

84. NUMBER LOOKUPS
# same lookup_table() as before…
def lookup
lookup_table.each do |number, character_and_action|
return number if yield character_and_action
end
nil
end
def character_number(msg)
lookup { |ca| msg.start_with? ca[/As*([^,]+(?:([^)]+))?)/, 1] }
end
def action_number(msg)
lookup { |ca| msg.end_with? ca[/As*[^,]+(?:([^)]+))?,s*(.+?)s*z/, 1] }
end

85. NUMBER LOOKUPS
# same lookup_table() as before…
def lookup
lookup_table.each do |number, character_and_action|
return number if yield character_and_action
end
nil
end
def character_number(msg)
lookup { |ca| msg.start_with? ca[/As*([^,]+(?:([^)]+))?)/, 1] }
end
def action_number(msg)
lookup { |ca| msg.end_with? ca[/As*[^,]+(?:([^)]+))?,s*(.+?)s*z/, 1] }
end

86. A DECODER

87. A DECODER
def decode(message)
message.to_s.strip.split(/.(?:s{2}|n|z)/).map { |chunk|
chunk.gsub!(/s+/, " ")
character = character_number(chunk)
action = action_number(chunk)
character == action ? character : "#{character}#{action}"
}.join
end

88. THE MIDDLE PIPE

89. THE MIDDLE PIPE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
down = zmq.socket(ZMQ::DOWNSTREAM)
up.bind("tcp://127.0.0.1:5001")
down.connect("tcp://127.0.0.1:5000")
while message = up.recv
down.send(decode(message))
end

90. THE MIDDLE PIPE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
down = zmq.socket(ZMQ::DOWNSTREAM)
up.bind("tcp://127.0.0.1:5001")
down.connect("tcp://127.0.0.1:5000")
while message = up.recv
down.send(decode(message))
end

91. THE MIDDLE PIPE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
down = zmq.socket(ZMQ::DOWNSTREAM)
up.bind("tcp://127.0.0.1:5001")
down.connect("tcp://127.0.0.1:5000")
while message = up.recv
down.send(decode(message))
end

92. THE MIDDLE PIPE
require "zmq"
zmq = ZMQ::Context.new
up = zmq.socket(ZMQ::UPSTREAM)
down = zmq.socket(ZMQ::DOWNSTREAM)
up.bind("tcp://127.0.0.1:5001")
down.connect("tcp://127.0.0.1:5000")
while message = up.recv
down.send(decode(message))
end

93. THE PIPELINE OPENING

94. THE PIPELINE OPENING
require "zmq"
zmq = ZMQ::Context.new
down = zmq.socket(ZMQ::DOWNSTREAM)
down.connect("tcp://127.0.0.1:5001")
down.send(ARGF.read)

95. A DATA PIPELINE IN ACTION

96. A DATA PIPELINE IN ACTION
$ ruby queue_message.rb
Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!).
Stuart Little (tiny fictional mouse wearing street clothes) releasing
helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into
the Wardrobe.
^d

97. A DATA PIPELINE IN ACTION
$ ruby queue_message.rb
Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!).
Stuart Little (tiny fictional mouse wearing street clothes) releasing
helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into
the Wardrobe.
^d
$ ruby decoder.rb

98. A DATA PIPELINE IN ACTION
$ ruby queue_message.rb
Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!).
Stuart Little (tiny fictional mouse wearing street clothes) releasing
helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into
the Wardrobe.
^d
$ ruby decoder.rb
$ ruby decoder_display.rb
4052850536

99. 0MQ PROS

100. 0MQ PROS
• Asynchronous messaging primitives for constructing networks

101. 0MQ PROS
• Asynchronous messaging primitives for constructing networks
• Extremely versatile (can be made to fit most any model)

102. 0MQ PROS
• Asynchronous messaging primitives for constructing networks
• Extremely versatile (can be made to fit most any model)
• Supports many to many connections

103. 0MQ PROS
• Asynchronous messaging primitives for constructing networks
• Extremely versatile (can be made to fit most any model)
• Supports many to many connections
• Automatically load balanced

104. 0MQ CONS

105. 0MQ CONS
• Very low-level (really just a platform to build on)
• Missing all the conveniences: logging, message formats, etc.

106. 0MQ CONS
• Very low-level (really just a platform to build on)
• Missing all the conveniences: logging, message formats, etc.
• Best for asynchronous networks you design and build

108. THANKS!