An experience report from 3 years of using Clojure

1. Life of an immutant
3 years with Clojure

2. About me
• @michaelklishin
• github.com/michaelklishin

3. Setting expectations

4. Setting expectations
• This is an experience report

5. Setting expectations
• This is an experience report
• Not a language primer

6. Setting expectations
• This is an experience report
• Not a language primer
• Assumes no familiarity except basic facts

7. Setting expectations
• This is an experience report
• Not a language primer
• Assumes no familiarity except basic facts
• ~200 slides.This is gonna be fast.

8. Background

9. Background
• Data processing

10. Background
• Data processing
• Scheduling systems

11. Background
• Data processing
• Scheduling systems
• Specialized Web crawlers

12. Background
• Data processing
• Scheduling systems
• Specialized Web crawlers
• Billing, accounting

13. Background
• Data processing
• Scheduling systems
• Specialized Web crawlers
• Billing, accounting
• Infrastructure automation

14. Wanted

15. Wanted
• An environment that does not suck

16. Wanted
• An environment that does not suck is
operationally sane

17. Wanted
• An environment that does not suck is
operationally sane
• A runtime that has decent tooling
(profilers, concurrency-related metrics, GC
metrics)

18. Wanted
• An environment that does not suck is
operationally sane
• A runtime that has decent tooling
(profilers, concurrency-related metrics, GC
metrics)
• A runtime that yields good baseline
performance

19. Wanted
• An environment that does not suck is
operationally sane
• A runtime that has decent tooling
(profilers, concurrency-related metrics, GC
metrics)
• A runtime that yields good baseline
performance
• Painless upgrade paths

20. Wanted
• Community

21. Wanted
• Community
• Benefit from existing libraries and tools

22. Wanted
• Community
• Benefit from existing libraries and tools
• Concurrency and parallelism

23. Wanted
• Community
• Benefit from existing libraries and tools
• Concurrency and parallelism
• A productive, concise language

24. Wanted
JVM

25. JVM

26. JVM
• One of the most mature runtimes you can
find

27. JVM
• One of the most mature runtimes you can
find
• Excellent tooling

28. JVM
• One of the most mature runtimes you can
find
• Excellent tooling (VisualVM,YourKit, jstack,
jhat,VM/+XX flags, BTrace, …)

29. JVM
• One of the most mature runtimes you can
find
• Excellent tooling
• Excellent baseline performance

30. JVM
• One of the most mature runtimes you can
find
• Excellent tooling
• Excellent baseline performance
• Multilingual: Clojure, Java, JRuby, Scala, JS
implementations

32. Clojure

33. Clojure
• Very high level, concise, expressive

34. Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story

35. Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance

36. Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
• Consistent, well designed

37. Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
• Consistent, well designed
• Decent tooling

38. Clojure
• Very high level, concise, expressive
• Excellent concurrency/parallelism story
• Very good baseline performance
• Consistent, well designed
• Decent tooling
• Fun to work with (motivation)

39. Clojure
• No bored academics, built by a practicioner

40. Clojure: concurrency/
parallelism story

41. Clojure: concurrency/
parallelism story
• Immutable data structures by default

42. Clojure: concurrency/
parallelism story
• Immutable data structures by default
• Identity/values separation

43. Clojure: concurrency/
parallelism story
• Immutable data structures by default
• Identity/values separation
• Concurrency-aware reference types

44. Clojure: concurrency/
parallelism story
• Immutable data structures by default
• Identity/values separation
• Concurrency-aware reference types
• java.util.concurrent

45. Clojure: concurrency/
parallelism story
When I find myself in times of trouble
Prof. Doug Lea comes to me
Coding lines of wisdom
j.u.c.

46. Clojure: concurrency/
parallelism story
• Immutable data structures by default
• Identity/values separation
• Concurrency-aware reference types
• java.util.concurrent
• Reactor,Akka, Disruptor, RxJava…

47. Clojure: concurrency/
parallelism story
• Async I/O: NIO, Netty,Vert.x…

48. Clojure: concurrency/
parallelism story
• You have choice

49. Clojure: concurrency/
parallelism story
• You have choice
• Runtime parallelism (on JVM, CLR)

50. Clojure: concurrency/
parallelism story
• You have choice
• Runtime parallelism (on JVM, CLR)
• Hundreds, thousands of threads at the
same time

52. Clojure: baseline
performance

53. Clojure: baseline
performance
• What matters the most is what bytecode
the compiler produces

54. contextneeded.com

55. Clojure: baseline
performance
• What matters the most is what bytecode
the compiler produces
• Java interop bytecode is typically the same
as produced by javac

56. Clojure: baseline
performance
• What matters the most is what bytecode
the compiler produces
• Java interop bytecode is typically the same
as produced by javac
• Watch out for reflective calls, primitive
boxing

57. Clojure: baseline
performance
• Optional type hints

58. Clojure: baseline
performance
• Optional type hints
• Type inference in simple cases

59. Clojure: baseline
performance
(let [d (java.util.Date.)]
d)

60. Clojure: baseline
performance
(let [d (java.util.Date.)]
d)

61. Clojure: baseline
performance
• Optional type hints
• Type inference in simple cases

62. Clojure: baseline
performance
• Optional type hints
• Type inference in simple cases
• If JVM can optimize your bytecode, things
run screaming fast

63. Clojure: baseline
performance
• Optional type hints
• Type inference in simple cases
• If JVM can optimize your bytecode, things
run screaming fast
• From 2% (Java lib wrappers) to 200% (naïve
numerics heavy code, tight loops) penalty
compared to Java

64. Clojure: baseline
performance
• Immutable data structures have O(log32 n)
access complexity

65. Clojure: baseline
performance
• Immutable data structures have O(log32 n)
access complexity
• Immutable data structures have associated
GC tax

66. Clojure: baseline
performance
• Immutable data structures have O(log32 n)
access complexity
• Immutable data structures have associated
GC tax
• Focus on exploiting parallelism over
squeezing µs from a single thread

67. Clojure: baseline
performance
• Poor startup time, excellent execution
performance

68. Clojure: baseline
performance
• Poor startup time, excellent execution
performance
• Anecdotal evidence: 10 — 50 times
throughput improvements compared to
Ruby in real apps

70. Clojure: stability

71. Clojure: stability
• Have been using 1.3 since alpha1

72. Clojure: stability
• Have been using 1.3 since alpha1
• Hit by 4 Clojure bugs in ~3 years

73. Clojure: stability
• Have been using 1.3 since alpha1
• Hit by 4 Clojure bugs in ~3 years
• 1 bug is an edge case that is not worth
fixing (I did something really stupid)

74. Clojure: stability
• Have been using 1.3 since alpha1
• Hit by 4 Clojure bugs in ~3 years
• 1 bug is an edge case that is not worth
fixing (I did something really stupid)
• Alphas more solid than GA releases of
almost all other languages I’ve worked with

75. Clojure: stability
• Hit by 4 Clojure bugs in ~3 years
• 1 bug is an edge case that is not worth
fixing (I did something really stupid)
• Alphas more solid than GA releases of
almost all other languages I’ve worked with
• Others (e.g.ThoughtWorks) report the
same experience

76. Clojure: stability
• Upgraded 28 apps & libraries from 1.3 to
1.4. Just 1 change due to a tricky 3rd party
library.The rest was a drop-in replacement.

77. Clojure: stability
• Upgraded 28 apps & libraries from 1.3 to
1.4. Just 1 change due to a tricky 3rd party
library.The rest was a drop-in replacement.
• When upgrading is this easy, people do it
happily

78. Clojure: stability
• Upgraded 28 apps & libraries from 1.3 to
1.4. Just 1 change due to a tricky 3rd party
library.The rest was a drop-in replacement.
• When upgrading is this easy, people do it
happily
• No library binary compatibility hell some
other JVM languages suffer from

79. Clojure: stability
• Many active developers are conservative
with versioning & calling things “done”

80. Clojure: stability
• Many active developers are conservative
with versioning & calling things “done”
• Fits JVM ecosystem maturity really well

81. Clojure: stability
• The real reason: Clojure is small and was
designed to be simple

82. Clojure: stability
• No monkey patching (there are features
that give you all the good parts of teh)

83. Clojure: consistency

84. Clojure: consistency
• Collections/sequences library is very
unified

85. Clojure: consistency
• Collections/sequences library is very
unified
• Easy to implement your own data types
that work just like core Clojure types

86. Clojure: consistency
• Collections/sequences library is very
unified
• Easy to implement your own data types
that work just like core Clojure types
• Almost no “bad baggage” in the core

87. Clojure: consistency
• Collections/sequences library is very
unified
• Easy to implement your own data types
that work just like core Clojure types
• Almost no “bad baggage” in the core
• The language was actually designed

88. Clojure: deployment

89. Clojure: deployment
• Pretty standard JVM deployment options

90. Clojure: deployment
• Pretty standard JVM deployment options
• Compile everything AOT into a single jar
(“überjar”), including dependencies, assets,
etc

91. Clojure: deployment
• Pretty standard JVM deployment options
• Compile everything AOT into a single jar
(“überjar”), including dependencies, assets,
etc
• A tiny shell script + start-stop-daemon

92. Clojure: deployment
• Pretty standard JVM deployment options
• Compile everything AOT into a single jar
(“überjar”), including dependencies, assets,
etc
• A tiny shell script + start-stop-daemon
• Chef or Capistrano

93. Clojure: deployment
• CloudFoundry

94. Clojure: deployment
• CloudFoundry
• Heroku

95. Clojure: deployment
• CloudFoundry
• Heroku
• Immutant/JBoss AS

96. Clojure: deployment
• CloudFoundry
• Heroku
• Immutant/JBoss AS
• OpenShift

97. Clojure: operations

98. Clojure: operations
• VisualVM, jstack, jheap, …

99. Clojure: operations
• VisualVM, jstack, jheap, …
• nREPL

100. Clojure: operations
• VisualVM, jstack, jheap, …
• nREPL
• Remote REPL over HTTP(S)

101. Clojure: operations
• VisualVM, jstack, jheap, …
• nREPL
• Remote REPL over HTTP(S)
• Not quite Erlang but still impressive

102. Clojure: operations
• VisualGC gives real insight into what is
going on with object allocation, tenuring,
death rate

103. Clojure: operations
• VisualGC gives real insight into what is
going on with object allocation, tenuring,
death rate
• JVM DTrace support (Solaris, SmartOS,
FreeBSD, OS X)

104. Clojure: operations
• VisualGC gives real insight into what is
going on with object allocation, tenuring,
death rate
• JVM DTrace support (Solaris, SmartOS,
FreeBSD, OS X)
• BTrace

105. Clojure: community

106. Clojure: community
• Full of intelligent engineers who don’t give
two shits about Hacker News

107. Clojure: community
• Full of intelligent engineers who don’t give
two shits about Hacker News
• Over 10 conferences to date

108. Clojure: community
• Full of intelligent engineers who don’t give
two shits about Hacker News
• Over 10 conferences to date
• ~10K people on the mailing list

109. Clojure: community
• Full of intelligent engineers who don’t give
two shits about Hacker News
• Over 10 conferences to date
• ~10K people on the mailing list
• Users from small and not-so-small startups
to Fortune 50 corporations in finance,
biotech, retail

110. Clojure: fun to work
with

111. Clojure: fun to work
with
• Profoundly changes the way you think

112. Clojure: changes the
way you think

113. Clojure: changes the
way you think
• About managing your program state

114. Clojure: changes the
way you think
• About managing your program state
• About the value of experimenting (REPL
driven development)

115. Clojure: changes the
way you think
• About managing your program state
• About the value of experimenting (REPL
driven development)
• About metaprogramming

116. Clojure: changes the
way you think
• About managing your program state
• About the value of experimenting (REPL
driven development)
• About metaprogramming
• About isolating side effects

117. Clojure: changes the
way you think

118. Clojure: changes the
way you think
• Program = pure core + parts that
communicate with the outside world
(network, disk I/O)

119. Clojure: changes the
way you think
• Program = pure core + parts that
communicate with the outside world
(network, disk I/O)
• Pure parts are ridiculously easy to reason
about, test, experiment with and reuse

120. Clojure: changes the
way you think
• Program = pure core + parts that
communicate with the outside world
(network, disk I/O)
• Pure parts are ridiculously easy to reason
about, test, experiment with and reuse
• Typical Rails app: 95% of methods have side
effects

121. Clojure: changes the
way you think
• Typical Clojure app: 40% of functions have
side effects

122. Clojure: changes the
way you think
• Typical Clojure app: 40% of functions have
side effects
• Anecdotal evidence: 3-4 times less time
spent on tests

123. Clojure: changes the
way you think
• Typical Clojure app: 40% of functions have
side effects
• Anecdotal evidence: 3-4 times less time
spent on tests
• Moral of the story: make more of your
code not have side effects

124. Clojure: fun to work
with
• Profoundly changes the way you think

125. Clojure: fun to work
with
• Profoundly changes the way you think
• Motivates you to learn

126. Clojure: fun to work
with
• Profoundly changes the way you think
• Motivates you to learn
• Easy things are easy, hard things are
possible

127. Clojure: fun to work
with
• Profoundly changes the way you think
• Motivates you to learn
• Easy things are easy, hard things are
possible
• Wheels are already invented in clojure.core
or JDK

128. Clojure: fun to work
with
• Stable: fewer distractions, get in the flow

129. Clojure: fun to work
with
• Stable: fewer distractions, get in the flow
• REPL driven development keeps you in the
flow for longer periods

130. Clojure: fun to work
with
• Stable: fewer distractions, get in the flow
• REPL driven development keeps you in the
flow for longer periods
• Polymorphism done right <3

131. Clojure: fun to work
with
“Clojure demands that you raise your game
and rewards you greatly for it…”

132. Clojure: fun to work
with
• The community is full of smart and down
to earth people.

134. Clojure: use cases

135. Clojure: use cases
• Data processing

136. Clojure: use cases
• Data processing
• Stream processing

137. Clojure: use cases
• Data processing
• Stream processing
• Micro-services

138. Clojure: use cases
• Data processing
• Stream processing
• Micro-services
• Machine learning

139. Clojure: use cases
• Data processing
• Stream processing
• Micro-services
• Machine learning
• Web development

141. Clojure: misc

142. Clojure: misc
• Leiningen (leiningen.org) is excellent

143. Clojure: misc
• Leiningen (leiningen.org) is excellent
• Deps management, build tool, custom tasks
are just Clojure functions

144. Clojure: misc
• Leiningen (leiningen.org) is excellent
• Deps management, build tool, custom tasks
are just Clojure functions
• Packages your code + all deps into a single
jar for deployment

145. Clojure: misc
• Leiningen (leiningen.org) is excellent
• Deps management, build tool, custom tasks
are just Clojure functions
• Packages your code + all deps into a single
jar for deployment
• Makes it trivial to develop against multiple
Clojure versions

146. Clojure: misc
• clojars.org

147. Clojure: misc
• clojars.org
• Deploy via SSH (SCP) or HTTPS

148. Clojure: misc
• clojars.org
• Deploy via SSH (SCP) or HTTPS
• Very easy to deploy snapshot releases

149. Clojure: misc
• clojars.org
• Deploy via SSH (SCP) or HTTPS
• Very easy to deploy snapshot releases
• Clojars and Leiningen are maintained by the
same core team

150. Clojure: misc
• travis-ci.org support

151. Clojure: misc
• travis-ci.org support
• Test against multiple JDKs

152. Clojure: ecosystem
• Feels a bit like Ruby circa 2009

153. Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your
reach

154. Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your
reach
• And Leiningen + clojars.org

155. Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your
reach
• And Leiningen + clojars.org
• Community ~ doubles every year in size

156. Clojure: ecosystem
• Feels a bit like Ruby circa 2009
• But with the entire JVM ecosystem at your
reach
• And Leiningen + clojars.org
• Community ~ doubles every year in size
• Google for “The State of Clojure” survey

157. Clojure: ecosystem
• ~35% of people come from Java, ~35% from
Ruby or Python

158. Clojure: ecosystem
• ~35% of people come from Java, ~35% from
Ruby or Python
• Libraries, not frameworks

159. Clojure: ecosystem
• ~35% of people come from Java, ~35% from
Ruby or Python
• Libraries, not frameworks
• Decent libraries just for almost every
problem

160. Clojure: ecosystem
• ~35% of people come from Java, ~35% from
Ruby or Python
• Libraries, not frameworks
• Decent libraries just for almost every
problem
• 15 books, 2 outdated (cover 1.2)

161. Clojure: ecosystem
• ~35% of people come from Java, ~35% from
Ruby or Python
• Libraries, not frameworks
• Decent libraries just for almost every
problem
• 15 books, 2 outdated (cover 1.2)
• clojure-doc.org, tryclj.com, …

162. Clojure: ecosystem

164. Clojure: what sucks

165. Clojure: what sucks
• clojure.org

166. Clojure: what sucks
• clojure.org
• Documentation (guides, the ref is mostly
OK, books are great)

167. Clojure: what sucks
• clojure.org
• Documentation (guides, the ref is mostly
OK, books are great)
• Compiler messages can be cryptic

168. Clojure: what sucks
• clojure.org
• Documentation (guides, the ref is mostly
OK, books are great)
• Compiler messages can be cryptic
• JVM startup time + code compilation on
each run (without REPL or tools such as
Drip)

169. Clojure: what sucks
• Paper contributor agreement

171. ClojureWerkz

172. ClojureWerkz
“A growing collection of open source
Clojure libraries…”

173. ClojureWerkz
“It just werkz…”

174. ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)

175. ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich

176. ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
• Well documented

177. ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
• Well documented
• Friendly license (EPL, the same as Clojure)

178. ClojureWerkz
• Modern targets (Clojure 1.5+, JDK 6)
• Feature rich
• Well documented
• Friendly license (EPL, the same as Clojure)
• Tested against multiple Clojure versions, 3
JDKs on travis-ci.org

179. ClojureWerkz
clojurewerkz.org

180. ClojureWerkz
• Projects reuse a lot of Java libraries

181. ClojureWerkz
• Projects reuse a lot of Java libraries
• Which are often officially supported (e.g.
RabbitMQ, ElasticSearch, MongoDB, Riak,
Reactor)

182. “ClojureWerkz stuff is really improving my
Clojure experience, which is rapidly becoming
my language of choice…”

184. Clojure: the takeaway

185. Clojure: the takeaway
• A lot of merit of its own

186. Clojure: the takeaway
• A lot of merit of its own
• Great stability

187. Clojure: the takeaway
• A lot of merit of its own
• Great stability
• Hosted (symbiotic) language is a great thing

188. Clojure: the takeaway
• A lot of merit of its own
• Great stability
• Hosted (symbiotic) language is a great thing
• Don’t fear the JVM

189. Clojure: the takeaway
• A lot of merit of its own
• Great stability
• Hosted (symbiotic) language is a great thing
• Don’t fear the JVM
• Immutability is essential for sane
concurrent programming

190. Clojure: the takeaway
• Clojure greatly rewards those who choose
to use it

192. The real reason?

194. The real reason?
???

195. clojure.core/lazy-cat

196. Thank you
• @michaelklishin
• github.com/michaelklishin
• clojurewerkz.org
• michael@clojurewerkz.org