This document summarizes a presentation about using algebraic structures like functors, applicative functors and monads to build asynchronous and concurrent programs in Clojure. It discusses using the imminent library to handle futures and parallelism in a more declarative way compared to core.async. Examples are provided that show how to aggregate data from multiple asynchronous sources using these concepts. It also touches on handling exceptions in an asynchronous setting.

1. The algebra of library
design
#cljsyd - October 2014
Leonardo Borges
@leonardo_borges
www.leonardoborges.com
www.thoughtworks.com

2. Algebra?

3. More specifically…
• Algebraic structures, studied in Abstract Algebra
• a set with one or more operations on it
• Category theory is another way to study these
structures

4. Overview
• The problem with Clojure futures
• A better futures library
• Functors, Applicative Functors and Monads
• What about core.async (and others?)

5. The beginning…
(def age (future (do (Thread/sleep 2000)!
31)))!
(prn "fetching age...")!
!
(def double-age (* 2 @age))!
(prn "age, doubled: " double-age)!
!
(prn "doing something else important...")

6. Do you see any issues?

7. The beginning…
(def age (future (do (Thread/sleep 2000)!
31)))!
(prn "fetching age...")!
!
(def double-age (* 2 @age))!
(prn "age, doubled: " double-age)!
!
(prn "doing something else important...")!
!
;; "fetching age..."!
;; "age, doubled: " 62!
;; "doing something else important..."!

8. It looks like we would like to execute
something once the Future has completed

9. A first glance at imminent
(require '[imminent.core :as i])!
!
(def age (i/future (do (Thread/sleep 2000)!
31)))!
(prn "fetching age...")!
(def double-age (i/map age #(* % 2)))!
(i/on-success double-age #(prn "age, doubled: " %))!
!
(prn "doing something else important...")

10. A first glance at imminent
(require '[imminent.core :as i])!
!
(def age (i/future (do (Thread/sleep 2000)!
31)))!
(prn "fetching age...")!
(def double-age (i/map age #(* % 2)))!
(i/on-success double-age #(prn "age, doubled: " %))!
!
(prn "doing something else important...")!
!
;; "fetching age..."!
;; "doing something else important..."!
;; "age, doubled: " 62!

11. Functor
class Functor f where!
fmap :: (a -> b) -> f a -> f b

12. Another example
(def age (future (do (Thread/sleep 2000)!
31)))!
(def name (future (do (Thread/sleep 2000)!
"Leonardo")))!
(prn "fetching name...")!
(prn (format "%s is %s years old" @name @age))!
(prn "more important things going on...")

13. Same as before, only this time we
want to execute the code once
both futures have completed

14. Rewriting it in imminent
(def age (i/future (do (Thread/sleep 2000)!
31)))!
(def name (i/future (do (Thread/sleep 2000)!
"Leonardo")))!
(prn "fetching name...")!
(def both (i/sequence [name age]))!
(i/on-success both !
(fn [[name age]]!
(prn (format "%s is %s years old" name age))))!
!
(prn "more important things going on...")!
!

15. Rewriting it in imminent
(def age (i/future (do (Thread/sleep 2000)!
31)))!
(def name (i/future (do (Thread/sleep 2000)!
"Leonardo")))!
(prn "fetching name...")!
(def both (i/sequence [name age]))!
(i/on-success both !
(fn [[name age]]!
(prn (format "%s is %s years old" name age))))!
!
(prn "more important things going on...")!
!
;; "fetching name..."!
;; "more important things going on..."!
;; "Leonardo is 31 years old"!

16. Monad
class Monad m where!
(>>=) :: m a -> (a -> m b) -> m b!
return :: a -> m a

17. Monad
class Monad m where!
(>>=) :: m a -> (a -> m b) -> m b!
return :: a -> m a
(>>=) is also called bind, flatmap, selectMany
and mapcat…

18. Monad - derived functions
liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r

19. Monad - derived functions
liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r
(defn mlift2!
[f]!
(fn [ma mb]!
(flatmap ma!
(fn [a]!
(flatmap mb!
(fn [b]!
(pure (f a b))))))))

20. Monad - derived functions
sequence :: Monad m => [m a] -> m [a]

21. Monad - derived functions
sequence :: Monad m => [m a] -> m [a]
(defn sequence!
[ms]!
(reduce (mlift2 conj)!
(pure [])!
ms))

22. Monad - derived functions
mapM :: Monad m => (a -> m b) -> [a] -> m [b]

23. Monad - derived functions
mapM :: Monad m => (a -> m b) -> [a] -> m [b]
(defn mmap!
[f vs]!
(sequence (map f vs)))

24. Monad - derived functions
mapM :: Monad m => (a -> m b) -> [a] -> m [b]
(defn mmap!
[f vs]!
(sequence (map f vs)))
plus a bunch of others…

25. Let’s look at a more concrete
example

26. Aggregating movie data
(defn cast-by-movie [name]!
(future (do (Thread/sleep 5000)!
(:cast movie))))!
!
(defn movies-by-actor [name]!
(do (Thread/sleep 2000)!
(->> actor-movies!
(filter #(= name (:name %)))!
first)))!
!
(defn spouse-of [name]!
(do (Thread/sleep 2000)!
(->> actor-spouse!
(filter #(= name (:name %)))!
first)))!
!
(defn top-5 []!
(future (do (Thread/sleep 5000)!
top-5-movies)))!

27. The output
({:name "Cate Blanchett",!
:spouse "Andrew Upton",!
:movies!
("Lord of The Rings: The Fellowship of The Ring - (top 5)"...)}!
{:name "Elijah Wood",!
:spouse "Unknown",!
:movies!
("Eternal Sunshine of the Spotless Mind"...)}!
...)

28. One possible solution
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (pmap movies-by-actor @cast)!
spouses (pmap spouse-of @cast)!
top-5 (top-5)]!
(prn "Fetching data...")!
(pprint (aggregate-actor-data spouses movies @top-5)))

29. One possible solution
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (pmap movies-by-actor @cast)!
spouses (pmap spouse-of @cast)!
top-5 (top-5)]!
(prn "Fetching data...")!
(pprint (aggregate-actor-data spouses movies @top-5)))
;; "Elapsed time: 10049.334 msecs"

30. We face the same problems
as before

31. Re-arranging the code
would improve it

32. But the point is not having to
do so

33. In imminent
(defn cast-by-movie [name]!
(i/future (do (Thread/sleep 5000)!
(:cast movie))))!
!
(defn movies-by-actor [name]!
(i/future (do (Thread/sleep 2000)!
(->> actor-movies!
(filter #(= name (:name %)))!
first))))!
!
(defn spouse-of [name]!
(i/future (do (Thread/sleep 2000)!
(->> actor-spouse!
(filter #(= name (:name %)))!
first))))!
!
(defn top-5 []!
(i/future (do (Thread/sleep 5000)!
top-5-movies)))

34. In imminent
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (i/flatmap cast #(i/map-future movies-by-actor %))!
spouses (i/flatmap cast #(i/map-future spouse-of %))!
result (i/sequence [spouses movies (top-5)])]!
(prn "Fetching data...")!
(pprint (apply aggregate-actor-data!
(i/dderef (i/await result)))))

35. In imminent
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (i/flatmap cast #(i/map-future movies-by-actor %))!
spouses (i/flatmap cast #(i/map-future spouse-of %))!
result (i/sequence [spouses movies (top-5)])]!
(prn "Fetching data...")!
(pprint (apply aggregate-actor-data!
(i/dderef (i/await result)))))
;; "Elapsed time: 7087.603 msecs"

36. Everything is asynchronous
by default

37. We can optionally block and wait
for a result using the “i/await”
function

38. Ok, that’s cool!
But what about Applicative Functors?

39. Writing this sucks
(defn int-f [n]!
(i/future (do (Thread/sleep 2000)!
(* 2 n))))!
!
!
(-> (i/bind!
(int-f 10)!
(fn [a] (i/bind!
(int-f a)!
(fn [b] (i/bind!
(int-f b)!
(fn [c] !
(i/pure (+ a b c)))))))

40. Monadic “do” notation
(i/mdo [a (int-f 10)!
b (int-f a)!
c (int-f b)]!
(i/pure (+ a b c)))

41. Monadic “do” notation
(i/mdo [a (int-f 10)!
b (int-f a)!
c (int-f b)]!
(i/pure (+ a b c)))
How long does this computation take?

42. Monadic “do” notation
(i/mdo [a (int-f 10)!
b (int-f a)!
c (int-f b)]!
(i/pure (+ a b c)))
How long does this computation take?
;; "Elapsed time: 6002.39 msecs"

43. What if the computations don’t
depend on each other?

44. Monadic “do” notation
(i/mdo [a (int-f 10)!
b (int-f 20)!
c (int-f 30)]!
(i/pure (+ a b c)))

45. Monadic “do” notation
(i/mdo [a (int-f 10)!
b (int-f 20)!
c (int-f 30)]!
(i/pure (+ a b c)))
How long does this take now?

46. Monadic “do” notation
(i/mdo [a (int-f 10)!
b (int-f 20)!
c (int-f 30)]!
(i/pure (+ a b c)))
How long does this take now?
;; "Elapsed time: 6002.39 msecs"

47. ?

48. The ‘do-notation’ is used to sequence monadic
steps so we remain serial, but still asynchronous

49. Applicative Functor
class Functor f => Applicative f where!
pure :: a -> f a!
(<*>) :: f (a -> b) -> f a -> f b!

50. Previous example, rewritten in applicative style
(i/<*> (i/map (int-f 10) (curry + 3))!
(int-f 20)!
(int-f 30))
;;"Elapsed time: 2001.509 msecs"

51. It turns out this pattern is a derived
function from Applicative Functors

52. Applicative Functor - derived functions
liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c

53. Applicative Functor - derived functions
liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c
(defn alift!
[f]!
(fn [& as]!
{:pre [(seq as)]}!
(let [curried (curry f (count as))]!
(apply <*>!
(fmap curried (first as))!
(rest as)))))

54. “alift” assumes every Applicative
Functor is also Functor

55. Previous example, rewritten using “alift"
;;"Elapsed time: 2001.509 msecs"
((i/alift +) (int-f 10) (int-f 20) (int-f 30))

56. It looks a lot like function application
:)

57. Applicatives give us the parallel
semantics without giving up convenience

58. Ok, this is awesome! But we have
core.async!

59. The movies example, revisited
(defn cast-by-movie [name]!
(let [c (chan)]!
(go (<! (timeout 5000))!
(>! c (:cast movie))!
(close! c))!
c))!
!
(defn movies-by-actor [name]!
(let [c (chan)]!
(go (<! (timeout 2000))!
(>! c (->> actor-movies!
(filter #(= name (:name %)))!
first))!
(close! c))!
c))

60. The movies example, revisited
(defn spouse-of [name]!
(let [c (chan)]!
(go (<! (timeout 2000))!
(>! c (->> actor-spouse!
(filter #(= name (:name %)))!
first))!
(close! c))!
c))!
!
(defn top-5 []!
(let [c (chan)]!
(go (<! (timeout 5000))!
(>! c top-5-movies)!
(close! c))!
c))!
!
!
(defn async-pmap [f source]!
(go (->> (map f (<! source))!
async/merge!
(async/into [])!
<!)))

61. The movies example, revisited
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
m-cast (mult cast)!
movies (async-pmap movies-by-actor (tap m-cast (chan)))!
spouses (async-pmap spouse-of (tap m-cast (chan)))!
top-5 (top-5)]!
(prn "Fetching data...")!
(pprint (<!! (go (aggregate-actor-data (<! spouses)!
(<! movies)!
(<! top-5))))))!
!
;; "Elapsed time: 7088.834 msecs"

62. “mult” and “tap” have no business
being in that code

63. they are necessary because
channels are single-take containers

64. Exceptions
core.async swallows them by default
(defn movies-by-actor [name]!
(let [c (chan)]!
(go (<! (timeout 2000))!
(throw (Exception. (str "Error fetching movies for actor " name)))!
(>! c (->> actor-movies!
(filter #(= name (:name %)))!
first))!
(close! c))!
c))

65. Exceptions
core.async swallows them by default
(defn movies-by-actor [name]!
(let [c (chan)]!
(go (<! (timeout 2000))!
(throw (Exception. (str "Error fetching movies for actor " name)))!
(>! c (->> actor-movies!
(filter #(= name (:name %)))!
first))!
(close! c))!
c))
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
m-cast (mult cast)!
movies (async-pmap movies-by-actor (tap m-cast (chan)))!
spouses (async-pmap spouse-of (tap m-cast (chan)))!
top-5 (top-5)]!
(prn "Fetching data...")!
(pprint (<!! (go (aggregate-actor-data (<! spouses)!
(<! movies)!
(<! top-5))))))!
!
;; nil - WTF???"

66. Exceptions - workaround
(defn throw-err [e]!
(when (instance? Throwable e) (throw e))!
e)!
!
(defmacro <? [ch]!
`(throw-err (<! ~ch)))!
!
!
(defn movies-by-actor [name]!
(let [c (chan)]!
(go (try (do (<! (timeout 2000))!
(throw (Exception. (str "Error fetching movies for actor " name)))!
(>! c (->> actor-movies!
(filter #(= name (:name %)))!
first))!
(close! c))!
(catch Exception e!
(do (>! c e)!
(close! c)))))!
c))

67. Exceptions - workaround
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
m-cast (mult cast)!
movies (async-pmap movies-by-actor (tap m-cast (chan)))!
spouses (async-pmap spouse-of (tap m-cast (chan)))!
top-5 (top-5)]!
(prn "Fetching data...")!
(pprint (<!! (go (try!
(aggregate-actor-data (<? spouses)!
(<? movies)!
(<? top-5))!
(catch Exception e!
e)))))))

68. A lot of effort for not much benefit

69. Exceptions - the imminent way *
* not really unique to imminent. Other libraries use this same approach
(defn movies-by-actor [name]!
(i/future (do (Thread/sleep 2000)!
(throw (Exception. (str "Error fetching movies for actor
" name)))!
(->> actor-movies!
(filter #(= name (:name %)))!
first))))

70. Exceptions - the imminent way # 1
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (i/flatmap cast #(i/map-future movies-by-actor %))!
spouses (i/flatmap cast #(i/map-future spouse-of %))!
results (i/sequence [spouses movies (top-5)])!
_ (prn "Fetching data...")!
result (deref (i/await results))]!
!
(i/map result #(pprint (apply aggregate-actor-data %)))!
(i/map-failure result #(pprint (str "Oops: " %)))))
;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett

71. Exceptions - the imminent way # 1
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (i/flatmap cast #(i/map-future movies-by-actor %))!
spouses (i/flatmap cast #(i/map-future spouse-of %))!
results (i/sequence [spouses movies (top-5)])!
_ (prn "Fetching data...")!
result (deref (i/await results))]!
!
(i/map result #(pprint (apply aggregate-actor-data %)))!
(i/map-failure result #(pprint (str "Oops: " %)))))
imminent results are themselves functors :)
;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett

72. Exceptions - the imminent way # 2
(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")!
movies (i/flatmap cast #(i/map-future movies-by-actor %))!
spouses (i/flatmap cast #(i/map-future spouse-of %))!
result (i/sequence [spouses movies (top-5)])]!
(prn "Fetching data...")!
(i/on-success result #(prn-to-repl (apply aggregate-actor-data %)))!
(i/on-failure result #(prn-to-repl (str "Oops: " %))))!
;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett

73. Final Thoughts
Category Theory helps you design libraries with higher
code reuse due to well known and understood
abstractions

74. Final Thoughts
Imminent takes advantage of that and provides an
asynchronous and composable Futures library for
Clojure

75. Final Thoughts
Even when compared to core.async, imminent still makes
sense unless you need the low level granularity of
channels and/or coordination via queues

76. Questions?
Leonardo Borges
@leonardo_borges
www.leonardoborges.com
www.thoughtworks.com