There are plenty of patterns and literature about how to organize systems in traditional languages like Java or C#. The same isn’t exactly true for functional programming languages, specially Clojure. At Nubank we had to figure it out over 3 years of writing tens of microservices in Clojure, using some of our knowledge from other languages and some creativity. Several patterns emerged, several mistakes were made and eventually we came up with a very sustainable and scalable way to write new code even for developers which are new to Clojure. This talk will explore those learnings. Presented at Clojure Remote 2017

1. Lucas Cavalcanti
@lucascs
Microservices in
Clojure

2. Context
Microservices
~80 Clojure services
~60 engineers
~10 teams
3.5 years old

3. OOP
Objects, the mainstream abstraction
Image @ http://www.eduardopires.net.br/2015/01/solid-teoria-e-pratica/

4. What about Functional Programming?
SÃO PAULO, BRASIL

5. TABLE OF CONTENTS
Immutability
Components
Pure Functions
Schemas
Ports and Adapters
SÃO PAULO, BRASIL

6. Immutability
SOUTHEAST BRAZIL REGION FROM SPACE

7. Immutability
Definition
“If I’m given a value, it’s guaranteed that it won’t ever change”

8. Technology choices
Immutability

9. Clojure
Immutability
All default data structures are immutable:
-Maps, Lists, Sets
-Records
Mutability is explicit: atoms/refs @, dynamic vars *…*

10. Datomic
Immutability
Datomic stores the changes/transactions, not just the
data
-append only
-db as a value
-everything is data (transaction, schema, entities)

11. Kafka
Immutability
Persistent Queues/Topics
-each consumer has its offset
-ability to replay messages

12. AWS + Docker
Immutability
Ability to spin machines with a given image/
configuration
-Each build generates a docker image
-Each deploy spins a new machine with the new
version
-As soon as the new version is healthy, old version is
killed. (blue-green deployment)

13. Components
SOUTHEAST BRAZIL REGION FROM SPACE

14. Components
https://github.com/stuartsierra/component
(defprotocol Database
(query [this query-str]))
(defrecord SomeDatabase [config-1 config-2 other-components]
component/Lifecycle
(start [this]
(assoc this :connection (connect! config-1 config-2 other-components)))
(stop [this]
(release! (:connection this))
(dissoc this :connection))
Database
(query [this query-str] (do-query! (:connection this) query-str)))

15. System map
Components
{:database #SomeDatabase{...}
:http-client #HttpClient{...}
:kafka #Kafka{...}
:auth #AuthCredentials{...}
...}
-Created at startup
-Entrypoints (e.g http server or kafka consumers) have access to all
components the business flows need
-dependencies of a given flow are threaded from the entry point until
the end, one by one if possible
-Thus no static access to system map! (e.g via a global atom)
-Any resemblance to objects and classes is just coincidence ;)

16. Pure functions
SOUTHEAST BRAZIL REGION FROM SPACE

17. Pure functions
Definition
"Given the same inputs, it will always produce the same output"

18. Simplicity
Pure functions
-easier to reason about, fewer moving pieces
-easier to test, less need for mocking values
-parallelizable by default, no need for locks or STMs

19. Datomic
Pure functions
-Datomic’s db as a value allows us to consider a function
that queries the database as a pure function
-db is a snapshot of the database at a certain point in time.
-So, querying the same db instance will always produce the
same result

20. Impure functions
Pure functions
-functions that produce side effects should be marked as
such. We use `!` at the end.
-split code which handles and transforms data from code
that handles side effects
-should be moved to the borders of the flow, if possible
-Consider returning a future/promise like value, so side
effect results can be composed (e.g with manifold or
finagle)
https://github.com/ztellman/manifold
https://github.com/twitter/finagle

21. Schema/Spec
SOUTHEAST BRAZIL REGION FROM SPACE

22. Schema
Legacy
Majority of our code base was written before clojure.spec existed,
so I’ll be talking about the Schema library instead. Most principles
apply to clojure.spec as well.

23. Schema/Spec
Documentation
-Clojure doesn’t force you to write types
-parameter names are not enough
-declaring types helps a lot when glancing at the function
-values can be verified against a schema

24. Function declaration
Schema/spec
-All pure functions declare schemas for parameters and
return value
-All impure functions declare for parameters and don’t
declare output type if it’s not relevant.
-Validated at runtime in dev/test environments, on every
function call
-Validation is off on production.

25. Wire formats
Schema/Spec
-Internal schemas are your domain models
-Wire schemas are how you expose data to other services/
clients
-If they are different, you can evolve internal schemas
without breaking clients
-Need an adapter layer
-wire schemas are always validated on entry/exit points,
specially in production
-single repository for all wire schemas (for all 60+ services)
-caveat: this repository has a really high churn. Beware

26. Growing Schemas
Spec-ulation
Please watch Rich Hickey’s talk at Clojure Conj 2016
Spec-ulation:
https://www.youtube.com/watch?v=oyLBGkS5ICk

27. Ports and Adapters
(a.k.a Hexagonal Architecture)
SOUTHEAST BRAZIL REGION FROM SPACE

28. Ports and Adapters
Definition
Core logic is independent to how we can call it (yellow)
A port is an entry-point of the application (blue)
An adapter is the bridge between a port and the core logic (red)
http://www.dossier-andreas.net/software_architecture/ports_and_adapters.html
http://alistair.cockburn.us/Hexagonal+architecture

29. Ports and Adapters (Nubank version)
Extended Definition
Pure business logic (green)
Controller logic wires the flow between the ports (yellow)
A port is an entry-point of the application (blue)
An adapter is the bridge between a port and the core logic (red)

30. Ports (Components)
Ports and Adapters
-Ports are initialised at startup
-Each port has a corresponding
component
-Serializes data to a transport
format (e.g JSON, Transit)
-Usually library code shared by
all services
-Tested via integration tests
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail

31. Adapters (Diplomat)
Ports and Adapters
-Adapters are the interface to
ports
-Contain HTTP and Kafka
consumer handlers
-Adapt wire schema to
internal schema
-Calls and is called by
controller functions
-Tested with fake versions of
the port components, or
mocks
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail

32. Controllers
Ports and Adapters
-Controllers wires the flow
between entry-point and the
side effects
-Only deals with internal
schemas
-Delegates business logic to
pure functions
-Composes side effect results
-Tested mostly with mocks
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail

33. Business Logic
Ports and Adapters
-Handles and transforms
immutable data
-Pure functions
-Best place to enforce
invariants and type checks
(e.g using clojure.spec)
-Can be tested using
generative testing
-Should be the largest part of
the application
HTTP
Kafka
Datomic
File Storage
Metrics
E-mail

34. Microservices
Ports and Adapters
-Each service follows about
the same design
-Services communicate with
each other using one of the
ports (e.g HTTP or Kafka)
-Services DON’T share
databases
-HTTP responses contain
hypermedia, so we can
replace a service without
having to change clients
-Tested with end to end tests,
with all services deployed

35. Clojure is simple
Keep your design simple
Keep your architecture simple
SÃO PAULO, BRASIL

36. Lucas Cavalcanti
@lucascs
Thank you