2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report

@RealGeneKim
Session ID:
Gene Kim
My Love Letter to Clojure:
And A Datomic Experience Report

@RealGeneKim
Why I’m Presenting and Who It’s For
 Alex Miller emailed me if I’d be willing:
 Share why I’m so grateful to Clojure
 Present an experience report on Datomic
 Intended audience:
 People who have been afraid to jump into Datomic,
because it’s so alien and unfamiliar (perhaps even
more alien than Clojure!)
 People who want to spread more widely the use of
Datomic, or make it easier to explain/adopt/use — I’m
part of that target market

@RealGeneKim
Definition: Experience Report
 Noun
 “Experience report is a paper written by a person (persons)
who participated in the work completed (being completed) in
a context of a real-life organization”
 In the form of: here’s what I wanted to do, here’s what
happened, here’s what I learned, here’s problem that still
remains
 What’s they’re also good for
 Closely resemble the scientific method (hypothesis, test
hypothesis)
 Helps uncover demographics and psychographics (goals
and ideals) of a population
Source: http://processplatsen.ibissoft.se/node/72

@RealGeneKim
Agenda
 Who am I, what I’ve learned in the DevOps community,
and why I’m so grateful to Clojure and the Clojure
community (15m)
 Datomic Experience Report (25m)
 What I’ve used Clojure for
 Why Datomic scared me away
 What made me finally take the plunge
 What it felt like to solve the problem
 Help I’m Looking For (5m)

@RealGeneKimSource: https://itrevolution.com/love-letter-to-clojure-part-1/

@RealGeneKim
My Definition of DevOps
The architecture, technical practices, and cultural norms
that enable us to…
increase our ability to deliver applications and services...
quickly and safely, which enables rapid experimentation
and innovation, and the fastest delivery of value to our
customers…
while ensuring world-class security, reliability, and stability...
…so that we can win in the marketplace.

@RealGeneKim
Another Definition of DevOps
Better Value,
Sooner, Safer, Happier
Jon Smart, Partner, Enterprise Agility, Deloitte

@RealGeneKim
State of DevOps Research
 State of DevOps Report (2013-2019)
 Dr. Nicole Forsgren, Jez Humble, Gene
Kim
 Cross population study spanning over 35K
respondents
 Identified “IT performance” and the factors
that predicts:
 Deployment Frequency
 Deployment Lead Time
 Deploy Success Rate
 Mean Time to RestoreSource: Google/DORA: 2018 State Of DevOps Report:
https://cloudplatformonline.com/2018-state-of-devops.html

@RealGeneKim
Elite Low Difference
Deployment Frequency
On-demand
(multiple times per day)
Monthly or quarterly 208x
Deployment Lead Time < 1 hour 1 day to 1 week 2,555x
Deploy Success Rate 0-15% 46-60% 7x
Mean Time to Restore < 1 hour 1 week to 1 month 2,604x
Elite vs. Low Performers
Source: Google/DORA: 2019 State Of DevOps Report: https://cloud.google.com/devops/state-of-devops/

@RealGeneKim
On-demand
Deployment Lead Time < 1 hour 1 week to 1 month 106x
Mean Time to Restore < 1 hour 1 week to 1 month 2,604x

@RealGeneKim
On-demand
Deployment Lead Time < 1 hour 1 week to 1 month 106x
Mean Time to Restore < 1 hour Less than one day 2,604x

@RealGeneKim
High Performers Are More Secure And
Controlled
2x 29%
less time spent
remediating
security issues
more time spent
on new work
Source: Google/DORA: 2018 State Of DevOps Report: https://cloudplatformonline.com/2018-state-of-devops.html

@RealGeneKim
High Performers Win In The Marketplace
2x 2xmore likely to
exceed profitability,
market share &
productivity goals
more likely to achieve
organizational and
mission goals, customer
satisfaction, quantity &
quality goals

@RealGeneKim
High Performers Win In The Marketplace
2.2xhigher employee
Net Promoter Score
50%higher market
capitalization growth
over 3 years*

@RealGeneKim
The Opposite Of
Technical Debt Is…

@RealGeneKim
When we can safely, quickly,
reliably, securely (and happily)
achieve all the goals, dreams and
aspirations of our business…

@RealGeneKim
Clojure Brought Joy Of Coding Back
 For decades, I self-identified as an Ops person…
 I now self-identify as a developer
 Development is so fun, and these days, you can do
miraculous things with so little effort
 I wrote about my 30 year habits of self-sabotaging
my own code. One app I’ve written three times:
 3000 lines of Objective C
 1500 lines of TypeScript/React
 500 lines of ClojureScript/Re-frame

@RealGeneKim
Cumulative LoC Written: Three Decades
“I’m going to make the case
that me learning Clojure is
sort of like Encino Man
learning to drive—you know,
the movie about the
caveman who was frozen in
ice and wakes up in the
completely changed,
modern world.”
Source: https://itrevolution.com/love-letter-to-clojure-part-1/
Little coding

@RealGeneKim
The Problems That Still Remain
 Absence of all the invisible structures needed to
enable developer productivity
 The orthogonal problem of getting data from
where it resides to where it needs to be used
 Strong opposition to support new ways of
working
 Ambiguity on what behaviors needed to support
during a transformation

@RealGeneKim
Rich Hickey (STL 2011, JavaOne 2015)

@RealGeneKim
The Five Ideals
1. Locality and Simplicity
2. Focus, Flow, and Joy
3. Improvement of Daily Work
4. Psychological Safety
5. Customer Focus

@RealGeneKim
Session ID:
Ideal #1:
Locality and Simplicity

@RealGeneKim
The First Ideal: A Measure
 Bus factor
 Lunch factor

@RealGeneKim
How Many People Do You Need To Feed?
 Two pizza team
 Feeding everyone in the building
 Schedule lunch with 43 different people

@RealGeneKim
The First Ideal: Code
 Ideal: anyone can implement what they need by
looking at one file or module, and make the
needed change
 Kubernetes sidecars
 Spring (http-retry, Dependency Injection)
 Aspect Oriented Programming
 Not Ideal: to make your needed change, you
have to understand and change all the files and
modules

@RealGeneKim
The First Ideal: Code
 Ideal: changes can be independently
implemented and tested, isolated from other
components (composability)
 Not Ideal: in order for changes to be
implemented and tested, the entire system must
be present (e.g., integrated test environment)

@RealGeneKim
The First Ideal: Organization
 Ideal: every team has the expertise, capability
and authority to satisfy customer needs
 Not Ideal: in order to satisfy customer needs,
every team must escalate up two levels (and over
two, and down two)

@RealGeneKimSource: Manu Cornet: Bonkersworld

@RealGeneKim
Session ID:
Ideal #2:
Focus, Flow, and Joy

@RealGeneKim
Why Functional Programming
 The famous French philosopher Claude Lévi-
Strauss would say of certain tools, ‘is it good to
think with?’

@RealGeneKim
Interestingly, It Portends Future Of Ops
 Core concepts
 Immutability
 Pure functions
 Composability
 Look at…
 Docker, Docker Compose
 Kubernetes
 Kubernetes sidecars
 Event streams: Apache Kafka
 Git

@RealGeneKim
The Second Ideal: Focus and Flow
 Ideal: your energy and time is focused on solving
the business problem, and you’re having fun
 Not Ideal: all your time is spent trying to solve
problems you don’t even want to solve (e.g.,
YAML files, Makefile and spaces in filenames,
bash)

@RealGeneKim
Never Have I Valued Infrastructure More
 Things I detest now
 Everything outside of my application
 Connecting anything to anything
 Updating dependencies
 Secrets management
 Bash
 YAML
 Patching
 Building kubernetes deployment files (mostly by Googling)
 Why my cloud costs are so high

@RealGeneKim
The Value Of Platforms
 Enable developer productivity
 Self-service
 On-demand
 Immediacy and fast feedback
 Focus and flow
 Joy
 Monitoring, deployment, environment creation,
security scans, orchestration…

@RealGeneKim
Two Types Of Learning
● Procedural Learning
● One-shot Learning
Source: Dr. Mihaly Csikszentmihalyi

@RealGeneKim
In Clojure Love Letter
 I want to tell a story that has the same ending as
Rich Hickey’s “If I had to keep programming in
C++ and Java, I think I would need to find a new
career.”
 I’ve heard similar things many times…
 I wondered if they were a bit… Hysterical? Manic?
Melodramatic?

@RealGeneKim
 Rich Hickey
 “Solve problems, not solve puzzles”
 Category theory
 Me
 Figuring out how to escape spaces in filenames inside
of Makefile
 SQL
More Examples

@RealGeneKim
Session ID:
Experience Report:
Or When I Decided I Was Done
With SQL *

@RealGeneKim
A Book Tracking Application
 Four years ago, Tom Limoncelli (StackOverflow) and
I wrote some tools to track sales ranks from various
e-commerce sites:
 Python
 Ruby and ActiveRecord
 MySQL
 Some bash, Makefiles, and JSON files
 Kubernetes
 Cron (or K8S CronJobs or Rundeck)

@RealGeneKim
E-commerce
Site 1
E-commerce
Site 1
2015
User
Google
StackDriver
E-commerce
Sites...
API Retrieve

@RealGeneKim
E-commerce
Site 1
E-commerce
Site 1
2016-2017
API Retrieve
ActiveRecord/
MySQL
Dispatcher
User
Google
StackDriver
Slack
Hosted
Graphite
E-commerce
Sites...
Cloud SQL
MySQL
Bash script

@RealGeneKim
API Retrieve
ActiveRecord/
MySQL
Dispatcher
E-commerce
Site 1
E-commerce
Site 1
2016
User
Google
StackDriver
Hosted
Graphite
Slack
Hosted
Graphite
E-commerce
Sites...
Bash script
- Program error?
- Environment error?
- Interface error?
- 503 enforcement?
Cloud SQL
MySQL

@RealGeneKim
3GB Docker Image / Kubernetes (GKE)
E-commerce
Site 1
E-commerce
Site 1
2016-2018
User
Hosted
Graphite
Slack
Hosted
Graphite
E-commerce
Sites...
Cloud SQL
MySQL
Bash script
API Retrieve
Get Reviews
(Scrapy)
ActiveRecord/
MySQL
Dispatcher

@RealGeneKim
100MB Docker Image / Kubernetes (GKE)
E-commerce
Site 1
E-commerce
Site 1
2019: 10x LoC
Reduction
API Retrieve
Dispatcher
User
Slack
E-commerce
Sites...
Cloud SQL
MySQL
Google Pub/Sub
Get Reviews
Google Data
Studio

@RealGeneKim
Google Data Studio Is Amazing

@RealGeneKim
Sales Rank (lower is better)
Unicorn Project: Kindle
Phoenix Project: Kindle
Yay! Promotion activities are working!

@RealGeneKim
But I’m Not Very Good At Databases…
WHAT HAPPENED?!?
Oops. 2 weeks of all new records with NULL date/times.
ARGH! Analyzing this critical launch period is why I wrote the app in the first
place, and I was totally blind to what happened!!!

@RealGeneKim
How Bad Am I At Databases
 I wrote an SQL database as an undergraduate CS
student
 I’ve spent time as an (unwilling) Oracle DBA
 I’ve used MySQL for 15 years
 I’m still scared of schema changes and database
migrations
 I almost never get database schema changes right
 (insert screenshot of 10 git commits to fix Ruby
ActiveRecord migrations)

@RealGeneKim
I am afraid of MySQL
because I’m so bad at it

@RealGeneKim
What Is The One Question That
Predicts Performance With
Startling Accuracy?

@RealGeneKim
“To what degree do we fear
doing deployments?”
Source: Puppet Labs 2015 State Of DevOps: https://puppetlabs.com/2015-devops-report

@RealGeneKim
My Final MySQL Indignity
 This query takes 8.1s!!!
 I need this for each book I’m
tracking — for 150 books, that’s
2 minutes!
 No problem, I’ll just run queries on a
background thread…
 Or maybe cache results, or use
Firebase Realtime DB? (and now I
have two problems)

@RealGeneKim
Source: https://twitter.com/b0rk/status/1179054952938913794

@RealGeneKim

@RealGeneKim
Adding Indices Did Not Work
 CREATE INDEX time ON salesranks(time);
 CREATE INDEX book_id ON
salesranks(book_id);
 CREATE INDEX id ON books(id);
 ALTER TABLE salesranks
MODIFY COLUMN
time DATETIME NOT NULL;

@RealGeneKim
😱 😱 😱 😱

@RealGeneKim
😱 😱 😱 😱 😱 😱

@RealGeneKim
🤮🤮🤮 When I Knew I Was Done With
MySQL (Because I’m So Bad At It)
BEFORE: 8,100 ms

@RealGeneKim
New Rule:
No More SQL, Because
I Am Too Stupid To Use It…
(And NoSQL, Too…)

@RealGeneKim
Clojure helped me realized the
Second Ideal
MySQL deprived me of it
(Because I’m so bad at it)

@RealGeneKim
Story Line
 For 2 years, I’ve wanted to extend database
schema to encompass notion of “publisher” —
but conclusion was always “maybe next year”
 Datomic looked so interesting: kept reading and
watching videos

@RealGeneKim
Great Articles on Datomic
 Carin Meier:
Conversations With
Datomic (Part 1 and 2)
 Datalog is intimidating,
but its link to core.logic
and Prolog link made it
seem incredibly
worthwhile
 I’m seeking a more
declarative mode of
thinking…
 OTOH: Schema was…
whoa.. Difficult for me to
understand…

@RealGeneKim
Book Publisher: But Never Used In Anger

@RealGeneKim
Definition: “Use In Anger”
 “To use in anger”:
“In a military context, 'to fire in anger' means to
shoot for a purpose in war. For example, a
submarine that 'fires in anger' shoots missiles at
an enemy ship.
“Shots fired in anger are never just for practice;
they're fired to deliberately cause damage or
harm.”
Source: http://www.bbc.co.uk/worldservice/learningenglish/radio/specials/1837_aae/page22.shtml

@RealGeneKim
Definition: “Use In Anger”
 “To use in anger”:
“In a military context, 'to fire in anger' means to
shoot for a purpose in war. For example, a
submarine that 'fires in anger' shoots missiles at
an enemy ship.
“Shots fired in anger are never just for practice;
they're fired to achieve a real mission
objective.”
Source: http://www.bbc.co.uk/worldservice/learningenglish/radio/specials/1837_aae/page22.shtml

@RealGeneKim
An Urgent Mission: Use In Anger: Book Launch
 Book and product launches are a game of activations
 In the Twitter relationship graph around me, where were
points of maximum amplification?
 Among my friends, who had the most friends and followers?
 Among my followers, who had the most friends and followers?
 For a set of people, who were common friends and followers?
 I looked for many weeks for commercial offerings, but
didn’t find any — often after a week of waiting before my
account became fully activated.
 More on this later…

@RealGeneKim
Initial Challenges And Intimidating Things
 Okay, I think I can do this, with Hodur helping create
schemas
 Datomic Cloud made it feasible — my one attempt to
install Datomic and license 1.5 years ago was not so
good. I never got past the first page. (It didn’t help
that I didn’t even know what a classpath was.)
 I’ve never used anything in AWS besides EC2.
Involved using things I’ve never used (DynamoDB,
API Gateway, AWS Lambda, CloudWatch)

@RealGeneKim
This Project Was Important To Me
 I usually love experience reports that describe “here’s
how we scaled our multiuser chat service to 200MM
simultaneous users overnight without a hitch”
 My experience report is different — it was just for me,
it never even ran anywhere besides my laptop
 It was important to me and time-critical
 I searched weeks for a tool that could do it for me
 It completely changed how I think about databases, and
what I expect for one

@RealGeneKim
Most Amazing Thing Ever

@RealGeneKim
4 Chapters On Datomic
 Jeremy Anderson
 Michael Gaare
 Justin Holguín
 Nick Bailey
 Timothy Pratley

@RealGeneKim
Schema Changes Were So Easy…

@RealGeneKim
First Goals
 Used chbrown’s fantastic “twttr” library to access Twitter API
 Retrieve details for a Twitter account
 Retrieve details about all my friends and followers
 Uh oh… Rate limits…
 Final N = 500K+
 Only 15 requests allowed every 15 minutes (200 users returned upon each request)
 It took 96 hours (4 days), but this was unknowable until program finally finished running
 I remember writing/running programs like this; so many things can go wrong
 Inability to resume after unexpected program termination
 Unwanted data duplication
 Unwanted retrieval duplication And THIS is why it takes multiple days for
social media tools to work after sign-up

@RealGeneKim
Calling Twitter API w/Rate Limits

@RealGeneKim
Initial Challenges
 Datomic Cloud made it even feasible — my one
attempt to install Datomic and license 1.5 years
ago was not so good. I completed only first 3
steps
 I’ve never used anything in AWS besides EC2.
Involved using things I’ve never used
(DynamoDB, API Gateway, AWS Lambda,
CloudWatch)
 Video tutorials were great

@RealGeneKim
SUCCESS
 In a week, had collected 500K users, able to
execute all desired queries
 Focused on the top people who have got the
word out about DevOps and Phoenix Project
 Copied into Google Sheet and operationalized

@RealGeneKim
Problems Encountered

@RealGeneKim
Problems Encountered: Out of Memory
 String case insensitive comparison consumed all
memory (remember, I never really used Java)

@RealGeneKim
AWS CloudWatch: what is all this?!

@RealGeneKim
Observations
 Queries were so easy
 It felt like the database was part of my program
 Immediacy and fast feedback was incredible
 Schema changes were incredibly easy and
painless
 Hodur schema changes are amazing
 Ghostwheel for type checking so incredible

@RealGeneKim
If I Could Wave A Magic Wand
 Help upgrading (I’m so scared)
 Better understanding what all the CloudWatch
events mean and whether I should care
 Ideal use case
 Access it from apps in Heroku
 Not have to make change API Gateway each time
 (I got one set of Datomic calls working in Heroku,
thanks to Clojurians Slack)

@RealGeneKim
Exposing Datomic as JDBC: Whoa…
 This would be
amazing
 I could connect
Datomic into Google
Data Studio via JDBC

@RealGeneKim
Dr. John Launchbury
 One of earliest collaborators w/Dr. Simon Peyton-
Jones on Glasgow Haskell Compiler (ghc)

@RealGeneKim
 Lessons
 Define inputs
and output types
 Define what your
terminating case
 Use
gnl/ghostwheel

@RealGeneKim
Session ID:
The Opportunity: Study the Java Maven Ecosystem!

@RealGeneKim
Session ID:
The Opportunity: Study the Java Maven Ecosystem!
Clojure
Haskell

@RealGeneKim
Dr. Stephen Magill (Galois)
Gene Kim (IT Revolution)
Bruce Mayhew (Sonatype)
Gazi Mahmud (Sonatype)
Brian Fox (Sonatype, Maven
Co-Inventor)
Thanks also to:
Kevin Witten, Derek Weeks,
and Matt Howard

@RealGeneKim
2020 goals
 Use with SSC
 Describe project and data set sizes
 100MM records
 Components, dependencies, updates, and
vulnerabilities

@RealGeneKim
Dr. Stephen Magill’s Endless SQL

@RealGeneKim
Quote
 "If you think that ‘software project’ is
pronounced ‘slow-moving train wreck,’ you
should read The Unicorn Project. Gene Kim
shares the hilarious and sad ways that people
are ground down by failing projects -- and he
has a proven plan to fix it."
-- Stuart Halloway, President, Cognitect, Inc.
 “This book describes how to rescue
developers from the tundra of technical debt
and bureaucracy that plagues the most
important missions in almost every
enterprise.”
— Eric Normand, PurelyFunctional.tv

@RealGeneKim
In Conclusion
 THANK YOU! Clojure brought the joy of coding
back into my life!
 And Datomic, even though it looks even more
alien than Clojure, made me so dissatisfied with
SQL databases, I wondered how I’d survive
without databases in my life.
And like Clojure, I’m amazed at how much I can
do with Datomic, easier, safer, and happier!

@RealGeneKim
Help I’m Looking For
 Help Datomic win the bake-off for SSC Analysis, and then
help to make sure I don’t screw it up (me being the
Datomic expert on the team seems like a recipe for
disaster)
 I’m interested in analyzing the clojars ecosystem and
data, too.
 I’m interested in wondering what markers there are that
APIs are stable and won’t break consumers.
 I’d love to work with anyone on getting more compile-time
checking of types — spectrum? clj-kondo?

2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report

Similar to 2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report (20)

More from Gene Kim

More from Gene Kim (7)

Recently uploaded

Recently uploaded (20)

2019 12 Clojure/conj: Love Letter To Clojure, and A Datomic Experience Report

Editor's Notes