Our journey and experience in dealing with the collection/analysis of Continuous Delivery log events using Gerrit Code Review, Jenkins with Apache Flume, ElasticSearch, Kibana and Spark

1. @gitenterprise#DevoxxPL
Platinum Sponsors:
Speed-up your Continuous Delivery
with BigData
Luca Milanesio
GerritForge

2. @gitenterprise#DevoxxPL
• Founded in the UK
• HQ in London, Offices in
the UK and Italy
• Committed to
OpenSource
About GerritForge

3. @gitenterprise#DevoxxPL
• Co-founder and
Director of GerritForge
• over 20 years in Agile
Development and ALM
• Contributor to many
OpenSource projects
• Mind and hands of
@gitenterprise and
@gerritreview
About Luca Milanesio

4. @gitenterprise#DevoxxPL
NOT a lecture on Continuous Delivery
NOT a lecture on Code Review
NOT a lecture on BigData
What NOT to expect?

5. @gitenterprise#DevoxxPL
SO WHAT?
#!@$!

6. @gitenterprise#DevoxxPL
SHOW
a scenario of Continuous Delivery

7. @gitenterprise#DevoxxPL
LEARN
together about the problems

8. @gitenterprise#DevoxxPL
EXPERIMENT
my approach leveraging BigData

9. @gitenterprise#DevoxxPL
DISCUSS
and evolve the approach together

10. @gitenterprise#DevoxxPL
Continuous Delivery Pipeline
Push Build
Unit Tests
Push Build
Unit Tests
Automated
Tests
Push Build
Unit Tests
Automated
Tests
Beta Users
Tests
User's feedback Publish to all Users
Failed
Failed

11. @gitenterprise#DevoxxPL
Code Review & CD Validation
A1
refs/for/master
master
C2 C2*
A1 C2 (local)
master
C2*
push
Verified: -1
(build failed)
push
Verified: +1
(build OK)
Reviewed: +2
CI Build: OK
C2*
CI Build: OK
amend

12. @gitenterprise#DevoxxPL
Git/Gerrit
(Code Review)
Jenkins
(Continuous
Integration & Delivery)
Docker
(Release and Service
Packaging)
Mesos &
Marathon
(Delivery and Resource
Manangement)
Components

13. @gitenterprise#DevoxxPL
DEMO
See the pipeline in action

14. @gitenterprise#DevoxxPL
We use micro-services
• 100+ Components
• 200+ Repositories
• 300+ People and 100,000+ of review comments
• Cross-components [implicit] dependencies
• Lots of build chains
• 10,000+ of builds
Now feel the pain :-(

15. @gitenterprise#DevoxxPL
"It was working until 5
mins ago … then
someone changed
something"
 Rings any
bell?
Problem #1 : Ripple effect

16. @gitenterprise#DevoxxPL
In order to deliver Feature ABC
I NEED
Ver. N of Service-A with
Ver. M of Service-B and
Ver. K of Service C
Problem #2 : Dependencies

17. @gitenterprise#DevoxxPL
"Where can I find an environment
to
run ALL my Micro-Services
integration tests?"
Problem #3 : Resources

18. @gitenterprise#DevoxxPL
Release small changes is GOOD
Release often every change is GOOD
BUT
How long it takes the "Big Picture" and End-to-End?
Where is my bottleneck?
Problem #4 : Cycle time

19. @gitenterprise#DevoxxPL
Micro-Services are
distributed
Builds are distributed
Deployments are
distributed
 Tackling THIS
have you ever felt
like drowning in
your logs?
Problem #5 : Lots of logs

20. @gitenterprise#DevoxxPL
Problems #1 + #2 + #3 + #4 + #5 =
lead to your daily
Micro-Services mental meltdown
Problem #6 : all together

21. @gitenterprise#DevoxxPL
• Collect all review events
• Collect all logs
• Channel them to a
central store
• Crunch and Crunch
continuously
• Never delete
• Process, inspect and
learn
BigData to the rescue

22. @gitenterprise#DevoxxPL
DEMO
Continuous Delivery Analytics

23. @gitenterprise#DevoxxPL
•Problem #1 – Ripple effect?
•Problem #2 – Dependencies?
•Problem #3 – Resources?
•Problem #4 – Cycle Time?
•Problem #5 – Logs aggregation?
•Problem #6 – Avoid mental meltdown?
Did we solve the problems?

24. @gitenterprise#DevoxxPL
• Collect more data (System, JVM, Review
scores)
• Real-time statistics
• Events correlation
• Feedback loop to Code-Review
• Production data harvest
• More?
Future directions

25. @gitenterprise#DevoxxPL
• My blog
www.gitenterprise.me
• Gerrit Code Review
www.gerritcodereview.co
m
• Jenkins
www.jenkins.io
• RabbitMQ
rabbitmq.com
• Apache Flume
flume.apache.org
• Hadoop HDFS
hadoop.apache.org
• Apache Spark
spark.apache.org
• Kibana
github.com/elastic/kibana
Resources