Slides from the online Picnic meetup of April 23rd 2020: https://www.meetup.com/Meetinup-Picnic/events/269970359/

1. Phil Leonard / Sander Mak
Scalable Application Development
@phillyleonard @sander_mak

2. 2 | Confidential | 07 November
2018
The Picnic Application Landscape1
Observability2
Lessons Learned3
Agenda

3. 3 | Confidential | 07 November
2018
Bring your questions!
menti.com
code: 91 76 42

4. 4 | Confidential | 07 November
2018
The Picnic Application Landscape1
Observability2
Lessons Learned3
Agenda

5. 5 | Confidential | 07 November
2018
Services
The Picnic Application Landscape
Storefront
Stock
Orders
Catalog
Finance
...
...

6. 6 | Confidential | 07 November
2018
Tech Stack
The Picnic Application Landscape
Services
▪ Java 11
▪ Spring Boot
▪ WebFlux/RxJava
1
Platform3
Data2
▪ MongoDB (Atlas)
▪ Postgres (Amazon RDS)
▪ RabbitMQ (CloudAMQP)
▪ AWS
▪ Kubernetes (EKS)

7. 7 | Confidential | 07 November
2018
The Picnic Application Landscape1
Observability2
Lessons Learned3
Agenda

8. 8 | Confidential | 07 November
2018
Monitoring vs Observability: what is the difference?
Observability
Monitoring is the tool,
observability is the measure.

9. 9 | Confidential | 07 November
2018
Logs, Traces and Metrics.
3 Pillars of Observability
Logs, metrics, and traces are the sources of information that
make systems more observable.
Scalyr
TracesLogs Metrics

10. 10 | Confidential | 07 November
2018
SLF4J for Metrics
Micrometer
GrafanaPrometheus

11. 11 | Confidential | 07 November
2018
Increasing visibility and therefore observability
Micrometer Instrumentation
● Apache HTTP client connection pools
● Mongo client connection pools
● Postgres connection pools
● Reactive schedulers
● jOOQ instrumentation
Where is my bottleneck? Homogeneous service instrumentation:
Picnic application domain specific metrics:
● Service KPIs
● Service health

12. 12 | Confidential | 07 November
2018

13. 13 | Confidential | 07 November
2018
The Picnic Application Landscape1
Observability2
Lessons Learned3
Agenda

14. 14 | Confidential | 07 November
2018
Slow, but not slow?
Internal HTTP Service Calls
✅ ✅
Hypothesis: resource contention on
HttpClient connection pool?
Similar issue with Mongo connection pool

15. 15 | Confidential | 07 November
2018
The best HTTP call is the one you never make
Reducing Service Dependencies
●
● Finding duplicate calls during request lifecycle
✅ ✅ ✅
?
● Listening to events rather than calling for
information
● Caching results

16. 16 | Confidential | 07 November
2018
Thundering herd problem
Caching
● In-flight caching
● Pre-loaded/polling caches

17. 17 | Confidential | 07 November
2018
MongoDB: Approaching Physical Limits
Peak load of 40k queries per second

18. 18 | Confidential | 07 November
2018
Solutions
MongoDB: Optimisations
○ Read from secondaries, where you can.
○ Reduce bytes on the wire, .projection() and don’t
always read-your-writes
○ Splitting into service-based Mongo clusters.
Mongomirror
● 1.5 TB+* data mirrored,
● 9 new clusters,
● and two late night
migrations later....
* uncompressed

19. 19 | Confidential | 07 November
2018
MongoDB: Non Paintbrush Style Optimisations

20. 20 | Confidential | 07 November
2018
Building the best milkman on
earth serving millions of families
Climbing a mountain
One step at a time
https://join.picnic.app