How to monitor unknown third party code? One of the hardest challenges we face running Clever Cloud, apart from the impressive scale we face with hundreds of new applications per week, is the monitoring of unknown tech stacks. The first goal of rebuilding the monitoring platform was to accommodate the immutable infrastructure pattern that generates lots of ephemeral hosts every minute. The traditional approach is to focus on VMs or hosts, not applications. We needed to shift this into an approach of auto-discovery of metrics to monitor, allowing third party code to publish new items. This talk explains our journey in building Clever Cloud Metrics stack, heavily based on Warp10 (Kafka/Hadoop/Storm based) to deliver developer efficiency and trustability to our clients applications.

1. Monitoring the unknown,
1000*100 series a day
Quentin ADAM - @waxzce
2017

2. Who am I ?
Quentin ADAM from the Clever Cloud
@waxzce on twitter – github- soundcloud – instagram ….

3. My day to day work :
, the IT automation company

4. Keep your apps online. made with node.js,
scala, java, ruby, php, python, go…
Cloud & on premise

5. And learn a lot of things
about your code, apps,
and good/bad design…

6. Give back to the
community
NEVER GONNA
LET YOU DOWN
clever-cloud.com

7. Automate infrastructure

8. What are we talking about?

9. Immutable infrastructure
FR : https://www.youtube.com/watch?v=WrZCbgQsPVU
EN : https://www.infoq.com/presentations/server-cloud

10. Ephemeral instances

11. Pet
!=
Cattle

12. Monitoring service
!=
Monitoring host

13. Monitoring the unknow
Part 1

14. Which tech stack
peoples will use?

15. Monitoring the unknow
Part 2

16. There is no ready solution…

17. Let’s build something

18. Data needs

19. Store measure in time

20. Analyze it
Most of the time using a micro batching approach

21. Traditional needs not required
• No ACID
• No Transaction
• Just a few full text
• Eventual consistency if enough

22. SCALE

23. Time Series DB

24. Influx DB

25. Prometeus

26. Open TSDB

27. Cassandra + Akka persistence/stream

28. Warp10

29. Standalone
Java fat jar

30. Distributed
• Consensus using Zookeepers
• Store and analysis on Hadoop
• Event bus with Kafka
• Several roles in a single JAR, all stateless

31. So, We can now store data

32. Agent: Telegraf

33. Agent goals

34. Collect data
• System
• Many connector to specific softs
• Open protocols…

35. JMX

36. Statsd

37. Prometeus

38. Buffer to send batched data
~100 metric point / minutes

39. Token based security

40. More tokens: JWT and Macaroon
EN : https://www.youtube.com/watch?v=SO5G0Gsm5-w
FR : https://www.youtube.com/watch?v=mvKeCsxGZhE

41. ✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
❔ Analyze data
❔ Trigger data
❔ Display data

42. WarpScript, distributed on hadoop

43. Warp10 roles
• Ingress
• Directory
• Store
• Egress
• Webcall
• Plasma
• Runner
• Fetcher

44. µBatching WarpScriptµBatching WarpScriptµBatching WarpScript

45. Build a new series

46. Listen on Kafka or Plasma for triggers

47. Immutable DataBase

48. ✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
✅ Analyze data
✅ Trigger data
❔ Display data

49. Quantum
Tool
+
PolymerJS lib

50. Quantum
Tool
+
PolymerJS lib

51. Which data expose?

52. Templating the data

53. Templating the data

54. Templating the data

57. Give access
to the API

58. Enhance developer work, happiness and
productivity

59. Give access to metrics and logs

60. Thank you
find me on twitter
@waxzce
Gift coupon for
clever-cloud.com:
devoxxMA2017