ContainerSched 2017 talk

1. Timothy Chen
Co-founder / CTO
Scale Containers Operations
With AIOps

2. About me
2
Apache PMC Mesos, Drill
Previously container runtime lead at Mesosphere
Help community effort and design of Spark on Kubernetes
@tnachen

3. Agenda
3
Challenges running containers at scale
What is AIOps?
How can AIOps help ops?

4. Adoption

5. Infrastructure Maturity

6. Microservices

7. Microservices at scale

8. Open source software

9. Container Production Stack
Orchestration
Security Internal Network Storage
Runtime
OS
VM/HW
External Network

11. Infrastructure Monitoring
Orchestration
Network
Storage
Runtime
OS
Image
Users:
SRE
System Engineers
Ops
DevOps

12. Application Monitoring
Kafka
Microservice #1
Spark
Microservice #2
Microservice #3
Users:
SRE
DevOps
Developers

13. Metrics Explosion

14. Reactive monitoring new challenges
Containers != VM
Cannot focus on host metrics anymore with dynamic container
changes
Which metric(s) to track?
10000s of metrics, massive amounts of alert
How to identify or isolate the problem (correlate metrics)?
HW/Container/Host/Neighbor/Cluster/Dependency….
How to resolve these performance problems? (RCA)
Unmanaged resources -> Manual tuning

15. Proactive challenges
Goal: Maximize performance with minimum cost!
Capacity Planning / What-If Analysis
Cluster scheduling (interference, heterogenous, etc)
Dynamic configuration
-
Orchestration
Network Storage
Runtime
OS
Application
VM

16. Gartner definition
AIOps platforms utilize big data, modern machine learning
and other advanced analytics technologies to directly and
indirectly enhance IT operations (monitoring, automation
and service desk) functions with proactive, personal and
dynamic insight. AIOps platforms enable the concurrent use
of multiple data sources, data collection methods, analytical
(real-time and deep) technologies, and presentation
technologies.

17. https://netman.cs.tsinghua.edu.cn/contacts/projects/

18. AIOps is not a new concept
Decades of academic research around managing
performance in computing….
- Cluster Scheduling
- SLA-driven interference, storage, network, caching...

21. https://www.youtube.com/watch?v=dJxGtfTPVCg

26. The Difficulty of using the Public Cloud
Many instance types
Reserved, on-demand, spot instances
Many instant sizes (10s)
Application churn
Variability in load
Can we guarantee SLOs & minimize cost automatically?

27. Choosing the right VM/HW

28. Exhaustive testing

29. Confidential: do not distribute
Results in a couple hrs, with 20% sampling
Optimal Solution:
Highest Perf / Cost
ratio
...and there is room to further improve on speed & sampling
2nd Highest
Perf / Cost ratio
with higher TPM

30. Performance tuning

32. Live traffic load testing
https://engineering.linkedin.com/blog/2017/02/redliner--how-linkedin-determines-the-capacity-limits-of-its-ser

33. Live traffic bottleneck
https://research.fb.com/publications/kraken-leveraging-live-traf%EF%AC%81c-tests-to-id
entify-and-resolve-resource-utilization-bottlenecks-in-large-scale-web-services/

35. Batch scheduling
Improve deadline by 5x - 13x, increase utilization 14 - 28%

36. Resource selection

38. Variability in user load
Bloated reservations due performance variability
TwitterGoogle

39. Oversubscription
Google search + Google brain (deep neural nets)
>90% hardware utilization
No latency violations for search
[Lo et al’15]
>90% HW utilization No tail latency
problems

41. Oversubscription in Action

42. Hyperpilot
42
Using ML to build the intelligence to
help operators to gain deep insights,
and further enable dynamic infrastructure.
Processors, Memory, Flash, Disk, NIC
Operating system
Cluster resource manager
Distributed
analytics
Hadoop,
Spark, …
IaaS
Online,
data-intensive
(OLDI)
Search, social nets,
SaaS

43. tim@hyperpilot.io
Thank you!