Linux con berlin-2016-presentation-deitchera
•
2 likes•803 views
LinuxCon/ContainerCon Berlin 2016 - Networking (Containers) in Ultra-Low-Latency Environments
Recommended
More Related Content
Viewers also liked
Viewers also liked (15)
Similar to Linux con berlin-2016-presentation-deitchera
Similar to Linux con berlin-2016-presentation-deitchera (20)
Recently uploaded
Recently uploaded (20)
Linux con berlin-2016-presentation-deitchera
- 5. Who Am I? • Life in tech business: – 10 yrs large-scale mission-criKcal IT – 10+ yrs consulKng & training – Some startups on the way • Avid (if not very good) ice hockey player • Long-Kme lover of great engineering…. when used to make a real difference • Atomic Inc: – Generalist pracKKoner • Network just one element – Product : engineering : operaKons Avi Deitcher avi@atomicinc.com (not 24601)
- 7. A LiVle History Summer 2015 • Fintech X: “Containerize us!” – Hint: It is harder than you think… and worth it – Culture/process > technology • QuesKon: Networking? • Answer: ScienKfic method Avi Deitcher avi@atomicinc.com
- 8. A LiVle History Summer 2015 • Fintech X: “Containerize us!” – Hint: It is harder than you think… and worth it – Culture/process > technology • QuesKon: Networking? • Answer: ScienKfic method Fall 2016 • Good pracKce demands: 1. Redo tests with new opKons and versions 2. Make tests available 3. Explain it all well Avi Deitcher avi@atomicinc.com
- 19. Our Tests What We Tested • netperf ⇒ netserver • UDP & TCP round-robin • Sizes: 300, 500, 1024, 2048 • No orchestraKon = complete control • 50000 iteraKons – Law of large numbers • Latency (Avg, %iles), CPU • DifferenQals, not absolutes How We Tested • .net – Because it had to be metal – Wicked smart team • Complete test run – Network changes – Hardware variaKons, errors hVps://github.com/deitch/network-tests Avi Deitcher avi@atomicinc.com
- 24. Local Networking Summary • SR-IOV horrible latency but great CPU – Hold that thought… • net=host on par with metal • macvlan closest virtualized to metal • Rest in same range: – Latency: 5-10 𝓊-sec overhead – CPU: negligible difference • Calico (IPIP & naKve) & Docker overlay slightly more performant (margin of error?) • Watch out for very large TCP packets Avi Deitcher avi@atomicinc.com
- 28. Remote Networking Summary • Weave (sleeve) adds latency and CPU – Reason for “fast datapath” • Again, macvlan best virtualized • All the rest: – Latency: within 50 𝓊-sec of each other, except SR- IOV with very large TCP packets – CPU: similar, but keep an eye on Flannel (UDP) Avi Deitcher avi@atomicinc.com
- 30. SR-IOV SR-IOV does not automaQcally mean beWer • Switch in network card • Trades host CPU for card processor • Quality varies drama3cally – Even Mellanox far worse locally • My 2€: SR-IOV falls further behind due to: – Speed of iteraKon – Open-source – Sosware + CPU Avi Deitcher avi@atomicinc.com
- 31. What else could we do? Ø Other hardware types Ø Other network fabrics Ø Other network overlay versions (we have the data…) Ø Docker macvlan network driver Ø ipvlan Ø IPv6 Ø Kernel and network stack tuning Ø Distant networks Ø Other traffic paVerns (mulKcast vs unicast) Ø Other host-to-host encrypKon Ø Other kernel versions Ø Other OSes (Illumos-based?) Ø A whole lot more… Avi Deitcher avi@atomicinc.com
- 32. Headaches (and Thanks) • Headaches – Weave SYN-(nothing) – etcd is “touchy” – Packet L3 network is powerful but… unique • Macvlan, weave, flannel: all required pings for mac • Sexng up bridge w/o NAT, Calico, macvlan was “different” – SR-IOV is complicated and flaky, especially Mellanox – netperf with UDP packets can get stuck (Calico-ipip) – And a whole lot more (ask me offline) • And thanks: – Bryan Boreham, Adam Harrison at weave.works – Zac Smith, Adam, Aaron, Andy, Lucas, everyone at Packet Avi Deitcher avi@atomicinc.com
- 33. Conclusions • SR-IOV: most of the Kme, just not worth it • Performance: – Metal (+ net=host): always performs best – Direct network++: macvlan is your friend – Others: Roughly similar, careful of Weave (sleeve) • What’s your use case? – ULL: Metal/net=host > macvlan > calico > overlay – Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple Avi Deitcher avi@atomicinc.com
- 34. Conclusions • SR-IOV: most of the Kme, just not worth it • Performance: – Metal (+ net=host): always performs best – Direct network++: macvlan is your friend – Others: Roughly similar, careful of Weave (sleeve) • What’s your use case? – ULL: Metal/net=host > macvlan > calico > overlay – Everything else: Focus on your architecture and skills Pick intelligently: easier, not simple Avi Deitcher avi@atomicinc.com