Building a Billion User Load Balancer
•
7 likes•5,630 views
Want to learn how Facebook scales their load balancing infrastructure to support more than 1.3 billion users? We will be revealing the technologies and methods we use to global route and balance Facebook's traffic. The Traffic team at Facebook has built several systems for managing and balancing our site traffic, including both a DNS load balancer and a software load balancer capable of handling several protocols. This talk will focus on these technologies and how they have helped improve user performance, manage capacity, and increase reliability.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Building a Billion User Load Balancer
Similar to Building a Billion User Load Balancer (20)
Recently uploaded
Recently uploaded (20)
Building a Billion User Load Balancer
- 2. Production Engineer, Traffic Team Building a Billion User Load Balancer Patrick Shuff
- 4. We’ll be talking about © 2015 Facebook | Dublin | Credits: network icon by Daniel Gamage https://thenounproject.com/term/network/49138/, tools icon by Kiryl Sytsko https://thenounproject.com/term/tools/122204/, chess icon by Dream Icons, https://thenounproject.com/term/chess/127867/, Paraglider by Matt Brooks https://thenounproject.com/term/paraglider/114410/, Question by Henrik Lund Mikkelsen https://thenounproject.com/term/question/8325/ https://creativecommons.org/licenses/by/3.0/ https://thenounproject.com/term/web-search/18390/ Serving Dynamic Facebook Requests L4/L7 Load Balancing Edge PoP and Reducing Latency Global DNS Load Balancing Q&A
- 5. Traffic @ fb
- 6. Facebook scale as of March 2015 936 million daily active people on average 798 million mobile daily active people on average 1.25 billion mobile monthly active people 1.44 billion monthly active people © 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain
- 7. Facebook scale as of March 2015 936 million daily active people on average 798 million mobile daily active people on average 1.25 billion mobile monthly active people 1.44 billion monthly active people Approximately 82.4% of our daily active users are outside the US and Canada © 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain
- 8. What is facebook? Photos Videos Newsfeed Likes © 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain Dynamic Requests Javascript/CSSStatus Updates Static Requests Messaging (from traffic's perspective)
- 9. What is facebook? Photos Videos Newsfeed Likes © 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain Dynamic Requests Javascript/CSSStatus Updates Static Requests Messaging (from traffic's perspective) Terabits of egress (outgoing bits per second)
- 10. What is facebook? Photos Videos Newsfeed Likes © 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain Dynamic Requests Javascript/CSSStatus Updates Static Requests Messaging (from traffic's perspective)
- 11. © 2015 Facebook | Menlo Park | Credit: What are we talking about? HHVM
- 12. © 2015 Facebook | Menlo Park | Credit: What are we not talking about? HHVM MySQL Cache Msgs Feed Ads
- 14. 7 Days Egress Ingress Weekly egress cycle
- 15. 11 AM 24 hours 3 PM Diurnal egress Cycle Time zone == Pacific (-0800 GMT)
- 16. United Kingdom Canada Indonesia 7 PM 24 hours 9 AM Sum of timezones Time zone == Pacific (-0800 GMT)
- 17. TCP/IP Review
- 18. © 2015 Facebook | Menlo Park | Credit: OSI Model Layer Purpose Ex 7: Application High-Level API HTTP, SPDY, MQTT 6: Presentation Data Translation ASCII, JPEG 5: Session Communication Session RPC 4: Transport Transmission TCP, UDP 3: Network Address, Routing, Flow IPv6, IPv4 2: Data Link Reliable Physical Comm. IEEE, 802.2 1: Physical Raw bit transmission DSL, USB
- 19. © 2015 Facebook | Menlo Park | Credit: IP Header (OSI Layer 3) Version DSCP ECN Flow Label Payload Length Next Header Hop limit Source Address Destination Address Data
- 20. © 2015 Facebook | Menlo Park | Credit: TCP Header (OSI Layer 4) Source Port Destination Port Sequence Number Acknowledgement Number …. Application Payload
- 21. © 2015 Facebook | Menlo Park | Credit: HTTP Request (OSI Layer 7) GET / HTTP/1.1 host: www.facebook.com
- 22. © 2015 Facebook | Menlo Park | Credit: Putting it all together Version DSCP ECN Flow Label Payload Length Next Header Hop limit Source Address Destination Address Source Port Destination Port Sequence Number Acknowledgement Number …. GET / HTTP/1.1 host: www.facebook.com L3 L4 L7
- 23. © 2015 Facebook | Menlo Park | Credit: Putting it all together IP Packet TCP Segment HTTP Request
- 25. © 2015 Facebook | Menlo Park | Credit: FB Request -- one web server low rps how do we get more rps?! GET / HHVM AAAA www.facebook.com DNS rps = requests per second <html>... AAAA 2a03:2880:2130:cf05:face:b00c::1
- 26. © 2015 Facebook | Menlo Park | Credit: Add a load balancer! www ? GET / <html>... L7LB (proxygen) PHP PHP HHVM GET / <html>... AAAA 2a03:2880:2130:cf05:face:b00c::1 DNS rps = requests per second how do we get more rps?! lots more rps low rps AAAA www.facebook.com
- 27. © 2015 Facebook | Menlo Park | Credit: Add another load balancer! www ? GET / <html>... L7LB (proxygen) PHPL4LB (ipvs) PHP PHP GET / <html>... L7LB (proxygen) L7LB (proxygen) GET / <html>... DNS how do we get more rps?! network bound lots more rps low rps PHP HHVM AAAA 2a03:2880:2130:cf05:face:b00c::1 AAAA www.facebook.com
- 28. © 2015 Facebook | Menlo Park | Credit: Add another load balancer! GET / <html>... L7LB (proxygen) L4LB (shiv) GET / <html>... L7LB (proxygen) L7LB (proxygen) GET / <html>... L4LB (ipvs) DNS network bound lots more rps low rps L7LB (proxygen) PHP PHP PHP PHP HHVM PHP PHP PHP PHP HHVM ECMP AAAA 2a03:2880:2130:cf05:face:b00c::1 AAAA www.facebook.com
- 29. © 2015 Facebook | Menlo Park | Credit: Front end Web Cluster ~10 ~100 Thousands
- 30. © 2015 Facebook | Menlo Park | Credit: cont. x 10 or more
- 31. © 2015 Facebook | Menlo Park | Credit: More RPS? Add another cluster! w GET / <html>... L7LB (proxygen) PHP L4LB (shiv) PHP HHVM how do we get more rps?! GET / <html>... L7LB (proxygen) L7LB (proxygen) GET / <html>... L4LB (ipvs) GET / <html>... L7LB (proxygen) PHP L4LB (shiv) PHP HHVM GET / <html>... L7LB (proxygen) L7LB (proxygen) GET / <html>... L4LB (ipvs) DNS AAAA www.facebook.com AAAA 2a03:2880:2130:cf05:face:b00c::1
- 32. © 2015 Facebook | Menlo Park | Credit: Add another datacenter! w G < G < G < G < G < G < G < G < G < G < G < G < G < G < G < G < G < G < DNS AAAA www.facebook.com AAAA 2a03:2880:2130:cf05:face:b00c::1
- 33. Not really top down 1.2.3.4 L4LB L7LB HHVM
- 34. © 2015 Facebook | Menlo Park | Credit: Datacenter Locations
- 36. © 2015 Facebook | Menlo Park | Credit: Let's break it down GET / <html>... L7LB (proxygen) PHP L4LB (shiv) PHP HHVM GET / <html>... L7LB (proxygen) L7LB (proxygen) GET / <html>... L4LB (ipvs) www? A 1.2.3.4 DNS L4 = IP/port L7 = HTTP
- 37. © 2015 Facebook | Menlo Park | Credit: OSI Model: What is L4/L7? L7LB (proxygen) L4LB (ipvs) Layer Purpose Ex 7: Application High-Level API HTTP, SPDY, MQTT 6: Presentation Data Translation ASCII, JPEG 5: Session Communication Session RPC 4: Transport Transmission TCP, UDP 3: Network Address, Routing, Flow IPv6, IPv4 2: Data Link Reliable Physical Comm. IEEE, 802.2 1: Physical Raw bit transmission DSL, USB ECMP
- 38. L4LB L4LB L4LB ECMP Hash BGP L7LB L7LB L7LB L7LB NOTE: L7 (proxygen) Listens to the VIP on loopback (lo) interface. Not eth0. face:b00c::1 face:b00c::1 face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
- 39. L4LB State Table + Hash ECMP Hash BGP L4LB L4LB L7LB L7LB L7LB L7LB NOTE: L7 (proxygen) Listens to the VIP on loopback (lo) interface. Not eth0. Service Discovery (Zookeeper) face:b00c::1 face:b00c::1 face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
- 40. L7LB L7LB L7LB L7LB L4LB L4LB L4LB Example State Table + Hash ECMP Hash A face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 face:b00c::1
- 41. L7LB L7LB L7LB L7LB L4LB L4LB L4LB Routing D State Table + Hash ECMP Hash A A face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 face:b00c::1
- 42. L7LB L7LB L7LB L7LB L4LB L4LB L4LB Routing B State Table + Hash ECMP Hash face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 face:b00c::1
- 43. L7LB L7LB L7LB L7LB L4LB L4LB L4LB Routing B State Table + Hash ECMP Hash face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 face:b00c::1
- 44. Direct Server Return TCP Routing Facebook HHVM HHVM L4LB HHVM HHVM L7LBL7LBL7LBL7LB TCP SSL HTTP face:b00c::1 face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
- 45. © 2015 Facebook | Menlo Park | Credit: Remember this? Original IP Packet TCP Segment HTTP Request
- 46. © 2015 Facebook | Menlo Park | Credit: IP Packet from IPVS IP in IP encapsulation Original IP Packet TCP Segment HTTP Request
- 47. Direct Server Return Original IP Packet TCP Segment HTTP Request L4LB L7LB Response IP Pckt TCP Segment HTTP Response face:b00c::1 (lo) IP Packet from IPVS Original IP Packet TCP Segment HTTP Request L4LB L7LB TCP/SSL Term Forward HTTP RequestIP Packet from IPVS Original IP Packet TCP Segment HTTP Request face:b00c::1 (lo)face:b00c::1 face:b00c::1
- 48. Edge PoP’s & Reducing Latency
- 50. Seoul -> Oregon TCP Connect: 150ms
- 51. HTTPS Seoul -> Oregon GET HTTP 1.1 ChangeCipherSpec ChangeCipherSpec ACK ServerHello SYN+ACK SYN ClientHello TCP conn established: 150 ms SSL session established: 450 ms Response Received 600 ms 75ms
- 52. Seoul -> Tokyo -> Oregon TCP Connect: 30ms SSL Session: ?? HTTP Response: ?? NRT
- 53. HTTPS Seoul->Tokyo->Oregon GET HTTP 1.1 Sessions established: 90 ms (vs 450 ms) Response Received: 240 ms 60ms GET HTTP 1.1 200 15ms Request Received
- 54. Seoul -> Oregon NRT TCP Connect: 150ms SSL Session: 450ms HTTP Response: 600ms 30ms 90ms 240ms
- 55. © 2015 Facebook | Menlo Park | Credit: Edge POP Locations *POP = points of presence.
- 56. © 2015 Facebook | Menlo Park | Credit: How do the LB’s in PoP’s work? "Flag of Sweden". Licensed under Public Domain via Wikipedia - http://en.wikipedia.org/wiki/File:Flag_of_Sweden.svg#/media/File:Flag_of_Sweden.svg TCP Routing (ip/port) Facebook L4LB L7LB TCP/SSL HTTP HHVM face:b00c::1 face:b00c::1 (lo) HHVM L7LB L4LB face:b00c::1 face:b00c::1 (lo)
- 58. © 2015 Facebook | Menlo Park | Credit: Edge POP Locations *POP = points of presence.
- 59. DNS LB Decision Considerations: •Closest Edge** to user •Capacity •Health •Geo data ???
- 61. Cartographer Architecture Internet ISP DNS resolver Facebook Infrastructure DNS mapcartographer offline processing DNS Serverresolvers/latency capacity routing health
- 66. Monitoring
- 67. Monitoring (Not enough time)
- 68. Open Source
- 69. Open Source Components • Proxygen Libs https://github.com/facebook/proxygen • HHVM https://hhvm.com • TinyDNS https://cr.yp.to/djbdns/tinydns.html • IPVS (IP Virtual Server) http://www.linuxvirtualserver.org/software/ipvs.html • ExaBGP https://github.com/Exa-Networks/exabgp • Python https://python.org • Zookeeper https://zookeeper.apache.org