Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CHUG: Datacenter Computing with Apache Mesos

Mesos overview, presented at the Chicago Hadoop Users Group meetup at Sears Holdings Corp, 2013-11-06



CHUG: Datacenter Computing with Apache Mesos

  1. 1. Datacenter Computing with Apache Mesos CHUG @ Sears Holdings Corp, 2013-11-06 Paco Nathan, chief scientist @pacoid
  2. 2. The Modern Kernel: Top Linux Contributors…
  3. 3. Beyond Hadoop Google has been doing datacenter computing for years, to address the complexities of large-scale data workflows: • leveraging the modern kernel: isolation in lieu of VMs • “most (>80%) jobs are batch jobs, but the majority of resources (55–80%) are allocated to service jobs” • mixed workloads, multi-tenancy • relatively high utilization rates • JVM? not so much… • reality: scheduling batch is simple; scheduling services is hard/expensive
  4. 4. Beyond Hadoop Hadoop – an open source solution for fault-tolerant parallel processing of batch jobs at scale, based on commodity hardware… however, other priorities have emerged for the analytics lifecycle: • • • apps require integration beyond Hadoop • • • • higher utilization multiple topologies, mixed workloads, multi-tenancy significant disruptions in h/w cost/performance curves lower latency highly-available, long running services more than “Just JVM” – e.g., Python growth keep in mind the priority for multi-disciplinary efforts, to break down silos – extending beyond a de facto “priesthood” of data engineering
  5. 5. “Return of the Borg” Return of the Borg: How Twitter Rebuilt Google’s Secret Weapon Cade Metz The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines Luiz André Barroso, Urs Hölzle 2011 GAFS Omega John Wilkes, et al.
  6. 6. “Return of the Borg” Omega: flexible, scalable schedulers for large compute clusters Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, John Wilkes Schwarzkopf.pdf
  7. 7. Google describes the business case… Taming Latency Variability Jeff Dean
  8. 8. Mesos – definitions a common substrate for cluster computing heterogenous assets in your datacenter or cloud made available as a homogenous set of resources • • • • • • • • top-level Apache project scalability to 10,000s of nodes obviates the need for virtual machines isolation (pluggable) for CPU, RAM, I/O, FS, etc. fault-tolerant leader election based on ZooKeeper APIs in C++, Java, Python web UI for inspecting cluster state available for Linux, OpenSolaris, Mac OSX
  9. 9. Mesos – architecture given the use of Mesos as a Datacenter OS kernel… • Chronos provides complex scheduling capabilities, much like a distributed Unix “cron” • Marathon provides highly-available long-running services, much like a distributed Unix “init.d” • next time you need to build a distributed app, consider using these as building blocks a major lesson learned from Spark: • leveraging these kinds of building blocks, one can rebuild Hadoop 100x faster, in much less code
  10. 10. Mesos – architecture services batch Workloads Apps Scalding MPI Impala Hadoop Shark Spark MySQL Kafka JBoss Django Chronos Storm Rails Frameworks Py th on Marathon C ++ JV M Kernel distributed file system distributed resources: CPU, RAM, I/O, FS, rack locality, etc. DFS Cluster
  12. 12. Prior Practice: Dedicated Servers DATACENTER • • low utilization rates longer time to ramp up new services
  13. 13. Prior Practice: Virtualization DATACENTER PROVISIONED VMS • • even more machines to manage • VM licensing costs substantial performance decrease due to virtualization
  14. 14. Prior Practice: Static Partitioning DATACENTER STATIC PARTITIONING • • even more machines to manage • • VM licensing costs substantial performance decrease due to virtualization static partitioning limits elasticity
  15. 15. Mesos: One Large Pool Of Resources DATACENTER MESOS “We wanted people to be able to program for the datacenter just like they program for their laptop." Ben Hindman
  16. 16. Arguments for Datacenter Computing rather than running several specialized clusters, each at relatively low utilization rates, instead run many mixed workloads obvious benefits are realized in terms of: • • • scalability, elasticity, fault tolerance, performance, utilization reduced equipment cap­ex, Ops overhead, etc. reduced licensing, eliminating need for VMs or potential vendor lock­in subtle benefits – arguably, more important for Enterprise IT: • • reduced time for engineers to ramp­up new services at scale • enables Dev/Test apps to run safely on a Production cluster reduced latency between batch and services, enabling new high­ROI use cases
  17. 17. What are the costs of Virtualization? benchmark type OpenVZ improvement mixed workloads 210%-300% LAMP (related) 38%-200% I/O throughput 200%-500% response time order magnitude more pronounced at higher loads
  18. 18. What are the costs of Single Tenancy? MEMCACHED CPU LOAD RAILS CPU LOAD HADOOP CPU LOAD 100% 100% 100% 75% 75% 75% 50% 50% 50% 25% 25% 25% 0% 0% 0% t t COMBINED CPU LOAD (RAILS, MEMCACHED, HADOOP) 100% 75% 50% 25% 0% Hadoop Memcached Rails
  19. 19. Example: Resource Offer in a Two-Level Scheduler
  20. 20. Example: Docker on Mesos Mesos master servers Master Docker Registry Mesos slave servers S M S docker S … S Marathon can launch and monitor service containers from one or more Docker registries, using the Docker executor for Mesos M S docker S … marathon S M … … S S … docker … Local Docker Registry S S S … ( optional )
  21. 21. Example: Docker on Mesos Mesos Master Server init | + mesos-master | + marathon | The executor, monitored by the Mesos slave, delegates to the local Docker daemon for image discovery and management. The executor communicates with Marathon via the Mesos master and ensures that Docker enforces the specified resource limitations. Mesos Slave Server init | + docker | | | + lxc | | | + (user task, under container init system) | | | + mesos-slave | | | + /var/lib/mesos/executors/docker | | | | | + docker run … | | |
  22. 22. Example: Docker on Mesos Mesos Master Server init | + mesos-master | + marathon | Mesos, LXC, and Docker are tied together for launch Mesos Slave Server 3 2 1 6 When a user requests a container… Docker Registry 7 5 init | + docker 8 | | | + lxc | | | + (user task, under container init system) | | | 4 + mesos-slave | | | + /var/lib/mesos/executors/docker | | | | | + docker run … | | |
  23. 23. Production Deployments (public)
  24. 24. Opposite Ends of the Spectrum, One Common Substrate Solaris Zones Built-in / bare metal Linux CGroups Hypervisors
  25. 25. Opposite Ends of the Spectrum, One Common Substrate Request / Response Batch
  26. 26. Case Study: Twitter (bare metal / on premise) “Mesos is the cornerstone of our elastic compute infrastructure – it’s how we build all our new services and is critical for Twitter’s continued success at scale. It's one of the primary keys to our data center efficiency." Chris Fry, SVP Engineering • key services run in production: analytics, typeahead, ads • Twitter engineers rely on Mesos to build all new services • instead of thinking about static machines, engineers think about resources like CPU, memory and disk • allows services to scale and leverage a shared pool of servers across datacenters efficiently • reduces the time between prototyping and launching
  27. 27. Case Study: Airbnb (fungible cloud infrastructure) “We think we might be pushing data science in the field of travel more so than anyone has ever done before… a smaller number of engineers can have higher impact through automation on Mesos." Mike Curtis,VP Engineering • improves resource management and efficiency • helps advance engineering strategy of building small teams that can move fast • key to letting engineers make the most of AWS-based infrastructure beyond just Hadoop • allowed company to migrate off Elastic MapReduce • enables use of Hadoop along with Chronos, Spark, Storm, etc.
  28. 28. Media Coverage Mesosphere Adds Docker Support To Its Mesos-Based Operating System For The Data Center Frederic Lardinois TechCrunch (2013-09-26) Play Framework Grid Deployment with Mesos James Ward, Flo Leibert, et al. Typesafe blog (2013-09-19) Mesosphere Launches Marathon Framework Adrian Bridgwater Dr. Dobbs (2013-09-18) New open source tech Marathon wants to make your data center run like Google’s Derrick Harris GigaOM (2013-09-04) Running batch and long-running, highly available service jobs on the same cluster Ben Lorica O’Reilly (2013-09-01)
  29. 29. Resources Apache Mesos Project Mesosphere Tutorials Documentation 2011 USENIX Research Paper Hindman_new.pdf Collected Notes/Archives
  30. 30. Join us! Mesos Unconference @Twitter HQ, SF Nov 19 Data Day Texas, Austin Jan 11 (incl. Mesos talk+BOF) O’Reilly Strata, Santa Clara Feb 11-13 (incl. Mesos talk+BOF+workshop)
  31. 31. Enterprise Data Workflows with Cascading O’Reilly, 2013 monthly newsletter for updates, events, conference summaries, etc.: