Pachyderm: Building a Big Data Beast On Kubernetes

•

5 likes•2,046 views

Pachyderm is a containerized data analytics solution that's completely deployed using Kubernetes. We take all the amazing tools and potential in the container ecosystem and unlock that power for massive-scale data processing. In this talk we'll show you how to leverage Docker, Kubernetes, and Pachyderm, to build incredibly robust and scalable data infrastructure. We'll start by discussing the key components of a modern data-drive company and how your infrastructure choices can have a massive impact on your product and scalability roadmap. We'll then dive into some architecture details to show how Kubernetes, Docker, and Pachyderm all work in tandem to create a cohesive data infrastructure stack. Finally, we will demonstrate some high-level use cases and powerful benefits you get from the architecture we've outlined. KubeCon schedule link: http://sched.co/4WWA

Technology

Pachyderm
Building a Big Data Beast on Kubernetes
Joe Doliner
Founder & CEO
jd@pachyderm.io

The origin story
Wanted to analyze chess games with Hadoop

Let’s build a modern Hadoop!
Oh shit!
First I need to build 15 years of
distributed systems…

Distributed systems are hard
Bet it all on the container ecosystem

Pachyderm’s Architecture
Kubernetes
User
Analysis
Pachyderm
Pipeline
System
Services Jobs
Pachyderm
File
System
User
Data

Pachyderm File System
A copy-on-write distributed file system
Copy-on-write is the paradigm
that “powers” technologies like
Docker and Spark
Core storage for Pachyderm

Why is this cool?
• View diffs
• Instant Revert
• Reduce storage needs
• Reliability
Commit
0
Commit
1
Commit
2
Commit
3
Commit
4
Git for huge data sets

Pachyderm Pipeline System
• Runs k8s jobs over PFS
• Jobs triggered by commits
• Understands job dependencies
• Leverages copy-on-write storage
Task 1
Task 2 Task 3
Task 4
Dashboard
Task 5
Task 6
Data-aware container scheduler

Pachyderm is…
Task 1
Task 2 Task 3
Task 4
Dashboard
Task 5
Task 6
$ Task 2 failed
$ Task 4 and 6 waiting…
… Fixing code …
$ Task 2 resuming...
$ Task 2 complete
$ Task 4 starting…
Monitoring
Resilient: K8s jobs can be restarted

Efficient: incremental processing
3
2
1
0
Data Analysis
Task 4
DashboardTask 6
Task 1
Task 2 Task 3
Task 5
1% more
data
Task 4
DashboardTask 6
Pachyderm is…

PFS storage
nodes
PPS
Copy-on-write storage
nodes
Elastically scaling
computation nodes

d2.8xlarge
PPSPPS
PPS
Spot
SpotSpot
Cost-effective: resource management
Pachyderm is…

Summary
Kubernetes is a game-changer for
distributed systems
Copy-on-write data is really powerful
Pachyderm unlocks the power of
Kubernetes for big data

Thank You!
Questions?
pachyderm.io
jd@pachyderm.io
github.com/pachyderm/pachyderm

What's hot

Serverless Data Platform

Shu-Jeng Hsieh

Data relay introduction to big data clusters

Chris Adkin

Spark day 2017 - Spark on Kubernetes

Yousun Jeong

Data weekender deploying prod grade sql 2019 big data clusters

Chris Adkin

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015

Datadog

There is growing interest in running Apache Spark natively on Kubernetes. lan Filonenko explains the design idioms, architecture and internal mechanics of Spark orchestrations over Kubernetes. Since data for Spark analytics is often stored in HDFS, Ilan will also explain how to make Spark on Kubernetes work seamlessly with HDFS by addressing challenges such as data locality and security through the use of Kubernetes constructs such as secrets and RBAC rules

Apache Spark on K8s and HDFS Security

Databricks

Gocd – Kubernetes/Nomad Continuous Deployment

Leandro Totino Pereira

In this deck from 2018 Swiss HPC Conference, Christian Kniep from Docker Inc. presents: State of Containers and the Convergence of HPC and BigData. "This talk will recap the history of and what constitutes Linux Containers, before laying out how the technology is employed by various engines and what problems these engines have to solve. Afterward Christian will elaborate on why the advent of standards for images and runtimes moved the discussion from building and distributing containers to orchestrating containerized applications at scale. In conclusion attendees will get an update on how containers foster the convergence of Big Data and HPC workloads and the state of native HPC containers." Learn more: http://docker.com and http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/agenda.php Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

State of Containers and the Convergence of HPC and BigData

inside-BigData.com

Is your data infrastructure architected to withstand anything —including a disaster suddenly taking down an entire data center? When a cloud provider’s data center burned to the ground, 3.6 million websites went dark. But one leading travel service kept running without a hitch, thanks to the design of their environment-aware distributed database. Learn how they architected their data infrastructure for extreme resiliency, how their strategy held up and what lessons they learned.

What we Learned About Application Resiliency When the Data Center Burned Down

ScyllaDB

Organizations commonly use Apache Spark to gain actionable insight from their large amounts of data. Often, these analytics are in the form of data processing pipelines, where there are a series of processing stages, and each stage performs a particular function, and the output of one stage is the input of the next stage. There are several examples of pipelines, such as log processing, IoT pipelines, and machine learning. The common attribute among different pipelines is the sharing of data between stages. It is also common for Spark pipelines to process data stored in the public cloud, such as Amazon S3, Microsoft Azure Blob Storage, or Google Cloud Storage. The global availability and cost effectiveness of these public cloud storage services make them the preferred storage for data. However, running pipeline jobs while sharing data via cloud storage can be expensive in terms of increased network traffic, and slower data sharing and job completion times. Using Alluxio, a memory speed virtual distributed storage system, enables sharing data between different stages or jobs at memory speed. By reading and writing data in Alluxio, the data can stay in memory for the next stage of the pipeline, and this result in great performance gains. In this talk, we discuss how Alluxio can be deployed and used with a Spark data processing pipeline in the cloud. We show how pipeline stages can share data with Alluxio memory for improved performance benefits, and how Alluxio can improves completion times and reduces performance variability for Spark pipelines in the cloud.

Spark Pipelines in the Cloud with Alluxio with Gene Pang

Spark Summit

K8ssandra has made it effortless to deploy Apache Cassandra on Kubernetes. Long a simple means of deploying stateless applications, modern tooling and APIs has facilitated the move of databases to this pervasive platform. Join Chris Bradford in deploying the K8ssandra stack to Kubernetes. Learn how it packages a production Cassandra deployment with supporting tooling alongside Stargate, a next generation data gateway. We will explore everything from the management interfaces leveraged by DevOps teams to performant, highly available, REST, Graph, and Document APIs for developers.

Building a Data Plane with K8ssandra, Apache Cassandra on Kubernetes

Christopher Bradford

Alluxio Community Office Hour Apr 28, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Adit Madan Bin Fan Today’s conventional wisdom states that network latency across the two ends of a hybrid cloud prevents you from running analytic workloads in the cloud with the data on-prem. As a result, most companies copy their data into a cloud environment and maintain that duplicate data. All of this means that it is challenging to make both on-prem HDFS data accessible with the desired application performance. In this talk, we will show you how to leverage any public cloud (AWS, Google Cloud Platform, or Microsoft Azure) to scale analytics workloads directly on on-prem data without copying and synchronizing the data into the cloud. In this Office Hour, we will go over: - A strategy to embrace the hybrid cloud, including an architecture for running ephemeral compute clusters using on-prem HDFS. - An example of running on-demand Presto, Spark, and Hive with Alluxio in the public cloud. - An analysis of experiments with TPC-DS to demonstrate the benefits of the given architecture.

Burst Presto & Spark workloads to AWS EMR with no data copies

Alluxio, Inc.

A look at some of the ways available to deploy Postgres in a Kubernetes cloud environment, either in small scale using simple configurations, or in larger scale using tools such as Helm charts and the Crunchy PostgreSQL Operator. A short introduction to Kubernetes will be given to explain the concepts involved, followed by examples from each deployment method and observations on the key differences.

Deploying PostgreSQL on Kubernetes

Jimmy Angelakos

Since April 2016, Spark-as-a-service has been available to researchers in Sweden from the Swedish ICT SICS Data Center at www.hops.site. Researchers work in an entirely UI-driven environment on a platform built with only open-source software. Spark applications can be either deployed as jobs (batch or streaming) or written and run directly from Apache Zeppelin. Spark applications are run within a project on a YARN cluster with the novel property that Spark applications are metered and charged to projects. Projects are also securely isolated from each other and include support for project-specific Kafka topics. That is, Kafka topics are protected from access by users that are not members of the project. In this talk we will discuss the challenges in building multi-tenant Spark streaming applications on YARN that are metered and easy-to-debug. We show how we use the ELK stack (Elasticsearch, Logstash, and Kibana) for logging and debugging running Spark streaming applications, how we use Graphana and Graphite for monitoring Spark streaming applications, and how users can debug and optimize terminated Spark Streaming jobs using Dr Elephant. We will also discuss the experiences of our users (over 120 users as of Sept 2016): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and our novel solutions for helping researchers debug and optimize Spark applications. To conclude, we will also give an overview on our course ID2223 on Large Scale Learning and Deep Learning, in which 60 students designed and ran SparkML applications on the platform.

Spark-Streaming-as-a-Service with Kafka and YARN: Spark Summit East talk by J...

Spark Summit

Save 60% of Kubernetes storage costs on AWS & others with OpenEBS

MayaData Inc

Kubernetes As of Spark 2.3, Spark can run on clusters managed by Kubernetes. we will describes the best practices about running Spark SQL on Kubernetes upon Tencent cloud includes how to deploy Kubernetes against public cloud platform to maximum resource utilization and how to tune configurations of Spark to take advantage of Kubernetes resource manager to achieve best performance. To evaluate performance, the TPC-DS benchmarking tool will be used to analysis performance impact of queries between configurations set. Speakers: Junjie Chen, Junping Du

Apache Spark on K8S Best Practice and Performance in the Cloud

Databricks

Greenplum Overview for Postgres Hackers - Greenplum Summit 2018

VMware Tanzu

Bdc from bare metal to k8s

Chris Adkin

Nicolas Trésegnie, Chief Architect at SuperAwesome Abstract: SuperAwesome's mission is to make the internet safer for kids. At the core of SuperAwesome's analytics is Druid. In this talk, we walk through how we run Druid on spot instances. We explain the consequences in terms of cost and reliability, how we managed to build a reliable system despite the risks, and how you could do the same. Nicolas works as Chief Architect at SuperAwesome, where is is looking after the overall architecture of the systems and the infrastructure. He is all about automation and how technology can be used to achieve business goals. Nicolas studied Computer Science and Bioinformatics, and he is now pursuing an MBA at Imperial.

Druid in Spot Instances

Imply

When low latency (P99) and high performance are core requirements, what NoSQL database attributes should you consider, and what tradeoffs are key? While we live in a world of multi-CPU, multi-core servers capable of storing tens of terabytes of data, if your database isn’t architected to take advantage of this, you’re being penalized on performance or cost. Join this webinar to learn about the critical elements for a high-performance, low-latency NoSQL database. ScyllaDB’s engineers will discuss how they addressed core database performance challenges, including the pros and cons of each, and provide a detailed explanation of the architectural principles they applied to achieve their performance objectives. We’ll take a deep dive into the strategies applied to: Achieve precise control over I/O and compute-intensive workloads Avoid locks and contention on the CPU level Bypass kernel bottlenecks Squeeze the most out of modern multi-core hardware Satisfy SLAs while maintaining system stability

Critical Attributes for a High-Performance, Low-Latency Database

ScyllaDB

What's hot (20)

Serverless Data Platform

Data relay introduction to big data clusters

Spark day 2017 - Spark on Kubernetes

Data weekender deploying prod grade sql 2019 big data clusters

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015

Apache Spark on K8s and HDFS Security

Gocd – Kubernetes/Nomad Continuous Deployment

State of Containers and the Convergence of HPC and BigData

What we Learned About Application Resiliency When the Data Center Burned Down

Spark Pipelines in the Cloud with Alluxio with Gene Pang

Building a Data Plane with K8ssandra, Apache Cassandra on Kubernetes

Burst Presto & Spark workloads to AWS EMR with no data copies

Deploying PostgreSQL on Kubernetes

Spark-Streaming-as-a-Service with Kafka and YARN: Spark Summit East talk by J...

Save 60% of Kubernetes storage costs on AWS & others with OpenEBS

Apache Spark on K8S Best Practice and Performance in the Cloud

Greenplum Overview for Postgres Hackers - Greenplum Summit 2018

Bdc from bare metal to k8s

Druid in Spot Instances

Critical Attributes for a High-Performance, Low-Latency Database

Viewers also liked

Pachyderm: Data Storage and Processing with Docker

Joseph Zwicker

Big Data Applications

Joseph Zwicker

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...

Radhika Puthiyetath

Kubernetes can run application containers on clusters of physical or virtual machines. It can also do much more than that. Kubernetes satisfies a number of common needs of applications running in production, such as co-locating helper processes, mounting storage systems, distributing secrets, application health checking, replicating application instances, horizontal auto-scaling, load balancing, rolling updates, and resource monitoring. However, even though Kubernetes provides a lot of functionality, there are always new scenarios that would benefit from new features. Ad hoc orchestration that is acceptable initially often requires robust automation at scale. Application-specific workflows can be streamlined to accelerate developer velocity. This is why Kubernetes was also designed to serve as a platform for building an ecosystem of components and tools to make it easier to deploy, scale, and manage applications. The Kubernetes control plane is built upon the same APIs that are available to developers and users, implementing resilient control loops that continuously drive the current state towards the desired state. This design has enabled Apache Stratos and a number of other Platform as a Service and Continuous Integration and Deployment systems to build atop Kubernetes. This presentation introduces Kubernetes’s core primitives, shows how some of its better known features are built on them, and introduces some of the new capabilities that are being added.

WSO2Con US 2015 Kubernetes: a platform for automating deployment, scaling, an...

Brian Grant

Analyzing data with docker v4

Andreas Dewes

Pachyderm big data de l'ère docker

Enguerran Delahaie

Ansible & CloudStack - Configuration Management

ShapeBlue

Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...

Lucidworks

Ansible and CloudStack

ShapeBlue

Modern Distributed Messaging and RPC

Max Alexejev

Multi tenancy for docker

Ananth Padmanabhan

In this talk we share the lessons learned while building out the Baqend Cloud platform on AWS and Docker. Baqend’s AWS-hosted architecture consists of a caching CDN-Layer, global and local load balancing, a group of REST and Node.js servers and a database cluster with Redis and MongoDB. As customers have their own set of containerized REST and Node servers, we needed a cluster that on the one hand is horizontally scalable and on the other hand easily manageable and fault-tolerant from an operational perspective. Today there are at least 4 popular systems that claim to support this: - Kubernetes - Apache Mesos - Docker Swarm - AWS Elastic Container Service (ECS) Thinking that ECS would certainly be the easiest option on AWS, we started building our cluster on it. We quickly came to realize that while ECS was astoundingly stable and easy to use there were inherent limitations that could not be worked around. An old Docker version, missing network isolation, no means of parameterizing task and forced memory constraints are major limitations of ECS we will talk about. Seeing the daunting operational overhead of running Kubernetes or Mesos in practice we turned to Docker’s native clustering solution Swarm. We will present how Swarm works with both Docker and AWS and highlight the advantages and downsides compared to Amazon’s ECS.

Building a Global-Scale Multi-Tenant Cloud Platform on AWS and Docker: Lesson...

Felix Gessert

KubeCon EU 2016: Multi-Tenant Kubernetes

Pachyderm: Building a Big Data Beast On Kubernetes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (18)

Similar to Pachyderm: Building a Big Data Beast On Kubernetes

Similar to Pachyderm: Building a Big Data Beast On Kubernetes (20)

More from KubeAcademy

More from KubeAcademy (20)

Recently uploaded

Recently uploaded (20)

Pachyderm: Building a Big Data Beast On Kubernetes