Couchbase Meetup Jan 2016

•Download as PPTX, PDF•

3 likes•991 views

LinkedIn’s Big Data Pipeline with Kafka, Hadoop and Couchbase

Michael Kehoe
Senior Site Reliability Engineer
LinkedIn
LinkedIn’s Big Data Pipeline
with Kafka, Hadoop and
Couchbase

3
$ whoami
Michael Kehoe
• Sr Site Reliability Engineer
(SRE)
• Member of CBVT
• B.E. (Electrical Engineering)
from
the University of Queensland,
Australia

4
Kafka @ LinkedIn
• Kafka was created by LinkedIn
• Kafka is a publish-subscribe
system as a distributed commit
log
• Processes 500+ TB/ day (~500
billion messages)

5
LinkedIn’s use of Kafka
• Monitoring
• Pub-Sub Messaging
• Analytics
• Building block for (log) distributed application
• Samza
• Espresso
• Pinot

Kafka to Hadoop (Analytics)
6
Use Case
• LinkedIn tracks data to better understand how members use our
products
• Information such as which page got viewed and which content got
clicked on are sent into a Kafka cluster in each data center
• Some of these events are all centrally collected and pushed onto
our Hadoop grid for analysis and daily report generation

7
Couchbase @ LinkedIn
• About 80 separate services with one or more clusters in multiple data
centers
• Up to ~70 servers in a cluster
• Single & Multi-tenant clusters

8
Hadoop to Couchbase
• Our primary use-case for Hadoop  Couchbase is for building
(warming) / restoring Couchbase buckets
• LinkedIn built it’s own in-house solution to work with our ETL
processes etc

Jobs Cluster
9
Clusters & Numbers
• Used for read-scaling, > 150k QPS, 27 node clusters
• We use Hadoop to pre-build data by partition
• Couchbase average latency is 2-3ms
• 99th percentile is ~8 - 12ms

Questions?
10
Thank You

©2014 LinkedIn Corporation. All Rights Reserved.

More Related Content

What's hot

Large networks consist of a diverse range of equipment, across private, public, hybrid clouds and partner networks. A hierarchical network has layers of infrastructure, catering to access, core, or distribution roles, managed by different organizations specialized to architect the right network hardware, software, and features for that network layer. The nature of data generated by each component can vary in type and form, including logs, events, metrics, or alarms. The diversity of data generated by a large network is beyond human scale. Apache Kafka® is a critical hub in large networks, empowering AIOps to enhance decision making, improve analysis and insights by contextualizing large volumes of operational data. Kafka solved the big problem of collecting, processing, storing and normalizing data at scale, allowing us to focus on building the AIOps pipeline. Our platform connects the dots across relevant operations data and provides operations teams with simple and powerful access to insights, from within increasingly popular collaboration environments like Slack and Microsoft teams. The pipeline must also integrate with automation solutions. This session will cover how large volumes of streaming messages can be received by parallel Kafka consumers, and turned into action by network operations teams, dramatically reducing downtime and improving performance.

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

Kafka Connectors are used extensively in data migration solutions, serving as a middle tier when migrating data across databases. In addition, microservice architectures also use Kafka Connectors heavily when communicating with one another while still operating independently on their own data stores. In this talk, we cover these use cases in more detail along with a deep dive into the architecture of the source and sink Kafka Connectors for Cosmos DB.

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

HostedbyConfluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

Speaker: Jun Rao, Co-founder, Confluent In 2010, LinkedIn began developing Apache Kafka®. In 2011, Kafka was released an Apache open source project. Since then, the use of Kafka has grown rapidly in a variety of businesses. Now more than 30% of Fortune 500 companies are already using Kafka. In this 60-minute online talk, Confluent Co-founder Jun Rao will: -Explain how Kafka became the predominant publish/subscribe messaging system that it is today -Introduce Kafka's most recent additions to its set of enterprise-level features -Demonstrate how to evolve your Kafka implementation into a complete real-time streaming data platform that functions as the central nervous system for your organization Watch the recording: https://cnfl.io/kafka-past-present-future-on-demand

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost. in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

HostedbyConfluent

The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

HostedbyConfluent

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

At Gloo.us, we face a challenge in providing platform data to heterogeneous applications in a way that eliminates access contention, avoids high latency ETLs, and ensures consistency for many teams. We're solving this problem by adopting Data Mesh principles and leveraging Kafka, Kafka Connect, and Kafka streams to build an event driven architecture to connect applications to the data they need. A domain driven design keeps the boundaries between specialized process domains and singularly focused data domains clear, distinct, and disciplined. Applying the principles of a Data Mesh, process domains assume the responsibility of transforming, enriching, or aggregating data rather than relying on these changes at the source of truth -- the data domains. Architecturally, we've broken centralized big data lakes into smaller data stores that can be consumed into storage managed by process domains. This session covers how we’re applying Kafka tools to enable our data mesh architecture. This includes how we interpret and apply the data mesh paradigm, the role of Kafka as the backbone for a mesh of connectivity, the role of Kafka Connect to generate and consume data events, and the use of KSQL to perform minor transformations for consumers.

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

HostedbyConfluent

Flipp is an e-commerce company that promotes weekly shopping opportunities. We began our migration to event-driven microservices in November 2016, and have since moved to nearly 300 Kafka-powered microservices. In this presentation we will explore the major strategies we have used in our migration from distributed monoliths to event-driven microservices. There have been a number of painful learnings and pitfalls along the way that we will share with you. Lastly, we will provide recommendations for each step of the way on your journey from monoliths to effective event-driven microservices. The first major section of this presentation deals with the liberation of data from monolithic services. In this section we will cover: Kafka Connect vs System Production, Event Schematization, Entities and Events, The importance of the Single Source of Truth, Consumption patterns and Event update verbosity. The second major section of this presentation discusses the usage of liberated event data in conjunction with other event streams.In this section we will cover common access patterns, handling (lots) of relational data, Stateful Foreign-Key Joins in Kafka Streams (See Kafka KIP-213), High frequency updates (price, stock) vs static properties and how to handle too many data streams. The third major section details how to abstract event complexity away, leverage the single source of truth and the usage of Core Events across a company. In this section we cover abstracting data streams, Core Events as detailed by the Single Source of Truth, Core Events in relation to bounded contexts and using Core Events successfully as a business.

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Oak Ridge Leadership Facility (OLCF) in the National Center for Computational Sciences (NCCS) division at Oak Ridge National Laboratory (ORNL) houses world-class high-performance computing (HPC) resources and has a history of operating top-ranked supercomputers on the TOP500 list, including the world's current fastest, Summit, an IBM AC922 machine with a peak of 200 petaFLOPS. With the exascale era rapidly approaching, the need for a robust and scalable big data platform for operations data is more important than ever. In the past when a new HPC resource was added to the facility, pipelines from data sources spanned multiple data sinks which oftentimes resulted in data silos, slow operational data onboarding, and non-scalable data pipelines for batch processing. Using Apache Kafka as the message bus of the division's new big data platform has allowed for easier decoupling of scalable data pipelines, faster data onboarding, and stream processing with the goal to continuously improve insight into the HPC resources and their supporting systems. This talk will focus on the NCCS division's transition to Apache Kafka over the past few years to enhance the OLCF's current capabilities and prepare for Frontier, OLCF's future exascale system; including the development and deployment of a full big data platform in a Kubernetes environment from both a technical and cultural shift perspective. This talk will also cover the mission of the OLCF, the operational data insights related to high-performance computing that the organization strives for, and several use-cases that exist in production today.

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

(Preston Thompson, Braze) Kafka Summit SF 2018 If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy. We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors. In this talk, I will discuss: -How we started our journey in designing this large-scale streaming architecture -Why streaming technologies were necessary to solve our technology and business issues -The lessons we learned along the way that can help you with your Kafka-based architecture

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

The challenge with today’s “data explosion” is finding the most appropriate answer to the question, “So where do I put my data?” while avoiding the longer-term problem: data warehouses, data lakes, cloud storage, NoSQL databases, … are often the places where “big” data goes to die. Enter Physics 101, and my corollary to Newton’s First Law of Motion: Data in motion tends to stay in motion until it comes rest on disk. Similarly, if data is at rest, it will remain at rest until an external “force” puts it in motion again. Data inevitably comes to rest at some point. Without “external forces”, data often gets lost or becomes stale where it lands. “Modern” architectures tend to involve data pipelines where downstream consumers of data make use of data generated upstream, often with built-for-purpose repositories at each stage. This session will explore how data that has come to rest can be put in motion again; how Kafka can keep it in motion longer; and how pipelined architectures might be created to make use of that data.

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

HostedbyConfluent

Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

HostedbyConfluent

What's hot (20)

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Apache Kafka: Past, Present and Future

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

Developing custom transformation in the Kafka connect to minimize data redund...

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

Bootstrap SaaS startup using Open Source Tools

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Sum...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

Introduction to Apache Kafka and why it matters - Madrid

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Real-Time Dynamic Data Export Using the Kafka Ecosystem

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

Viewers also liked

Just over two years ago, I completed college, moved countries and started as a SRE at LinkedIn. Over those two years, I’ve gone from a Junior engineer to a Senior Engineer, however, I wouldn’t have been able to do it without a great onboarding experience and the mentors that have guided me This session will take the perspective of myself as a New College Graduate and Nina Mushiana (SRE manager) on how to onboard new SREs and help them reach their full potential. We will deep dive into onboarding, mentoring and training topics and reflect on lessons learnt over my two year experience. Finally I’ll present advice for ELTs on how to become an important and successful part of their organization.

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

Good monitoring can be the difference between a great night's sleep or hearing your phone go off at 2:37 a.m. because of a production outage. Couchbase Server provides a large number of metrics which can be overwhelming if you do not know the critical things to focus on or how to expose that information to your monitoring system. In this talk we will look at example production incidents, going in depth around specific things to monitor, and how this information can be used to find issues, work out root cause, and discover trends.

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

Couchbase Connect 2016

Couchbase Connect 2016

Couchbase Connect 2016

LinkedIn serves traffic for its 467 million members from four data centers and multiple PoPs spread geographically around the world. Serving live traffic from from many places at the same time has taken us from a disaster recovery model to a disaster avoidance model where we can take an unhealthy data center or PoP out of rotation and redistribute its traffic to a healthy one within minutes, with virtually no visible impact to users. The geographical distribution of our infrastructure also allows us to optimize the end-user's experience by geo routing users to the best possible PoP and datacenter. This talk provide details on how LinkedIn shifts traffic between its PoPs and data centers to provide the best possible performance and availability for its members. We will also touch on the complexities of performance in APAC, how IPv6 is helping our members and how LinkedIn stress tests data centers verify its disaster recovery capabilities.

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

LinkedIn’s production stack is made up of over 900 applications and over 2200 internal API’s. With any given application having many interconnected pieces, it is difficult to escalate to the right person in a timely manner. In order to combat this, LinkedIn built an Event Correlation Engine that monitors service health and maps dependencies between services to correctly escalate to the SRE’s who own the unhealthy service. We’ll discuss the approach we used in building a correlation engine and how it has been used at LinkedIn to reduce incident impact and provide better quality of life to LinkedIn’s oncall engineers.

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Software reliability tools and common software errors

Software reliability tools and common software errors

Software reliability tools and common software errors

How TPM saves the day

How TPM saves the day

How TPM saves the day

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Software Reliability Engineering

Software Reliability Engineering

Software Reliability Engineering

Software reliability growth model

Software reliability growth model

Software reliability growth model

Kafka is a scalable, distributed publish subscribe messaging system that's used as a data transmission backbone in many data intensive digital businesses. Couchbase Server is a scalable, flexible document database that's fast, agile, and elastic. Because they both appeal to the same type of customers, Couchbase and Kafka are often used together. This presentation from a meetup in Mountain View describes Kafka's design and why people use it, Couchbase Server and its uses, and the use cases for both together. Also covered is a description and demo of Couchbase Server writing documents to a Kafka topic and consuming messages from a Kafka topic. using the Couchbase Kafka Connector.

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

This is a lightning talk topic I presented at Triangle DevOps meetup to share ideas about operating load balancer by following SRE practices. It highlighted continuous delivery process and test driven development when we develop this feature. There're lots of online cloud-based services we used to achieve our goal including AWS, Google Cloud, Librato, Logentries, SignalSciences, BlueBoxCloud, Quay.io etc. And lots of open source software we love and use every day. Disclaimer: It represents personal opinions only, not represent any points of view of my employer.

Load balancing in the SRE way

Load balancing in the SRE way

Load balancing in the SRE way

The need for gleaning answers from data in real-time is moving from nicety to a necessity. There are few options to analyze the never-ending stream of unbounded data at scale. Let’s compare and contrast the core principles and technologies the different open source solutions available to help with this endeavor, and where in the future processing engines need to evolve to solve processing needs at scale. These findings are based on the experience of continuing to build a scalable solution in the cloud to process over 700 billion events at Netflix, and how we are embarking on the next journey to evolve unbounded data processing engines.

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

Amazon Web Services

Viewers also liked (19)

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

SRECon USA 2016: Growing your Entry Level Talent

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

SouthBay SRE Meetup Jan 2016

Couchbase Connect 2016

Couchbase Connect 2016

Couchbase Connect 2016

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

Using SaltStack to Auto Triage and Remediate Production Systems

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Reducing MTTR and False Escalations: Event Correlation at LinkedIn

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Software reliability tools and common software errors

Software reliability tools and common software errors

Software reliability tools and common software errors

How TPM saves the day

How TPM saves the day

How TPM saves the day

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Cloud Native, Microservices and SRE/Chaos Engineering: The new Rules of The G...

Software Reliability Engineering

Software Reliability Engineering

Software Reliability Engineering

Software reliability growth model

Software reliability growth model

Software reliability growth model

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Real time Messages at Scale with Apache Kafka and Couchbase

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

Feedback loops: How SREs benefit and what is needed to realize their potential

Load balancing in the SRE way

Load balancing in the SRE way

Load balancing in the SRE way

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

Unbounded bounded-data-strangeloop-2016-monal-daxini

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

Similar to Couchbase Meetup Jan 2016

CA Technologies Customer Presentation

CA Technologies Customer Presentation

CA Technologies Customer Presentation

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

___________________________________________ Meetup#7 | Session 2 | 21/03/2018 | Taboola _____________________________________________ In this talk, we will present our multi-DC Kafka architecture, and discuss how we tackle sending and handling 10B+ messages per day, with maximum availability and no tolerance for data loss. Our architecture includes technologies such as Cassandra, Spark, HDFS, and Vertica - with Kafka as the backbone that feeds them all.

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

This presentation will describe how to go beyond a "Hello world" stream application and build a real-time data-driven product. We will present architectural patterns, go through tradeoffs and considerations when deciding on technology and implementation strategy, and describe how to put the pieces together. We will also cover necessary practical pieces for building real products: testing streaming applications, and how to evolve products over time. Presented at highloadstrategy.com 2016 by Øyvind Løkling (Schibsted Products & Technology), joint work with Lars Albertsson (independent, www.mapflat.com).

Building real time data-driven products

Building real time data-driven products

Building real time data-driven products

Lars Albertsson

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Future of Data Engineering

Future of Data Engineering

Future of Data Engineering

Designing a social network offers some exciting challenges to engineers. The system needs to operate at scale, to provide a responsive user experience and to be able to inspect user activity in order to both generate new content and improve how the existing content is delivered. Event Driven Architectures are particularly suitable to handle these kind of challenges and highly scalable messaging systems such as Apache Kafka have been designed specifically to support the requirements of modern high volume applications. In this talk we are describing how the Crowdmix back-end has been designed as an Event Based system running on top of Kafka. We are going to present the overall system architecture and discuss in more detail some of the different sub components processing those events in different fashions, from streaming based processing to batch processing passing through a lambda-style batch and stream cooperation. We are going to conclude describing some lessons learned from our one-year journey in implementing and operating the system

Music streams

Stefano Galarraga

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

We consider a microservices architecture to achieve an end goal, not because it's "the cool thing to do". Every organization looking to adopt this architecture must realize (and adhere) to a set of foundational principles. Guided by those principles, we can correctly choose the technology to help support a microservices architecture and meet our end goals. This talk explains those core principles and gives you the tools needed for your microservices journey.

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Christian Posta

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

(Bruno Simic, Solutions Engineer, Couchbase) Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Tras la compra de MySQL por el gigante de las bases de datos se ha experimentado una inevitable evolución. Exposición de algunos de los principales cambios acontecidos por este hecho. La llegada de más inversión, nuevos productos, las relaciones con la comunidad, la colaboración con openstack son algunos de los puntos clave. Además, explicación de algunos de los principales casos de éxito mundiales en diferentes sectores con Wikipedia, NASA Jet Propulsion Laboratory, BBC News, Bank of Finland, Sahlgrenska University Hospital o Virgin América. Autor: Abel Flórez Loranca (Oracle MySQL). Librecon.io

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Similar to Couchbase Meetup Jan 2016 (20)

CA Technologies Customer Presentation

CA Technologies Customer Presentation

CA Technologies Customer Presentation

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Data Stream Processing for Beginners with Kafka and CDC

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Apache Kafka at LinkedIn

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

Unlocking big data with Hadoop + MySQL

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Distributed Kafka Architecture Taboola Scale

Building real time data-driven products

Building real time data-driven products

Building real time data-driven products

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Future of Data Engineering

Future of Data Engineering

Future of Data Engineering

Music streams

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

Customer Feedback Analytics for Starbucks

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Microservices Journey Summer 2017

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

Modern ETL: Azure Data Factory, Data Lake, and SQL Database

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

LinkedIn Infrastructure (analytics@webscale, at fb 2013)

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

LINE's messaging service architecture underlying more than 200 million monthl...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

MySQL en el mundo real. Evolución desde la compra por Oracle

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

More from Michael Kehoe

eBPF Workshop

eBPF Basics

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

In 2016, Susan Fowler released the 'Production Ready Microservices' book. This book sets an industry benchmark on explaining how microservices should be conceived, all the way through to documentation. So how does this translate actionable items? This session will explore how to expertly deploy your microservice to production. The audience will learn best practice for designing, deploying, monitoring & documenting application. By the end of the session, attendees should feel confident that they have the knowledge to deploy a service that will be reliable and scalable.

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

All engineering teams run into trouble from time to time. Alert fatigue, caused by technical debt or a failure to plan for growth, can quickly burn out SREs, overloading both development and operations with reactive work. Layer in the potential for communication problems between teams, and we can find ourselves in a place so troublesome we cannot easily see a path out. At times like this, our natural instinct as reliability engineers is to double down and fight through the issues. Often, however, we need to step back, assess the situation, and ask for help to put the team back on the road to success. We will look at the process for Code Yellow, the term we use for this process of “righting the ship”, and discuss how to identify teams that are struggling. Through a look at three separate experiences, we will examine some of the root causes, what steps were taken, and how the engineering organization as a whole supports the process.

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

The National Transport Safety Bureau is one of the most widely known Government bodies in the world. It’s their role to run into an incident, secure the scene and understand everything that happened. Given the important and unpredictable nature of their work, they have an extensive manual that sets out how incidents should be attended to and how the investigation should progress. This session will detail how the NTSB’s approach to its work and the procedure that drives it, is transferable to us as incident responders. We’ll talk about the NTSB’s pre-incident preparation, incident notification, attending it, collecting information from the field and writing up a report and holding hearings. We’ll consistently draw parallels to IT incident management and how to create applicable process and procedures that mimic those of the NTSB.

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

Linux Container Basics

Linux Container Basics

Linux Container Basics

Network failures continue to plague datacenter operators as their symptoms may not have direct correlation with where or why they occur. We introduce 007, a lightweight, always-on diagnosis application that can find problematic links and also pinpoint problems for each TCP connection. 007 is completely contained within the end host. During its two month deployment in a tier-1 datacenter, it detected every problem found by previously deployed monitoring tools while also finding the sources of other problems previously undetected.

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

The National Transport Safety Bureau is one of the most widely known Government bodies in the world. It’s their role to run into an incident, secure the scene and understand everything that happened. Given the important and unpredictable nature of their work, they have an extensive manual that sets out how incidents should be attended to and how the investigation should progress. This session will detail how the NTSB’s approach to its work and the procedure that drives it, is transferable to us as incident responders. We’ll talk about the NTSB’s pre-incident preparation, incident notification, attending it, collecting information from the field and writing up a report and holding hearings. We’ll consistently draw parallels to IT incident management and how to create applicable process and procedures that mimic those of the NTSB.

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

SRE teams can sometimes run into periods of time where they have staff burnout, technical debt or poor reliability. As SRE’s, we’re programmed to keep fighting through the issues, when sometimes it’s best to step back, assess the situation; and ask for help to put the team back on a successful pathway. This talk will discuss three separate experiences where teams needed some extra help to stabilize their services and oncall. We’ll discuss how to identify struggling teams; get the right assistance; and build a strategy for the team to succeed.

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

In 2018, Site Reliability Engineering (SRE) will turn 15 years old. Since Google's inception of the term SRE, companies across the world have adopted a new operations mindset along with automation, deployment and monitoring principals. Most of what SRE does now is well established throughout the industry, so what is the next-wave of reliability principals and automation frameworks? This session will dive into what the future holds for reliability engineering as a field and what will be the next areas of investment and improvement for reliability teams.

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

How often have you heard stories where someone thought they had a disaster strategy, never tested it and it fails when you need it the most? LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken the company from a disaster recovery model to a disaster avoidance model, where an unhealthy data center can be taken out of rotation and its traffic redistributed to the healthy data centers within minutes, with virtually no visible impact to users. As LinkedIn transitioned from big monolithic applications to microservices, it was difficult to determine capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex microservices architecture wasn’t sufficient to provide enough confidence in data center’s capacity. To solve this problem, LinkedIn moves live traffic to services site-wide by shifting traffic between datacenters to simulate a disaster every business day!

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

LinkedIn’s production stack is made up of over 900 applications, 2200 internal API’s and hundreds of databases. With any given application having many interconnected pieces, it is difficult to escalate to the right person in a timely manner. In order to combat this, LinkedIn built an Event Correlation Engine that monitors service health and maps dependencies between services to correctly escalate to the SRE’s who own the unhealthy service. We’ll discuss the approach we used in building a correlation engine and how it has been used at LinkedIn to reduce incident impact and provide better quality of life to LinkedIn’s oncall engineers.

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

All of us depend on the underlying network to be stable whether in the datacenter or in the cloud. We all have a basic knowledge of how traditional networks run, however in the past 10 years, we’ve moved to building redundant physical topologies in our networks, optimized the routing methodologies accordingly, moved into the cloud and gotten greater visibility and tuneables in the Linux kernel network stack. A lot has changed! However, the way we troubleshoot the network in relation to the applications we support hasn’t adapted. In this session, we’ll review the progress that network infrastructure has made look at specific examples where traditional troubleshooting responses fail us and demonstrate our need to rethink our approach to making applications and the network interact harmoniously.

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

LinkedIn has evolved from serving live traffic out of one data center to four data centers spread geographically. Serving live traffic from four data centers at the same time has taken the company from a disaster recovery model to a disaster avoidance model, where an unhealthy data center can be taken out of rotation and its traffic redistributed to the healthy data centers within minutes, with virtually no visible impact to users. As LinkedIn transitioned from big monolithic applications to microservices, it was difficult to determine capacity constraints of individual services to handle extra load during disaster scenarios. Stress testing individual services using artificial load in a complex microservices architecture wasn’t sufficient to provide enough confidence in data center’s capacity. To solve this problem, LinkedIn leverages live traffic to stress services site-wide by shifting traffic to simulate a disaster load. Michael Kehoe and Anil Mallapur discuss how LinkedIn uses traffic shifts to mitigate user impact by migrating live traffic between its data centers and stress test site-wide services for improved capacity handling and member experience.

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

More from Michael Kehoe (17)

eBPF Workshop

eBPF Basics

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

QConSF 2018: Building Production-Ready Applications

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

AllDayDevops: What the NTSB teaches us about incident management & postmortems

Linux Container Basics

Linux Container Basics

Linux Container Basics

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

What the NTSB teaches us about incident management & postmortems

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

PyBay 2018: Production-Ready Python Applications

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

Helping operations top-heavy teams the smart way

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

The Next Wave of Reliability Engineering

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

Building Production-Ready Microservices: DevopsExchangeSF

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

SRECon-Europe-2017: Networks for SREs

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale

Recently uploaded

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

MS Copilot expands with MS Graph connectors

MS Copilot expands with MS Graph connectors

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Understanding the FAA Part 107 License ..

Understanding the FAA Part 107 License ..

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Recently uploaded (20)

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

MS Copilot expands with MS Graph connectors

MS Copilot expands with MS Graph connectors

MS Copilot expands with MS Graph connectors

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Why Teams call analytics are critical to your entire business

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Understanding the FAA Part 107 License ..

Understanding the FAA Part 107 License ..

Understanding the FAA Part 107 License ..

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

Couchbase Meetup Jan 2016

1.

2. Michael Kehoe Senior Site Reliability Engineer LinkedIn LinkedIn’s Big Data Pipeline with Kafka, Hadoop and Couchbase

3. 3 $ whoami Michael Kehoe • Sr Site Reliability Engineer (SRE) • Member of CBVT • B.E. (Electrical Engineering) from the University of Queensland, Australia

4. 4 Kafka @ LinkedIn • Kafka was created by LinkedIn • Kafka is a publish-subscribe system as a distributed commit log • Processes 500+ TB/ day (~500 billion messages)

5. 5 LinkedIn’s use of Kafka • Monitoring • Pub-Sub Messaging • Analytics • Building block for (log) distributed application • Samza • Espresso • Pinot

6. Kafka to Hadoop (Analytics) 6 Use Case • LinkedIn tracks data to better understand how members use our products • Information such as which page got viewed and which content got clicked on are sent into a Kafka cluster in each data center • Some of these events are all centrally collected and pushed onto our Hadoop grid for analysis and daily report generation

7. 7 Couchbase @ LinkedIn • About 80 separate services with one or more clusters in multiple data centers • Up to ~70 servers in a cluster • Single & Multi-tenant clusters

8. 8 Hadoop to Couchbase • Our primary use-case for Hadoop  Couchbase is for building (warming) / restoring Couchbase buckets • LinkedIn built it’s own in-house solution to work with our ETL processes etc

9. Jobs Cluster 9 Clusters & Numbers • Used for read-scaling, > 150k QPS, 27 node clusters • We use Hadoop to pre-build data by partition • Couchbase average latency is 2-3ms • 99th percentile is ~8 - 12ms

10. Questions? 10 Thank You

11. ©2014 LinkedIn Corporation. All Rights Reserved.