Apache Druid®: A Dance of Distributed Processes

•

0 likes•279 views

This document summarizes the key components and collaborations in Apache Druid. It describes Zookeeper's role in coordination, the Overlord's role in task management, the Broker's role in query routing, and the Middle Manager's role in ingestion and indexing. It provides diagrams illustrating how these components work together to ingest and store distributed data, and answer queries in a scalable way.

Technology

peter.marshall@imply.io
20 years in Enterprise Architecture
CRM, EDRM, ERP, EIP, Digital Services,
Security, BI, RI, and MDM
BA Theology (!) and Computer Studies
TOGAF certiﬁed
Book collector & A/V buyer
Prime Timeline = proper timeline
#werk
petermarshall.io

What the collaborations do
Some principles
Each collaboration
Wrap-up
Questions!

Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

● A job to do
● Compute to do the ingestion
● A place to store optimised data
● A question to answer
● Data to process!
● Compute to answer queries
● Somewhere to put the data that’s
near to the query process
● Some rules to follow
Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

Zookeeper
Overlord
Ingest
Spec
Task
Log
S3, GCS, ABS, HFDS, HTTP, Local, MySql, Postgres, Druid
Kafka Kinesis
Middle Manager
(Indexer)

Zookeeper
Overlord
Middle Manager
(Indexer)

Deep Store
Zookeeper
Overlord
Middle Manager
(Indexer) columnarise
index & encode
time-shardDeep Store

Deep Store
Metadata
Store
Zookeeper
Overlord
Middle Manager
(Indexer)

Metadata
Store
Broker
Middle Manager
(Indexer)
Overlord
Deep Store
Zookeeper
SQL
and
Native

Metadata
Store
Zookeeper
Broker
Middle Manager
(Indexer)
Overlord
Deep Store

Metadata
Store
Zookeeper
Broker
Middle Manager
(Indexer)
Overlord
Deep Store Historical

Middle Manager
(Indexer)
Metadata
Store
Zookeeper
Historical
BrokerOverlord
Deep Store

OverlordOverlordCoordinator
Metadata
Store
Historical
Zookeeper
BrokerOverlord
Middle Manager
(Indexer)
Deep Store

Zookeeper
Historical
OverlordOverlordCoordinator
Metadata
Store
Historical
BrokerOverlord
Middle Manager
(Indexer)
Deep Store
Historical

Zookeeper
Historical
Historical
OverlordOverlordCoordinator
Metadata
Store
Historical
BrokerOverlord
Middle Manager
(Indexer)
Deep Store

Zookeeper
Coordinator
Overlord
Broker
Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

http://druid.apache.org
Imply Distribution
https://imply.io/get-started
@druidio
Add Apache Druid
as a skill
Apache Distribution
https://github.com/apache/druid
ASF Slack
#druid
Druid Community
https://druid.apache.org/community/
Meetup Groups
https://www.meetup.com/pro/apache-druid/
Google Groups Druid User Forum
https://groups.google.com/

What's hot

Apache Pinot Meetup Sept02, 2020

Mayank Shrivastava

Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale

Seunghyun Lee

We went through the journey of deploying Apache Druid clusters on Kubernetes(K8s) and created a druid-operator (https://github.com/druid-io/druid-operator). This talk introduces the druid kubernetes operator, how to use it to deploy druid clusters and how it works under the hood. We will share how we use this operator to deploy Druid clusters at Splunk. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Druid is a complex stateful distributed system and a Druid cluster consists of multiple web services such as Broker, Historical, Coordinator, Overlord, MiddleManager etc each deployed with multiple replicas. Deploying a single web service on K8s requires creating few K8s resources via YAML files and it multiplies due to multiple services inside of a Druid cluster. Now doing it for multiple Druid clusters (dev, staging, production environments) makes it even more tedious and error prone. K8s enables creation of application (such as Druid) specific extension, called “Operator”, that combines kubernetes and application specific knowledge into a reusable K8s extension that makes deploying complex applications simple.

Splunk: Druid on Kubernetes with Druid-operator

Imply

As Twitch grew, both the amount of data we received and the number of employees interested in the data grew rapidly. In order to continue empowering decision making as we scaled, we turned to using Druid and Imply to provide self service analytics to both our technical and non technical staff allowing them to drill into high level metrics in lieu of reading generated reports. In this talk, learn how Twitch implemented a common analytics platform for the needs of many different teams supporting hundreds of users, thousands of queries, and ~5 billion events each day. This session will explain our Druid architecture in detail, including: -The end-to-end architecture deployed on Amazon that includes Kinesis, RDS, S3, Druid, Pivot and Tableau -How the data is brought together to deliver a unified view of live customer engagement and historical trends -Operational best practices we learnt scaling Druid -An example walk through using the platform

Self Service Analytics at Twitch

Imply

Adding nodes at runtime (Upscale) to already running Spark-on-Yarn clusters is fairly easy. But taking away these nodes (Downscale) when the workload is low at some later point of time is a difficult problem. To remove a node from a running cluster, we need to make sure that it is not used for compute as well as storage. But on production workloads, we see that many of the nodes can’t be taken away because: Nodes are running some containers although they are not fully utilized i.e., containers are fragmented on different nodes. Example. – each node is running 1-2 containers/executors although they have resources to run 4 containers. Nodes have some shuffle data in the local disk which will be consumed by Spark application running on this cluster later. In this case, the Resource Manager will never decide to reclaim these nodes because losing shuffle data could lead to costly recomputation of stages. In this talk, we will talk about how we can improve downscaling in Spark-on-YARN clusters under the presence of such constraints. We will cover changes in scheduling strategy for container allocation in YARN and Spark task scheduler which together helps us achieve better packing of containers. This makes sure that containers are defragmented on fewer set of nodes and thus some nodes don’t have any compute. In addition to this, we will also cover enhancements to Spark driver and External Shuffle Service (ESS) which helps us to proactively delete shuffle data which we already know has been consumed. This makes sure that nodes are not holding any unnecessary shuffle data – thus freeing them from storage and hence available for reclamation for faster downscaling.

Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters

Databricks

We built Apache Pinot - a real-time distributed OLAP datastore - for low-latency analytics at scale. This is heavily used at companies such as LinkedIn, Uber, Slack, where Kafka serves as the backbone for capturing vast amounts of data. Pinot ingests millions of events per sec from Kafka, builds indexes in real-time and serves 100K+ queries per second while ensuring latency SLA of millisecond to sub second. In the first implementation, we used the Consumer Group feature to manage the offsets and checkpoints across multiple Kafka Consumers. However, to achieve fault tolerance and scalability, we had to run multiple consumer groups for the same topic. This was our initial strategy to maintain the SLA at high query workload. But this model posed other challenges - since Kafka maintains offset per consumer group, achieving data consistency across multiple consumer groups was not possible. Also, a failure of a single node in a consumer group meant the entire consumer group was unavailable for query processing. Restarting the failed node needed lot of manual operations to ensure data is consumed exactly once. This resulted in management overhead and inefficient hardware utilization. While taking inspiration from the Kafka consumer group implementation, we redesigned the real-time consumption in Pinot to maintain consistent offset across multiple consumer groups. This allowed us to guarantee consistent data across all replicas. This enabled us to copy data from another consumer group during node addition, node failure or increasing the replication group. In this talk, we will deep dive into the various challenges faced and considerations that went into this design, and learn what makes Pinot resilient to failures both in Kafka Brokers and Pinot Components. We will introduce the new concept of ""lockstep"" sequencing where multiple consumer groups can synchronize checkpoints periodically and maintain consistency. We'll describe how we achieve this while maintaining strict freshness SLAs, and also withstanding high throughput and ingestion.

Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...

HostedbyConfluent

Extending Druid Index File

Navis Ryu

Premier Inside-Out: Apache Druid

Hortonworks

Apache Kafka is used as the primary message bus for propagating events and logs across Uber. In particular, it pairs with Apache Pinot, a real-time distributed OLAP datastore, to deliver real-time insights seconds after the messages produced to Kafka. One challenge we faced was to update existing data in Pinot with the changelog in Kafka, and deliver an accurate view in the real-time analytical results. For example, the financial dashboard can report gross booking with the corrected Ride fares. And restaurant owners can analyze the UberEats orders with their latest delivery status. Implementing upserts in an immutable real-time OLAP store like Pinot is nontrivial. We need to make architectural changes in how data is distributed via Kafka amongst the server nodes, how it's indexed and queried in a distributed fashion. In this talk I will discuss how we leveraged Kafka's partition-by-key feature to this end and how we added this ability in Pinot without any performance degradation.

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

HostedbyConfluent

History of Apache Pinot

Kishore Gopalakrishna

VictoriaLogs Preview - Aliaksandr Valialkin * Existing open source log management systems - ELK (ElasticSearch) stack: Pros & Cons - Grafana Loki: Pros & Cons * What is VictoriaLogs - Open source log management system from VictoriaMetrics - Easy to setup and operate - Scales vertically and horizontally - Optimized for low resource usage (CPU, RAM, disk space) - Accepts data from Logstash and Fluentbit in Elasticsearch format - Accepts data from Promtail in Loki format - Supports stream concept from Loki - Provides easy to use yet powerful query language - LogsQL * LogsQL Examples - Search by time - Full-text search - Combining search queries - Searching arbitrary labels * Log Streams - What is a log stream? - LogsQL examples: querying log streams - Stream labels vs log labels * LogsQL: stats over access logs * VictoriaLogs: CLI Integration * VictoriaLogs Recap

VictoriaLogs: Open Source Log Management System - Preview

VictoriaMetrics

Druid deep dive

Kashif Khan

Fluentd vs. Logstash for OpenStack Log Management

NTT Communications Technology Development

Apache Iceberg - A Table Format for Hige Analytic Datasets

Alluxio, Inc.

LinkedIn's is the most advantageous social networking tool available to job seekers and business professionals today, with 610+ million members creating millions of posts, videos, and articles that generate tens of millions of shares, comments, and likes per day. LinkedIn has leveraged this activity data to build rich interactive user-facing analytics applications like “Who Viewed My Profile”, Talent Insights, Ad Analytics, and Publisher Analytics, among others. These applications are all powered by Pinot, as are internal dashboards, anomaly detection and root cause analysis platform like ThirdEye. This talk will present how Pinot has become the de-facto solution for serving analytic queries in milliseconds, ad-hoc reporting, monitoring & Anomaly Detection on multidimensional data.

Building real time analytics applications using pinot : A LinkedIn case study

Kishore Gopalakrishna

Innodb에서의 Purge 메커니즘 deep internal (by 이근오)

I Goo Lee.

Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. With a data warehouse at this scale, it is a constant challenge to keep improving performance. This talk will focus on Iceberg, a new table metadata format that is designed for managing huge tables backed by S3 storage. Iceberg decreases job planning time from minutes to under a second, while also isolating reads from writes to guarantee jobs always use consistent table snapshots. In this session, you'll learn: • Some background about big data at Netflix • Why Iceberg is needed and the drawbacks of the current tables used by Spark and Hive • How Iceberg maintains table metadata to make queries fast and reliable • The benefits of Iceberg's design and how it is changing the way Netflix manages its data warehouse • How you can get started using Iceberg Speaker Ryan Blue, Software Engineer, Netflix

Iceberg: a fast table format for S3

DataWorks Summit

Cloud Native Landscape (CNCF and OCI)

Chris Aniszczyk

Postgres has a nice feature called Point-in-time Recovery (PITR) that would allow you to go back in time. In this talk, we will discuss what are the use-cases of PITR, how to prepare your database for PITR by setting good base backup and WAL shipping setups, with some examples. We will expand the discussion with how to achieve PITR if you have a distributed and sharded Postgres setup by mentioning challenges such as clock differences and ways to overcome them, such as two-phase commit and pg_create_restore_point.

Distributed Point-in-Time Recovery with Postgres | PGConf.Russia 2018 | Eren ...

Citus Data

Openstack Swift - Lots of small files

Alexandre Lecuyer

What's hot (20)

Apache Pinot Meetup Sept02, 2020

Pinot: Enabling Real-time Analytics Applications @ LinkedIn's Scale

Splunk: Druid on Kubernetes with Druid-operator

Self Service Analytics at Twitch

Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters

Apache Pinot Case Study: Building Distributed Analytics Systems Using Apache ...

Extending Druid Index File

Premier Inside-Out: Apache Druid

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

History of Apache Pinot

VictoriaLogs: Open Source Log Management System - Preview

Druid deep dive

Fluentd vs. Logstash for OpenStack Log Management

Apache Iceberg - A Table Format for Hige Analytic Datasets

Building real time analytics applications using pinot : A LinkedIn case study

Innodb에서의 Purge 메커니즘 deep internal (by 이근오)

Iceberg: a fast table format for S3

Cloud Native Landscape (CNCF and OCI)

Distributed Point-in-Time Recovery with Postgres | PGConf.Russia 2018 | Eren ...

Openstack Swift - Lots of small files

Similar to Apache Druid®: A Dance of Distributed Processes

Apache Druid ingests and enables instant query on many billions of events in real-time. But how? In this talk, each of the components of an Apache Druid cluster is described – along with the data and query optimisations at its core – that unlock fresh, fast data for all. Bio: Peter Marshall (https://linkedin.com/in/amillionbytes/) leads outreach and engineering across Europe for Imply (http://imply.io/), a company founded by the original developers of Apache Druid. He has 20 years architecture experience in CRM, EDRM, ERP, EIP, Digital Services, Security, BI, Analytics, and MDM. He is TOGAF certified and has a BA (hons) degree in Theology and Computer Studies from the University of Birmingham in the United Kingdom.

Druid: Under the Covers (Virtual Meetup)

Imply

20th Athens Big Data Meetup - 1st Talk - Druid: the open source, performant, ...

Athens Big Data

Peter Marshall, Technology Evangelist at Imply Abstract: Apache Druid® can revolutionise business decision-making with a view of the freshest of fresh data in web, mobile, desktop, and data science notebooks. In this talk, we look at key activities to integrate into Apache Druid POCs, discussing common hurdles and signposting to important information. Bio: Peter Marshall (https://petermarshall.io) is an Apache Druid Technology Evangelist at Imply (http://imply.io/), a company founded by original developers of Apache Druid. He has 20 years architecture experience in CRM, EDRM, ERP, EIP, Digital Services, Security, BI, Analytics, and MDM. He is TOGAF certified and has a BA degree in Theology and Computer Studies from the University of Birmingham in the United Kingdom.

Druid Adoption Tips and Tricks

Imply

August meetup - All about Apache Druid

Imply

"Building Data Warehouse with Google Cloud Platform", Artem Nikulchenko

Fwdays

Hadoop summit 2010 frameworks panel elephant bird

Kevin Weil

Salesforce & SAP Integration

Raymond Gao

Turning Relational Database Tables into Hadoop Datasources by Kuassi Mensah

Data Con LA

Next Generation Hadoop Introduction

Adam Muise

DBA's used to be Relational Database centric for instance managing Microsoft SQL Server or Oracle, in this changing world of polyglot database environments their role has expanded not just into new platforms other than SQL but also new legal governance, modelling techniques, architecture etc. They need to have a base knowledge of Kimball, Inmon, Data Vault, what CAP theorem is, LAMBDA, Big Data, Data Science etc.

Evolution of the DBA to Data Platform Administrator/Specialist

Tony Rogerson

Using the Power of Big SQL 3.0 to Build a Big Data-Ready Hybrid Warehouse

Rizaldy Ignacio

Slides for a talk. Talk abstract: In the dark of the night, if you listen carefully enough, you can hear databases cry. But why? As developers, we rarely consider what happens under the hood of widely used abstractions such as databases. As a consequence, we rarely think about the performance of databases. This is especially true to less widespread, but often very useful NoSQL databases. In this talk we will take a close look at NoSQL database performance, peek under the hood of the most frequently used features to see how they affect performance and discuss performance issues and bottlenecks inherent to all databases.

Why databases cry at night

Michael Yarichuk

Ibm info sphere datastage and hadoop two best-of-breed solutions together-f...

ArunshankarArjunan

Sql server enterprise edition awareness

Hamid J. Fard

Connecting Your Data Analytics Pipeline

Amazon Web Services

Designing big data analytics solutions on azure

Mohamed Tawfik

Azure Data Factory ETL Patterns in the Cloud

Mark Kromer

Data orchestration is the lifeblood of any successful data analytics solution. Take a deep dive into Azure Data Factory's data movement and transformation activities, particularly its integration with Azure's Big Data PaaS offerings such as HDInsight, SQL Data warehouse, Data Lake, and AzureML. Participants will learn how to design, build and manage big data orchestration pipelines using Azure Data Factory and how it stacks up against similar Big Data orchestration tools such as Apache Oozie. Video of presentation: https://channel9.msdn.com/Events/Ignite/Australia-2017/DA332

Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...

Lace Lofranco

Big Data Goes Airborne. Propelling Your Big Data Initiative with Ironcluster ...

Precisely

Hadoop at the Center: The Next Generation of Hadoop

Adam Muise

Similar to Apache Druid®: A Dance of Distributed Processes (20)

Druid: Under the Covers (Virtual Meetup)

20th Athens Big Data Meetup - 1st Talk - Druid: the open source, performant, ...

Druid Adoption Tips and Tricks

August meetup - All about Apache Druid

"Building Data Warehouse with Google Cloud Platform", Artem Nikulchenko

Hadoop summit 2010 frameworks panel elephant bird

Salesforce & SAP Integration

Turning Relational Database Tables into Hadoop Datasources by Kuassi Mensah

Next Generation Hadoop Introduction

Evolution of the DBA to Data Platform Administrator/Specialist

Using the Power of Big SQL 3.0 to Build a Big Data-Ready Hybrid Warehouse

Why databases cry at night

Ibm info sphere datastage and hadoop two best-of-breed solutions together-f...

Sql server enterprise edition awareness

Connecting Your Data Analytics Pipeline

Designing big data analytics solutions on azure

Azure Data Factory ETL Patterns in the Cloud

Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...

Big Data Goes Airborne. Propelling Your Big Data Initiative with Ironcluster ...

Hadoop at the Center: The Next Generation of Hadoop

Pivot 2.0 - The next generation visualization tool for your streaming data

Imply

Nicolas Trésegnie, Chief Architect at SuperAwesome Abstract: SuperAwesome's mission is to make the internet safer for kids. At the core of SuperAwesome's analytics is Druid. In this talk, we walk through how we run Druid on spot instances. We explain the consequences in terms of cost and reliability, how we managed to build a reliable system despite the risks, and how you could do the same. Nicolas works as Chief Architect at SuperAwesome, where is is looking after the overall architecture of the systems and the infrastructure. He is all about automation and how technology can be used to achieve business goals. Nicolas studied Computer Science and Bioinformatics, and he is now pursuing an MBA at Imperial.

Druid in Spot Instances

Imply

Druid has been the production workhorse for the past 2+ years at Zeotap powering the core Audience planning across our Connect and Targeting products. Though Druid is best suited for data having time as a dimension as it partitions data based on time first, we have used Druid to serve ML powered enhanced insights and Estimation of potential dataset sizes, to assist us with our core business case of Audience planning. These are datasets without timestamp a.k.a non-temporal with high scale and having nested dimensions. These have been achieved using nuanced data modelling to store the data sets and achieve millisecond latency retrieval on top of the same. The core of the presentation would be on the data modelling journey to achieve these use cases detailing the query access patterns. We also delve upon the architecture - ingestion into druid sink and processing including ML. In the end we go over the production setup and configurations and provide the performance tunings applied. The presentation would have the following heads: The presentation would have the following heads * Business case in Ad-Tech and Mar-Tech vertical * Audience Planner Usecase 1 - Insights -Lambda Architecture and data flow -Deep dive on data model -Takeaways *Audience Planner Usecase 2 - Estimator -Architecture and data flow -Stratified sampling explained -Data model to solve nested data - deep dive -Takeaways *Audience Planner Usecase 3 - Skew correction -Skew correction model -Query Access -Data model in Druid to accommodate output from ML models -Takeaways *Production setup, config and Tunings *Production Operation experience takeaways

Zeotap: Data Modeling in Druid for Non temporal and Nested Data

Imply

At Nielsen Identity, we leverage Druid to provide our customers with real-time analytics tools for various use-cases, including in-flight analytics, reporting and building target audiences. The common challenge of these use-cases is counting distinct elements in real-time at scale. We’ve been using Druid to solve these problems for the past 4 years, and gained a lot of experience with it. In this talk, we will share some of the best practices and tips we’ve gathered over the years, including: *Data modeling *Ingestion *Retention and deletion *Query optimization

Nielsen: Casting the Spell - Druid in Practice

Imply

One of the most popular use cases for Apache Druid is building data applications. Data applications exist to deliver data into the hands of everyone on a team in a business, and are used by these teams to make faster, better decisions. To fulfill this role, they need to support granular drill down, because the devil is in the details, but also be extremely fast, because otherwise people won't use them! In this talk, Gian Merlino will cover: *The unique technical challenges of powering data-driven applications *What attributes of Druid make it a good platform for data applications *Some real-world data applications powered by Druid

Building Data Applications with Apache Druid

Imply

Druid is a powerful real-time database, and part of that power is the level of control you get over cluster configuration, allowing you to get maximum performance for your specific data and query types. In this talk, Gian Merlino, one of the original authors of Druid and CTO and co-founder of Imply, will walk you through some advanced techniques that can provide a multiplier to your Druid performance. Afterwards, he’ll take your questions about performance, or anything else Druid-related.

Maximizing Apache Druid performance: Beyond the basics

Imply

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experi...

Imply

Target is one of the largest retailers in the United States, with brick-and-mortar stores in all 50 states and one of the most-visited ecommerce sites in the country. In addition to typical merchandising functions like assortment planning, pricing and inventory management, Target also operates a large supply chain, financial/banking operations and property management organizations. As a data-driven organization, we need a data analytics platform that can address the unique needs of each of these various business units, while scaling to hundreds of thousands of users and accommodating an ever-increasing amount of data. In this talk we’ll cover why Target chose to create our own analytics platform and specifically how Druid makes this platform successful. We’ll cover how we utilize key features in Druid, such as union datasources, arbitrary granularities, real-time ingestion, complex aggregation expressions and lightning-fast query response to provide analytics to users at all levels of the organization. We’ll also cover how Druid’s speed and flexibility allow us to provide interactive analytics to front-line, edge-of-business consumers to address hundreds of unique use-cases across several business units.

Building an Enterprise-Scale Dashboarding/Analytics Platform Powered by the C...

Imply

In this session, TrafficGuard’s Head of Data Science, Raigon Jolly, will discuss how TrafficGuard uses Druid and its partnership with Imply to: - Provide granular reporting to clients in near-real time - Monitor rules and concept drift - Staying ahead of the moving target that is ad fraud - Facilitate performance tuning and right-sizing infrastructure so our team can focus on innovation of our core product

How TrafficGuard uses Druid to Fight Ad Fraud and Bots

Imply

Talk abstract: Users are demanding access to large, multi-petabyte, multi-dimension, real-time datasets to answer business critical questions. Providing a self-service interface that meets the performance expectations of these users can be challenging. Enter Apache Druid: an open source analytics database powering real-time, ad hoc, lightning fast analytics. It is used for clickstream analytics, network telemetry, fraud detection, application monitoring and so much more by companies like Apple, Netflix, Twitter, and AirBnb. Druid can ingest millions of records per second and deliver sub-second response times on OLAP-style slice and dice queries. In this talk, we will start with an overview of Apache Druid followed by a look at several examples of how Druid is being used in the real-world. We'll finish up with Q&A and some virtual networking. Speaker Bio: Mike McLaughlin is a senior field engineer at Imply. He helps customers run and optimize Apache Druid in production. He has 20 years experience developing, architecting, and deploying software.

Apache Druid: Lightning Fast Analytics on Real-time and Historical Data (Atla...

Imply

Benchmarking Apache Druid

Imply

Check out the full webinar: https://imply.io/videos/why-data-warehouses-cannot-support-hot-analytics Today’s data warehouses - whether traditional, specialized or cloud-based - are good at supporting cold analytics, such as reporting, where query times can take minutes. But they cannot cost-effectively support hot analytics—interactive ad hoc analytics usually performed by larger groups of users against batch or streaming data. Examples of hot analytics include clickstream analytics; service, network and application performance monitoring; and risk analytics. Data warehouses struggle with hot analytics use cases because they are too slow, unable to scale, or too expensive. Learn how a new class of real-time data platforms overcome these limitations, and how companies implement a “temperature-based” approach to analytics.

Why data warehouses cannot support hot analytics

Imply

Check out the webinar: https://imply.io/videos/whats-new-imply-3-3-apache-druid-0-18 The most recent Imply 3.3 release, based on Apache 0.18 brings several major new features, including joins, query laning and Clarity Alerts. These new features deliver increased design flexibility during design, and provide improved ingestion performance, and sub-second response times to help accelerate data warehouse and data lake deployments, and add real-time analytics in general.

What’s New in Imply 3.3 & Apache Druid 0.18

Imply

Apache Druid Vision and Roadmap

Imply

MoPub, a Twitter company, provides monetization solutions for mobile app publishers and developers around the globe. MoPub receives over 33 Billion ad requests per day generating over 200TB of raw logs every day. We built MoPub Analytics as the analytics platform, using Druid + Imply for our end users who are Publishers, Demand side partners and Internal users. We will talk about the architecture of the analytics platform, our Druid cluster setup, hardware choices, monitoring, use cases, limiting factors, challenges with lookups and solutions we used. Watch video:https://imply.io/virtual-druid-summit/analytics-over-terabytes-of-data-at-twitter-apache-druid

Analytics over Terabytes of Data at Twitter

Imply

More from Imply (15)

Pivot 2.0 - The next generation visualization tool for your streaming data

Druid in Spot Instances

Zeotap: Data Modeling in Druid for Non temporal and Nested Data

Nielsen: Casting the Spell - Druid in Practice

Building Data Applications with Apache Druid

Maximizing Apache Druid performance: Beyond the basics

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experi...

Building an Enterprise-Scale Dashboarding/Analytics Platform Powered by the C...

How TrafficGuard uses Druid to Fight Ad Fraud and Bots

Apache Druid: Lightning Fast Analytics on Real-time and Historical Data (Atla...

Benchmarking Apache Druid

Why data warehouses cannot support hot analytics

What’s New in Imply 3.3 & Apache Druid 0.18

Apache Druid Vision and Roadmap

Analytics over Terabytes of Data at Twitter

Recently uploaded

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Real Time Object Detection Using Open CV

Khem

Manulife - Insurer Innovation Award 2024

The Digital Insurer

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Recently uploaded (20)

GenAI Risks & Security Meetup 01052024.pdf

Axa Assurance Maroc - Insurer Innovation Award 2024

A Domino Admins Adventures (Engage 2024)

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

AWS Community Day CPH - Three problems of Terraform

Why Teams call analytics are critical to your entire business

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

How to Troubleshoot Apps for the Modern Connected Worker

Strategies for Landing an Oracle DBA Job as a Fresher

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

presentation ICT roal in 21st century education

Top 10 Most Downloaded Games on Play Store in 2024

Powerful Google developer tools for immediate impact! (2023-24 C)

Real Time Object Detection Using Open CV

Manulife - Insurer Innovation Award 2024

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Partners Life - Insurer Innovation Award 2024

Apache Druid®: A Dance of Distributed Processes

1. Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

2. peter.marshall@imply.io 20 years in Enterprise Architecture CRM, EDRM, ERP, EIP, Digital Services, Security, BI, RI, and MDM BA Theology (!) and Computer Studies TOGAF certiﬁed Book collector & A/V buyer Prime Timeline = proper timeline #werk petermarshall.io

3. What the collaborations do Some principles Each collaboration Wrap-up Questions!

4. Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules

6. ● A job to do ● Compute to do the ingestion ● A place to store optimised data ● A question to answer ● Data to process! ● Compute to answer queries ● Somewhere to put the data that’s near to the query process ● Some rules to follow Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules

7. https://www.archimatetool.com/

8. What the collaborations do Some principles Each collaboration Wrap-up Questions!

9. Zookeeper Overlord Ingest Spec Task Log S3, GCS, ABS, HFDS, HTTP, Local, MySql, Postgres, Druid Kafka Kinesis Middle Manager (Indexer)

10. Zookeeper Overlord Middle Manager (Indexer)

11. Deep Store Zookeeper Overlord Middle Manager (Indexer) columnarise index & encode time-shardDeep Store

12. Deep Store Metadata Store Zookeeper Overlord Middle Manager (Indexer)

13. Metadata Store Broker Middle Manager (Indexer) Overlord Deep Store Zookeeper SQL and Native

14. Metadata Store Zookeeper Broker Middle Manager (Indexer) Overlord Deep Store

15. Metadata Store Zookeeper Broker Middle Manager (Indexer) Overlord Deep Store Historical

16. Middle Manager (Indexer) Metadata Store Zookeeper Historical BrokerOverlord Deep Store

17. OverlordOverlordCoordinator Metadata Store Historical Zookeeper BrokerOverlord Middle Manager (Indexer) Deep Store

18. OverlordOverlordCoordinator Metadata Store Historical Zookeeper BrokerOverlord Middle Manager (Indexer) Deep Store

19. Zookeeper Historical OverlordOverlordCoordinator Metadata Store Historical BrokerOverlord Middle Manager (Indexer) Deep Store Historical

20. Zookeeper Historical Historical OverlordOverlordCoordinator Metadata Store Historical BrokerOverlord Middle Manager (Indexer) Deep Store

21.

22.

23. What the collaborations do Some principles Each collaboration Wrap-up Questions!

24.

25.

26.

27.

28. https://www.archimatetool.com/

29. Zookeeper Coordinator Overlord Broker Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules

30. ★ A job to do ★ Compute to do the ingestion ★ A place to store optimised data ★ A question to answer ★ Data to process! ★ Compute to answer queries ★ Somewhere to put the data that’s near to the query process ★ Some rules to follow Query Distributed execution of SQL / Druid Native queries on the cluster Ingestion Ingestion tasks that bring data into Druid from storage and delivery services Distribution Replication and distribution of the ingested data according to rules

31. http://druid.apache.org Imply Distribution https://imply.io/get-started @druidio Add Apache Druid as a skill Apache Distribution https://github.com/apache/druid ASF Slack #druid Druid Community https://druid.apache.org/community/ Meetup Groups https://www.meetup.com/pro/apache-druid/ Google Groups Druid User Forum https://groups.google.com/

32. What the collaborations do Some principles Each collaboration Wrap-up Questions!

Apache Druid®: A Dance of Distributed Processes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Druid®: A Dance of Distributed Processes

Similar to Apache Druid®: A Dance of Distributed Processes (20)

More from Imply

More from Imply (15)

Recently uploaded

Recently uploaded (20)

Apache Druid®: A Dance of Distributed Processes