Submit Search
Upload
Ozone- Object store for Apache Hadoop
•
6 likes
•
4,196 views
Hortonworks
Follow
Slides from ApacheCon
Read less
Read more
Technology
Report
Share
Report
Share
1 of 46
Download Now
Download to read offline
Recommended
Ozone: An Object Store in HDFS
Ozone: An Object Store in HDFS
DataWorks Summit
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
Hive Does ACID
Hive Does ACID
DataWorks Summit
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
HostedbyConfluent
Building robust CDC pipeline with Apache Hudi and Debezium
Building robust CDC pipeline with Apache Hudi and Debezium
Tathastu.ai
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...
StreamNative
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilities
Nishith Agarwal
Hive: Loading Data
Hive: Loading Data
Benjamin Leonhardi
More Related Content
What's hot
Apache Hadoop In Theory And Practice
Apache Hadoop In Theory And Practice
Adam Kawa
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
Apache Hudi: The Path Forward
Apache Hudi: The Path Forward
Alluxio, Inc.
Time-Series Apache HBase
Time-Series Apache HBase
HBaseCon
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
DataWorks Summit
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
Ozone and HDFS's Evolution
Ozone and HDFS's Evolution
DataWorks Summit
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
Erik Krogen
Delta from a Data Engineer's Perspective
Delta from a Data Engineer's Perspective
Databricks
Dataflow with Apache NiFi
Dataflow with Apache NiFi
DataWorks Summit/Hadoop Summit
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3
DataWorks Summit
Apache doris (incubating) introduction
Apache doris (incubating) introduction
leanderlee2
Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0
Databricks
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Databricks
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
DataWorks Summit
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
Ryan Blue
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing
DataWorks Summit
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
DataWorks Summit
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the Beast
DataWorks Summit
Hive 3 - a new horizon
Hive 3 - a new horizon
Thejas Nair
What's hot
(20)
Apache Hadoop In Theory And Practice
Apache Hadoop In Theory And Practice
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Apache Hudi: The Path Forward
Apache Hudi: The Path Forward
Time-Series Apache HBase
Time-Series Apache HBase
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
Ozone and HDFS's Evolution
Ozone and HDFS's Evolution
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
Delta from a Data Engineer's Perspective
Delta from a Data Engineer's Perspective
Dataflow with Apache NiFi
Dataflow with Apache NiFi
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3
Apache doris (incubating) introduction
Apache doris (incubating) introduction
Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the Beast
Hive 3 - a new horizon
Hive 3 - a new horizon
Similar to Ozone- Object store for Apache Hadoop
Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objects
DataWorks Summit
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
DataWorks Summit/Hadoop Summit
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
Chris Nauroth
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
DataWorks Summit/Hadoop Summit
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
DataWorks Summit/Hadoop Summit
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
DataWorks Summit/Hadoop Summit
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
DataWorks Summit
Hadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
DataWorks Summit/Hadoop Summit
Running Services on YARN
Running Services on YARN
DataWorks Summit/Hadoop Summit
Big data spain keynote nov 2016
Big data spain keynote nov 2016
alanfgates
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
DataWorks Summit/Hadoop Summit
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
Cloudera, Inc.
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
Big Data Spain
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
DataWorks Summit/Hadoop Summit
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
Edelweiss Kammermann
Apache Hive on ACID
Apache Hive on ACID
Hortonworks
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
alanfgates
Apache Hive on ACID
Apache Hive on ACID
DataWorks Summit/Hadoop Summit
Similar to Ozone- Object store for Apache Hadoop
(20)
Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objects
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
Hadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
Running Services on YARN
Running Services on YARN
Big data spain keynote nov 2016
Big data spain keynote nov 2016
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
Apache Hive on ACID
Apache Hive on ACID
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
Apache Hive on ACID
Apache Hive on ACID
More from Hortonworks
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
Hortonworks
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Hortonworks
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
Hortonworks
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Hortonworks
HDF 3.2 - What's New
HDF 3.2 - What's New
Hortonworks
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Hortonworks
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Hortonworks
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
Hortonworks
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
Hortonworks
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
Hortonworks
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
Hortonworks
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Hortonworks
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Hortonworks
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
Hortonworks
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Hortonworks
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
Hortonworks
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
Hortonworks
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Hortonworks
More from Hortonworks
(20)
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
HDF 3.2 - What's New
HDF 3.2 - What's New
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Recently uploaded
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
Anna Loughnan Colquhoun
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
DianaGray10
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
Object Automation
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
YounusS2
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
Eric D. Schabell
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
infogdgmi
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
GDSC PJATK
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
Bachir Benyammi
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Aijun Zhang
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
francesco barbera
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
DianaGray10
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
David Newbury
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
Martin Humpolec
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
Udaiappa Ramachandran
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
IES VE
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
D Cloud Solutions
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
SANGHEE SHIN
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Commit University
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UbiTrack UK
Introduction to Quantum Computing
Introduction to Quantum Computing
GDSC PJATK
Recently uploaded
(20)
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
Introduction to Quantum Computing
Introduction to Quantum Computing
Ozone- Object store for Apache Hadoop
1.
1 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone – Object Store for Apache Hadoop Anu Engineer aengineer@apache.org Arpit Agarwal arp@apache.org
2.
2 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store ⬢ With workloads like IoT we are looking at scaling to trillions of objects. ⬢ Apache HDFS is designed for large objects – not for many small objects – Small files create memory pressure on namenode. ⬢ Each small file creates a block in the datanode. –Datanodes send all block information to namenode in BlockReports. ⬢ Both of these create scalability issues on Namenode. ⬢ Metadata in memory is the strength of the original GFS and HDFS design, but also its weakness in scaling number of files and blocks. ⬢ An object store has simpler semantics than a file system and is easier to scale Apache Hadoop, Hadoop, Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.
3.
3 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store (continued) ⬢ Ozone attempts to scale to trillions of objects – This presentation is about how we will get there. ⬢ Ozone is built on a distributed metadata store. ⬢ Avoids any single server becoming a bottleneck ⬢ More parallelism possible in both data and metadata operations ⬢ Build on well tested components and understood protocols –RAFT for consensus •RAFT is a protocol for reaching consensus between a set of machines in an unreliable environment where machines and network may fail. –Off-the-shelf Key-Value store like LevelDB •LevelDB is an open-source standalone key-value store built by Google.
4.
4 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Alternative solutions to NameNode scalability ⬢ HDFS federation aims to address namespace and Block Space scalability issues. –Federation deployment and planning adds complexity –Requires changes to other components in the Hadoop stack ⬢ HDFS-8286 - Partial Namespace In Memory. –Proposal to keep active working set of namespace in memory. ⬢ HDFS-5477 - Block Management as a Service. –Proposed solution for block space scalability issue. ⬢ Ozone borrows many ideas from these and is a super set of these approaches.
5.
5 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Presentation Outline Ozone Introduction Ozone Architectural Overview Containers Ozone - Bringing it all together Bonus Slides - if we have time.
6.
6 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - Introduction ⬢ An Ozone URL –http://hostname/myvolume/mybucket/mykey ⬢ An S3 URL –http://hostname/mybucket/mykey ⬢ An Azure URL –http://hostname/myaccount/mybucket/key
7.
7 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions ⬢ Storage Volume –A notion similar to an account –Allows admin controls on usage of the object store e.g. storage quota –Created and managed by admins only ⬢ Bucket –Consists of keys and objects –Similar to a bucket in S3 or a container in Azure –ACLs
8.
8 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions (continued) ⬢ Storage Key –Unique in a bucket ⬢ Object –Values in a bucket –Each corresponds to a unique key within a bucket
9.
9 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - REST API ⬢ POST - Creates Volumes and Buckets –Only Admin creates volumes –Bucket can be created by owner of the volume ⬢ PUT - Updates Volumes , Buckets and creates keys –Only admin can change some volume settings –Buckets have ACLs –Creates Keys
10.
1 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone - REST API (continued) ⬢ GET - Lists volumes and buckets and allows reading of keys –Lists Volumes –List Buckets –Get Keys ⬢ DELETE - Deletes volumes, buckets and keys. –Delete Volumes –Delete Buckets –Removes the Key
11.
1 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Components ⬢ Containers – Actual storage locations on Datanodes. –We acknowledge the term container is overloaded. No relation to YARN containers or LXC. –Assume container means a collection of blocks on a datanode for now. –Containers deep dive to follow. ⬢ Ozone Handler - REST frontend for the Ozone rest protocol - deployed on datanodes. ⬢ Storage Container Manager (SCM) - Manages the container life cycle. ⬢ Ozone Key Space Manager (KSM) - Maps Ozone entities to Containers. ⬢ Container Client - Talks to KSM to discover the location of a container and sends IO requests to the appropriate container.
12.
1 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Overview
13.
1 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager
14.
1 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ Key to container mapping service. ⬢ Keeps the key ranges to containers mapping in memory. –Θ(number of containers) - 1 Exabyte cluster = 200M containers x 5GB each. –Memory usage scales with number of containers and not number of keys. ⬢ KSM does NOT know about all the keys in the system. ⬢ KSM state is replicated via RAFT, NameNode-like snapshots.
15.
1 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ KSM knows about Ozone Volumes and Buckets. ⬢ KSM keeps a map of Volumes to container and buckets to containers. ⬢ KSM performs longest prefix match on a given string. ⬢ Example: The user wants to lookup a key - “/volume1/bucket1/key1” –KSM authenticates the user, maps this key to a container and looks up the container location. –Container client gets a token from the KSM and talks to the container on the data node. –Container client makes a getKey call to the datanode container with the full key path. –DataNode validates the access token and serves the value. ⬢ Contents of a bucket may span multiple containers.
16.
1 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved KSM - Bucket spanning multiple containers
17.
1 7 © Hortonworks Inc.
2011 – 2016. All Rights Reserved1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers
18.
1 8 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Container Framework ⬢ A shareable generic block service that can be used by distributed storage services. ⬢ Make it easier to develop new storage services - BYO storage format and naming scheme. ⬢ Design Goals –No single points of failure. All services are replicated. –Avoid bottlenecks •Minimize central state •No verbose Block Reports ⬢ Lessons learned from large scale HDFS clusters. ⬢ Ideas from earlier proposals in HDFS community.
19.
1 9 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Container Framework Components
20.
2 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers
21.
2 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers ⬢ A container is the unit of replication –Size bounded by how quickly it can be re-replicated after a loss. ⬢ Each container is an independent key-value store. –No requirements on the structure or format of keys/values. –Keys are unique only within a container. ⬢ E.g. key-value pair could be one of –An Ozone Key-Value pair –An HDFS block ID and block contents •Or part of a block, when a block spans containers.
22.
2 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers (continued) ⬢ Each container has metadata – Metadata consistency maintained via the RAFT protocol – Metadata consists of keys and references to chunks. ⬢ Container metadata stored in LevelDB. – Exact choice of KV store unimportant. LevelDB is already used by other Hadoop components. ⬢ A chunk is a piece of user data. – Chunks are replicated via a data pipeline. – Chunks can be of arbitrary sizes e.g. a few KB to GBs. – Each chunk reference is a (file, offset, length) triplet. ⬢ Containers may garbage collect unreferenced chunks. ⬢ Each container independently decides how to map chunks to files – Allow reauthoring files for performance, compaction and overwrites.
23.
2 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers (continued)
24.
2 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers support simple client operations ⬢ Write chunks - streaming writes ⬢ Put(key, List<ChunkReference>) –The value is a list of chunk references. –Putting a key makes previously written chunk data visible to readers. –Put overwrites previous versions of the key. ⬢ Get(key) –Returns a list of chunk references ⬢ Read chunks - streaming reads ⬢ Delete(key) ⬢ List Keys
25.
2 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager
26.
2 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager ⬢ A fault-tolerant replicated service ⬢ Replicates its own state using RAFT protocol ⬢ Provides Container Location Service to clients –Given a container ID, return a list of nodes with replicas –Mapping a container ID to Data Nodes (and vice versa) ⬢ Provides Cluster Membership Management –Maintain list of live Data Nodes in the cluster –Handle heartbeats from DataNodes
27.
2 7 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager (continued) ⬢ Provides Replication services –Detect lost container replicas and initiate re-replication –Containers send a container report. •Unlike HDFS block reports which include details about each block , a container report is a summary of information. •This is used by KSM for placement of containers ⬢ If a node suffers from disk failure or if a node is lost, the reconstruction is a local activity which is coordinated via RAFT running on the data nodes.
28.
2 8 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager ⬢ Maintains pre-created containers ⬢ Collects container operation statistics ⬢ Decides which Data Nodes form the replication set for a given container. –The number of replication sets in a cluster is bounded –Borrowing the work done by Facebook and RAMCloud (Copysets, Cidon et al. 2013). ⬢ Important - Knows nothing about keys –Does NOT provide Naming Service (mapping keys to containers) –e.g. KSM provides naming for Ozone.
29.
2 9 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Conceptual Representation of Ozone and Container State
30.
3 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Bringing it all together
31.
3 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Bringing it all together - Ozone createVolume operations Key Space Manager Replicated containers 1: createVolume Container Manager 2: Lookup(volName, Operation) 3: getContainer 4: putMetdata(VolumeName, Properties) Ozone HandlerClient Heartbeats & Reports DataNodes
32.
3 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Tracing an Ozone PutKey Key Space Manager Replicated containers 1: Ozone - putKey 2: Lookup(keyName, Operation) 4: putData(File, offset, Length, data) Client 5: putMetadata(key, List<chunks>) Ozone Handler DataNodes
33.
3 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Tracing an Ozone createVolume OzoneVolume vol = (new VolumeBuilder(pipeLine)) .setCreated(new Date()) .setOwnerName("bilbo") .setClient(client) .setName(“shire”) .build(); POST /shire keyData = {ContainerKeyData} keyName = "shire" containerName = "OzoneContainer" metadata = 0."Created" -> "1449533074362" 1."CreatedBy" -> "gandalf" 2."Key" -> "VOLUME" 3."Owner" -> "bilbo" Ozone REST Handler code Container wire and storage format
34.
3 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Metadata operations ⬢ Any metadata write to a container is Replicated via RAFT. ⬢ Machines forming the replication set for a container comprise a pipeline. ⬢ A createVolume call reduces to putKey operation on the container. ⬢ putKey is consistent, atomic and durable. ⬢ All metadata data operations are done via putKey, getKey and deleteKey. ⬢ Data is written to one or more chunks and a key is updated to point to those chunks. ⬢ Updating the key makes the data visible in the namespace.
35.
3 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Current State of Ozone ⬢ Stand alone container framework. ⬢ Single node ozone using container framework. ⬢ Full REST API -- Command Line Tools and Client Libs are fully functional. ⬢ Active development in branch HDFS-7240. ⬢ Work in progress: –SCM –KSM –Replication Pipeline –RAFT
36.
3 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Acknowledgements ⬢ Ozone is being designed and developed by Jitendra Pandey, Chris Nauroth, Tsz Wo (Nicholas) Sze, Jing Zhao, Suresh Srinivas, Sanjay Radia, Anu Engineer and Arpit Agarwal. ⬢ The Apache community has been very helpful and we were supported by comments and contributions from Kanaka Kumar Avvaru, Edward Bortnikov, Thomas Demoor, Nick Dimiduk, Chris Douglas, Jian Fang, Lars Francke, Gautam Hegde, Lars Hofhansl, Jakob Homan, Virajith Jalaparti, Charles Lamb, Steve Loughran, Haohui Mai, Colin Patrick McCabe, Aaron Myers, Owen O’Malley, Liam Slusser, Jeff Sogolov, Enis Soztutar, Andrew Wang, Fengdong Yu, Zhe Zhang & khanderao.
37.
3 7 © Hortonworks Inc.
2011 – 2016. All Rights Reserved3 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
38.
3 8 © Hortonworks Inc.
2011 – 2016. All Rights Reserved3 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bonus Slides
39.
3 9 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ KSM deals with dynamic partitioning of containers. ⬢ Let us say that a user starts by uploading all his photographs to a bucket in ozone. ⬢ Since all the photographs are called IMG_* (thanks Apple), we will soon overflow the 5GB capacity of the container. ⬢ At this point we need to split the container.
40.
4 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone KSM - Dynamic Container Partitioning ⬢ The container client attempts to write the Nth ozone key, IMG_N, gets a partition required error. ⬢ Container client will take that error and return that info to KSM. ⬢ That error contains the info -- about the proposed split -- That is IMG_0- IMG_200 will stay in this container and IMG_201-IMG_400 will move to next container. ⬢ Note: KSM initiates container partitioning but mechanics of the split are handled by the Container Layer
41.
4 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ One of the assumptions we have made about a container split is that the splits are on the same datanode as the original container. ⬢ This allows us to reduce a split operation to a copy of Keys from one LevelDB to another LevelDB. ⬢ if we need to move actual file data from one datanode to another -- we do support container moves. However they are slow. ⬢ A split on the other hand will complete in seconds in most cases. ⬢ The split point is chosen by the container so that we are able to pick the 50th percentile position that gives us reasonable chance at an equal partition of a container. ⬢ KSM does not know about the keys or the actual data sizes until much later. ⬢ So always relies on the container to tell it where the split should be.
42.
4 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ A container split is done in KSM via updating the Tree. The range partition key we maintain gets updated to reflect the fact that Keys - {IMG_0 - IMG_200} are on container C1, and keys {IMG_201-IMG_Z} are on C2. ⬢ Container will update the SCM when the split is done. ⬢ This information is learned and maintained by KSM.
43.
4 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Soft Quotas ⬢ In the HDFS world, a Quota is hard limit. It is actually conservative in quota management. ⬢ In the ozone world, Quotas are soft quotas. That is users can and will be able to violate it, but KSM/SCM will eventually learn about it and lock the volume out. ⬢ The key reason why this is different is because KSM/SCM is not involved in the allocation of chunks. ⬢ The containers have a partial -- that is an isolated view of the data in a volume. Since volumes can span many containers, it is possible for users to allocate chunks that violate the volume quota, but eventually KSM will learn and disable further writes to a volume.
44.
4 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Missing namespace problem ⬢ One great thing about HDFS is Namenode. –Despite scalability issues, in most cases Namenode does a wonderful job. ⬢ Ozone - Subtle problem if we lose the all replicas of a container. ⬢ We will not only lose data -- just as if HDFS lost its all 3 replicas, but we will also lose information about which keys have been lost. ⬢ To solve this issue, we propose to have a on-disk eventually consistent log maintained by a separate service. –Records information about the keys that exist in the cluster. ⬢ This Scribe service logs the state of the cluster.
45.
4 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone - Range reads ⬢ Ozone supports range reads and might support range writes like part upload in S3. –Ozone achieves this by using the chunk mechanism. –Chunks offer a stream like interface. –You can seek to any location and read as many bytes as you want. –This is used by ozone to support range reads ⬢ Periodic Scanner can reclaim unreferenced chunks.
46.
4 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Scalability - What HDFS does well ⬢ HDFS NN stores all namespace metadata in memory (as per GFS) –Scales to large clusters (5K) since all metadata in memory •60K-100K tasks can share the Namenode •Low latency –Large data if files are large ⬢ Proof points of large data and large clusters –Single Organizations have over 600PB in HDFS –Single clusters with over 200PB using federation –Large clusters of over 4K multi-core nodes hitting a single NN
Download Now