SlideShare a Scribd company logo
1 of 46
Download to read offline
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone – Object Store for Apache
Hadoop
Anu Engineer
aengineer@apache.org
Arpit Agarwal
arp@apache.org
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone – Why an Object Store
⬢ With workloads like IoT we are looking at scaling to trillions of objects.
⬢ Apache HDFS is designed for large objects – not for many small objects
– Small files create memory pressure on namenode.
⬢ Each small file creates a block in the datanode.
–Datanodes send all block information to namenode in BlockReports.
⬢ Both of these create scalability issues on Namenode.
⬢ Metadata in memory is the strength of the original GFS and HDFS design, but also its
weakness in scaling number of files and blocks.
⬢ An object store has simpler semantics than a file system and is easier to scale
Apache Hadoop, Hadoop, Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone – Why an Object Store (continued)
⬢ Ozone attempts to scale to trillions of objects
– This presentation is about how we will get there.
⬢ Ozone is built on a distributed metadata store.
⬢ Avoids any single server becoming a bottleneck
⬢ More parallelism possible in both data and metadata operations
⬢ Build on well tested components and understood protocols
–RAFT for consensus
•RAFT is a protocol for reaching consensus between a set of machines in an unreliable
environment where machines and network may fail.
–Off-the-shelf Key-Value store like LevelDB
•LevelDB is an open-source standalone key-value store built by Google.
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alternative solutions to NameNode scalability
⬢ HDFS federation aims to address namespace and Block Space scalability issues.
–Federation deployment and planning adds complexity
–Requires changes to other components in the Hadoop stack
⬢ HDFS-8286 - Partial Namespace In Memory.
–Proposal to keep active working set of namespace in memory.
⬢ HDFS-5477 - Block Management as a Service.
–Proposed solution for block space scalability issue.
⬢ Ozone borrows many ideas from these and is a super set of these approaches.
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Presentation Outline
Ozone Introduction
Ozone Architectural Overview
Containers
Ozone - Bringing it all together
Bonus Slides - if we have time.
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Introduction
⬢ An Ozone URL
–http://hostname/myvolume/mybucket/mykey
⬢ An S3 URL
–http://hostname/mybucket/mykey
⬢ An Azure URL
–http://hostname/myaccount/mybucket/key
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Definitions
⬢ Storage Volume
–A notion similar to an account
–Allows admin controls on usage of the object store e.g. storage quota
–Created and managed by admins only
⬢ Bucket
–Consists of keys and objects
–Similar to a bucket in S3 or a container in Azure
–ACLs
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Definitions (continued)
⬢ Storage Key
–Unique in a bucket
⬢ Object
–Values in a bucket
–Each corresponds to a unique key within a bucket
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - REST API
⬢ POST - Creates Volumes and Buckets
–Only Admin creates volumes
–Bucket can be created by owner of the volume
⬢ PUT - Updates Volumes , Buckets and creates keys
–Only admin can change some volume settings
–Buckets have ACLs
–Creates Keys
1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - REST API (continued)
⬢ GET - Lists volumes and buckets and allows reading of keys
–Lists Volumes
–List Buckets
–Get Keys
⬢ DELETE - Deletes volumes, buckets and keys.
–Delete Volumes
–Delete Buckets
–Removes the Key
1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Components
⬢ Containers – Actual storage locations on Datanodes.
–We acknowledge the term container is overloaded. No relation to YARN containers or LXC.
–Assume container means a collection of blocks on a datanode for now.
–Containers deep dive to follow.
⬢ Ozone Handler - REST frontend for the Ozone rest protocol - deployed on datanodes.
⬢ Storage Container Manager (SCM) - Manages the container life cycle.
⬢ Ozone Key Space Manager (KSM) - Maps Ozone entities to Containers.
⬢ Container Client - Talks to KSM to discover the location of a container and sends IO
requests to the appropriate container.
1
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Overview
1
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager
1
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager
⬢ Key to container mapping service.
⬢ Keeps the key ranges to containers mapping in memory.
–Θ(number of containers) - 1 Exabyte cluster = 200M containers x 5GB each.
–Memory usage scales with number of containers and not number of keys.
⬢ KSM does NOT know about all the keys in the system.
⬢ KSM state is replicated via RAFT, NameNode-like snapshots.
1
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager
⬢ KSM knows about Ozone Volumes and Buckets.
⬢ KSM keeps a map of Volumes to container and buckets to containers.
⬢ KSM performs longest prefix match on a given string.
⬢ Example: The user wants to lookup a key - “/volume1/bucket1/key1”
–KSM authenticates the user, maps this key to a container and looks up the container location.
–Container client gets a token from the KSM and talks to the container on the data node.
–Container client makes a getKey call to the datanode container with the full key path.
–DataNode validates the access token and serves the value.
⬢ Contents of a bucket may span multiple containers.
1
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
KSM - Bucket spanning multiple containers
1
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved1
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers
1
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Container Framework
⬢ A shareable generic block service that can be used by distributed storage services.
⬢ Make it easier to develop new storage services - BYO storage format and naming
scheme.
⬢ Design Goals
–No single points of failure. All services are replicated.
–Avoid bottlenecks
•Minimize central state
•No verbose Block Reports
⬢ Lessons learned from large scale HDFS clusters.
⬢ Ideas from earlier proposals in HDFS community.
1
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Container Framework Components
2
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers
2
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers
⬢ A container is the unit of replication
–Size bounded by how quickly it can be re-replicated after a loss.
⬢ Each container is an independent key-value store.
–No requirements on the structure or format of keys/values.
–Keys are unique only within a container.
⬢ E.g. key-value pair could be one of
–An Ozone Key-Value pair
–An HDFS block ID and block contents
•Or part of a block, when a block spans containers.
2
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers (continued)
⬢ Each container has metadata
– Metadata consistency maintained via the RAFT protocol
– Metadata consists of keys and references to chunks.
⬢ Container metadata stored in LevelDB.
– Exact choice of KV store unimportant. LevelDB is already used by other Hadoop components.
⬢ A chunk is a piece of user data.
– Chunks are replicated via a data pipeline.
– Chunks can be of arbitrary sizes e.g. a few KB to GBs.
– Each chunk reference is a (file, offset, length) triplet.
⬢ Containers may garbage collect unreferenced chunks.
⬢ Each container independently decides how to map chunks to files
– Allow reauthoring files for performance, compaction and overwrites.
2
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers (continued)
2
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Containers support simple client operations
⬢ Write chunks - streaming writes
⬢ Put(key, List<ChunkReference>)
–The value is a list of chunk references.
–Putting a key makes previously written chunk data visible to readers.
–Put overwrites previous versions of the key.
⬢ Get(key)
–Returns a list of chunk references
⬢ Read chunks - streaming reads
⬢ Delete(key)
⬢ List Keys
2
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager
2
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager
⬢ A fault-tolerant replicated service
⬢ Replicates its own state using RAFT protocol
⬢ Provides Container Location Service to clients
–Given a container ID, return a list of nodes with replicas
–Mapping a container ID to Data Nodes (and vice versa)
⬢ Provides Cluster Membership Management
–Maintain list of live Data Nodes in the cluster
–Handle heartbeats from DataNodes
2
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager (continued)
⬢ Provides Replication services
–Detect lost container replicas and initiate re-replication
–Containers send a container report.
•Unlike HDFS block reports which include details about each block , a container report is a
summary of information.
•This is used by KSM for placement of containers
⬢ If a node suffers from disk failure or if a node is lost, the reconstruction is a local
activity which is coordinated via RAFT running on the data nodes.
2
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Storage Container Manager
⬢ Maintains pre-created containers
⬢ Collects container operation statistics
⬢ Decides which Data Nodes form the replication set for a given container.
–The number of replication sets in a cluster is bounded
–Borrowing the work done by Facebook and RAMCloud (Copysets, Cidon et al. 2013).
⬢ Important - Knows nothing about keys
–Does NOT provide Naming Service (mapping keys to containers)
–e.g. KSM provides naming for Ozone.
2
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Conceptual Representation of Ozone and Container State
3
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Bringing it all together
3
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bringing it all together - Ozone createVolume operations
Key Space Manager
Replicated containers
1: createVolume
Container Manager
2: Lookup(volName, Operation) 3: getContainer
4: putMetdata(VolumeName, Properties)
Ozone HandlerClient
Heartbeats & Reports
DataNodes
3
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing an Ozone PutKey
Key Space Manager
Replicated containers
1: Ozone - putKey
2: Lookup(keyName, Operation)
4: putData(File, offset, Length, data)
Client
5: putMetadata(key, List<chunks>)
Ozone Handler
DataNodes
3
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing an Ozone createVolume
OzoneVolume vol =
(new VolumeBuilder(pipeLine))
.setCreated(new Date())
.setOwnerName("bilbo")
.setClient(client)
.setName(“shire”)
.build();
POST /shire
keyData = {ContainerKeyData}
keyName = "shire"
containerName = "OzoneContainer"
metadata =
0."Created" -> "1449533074362"
1."CreatedBy" -> "gandalf"
2."Key" -> "VOLUME"
3."Owner" -> "bilbo"
Ozone REST Handler code Container wire and storage format
3
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Metadata operations
⬢ Any metadata write to a container is Replicated via RAFT.
⬢ Machines forming the replication set for a container comprise a pipeline.
⬢ A createVolume call reduces to putKey operation on the container.
⬢ putKey is consistent, atomic and durable.
⬢ All metadata data operations are done via putKey, getKey and deleteKey.
⬢ Data is written to one or more chunks and a key is updated to point to those chunks.
⬢ Updating the key makes the data visible in the namespace.
3
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Current State of Ozone
⬢ Stand alone container framework.
⬢ Single node ozone using container framework.
⬢ Full REST API -- Command Line Tools and Client Libs are fully functional.
⬢ Active development in branch HDFS-7240.
⬢ Work in progress:
–SCM
–KSM
–Replication Pipeline
–RAFT
3
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Acknowledgements
⬢ Ozone is being designed and developed by Jitendra Pandey, Chris Nauroth, Tsz Wo
(Nicholas) Sze, Jing Zhao, Suresh Srinivas, Sanjay Radia, Anu Engineer and Arpit
Agarwal.
⬢ The Apache community has been very helpful and we were supported by comments
and contributions from Kanaka Kumar Avvaru, Edward Bortnikov, Thomas Demoor,
Nick Dimiduk, Chris Douglas, Jian Fang, Lars Francke, Gautam Hegde, Lars Hofhansl,
Jakob Homan, Virajith Jalaparti, Charles Lamb, Steve Loughran, Haohui Mai, Colin
Patrick McCabe, Aaron Myers, Owen O’Malley, Liam Slusser, Jeff Sogolov, Enis
Soztutar, Andrew Wang, Fengdong Yu, Zhe Zhang & khanderao.
3
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You
3
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bonus Slides
3
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Dynamic Container Partitioning
⬢ KSM deals with dynamic partitioning of containers.
⬢ Let us say that a user starts by uploading all his photographs to a bucket in ozone.
⬢ Since all the photographs are called IMG_* (thanks Apple), we will soon overflow the
5GB capacity of the container.
⬢ At this point we need to split the container.
4
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone KSM - Dynamic Container Partitioning
⬢ The container client attempts to write the Nth ozone key, IMG_N, gets a partition
required error.
⬢ Container client will take that error and return that info to KSM.
⬢ That error contains the info -- about the proposed split -- That is IMG_0- IMG_200 will
stay in this container and IMG_201-IMG_400 will move to next container.
⬢ Note: KSM initiates container partitioning but mechanics of the split are handled by
the Container Layer
4
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Dynamic Container Partitioning
⬢ One of the assumptions we have made about a container split is that the splits are on
the same datanode as the original container.
⬢ This allows us to reduce a split operation to a copy of Keys from one LevelDB to
another LevelDB.
⬢ if we need to move actual file data from one datanode to another -- we do support
container moves. However they are slow.
⬢ A split on the other hand will complete in seconds in most cases.
⬢ The split point is chosen by the container so that we are able to pick the 50th percentile
position that gives us reasonable chance at an equal partition of a container.
⬢ KSM does not know about the keys or the actual data sizes until much later.
⬢ So always relies on the container to tell it where the split should be.
4
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Dynamic Container Partitioning
⬢ A container split is done in KSM via updating the Tree. The range partition key we
maintain gets updated to reflect the fact that Keys - {IMG_0 - IMG_200} are on
container C1, and keys {IMG_201-IMG_Z} are on C2.
⬢ Container will update the SCM when the split is done.
⬢ This information is learned and maintained by KSM.
4
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Soft Quotas
⬢ In the HDFS world, a Quota is hard limit. It is actually conservative in quota
management.
⬢ In the ozone world, Quotas are soft quotas. That is users can and will be able to violate
it, but KSM/SCM will eventually learn about it and lock the volume out.
⬢ The key reason why this is different is because KSM/SCM is not involved in the
allocation of chunks.
⬢ The containers have a partial -- that is an isolated view of the data in a volume. Since
volumes can span many containers, it is possible for users to allocate chunks that
violate the volume quota, but eventually KSM will learn and disable further writes to a
volume.
4
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone Key Space Manager - Missing namespace problem
⬢ One great thing about HDFS is Namenode.
–Despite scalability issues, in most cases Namenode does a wonderful job.
⬢ Ozone - Subtle problem if we lose the all replicas of a container.
⬢ We will not only lose data -- just as if HDFS lost its all 3 replicas, but we will also lose
information about which keys have been lost.
⬢ To solve this issue, we propose to have a on-disk eventually consistent log maintained
by a separate service.
–Records information about the keys that exist in the cluster.
⬢ This Scribe service logs the state of the cluster.
4
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ozone - Range reads
⬢ Ozone supports range reads and might support range writes like part upload in S3.
–Ozone achieves this by using the chunk mechanism.
–Chunks offer a stream like interface.
–You can seek to any location and read as many bytes as you want.
–This is used by ozone to support range reads
⬢ Periodic Scanner can reclaim unreferenced chunks.
4
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scalability - What HDFS does well
⬢ HDFS NN stores all namespace metadata in memory (as per GFS)
–Scales to large clusters (5K) since all metadata in memory
•60K-100K tasks can share the Namenode
•Low latency
–Large data if files are large
⬢ Proof points of large data and large clusters
–Single Organizations have over 600PB in HDFS
–Single clusters with over 200PB using federation
–Large clusters of over 4K multi-core nodes hitting a single NN

More Related Content

What's hot

Apache Hadoop In Theory And Practice
Apache Hadoop In Theory And PracticeApache Hadoop In Theory And Practice
Apache Hadoop In Theory And PracticeAdam Kawa
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks
 
Apache Hudi: The Path Forward
Apache Hudi: The Path ForwardApache Hudi: The Path Forward
Apache Hudi: The Path ForwardAlluxio, Inc.
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBaseHBaseCon
 
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveDataWorks Summit
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
 
Ozone and HDFS's Evolution
Ozone and HDFS's EvolutionOzone and HDFS's Evolution
Ozone and HDFS's EvolutionDataWorks Summit
 
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneHadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneErik Krogen
 
Delta from a Data Engineer's Perspective
Delta from a Data Engineer's PerspectiveDelta from a Data Engineer's Perspective
Delta from a Data Engineer's PerspectiveDatabricks
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3DataWorks Summit
 
Apache doris (incubating) introduction
Apache doris (incubating) introductionApache doris (incubating) introduction
Apache doris (incubating) introductionleanderlee2
 
Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0Databricks
 
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...Databricks
 
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingApache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingDataWorks Summit
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Ryan Blue
 
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing DataWorks Summit
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataDataWorks Summit
 
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit
 
Hive 3 - a new horizon
Hive 3 - a new horizonHive 3 - a new horizon
Hive 3 - a new horizonThejas Nair
 

What's hot (20)

Apache Hadoop In Theory And Practice
Apache Hadoop In Theory And PracticeApache Hadoop In Theory And Practice
Apache Hadoop In Theory And Practice
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
Apache Hudi: The Path Forward
Apache Hudi: The Path ForwardApache Hudi: The Path Forward
Apache Hudi: The Path Forward
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBase
 
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
 
Ozone and HDFS's Evolution
Ozone and HDFS's EvolutionOzone and HDFS's Evolution
Ozone and HDFS's Evolution
 
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneHadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
 
Delta from a Data Engineer's Perspective
Delta from a Data Engineer's PerspectiveDelta from a Data Engineer's Perspective
Delta from a Data Engineer's Perspective
 
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataflow with Apache NiFi
Dataflow with Apache NiFi
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3
 
Apache doris (incubating) introduction
Apache doris (incubating) introductionApache doris (incubating) introduction
Apache doris (incubating) introduction
 
Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0Achieving Lakehouse Models with Spark 3.0
Achieving Lakehouse Models with Spark 3.0
 
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
 
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingApache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
 
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
 
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the Beast
 
Hive 3 - a new horizon
Hive 3 - a new horizonHive 3 - a new horizon
Hive 3 - a new horizon
 

Similar to Ozone- Object store for Apache Hadoop

Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objectsOzone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objectsDataWorks Summit
 
Ozone and HDFS’s evolution
Ozone and HDFS’s evolutionOzone and HDFS’s evolution
Ozone and HDFS’s evolutionDataWorks Summit
 
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage  object store integration in production (final)Hadoop & cloud storage  object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)Chris Nauroth
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionDataWorks Summit/Hadoop Summit
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionDataWorks Summit/Hadoop Summit
 
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage SubsystemEvolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage SubsystemDataWorks Summit/Hadoop Summit
 
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFSCBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFSDataWorks Summit
 
Big data spain keynote nov 2016
Big data spain keynote nov 2016Big data spain keynote nov 2016
Big data spain keynote nov 2016alanfgates
 
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseHBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseCloudera, Inc.
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
 
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...Big Data Spain
 
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for BeginnersThe Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for BeginnersEdelweiss Kammermann
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACIDHortonworks
 
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016alanfgates
 

Similar to Ozone- Object store for Apache Hadoop (20)

Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objectsOzone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objects
 
Ozone and HDFS’s evolution
Ozone and HDFS’s evolutionOzone and HDFS’s evolution
Ozone and HDFS’s evolution
 
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage SubsystemEvolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
 
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage  object store integration in production (final)Hadoop & cloud storage  object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
 
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in ProductionHadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
 
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage SubsystemEvolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
 
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFSCBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
 
Hadoop 3 in a Nutshell
Hadoop 3 in a NutshellHadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
 
Running Services on YARN
Running Services on YARNRunning Services on YARN
Running Services on YARN
 
Big data spain keynote nov 2016
Big data spain keynote nov 2016Big data spain keynote nov 2016
Big data spain keynote nov 2016
 
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage SubsystemEvolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
 
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBaseHBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
 
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and FutureApache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
 
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for BeginnersThe Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACID
 
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
 
Apache Hive on ACID
Apache Hive on ACIDApache Hive on ACID
Apache Hive on ACID
 

More from Hortonworks

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyHortonworks
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakHortonworks
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsHortonworks
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysHortonworks
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's NewHortonworks
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerHortonworks
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsHortonworks
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeHortonworks
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidHortonworks
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleHortonworks
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATAHortonworks
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Hortonworks
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseHortonworks
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseHortonworks
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationHortonworks
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementHortonworks
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHortonworks
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCHortonworks
 

More from Hortonworks (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's New
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
 

Recently uploaded

Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 

Recently uploaded (20)

Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 

Ozone- Object store for Apache Hadoop

  • 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone – Object Store for Apache Hadoop Anu Engineer aengineer@apache.org Arpit Agarwal arp@apache.org
  • 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store ⬢ With workloads like IoT we are looking at scaling to trillions of objects. ⬢ Apache HDFS is designed for large objects – not for many small objects – Small files create memory pressure on namenode. ⬢ Each small file creates a block in the datanode. –Datanodes send all block information to namenode in BlockReports. ⬢ Both of these create scalability issues on Namenode. ⬢ Metadata in memory is the strength of the original GFS and HDFS design, but also its weakness in scaling number of files and blocks. ⬢ An object store has simpler semantics than a file system and is easier to scale Apache Hadoop, Hadoop, Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.
  • 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store (continued) ⬢ Ozone attempts to scale to trillions of objects – This presentation is about how we will get there. ⬢ Ozone is built on a distributed metadata store. ⬢ Avoids any single server becoming a bottleneck ⬢ More parallelism possible in both data and metadata operations ⬢ Build on well tested components and understood protocols –RAFT for consensus •RAFT is a protocol for reaching consensus between a set of machines in an unreliable environment where machines and network may fail. –Off-the-shelf Key-Value store like LevelDB •LevelDB is an open-source standalone key-value store built by Google.
  • 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alternative solutions to NameNode scalability ⬢ HDFS federation aims to address namespace and Block Space scalability issues. –Federation deployment and planning adds complexity –Requires changes to other components in the Hadoop stack ⬢ HDFS-8286 - Partial Namespace In Memory. –Proposal to keep active working set of namespace in memory. ⬢ HDFS-5477 - Block Management as a Service. –Proposed solution for block space scalability issue. ⬢ Ozone borrows many ideas from these and is a super set of these approaches.
  • 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Presentation Outline Ozone Introduction Ozone Architectural Overview Containers Ozone - Bringing it all together Bonus Slides - if we have time.
  • 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Introduction ⬢ An Ozone URL –http://hostname/myvolume/mybucket/mykey ⬢ An S3 URL –http://hostname/mybucket/mykey ⬢ An Azure URL –http://hostname/myaccount/mybucket/key
  • 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions ⬢ Storage Volume –A notion similar to an account –Allows admin controls on usage of the object store e.g. storage quota –Created and managed by admins only ⬢ Bucket –Consists of keys and objects –Similar to a bucket in S3 or a container in Azure –ACLs
  • 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions (continued) ⬢ Storage Key –Unique in a bucket ⬢ Object –Values in a bucket –Each corresponds to a unique key within a bucket
  • 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - REST API ⬢ POST - Creates Volumes and Buckets –Only Admin creates volumes –Bucket can be created by owner of the volume ⬢ PUT - Updates Volumes , Buckets and creates keys –Only admin can change some volume settings –Buckets have ACLs –Creates Keys
  • 10. 1 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - REST API (continued) ⬢ GET - Lists volumes and buckets and allows reading of keys –Lists Volumes –List Buckets –Get Keys ⬢ DELETE - Deletes volumes, buckets and keys. –Delete Volumes –Delete Buckets –Removes the Key
  • 11. 1 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Components ⬢ Containers – Actual storage locations on Datanodes. –We acknowledge the term container is overloaded. No relation to YARN containers or LXC. –Assume container means a collection of blocks on a datanode for now. –Containers deep dive to follow. ⬢ Ozone Handler - REST frontend for the Ozone rest protocol - deployed on datanodes. ⬢ Storage Container Manager (SCM) - Manages the container life cycle. ⬢ Ozone Key Space Manager (KSM) - Maps Ozone entities to Containers. ⬢ Container Client - Talks to KSM to discover the location of a container and sends IO requests to the appropriate container.
  • 12. 1 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Overview
  • 13. 1 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager
  • 14. 1 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ Key to container mapping service. ⬢ Keeps the key ranges to containers mapping in memory. –Θ(number of containers) - 1 Exabyte cluster = 200M containers x 5GB each. –Memory usage scales with number of containers and not number of keys. ⬢ KSM does NOT know about all the keys in the system. ⬢ KSM state is replicated via RAFT, NameNode-like snapshots.
  • 15. 1 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ KSM knows about Ozone Volumes and Buckets. ⬢ KSM keeps a map of Volumes to container and buckets to containers. ⬢ KSM performs longest prefix match on a given string. ⬢ Example: The user wants to lookup a key - “/volume1/bucket1/key1” –KSM authenticates the user, maps this key to a container and looks up the container location. –Container client gets a token from the KSM and talks to the container on the data node. –Container client makes a getKey call to the datanode container with the full key path. –DataNode validates the access token and serves the value. ⬢ Contents of a bucket may span multiple containers.
  • 16. 1 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved KSM - Bucket spanning multiple containers
  • 17. 1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers
  • 18. 1 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Container Framework ⬢ A shareable generic block service that can be used by distributed storage services. ⬢ Make it easier to develop new storage services - BYO storage format and naming scheme. ⬢ Design Goals –No single points of failure. All services are replicated. –Avoid bottlenecks •Minimize central state •No verbose Block Reports ⬢ Lessons learned from large scale HDFS clusters. ⬢ Ideas from earlier proposals in HDFS community.
  • 19. 1 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Container Framework Components
  • 20. 2 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers
  • 21. 2 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers ⬢ A container is the unit of replication –Size bounded by how quickly it can be re-replicated after a loss. ⬢ Each container is an independent key-value store. –No requirements on the structure or format of keys/values. –Keys are unique only within a container. ⬢ E.g. key-value pair could be one of –An Ozone Key-Value pair –An HDFS block ID and block contents •Or part of a block, when a block spans containers.
  • 22. 2 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers (continued) ⬢ Each container has metadata – Metadata consistency maintained via the RAFT protocol – Metadata consists of keys and references to chunks. ⬢ Container metadata stored in LevelDB. – Exact choice of KV store unimportant. LevelDB is already used by other Hadoop components. ⬢ A chunk is a piece of user data. – Chunks are replicated via a data pipeline. – Chunks can be of arbitrary sizes e.g. a few KB to GBs. – Each chunk reference is a (file, offset, length) triplet. ⬢ Containers may garbage collect unreferenced chunks. ⬢ Each container independently decides how to map chunks to files – Allow reauthoring files for performance, compaction and overwrites.
  • 23. 2 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers (continued)
  • 24. 2 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers support simple client operations ⬢ Write chunks - streaming writes ⬢ Put(key, List<ChunkReference>) –The value is a list of chunk references. –Putting a key makes previously written chunk data visible to readers. –Put overwrites previous versions of the key. ⬢ Get(key) –Returns a list of chunk references ⬢ Read chunks - streaming reads ⬢ Delete(key) ⬢ List Keys
  • 25. 2 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager
  • 26. 2 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager ⬢ A fault-tolerant replicated service ⬢ Replicates its own state using RAFT protocol ⬢ Provides Container Location Service to clients –Given a container ID, return a list of nodes with replicas –Mapping a container ID to Data Nodes (and vice versa) ⬢ Provides Cluster Membership Management –Maintain list of live Data Nodes in the cluster –Handle heartbeats from DataNodes
  • 27. 2 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager (continued) ⬢ Provides Replication services –Detect lost container replicas and initiate re-replication –Containers send a container report. •Unlike HDFS block reports which include details about each block , a container report is a summary of information. •This is used by KSM for placement of containers ⬢ If a node suffers from disk failure or if a node is lost, the reconstruction is a local activity which is coordinated via RAFT running on the data nodes.
  • 28. 2 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storage Container Manager ⬢ Maintains pre-created containers ⬢ Collects container operation statistics ⬢ Decides which Data Nodes form the replication set for a given container. –The number of replication sets in a cluster is bounded –Borrowing the work done by Facebook and RAMCloud (Copysets, Cidon et al. 2013). ⬢ Important - Knows nothing about keys –Does NOT provide Naming Service (mapping keys to containers) –e.g. KSM provides naming for Ozone.
  • 29. 2 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Conceptual Representation of Ozone and Container State
  • 30. 3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Bringing it all together
  • 31. 3 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bringing it all together - Ozone createVolume operations Key Space Manager Replicated containers 1: createVolume Container Manager 2: Lookup(volName, Operation) 3: getContainer 4: putMetdata(VolumeName, Properties) Ozone HandlerClient Heartbeats & Reports DataNodes
  • 32. 3 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing an Ozone PutKey Key Space Manager Replicated containers 1: Ozone - putKey 2: Lookup(keyName, Operation) 4: putData(File, offset, Length, data) Client 5: putMetadata(key, List<chunks>) Ozone Handler DataNodes
  • 33. 3 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing an Ozone createVolume OzoneVolume vol = (new VolumeBuilder(pipeLine)) .setCreated(new Date()) .setOwnerName("bilbo") .setClient(client) .setName(“shire”) .build(); POST /shire keyData = {ContainerKeyData} keyName = "shire" containerName = "OzoneContainer" metadata = 0."Created" -> "1449533074362" 1."CreatedBy" -> "gandalf" 2."Key" -> "VOLUME" 3."Owner" -> "bilbo" Ozone REST Handler code Container wire and storage format
  • 34. 3 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Metadata operations ⬢ Any metadata write to a container is Replicated via RAFT. ⬢ Machines forming the replication set for a container comprise a pipeline. ⬢ A createVolume call reduces to putKey operation on the container. ⬢ putKey is consistent, atomic and durable. ⬢ All metadata data operations are done via putKey, getKey and deleteKey. ⬢ Data is written to one or more chunks and a key is updated to point to those chunks. ⬢ Updating the key makes the data visible in the namespace.
  • 35. 3 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Current State of Ozone ⬢ Stand alone container framework. ⬢ Single node ozone using container framework. ⬢ Full REST API -- Command Line Tools and Client Libs are fully functional. ⬢ Active development in branch HDFS-7240. ⬢ Work in progress: –SCM –KSM –Replication Pipeline –RAFT
  • 36. 3 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Acknowledgements ⬢ Ozone is being designed and developed by Jitendra Pandey, Chris Nauroth, Tsz Wo (Nicholas) Sze, Jing Zhao, Suresh Srinivas, Sanjay Radia, Anu Engineer and Arpit Agarwal. ⬢ The Apache community has been very helpful and we were supported by comments and contributions from Kanaka Kumar Avvaru, Edward Bortnikov, Thomas Demoor, Nick Dimiduk, Chris Douglas, Jian Fang, Lars Francke, Gautam Hegde, Lars Hofhansl, Jakob Homan, Virajith Jalaparti, Charles Lamb, Steve Loughran, Haohui Mai, Colin Patrick McCabe, Aaron Myers, Owen O’Malley, Liam Slusser, Jeff Sogolov, Enis Soztutar, Andrew Wang, Fengdong Yu, Zhe Zhang & khanderao.
  • 37. 3 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
  • 38. 3 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bonus Slides
  • 39. 3 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ KSM deals with dynamic partitioning of containers. ⬢ Let us say that a user starts by uploading all his photographs to a bucket in ozone. ⬢ Since all the photographs are called IMG_* (thanks Apple), we will soon overflow the 5GB capacity of the container. ⬢ At this point we need to split the container.
  • 40. 4 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone KSM - Dynamic Container Partitioning ⬢ The container client attempts to write the Nth ozone key, IMG_N, gets a partition required error. ⬢ Container client will take that error and return that info to KSM. ⬢ That error contains the info -- about the proposed split -- That is IMG_0- IMG_200 will stay in this container and IMG_201-IMG_400 will move to next container. ⬢ Note: KSM initiates container partitioning but mechanics of the split are handled by the Container Layer
  • 41. 4 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ One of the assumptions we have made about a container split is that the splits are on the same datanode as the original container. ⬢ This allows us to reduce a split operation to a copy of Keys from one LevelDB to another LevelDB. ⬢ if we need to move actual file data from one datanode to another -- we do support container moves. However they are slow. ⬢ A split on the other hand will complete in seconds in most cases. ⬢ The split point is chosen by the container so that we are able to pick the 50th percentile position that gives us reasonable chance at an equal partition of a container. ⬢ KSM does not know about the keys or the actual data sizes until much later. ⬢ So always relies on the container to tell it where the split should be.
  • 42. 4 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ A container split is done in KSM via updating the Tree. The range partition key we maintain gets updated to reflect the fact that Keys - {IMG_0 - IMG_200} are on container C1, and keys {IMG_201-IMG_Z} are on C2. ⬢ Container will update the SCM when the split is done. ⬢ This information is learned and maintained by KSM.
  • 43. 4 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Soft Quotas ⬢ In the HDFS world, a Quota is hard limit. It is actually conservative in quota management. ⬢ In the ozone world, Quotas are soft quotas. That is users can and will be able to violate it, but KSM/SCM will eventually learn about it and lock the volume out. ⬢ The key reason why this is different is because KSM/SCM is not involved in the allocation of chunks. ⬢ The containers have a partial -- that is an isolated view of the data in a volume. Since volumes can span many containers, it is possible for users to allocate chunks that violate the volume quota, but eventually KSM will learn and disable further writes to a volume.
  • 44. 4 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone Key Space Manager - Missing namespace problem ⬢ One great thing about HDFS is Namenode. –Despite scalability issues, in most cases Namenode does a wonderful job. ⬢ Ozone - Subtle problem if we lose the all replicas of a container. ⬢ We will not only lose data -- just as if HDFS lost its all 3 replicas, but we will also lose information about which keys have been lost. ⬢ To solve this issue, we propose to have a on-disk eventually consistent log maintained by a separate service. –Records information about the keys that exist in the cluster. ⬢ This Scribe service logs the state of the cluster.
  • 45. 4 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Range reads ⬢ Ozone supports range reads and might support range writes like part upload in S3. –Ozone achieves this by using the chunk mechanism. –Chunks offer a stream like interface. –You can seek to any location and read as many bytes as you want. –This is used by ozone to support range reads ⬢ Periodic Scanner can reclaim unreferenced chunks.
  • 46. 4 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Scalability - What HDFS does well ⬢ HDFS NN stores all namespace metadata in memory (as per GFS) –Scales to large clusters (5K) since all metadata in memory •60K-100K tasks can share the Namenode •Low latency –Large data if files are large ⬢ Proof points of large data and large clusters –Single Organizations have over 600PB in HDFS –Single clusters with over 200PB using federation –Large clusters of over 4K multi-core nodes hitting a single NN