Hadoop Everywhere
•Download as PPTX, PDF•
1 like•578 views
Hadoop Everywhere
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Hadoop Everywhere
Similar to Hadoop Everywhere (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Hadoop Everywhere
- 1. 1© Copyright 2015 EMC Corporation. All rights reserved. HADOOP EVERYWHERE GEO-DISTRIBUTED STORAGE FOR BIG DATA
- 2. 2© Copyright 2015 EMC Corporation. All rights reserved. ABOUT US Nikhil Joshi Consultant Product Manager EMC nikhil.joshi@emc.com @nikhilj0shi Vishrut Shah Director of Engineering EMC vishrut.shah@emc.com
- 3. 3© Copyright 2015 EMC Corporation. All rights reserved. CLOUD APPS & THEIR DATA ARE GLOBAL ParisDenver Beijing …BUT ANALYTICS IS CONFINED TO A SINGLE DATACENTER
- 4. 4© Copyright 2015 EMC Corporation. All rights reserved. Workflow – Real-time analytics on transactional system – Push to Risk-Analytics infra via distcp – Push to a cold-archive for compliance Scale – 3 datacenters, 7 tenants – 200gb a day. Expect 3x increase in 2 years – 27 Hadoop clusters (test, dev, prod, DR) – Regain control from the ‘shadow IT’ BEYOND YOUR FIRST HADOOP CLUSTER What we heard from a large European bank…
- 5. 5© Copyright 2015 EMC Corporation. All rights reserved. GEO-DISTRIBUTED ANALYTICS INFRASTRUCTURE Polyglot Hadoop-ready Storage 3x Replication Exabyte Scale Multi-tenant Large/Small Files Geo-accessible WAN-efficient Active-active Strong Consistency NEEDS TO ADDRESS SILOS, UNDERUTILIZATION, RUNAWAY REPLICATION
- 6. 6© Copyright 2015 EMC Corporation. All rights reserved. ECS: A THOROUGHBRED GEO-PLATFORM Global Namespace Denver Beijing Paris STORAGE EFFICIENCYGEO CACHINGSTRONG CONSISTENCY ACTIVE-ACTIVE w/ FAILOVER Geo-distributed Data Geo-distributed Data Apps
- 7. 7© Copyright 2015 EMC Corporation. All rights reserved. ECS AS A HADOOP STORAGE BACKEND HCFS* Vendor Agnostic Primary Filesystem Geo-distributed Kerberos Enabled Apache Hadoop 2.7 Ambari Compatible Multi-tenant
- 8. 8© Copyright 2015 EMC Corporation. All rights reserved. SHARED STORAGE Hadoop Compute Cluster Shared Storage Backend
- 9. 9© Copyright 2015 EMC Corporation. All rights reserved. HCFS CLIENT LIBRARY Client Library
- 10. 10© Copyright 2015 EMC Corporation. All rights reserved. HaaS Tenant 1 Tenant 2 Tenant 3
- 11. 11© Copyright 2015 EMC Corporation. All rights reserved. Hadoop as a Service Analysts Independently scalable primary storage to run Hadoop analytics Site Compute Grid COMMON CONFIGURATIONS Disaster Recovery Geo-distribution Site 1 Site 2 Compute Grid Archive/failover tier to Hadoop Multi-site analytics Geo-distribution Site 1 Site 2 Compute Grid Distributed storage to run in-place Hadoop analytics across sites Compute Grid
- 12. 12© Copyright 2015 EMC Corporation. All rights reserved. HORTONWORKS CERTIFIED HDP 2.3 – ECS 2.2 Certified and Validated Ambari 2.2 Joint Support Collateral Training
- 13. 13© Copyright 2015 EMC Corporation. All rights reserved. ENOUGH TALK. SHOW ME THE DEMO.
- 14. 14© Copyright 2015 EMC Corporation. All rights reserved.
- 15. 15© Copyright 2015 EMC Corporation. All rights reserved.
Editor's Notes
- Good afternoon everyone, welcome to the session on Geo-Distributed storage for Big Data. Hope you have enjoyed the conference so far. I would like to start our session today with a couple of “show of hands” questions: How many of you have clusters in more than than one region/GEO? How many of you use distcp today? Other tools apart from “distcp”?
- About us…
- 1. Today’s applications generate and consume data from all across the globe. But Hadoop has mostly remained confined to a single datacenter. 2. Hadoop has grown and matured around HDFS as its filesystem. HDFS did a great job in providing a petabyte scale, reliable storage system built on commodity hardware. 3. It’s storage layer is yet to evolve into a global filesystem with characteristics like geo-access, active-active access, selective replication. 4. Most vendor tools have relied on distcp based mechanisms… ---------- New age applications such as Mobile, social and IoT apps create and consume data across the globe. This gives rise to multiple apps creating silos of data that are accessed via different protocols like S3, NFS, Swift etc depending on the application. This data is then further copied into other silos specifically for analytics purposes. This leads to overall operational complexity as well higher storage costs.
- [Animated] Spend a lot of time on this slide. This will be framed as “desirable characteristics of a general geo-hadoop platform”. Stress of active-active and strong consistency. Not ECS specific yet.
- Reiterate the concept from previous slide – Mention WAN distances
- Spend time here. Hopefully, HCFS has already been explained before.
- Vishrut can take over from here. I will try to do a humorous hand off. Something like “Now, my technical colleague will show you that this is not all smoke and mirrors. He has also graciously agreed to field all hard questions”.
- [Animated] Point audience to some material available for further reading about this partnership
- Yet to decide if we want to move this inside our video demo, or do this via a screenshot-based walkthough
- [Animated] Simple animation to ensure that audience understands “global hadoop” only applies to storage. Apps (MR/Hbase) are not geo-accessible. And to demonstrate this solution only replaces the storage layer (HDFS)