Hortonworks Data Cloud for AWS 1.11 Updates

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Cloud Updates
Enterprise ready Hadoop on the cloud
蒋 逸峰（しょう いつほう／Yifeng Jiang）
SoluBons Engineer, Hortonworks
@uprush
March 22, 2017

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
à HDC Overview
à Enterprise Readness
à HDC 1.11 Feature Highlights
à DEMO

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Cloud Solu;ons
Microso= AWS Google
Managed Azure HDInsight
Marketplace
Hortonworks Data
Cloud for AWS
IaaS Hortonworks Data PlaDorm

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Cloud for AWS
à A new product from Hortonworks
powered by Hortonworks Data PlaRorm
à Delivered and sold via AWS Marketplace
as of November 15, 2016
à Focuses on business agility, rather than
infinite configurability and cluster
management
à Pre-tuned and configured for use with
AWS
à Addresses prescripBve, ephemeral use
cases around Apache Spark + Apache Hive
à Offers Pay-As-You-Go (PAYG) pricing

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Product Architecture
cloud
controller
(powered by Cloudbreak)
Master
Node
Worker
Node
Worker
Node
Worker
Node
Master
Node
Worker
Node
Worker
Node
Worker
Node
Master
Node
Worker
Node
Worker
Node
Worker
Node
HDP Cluster A
HDP Cluster B
HDP Cluster C
AMI-1 AMI-2 AMI-2 AMI-2 AMI-2
AMI-2 AMI-2 AMI-2 AMI-2
AMI-2 AMI-2 AMI-2 AMI-2
Docker

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Two AWS Marketplace Products
cloud
controller
(powered by Cloudbreak)
Master
Node
Worker
Node
Worker
Node
Worker
Node
Master
Node
Worker
Node
Worker
Node
Worker
Node
Master
Node
Worker
Node
Worker
Node
Worker
Node
HDP Cluster A
HDP Cluster B
HDP Cluster C
AMI-1 AMI-2 AMI-2 AMI-2 AMI-2
AMI-2 AMI-2 AMI-2 AMI-2
AMI-2 AMI-2 AMI-2 AMI-2
Docker
Hortonworks Data Cloud
Controller Service
Hortonworks Data Cloud
HDP Services
Hortonworks Data Cloud
HDP Services
Hortonworks Data Cloud
HDP Services
One for
Cloud Controller
One for
HDP Services

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Both Marketplace Lis;ngs are Updated

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Both Marketplace Lis;ngs are Updated
Updated to latest
Ambari 2.4.2.
Updated Cloud
Controller and
Cloudbreak engine
for features + fixes.

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Improving Enterprise
Readiness

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enterprise Readiness
Improving enterprise readiness in the cloud
à Easy administraBon
à Performance with Scalability
à Security
à Support by Hadoop and big data experts

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Smart Ranger
Configs Easy Administration

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Performance with Scalability
à Distcp
– OpBmized buffer management for transferring large files
– Randomize input to Distcp to avoid hot-spodng S3 nodes
à Improving S3 Access Performance
– Support fast random access reads (both direcBons) by avoiding tearing down S3
HTTP connecBons
– S3A fast upload: buffers, parallel uploads, etc.
h,ps://hortonworks.github.io/hdp-aws/s3-performance/index.html

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Securing data access via IAM Roles
à IntegraBon with cloud provider
– VPC, Security Groups, IAM Roles
à Provide an IAM role as instance profile for
a cluster
à Ahach policies for accessing S3 to the role
– E.g. Read-only access for BI cluster to
specific buckets

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Security Architecture
14

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Op;onal Free Community Support
à Registered users gain access to private Q&A
Track in HCC
à QuesBons handled by our support team in India;
Business Hours coverage with goal of 24-hour
iniBal response to quesBons posted to Track
during Biz Hours
– Not resoluBon, response only
– No explicit SLAs or response Bmes are specified
– No phone, No WebEx
à HOW CAN I GET ENTERPRISE SUPPORT?

16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
HDC 1.11
Feature Highlights

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What’s New in HDCloud 1.11
• Compute Nodes
• Spot Instances
• Node Recipes

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Compute Nodes and Spot Instances

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cluster Node Types
• Core Masters (NN, RM, ATS, JHS)
• Service Masters (HS2, SHS)
• Ambari Server
• Ambari Metrics Collector
• Simple Network Gateway (nginx)
• DataNode
• NodeManager
• Ambari Metrics Monitor
• NodeManager
• Ambari Metrics Monitor
MASTER WORKER COMPUTE

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Spot Instances: Background
This is an alternaBve to an “On-demand” EC2 instance.
AWS allows users to request EC2 instances based on a Spot price (or “bid”).
hhps://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html
The bid is the maximum you are willing to pay for that EC2 instance type.
If AWS can fulfill your request, you get the instances.
Once the price for those instances goes above your bid price, your instances are marked for terminaBon, you get a
“two-minute” noBce. Then, AWS terminates your instance.

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Spot Instances: Cluster Node Types
MASTER WORKER COMPUTE
Node Requirements on
Cluster Create
Always 1 Minimum 1 Minimum 0
Launch OpBon On-Demand only On-Demand only On-Demand or Spot
Resizing (On-demand) n/a Resize up or down Resize up or down
Resizing (Spot) n/a n/a
Resize up or down at
same Spot bid price

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Spot Instances: Sugges;ons (from AWS)
“One strategy is to launch a core group [“in our case, Worker nodes”] of On-Demand instances to maintain a
minimum level of guaranteed compute resources for your applicaBons, and supplement them with Spot instances
[“in our case, Compute nodes”] when the opportunity arises.”
“Store important data regularly to S3”
“Divide the work into smaller tasks”

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Node Recipes

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Execu;ng Node Recipes
Create Instances
and other AWS
Resources
Install Ambari
and Submit
Blueprint
Create Cluster
Cluster Ready
recipe1
recipe2
recipe3
recipe4
recipe5
recipe6
recipe7
recipe8
PRE-INSTALL
POST-INSTALL

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DEMO

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Summary

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Cloud - Summary
à Launch and manage clusters by workload type
– ETL / EDW, Data science, Business analyBcs
à Use highly scalable, durable storage for data (S3)
& metadata (RDS)
à Share data and metadata among mulBple
ephemeral clusters
à Scale up and down at the click of a buhon
à Secure clusters with IAM roles, security groups,
etc.

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Q&A

29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
THANK YOU