More Related Content
Similar to Introduction to Hortonworks Data Cloud for AWS (20)
More from Yifeng Jiang (16)
Introduction to Hortonworks Data Cloud for AWS
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Cloud - Summary
à Launch and manage clusters by workload type
– ETL / EDW, Data science, Business analytics
à Use highly scalable, durable storage for data (S3)
& metadata (RDS)
à Share data and metadata among multiple
ephemeral clusters
à Scale up and down at the click of a button
à Secure clusters with IAM roles, security groups,
etc.
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Performance with Scalability
à General strategy: Optimize by workload types
à ETL workloads
– Typical pipeline: Bring in data => Transform => Repair partitions => Compute statistics
– Multiple metadata calls: Batched and issued in parallel for performance gains
à Distcp
– Optimized buffer management for transferring large files
– Randomize input to Distcp to avoid hot-spotting S3 nodes
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Performance with Scalability
à Analytics workloads – ORC file related optimizations
– Support fast random access reads (both directions) by avoiding tearing down
S3 HTTP connections
– Pass index information to compute tasks as part of split data to avoid re-
computation
à Status: Available, but performance optimizations never stop J
https://hortonworks.github.io/hdp-aws/s3-performance/index.html
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Correctness with strong consistency
à Write operations followed by read may not return correct
results
– Issues for data pipelines, multi-stage jobs, etc.
à S3Guard project: Intermediate, consistent metadata store
à Write calls from S3AFileSystem update both S3 and metadata
store
à S3AFileSystem automatically tries to reconcile metadata
between S3 and metadata store on subsequent reads
– Inconsistencies are handled based on policy
à Status: In progress
16
https://issues.apache.org/jira/browse/HADOOP-13345