Cloudbreak - Technical Deep Dive

1. Cloudbreak – Technical Deep Dive
Janos Matyas & Krisztian Horvath
Hortonworks

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Presenters
Krisztian Horvath
Senior Member of technical staff, Cloudbreak
Former Co-Founder at SequenceIQ
Janos Matyas
Senior Director of Engineering, Cloudbreak
Former Co-Founder and CTO for SequenceIQ

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Goals and Motivations
Technology Stack + Deep Dive
Lessons Learned + Best Practices
Demo + Q & A

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Goals and Motivations – What We Wanted to Do…
 Declarative/full Hadoop stack provisioning in all major cloud providers
 Automate and unify the process
 Zero-configuration approach
 Same process through a cluster lifecycle (Dev, QA, UAT, Prod)
 Provide tooling - UI, REST API and CLI/shell
 Secure and multi-tenant
 SLA policy based autoscaling

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Goals and Motivations – What We Wanted to Do…
 All cloud providers are fundamentally different…
 Compute, network, security, performance
 We want to share what we found, and how we made it work!

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Goals and Motivations
Technology Stack + Deep Dive
Lessons Learned + Best Practices
Demo + Q & A

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Technology Stack
 Apache Ambari
 Cloud provider API
 Salt
 Docker
 Packer

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deep Dive - Overview
 Cloudbreak Deployer (CBD)
– Tool to deploy the Cloudbreak application
– Microservice architecture (using Docker)
– DevOps friendly
 Cloudbreak Application
– Extensible, available through UI, CLI, REST API
– SLA auto-scaling policy management
 Cluster deployed with Cloudbreak

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deep Dive – Cloudbreak Deployer
 Installation
– Single binary, written in Go
– Requires Docker 1.9.1+
– DIY installation on any RHEL / CentOS / Oracle Linux 7 (64-bit) distro
– Use one of the pre-built cloud images (AWS, Azure, GCP, OpenStack)
 Operations
– Easy upgrades/downgrades, automatic schema migration
 Cloud provider support
– AWS – generates IAM roles
– Azure – ARM and DASH config
 Utilities
– Cloudbreak shell support - interactive, remote, automated execution, OAuth2 token generation
– Local development environment setup

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deep Dive – Cloudbreak Application
 Installation
– Done with Cloudbreak Deployer (CBD)
 Operations
– Consistent feature set through UI, CLI and secure REST API
– Multi-tenant, ACL setup, usage reports
– Custom stack repositories, failure actions
– Event history, cluster management
– SLA based auto-scaling policy configs, enforcement
 Cloud provider support
– Agnostic API
– AWS, Azure, GCP, OpenStack, Mesos
– SPI interface – bring your own provider, stack under Cloudbreak management

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deep Dive – Cluster deployed with Cloudbreak
 Installation
– Managed by Cloudbreak using cloud provider API
– Default (optimized) configs – specific to cloud provider
 Operations
– Default, custom configs for stacks, services, network, storage, security
– Declarative Hadoop cluster
– Custom instance types (heterogeneous clusters)
– Different storage types
– Configurable network
– Security (access, Kerberos, SSSD, FreeIPA)
 Utilities
– Ambari Views
– Metadata/shared clusters support

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Goals and Motivations
Technology Stack + Deep Dive
Lessons Learned + Best Practices
Demo + Q & A

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned
 Not all cloud providers are the same
– Difference in performance, storage and functionality
 (Capacity) planning
– Based on workload type (batch / interactive and ad-hoc / long running)
– Use heterogeneous clusters
– Trial and error – mistakes are cheap, iterate until you find your best fit
– Leverage the cloud - scale your cluster on demand
 Number one consideration – storage
– Multiple choices (ephemeral, block storage and BLOB store)
– Bring compute to storage – might not work (everywhere) – in cloud everything is as a service
– Independently scale storage from compute, partition your data
 Security
– Consider using strict security rules (private subnets, access, etc) and use edge nodes

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - AWS
 Compute
– Find your instance types for the workload, use heterogeneous clusters
– Different instance types for transient (e.g. C4, M4) and long running (e.g. H2, D2) clusters
– Dedicated instances (to avoid noise, regulations e.g. HIPPA)
 Storage
– Use latest version of Hadoop (Hortonworks contributed cloud specific optimizations)
– Note that S3 gives you only eventual consistency
– Different driver implementation: S3n (native, jets3t based), S3a (successor of n) , S3 (block based)
 Network
– Use enhanced networking (Amazon Linux by default, RHEL based – apply patch)
– Placement groups
– Not all instance types can use the 10Gbit network (e.g. use 8x)
 Security
– Use instance roles to access S3, deploy in a private subnet/VPC

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - AWS
* D28xlarge used as instance type

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - AWS
* D28xlarge used as instance type

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - Azure
 Compute
– Find your instance types for the workload, use heterogeneous clusters
– Different instance types for transient (e.g. A and D family) and long running (e.g. Dv2) clusters
– Use ARM instead of old API
 Storage
– Use latest version of Hadoop (Hortonworks contributed cloud specific optimizations)
– Storage account scaling limitations
– Use WASB or WASB with DASH (default with Cloudbreak)
– Azure Data Lake Store – soon
– Ephemeral disk is faster than root disk – does not survive auto-updates
 Network
– No PTR record/reverse lookup support
 Security
– Integrate/sync with your corporate AD

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - Azure

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - GCP
 Compute
– Find your instance types for the workload, use heterogeneous clusters
– No template based provisioning
 Storage
– Use latest version of Hadoop (Hortonworks contributed cloud specific optimizations)
– Use Google Cloud Storage Connector
 Network
– Network isolation/DNS problem
 Security

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - OpenStack
 Compute
– Find your instance types for the workload, use heterogeneous clusters
– Use Heat templates instead of API calls (we support both)
 Storage
– Currently we support only Cinder volumes
– Swift and Ceph is planned
– Data locality through Cloudbreak – let us know your topology or rack/hypervisor mapping
 Network
– Configure DNS properly
– Use multiple network (Neutron) nodes in case of a large cluster
 Security
– Use Keystone 3 (support for OAuth, Federation, introduction of groups/domains)

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Lessons Learned - Mesos
 In Tech Preview
– come and talk to us after the talk
– Or @Hortonworks boot

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Goals and Motivations
Technology Stack + Deep Dive
Lessons Learned + Best Practices
Demo + Q & A

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You