A Continuously Deployed Hadoop Analytics Platform?

1. A Con&nuously Deployed
Hadoop Analy&cs pla2orm?
Graham Gear, Director, Systems Engineering, APJ

3. Logical Pilot Delivery Pipeline
Opera&ons
Monitor
Provision
Automate
Produc&on
Data Scien&sts
Hourly
100% Bugs
Produc&on
Data Science
Pre-Produc&on Development

4. Produc&on Worksta&on
Logical Nascent Delivery Pipeline
Data Engineers
Monthly
0% Bugs
Development User Acceptance Test
Backup Data
Produc&on Workload
Data Science
DS, Analysts, Apps
Monthly - Yearly
100% Bugs
Opera&ons
Monitor
Provision
Automate
Development Produc&on
Governance
Audit
Security
Lineage
Pre-Produc&on

5. SLA Worksta&on
Logical Staged Delivery Pipeline
Dev Ops
Weekly
10% Bugs
System Smoke Test
Data Engineers
Weekly
0% Bugs
Development
DS, Analysts, Apps
Monthly - Yearly
90% Bugs
Opera&ons
Monitor
Provision
Automate
Development Pre-Produc&on Produc&on
Governance
Audit
Security
Lineage
Produc&on
User Acceptance Test
Backup Data
Produc&on Workload
Data Science

6. Non-SLA SLA Worksta&on
Logical Manual Delivery Pipeline
Dev Ops
Weekly
10% Bugs
System Smoke Test
Data Engineers
Weekly
0% Bugs
Development User Acceptance Test
Backup Data
Disaster Recovery
Data Science
Data Scien&sts
Weekly - Monthly
60% Bugs
Opera&ons
Monitor
Provision
Automate
Development Pre-Produc&on Produc&on
Governance
Audit
Security
Lineage
SLA
Analysts, Apps
Monthly - Yearly
30% Bugs
Produc&on Workload

7. Logical Con&nuous Delivery Pipeline
Test
Ar&fact Repo
Build
Acceptance Test
Release Ar&fact
Unit Suite Test
Bake Ar&fact
Deploy Pipeline
Dev Ops
Hourly – Daily
15% Bugs
System Smoke Test
Worksta&on
Source Repo
Release Tag
Data Engineers
Hourly
70% Bugs
Light Unit Test
Development
Non-SLA
User Acceptance Test
Backup Data
Disaster Recovery
Data Science
Data Scien&sts
Weekly - Monthly
15% Bugs
Acceptance Test
Opera&ons
Monitor
Provision
Automate
Development Pre-Produc&on Produc&on
Governance
Audit
Security
Lineage
SLA
Analysts, Apps
Weekly - Monthly
0% Bugs
Produc&on Workload

8. Source Repo
Git
Gerrit
Physical Con&nuous Delivery Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Single-tenant
1 Laptop, Desktop
Physical
Development Pre-Produc&on Produc&on
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator

10. Source Repo
Git
Gerrit
Data Engineer Development Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Pre-Produc&on Produc&on
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
Single-tenant
1 Laptop, Desktop
Physical
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Development
1. Create a Maven module from a Maven
Archetype, providing a baseline project
encoding all corporate standards and and
targe&ng a specific produc&on version
2. Develop a dataset ingest and prepara&on
pipeline using Flume, Kaca, Hive and
MapReduce using Eclipse and Maven
3. Build a suite of unit tests and synthe&c data
to exercise the codebase, iden&fying and
resolving bugs

11. Source Repo
Git
Gerrit
Data Engineer Source Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Pre-Produc&on Produc&on
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
Single-tenant
1 Laptop, Desktop
Physical
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Development
1. Via Maven, Gerrit and
Git interac&ons, show
developer ini&ated
project source code
stages:
• Stage
• Review
• Commit
• Release

12. Source Repo
Git
Gerrit
Automated Bake Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Single-tenant
1 Laptop, Desktop
Physical
Produc&on
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
1. Simulate an automa&cally
triggered Jenkins unit test
suite, bake and smoke test
pipeline against a Director
provisioned Test cluster
served by Ar&fcatory and
Parcel repositories
Pre-Produc&on Development

13. Automated Deploy & Test Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Single-tenant
1 Laptop, Desktop
Physical
Development
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
Source Repo
Git
Gerrit
1. Show deploy, smoke and user
acceptance test stages, crea&ng the
opera&onal Manager dashboards
and Navigator meta-data
Pre-Produc&on Produc&on

14. Data Scien&st & Analyst Dev Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Single-tenant
1 Laptop, Desktop
Physical
Development Pre-Produc&on
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Test
CDH Cluster
Synthe&c Data
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
Source Repo
Git
Gerrit
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
1. Query dataset using Impala via
Hue, capture SQL logs and feed
them back into Op&mizer and
project, show dependency
verifica&on under schema
evolu&on
2. Analyse dataset using Python and
Ibis via the DS Workbench
applica&on feeding back into
project, show dependency
checking during dataset evolu&on
Produc&on

15. Source Repo
Git
Gerrit
Applica&on Delivery Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Single-tenant
1 Laptop, Desktop
Physical
Development Pre-Produc&on Produc&on
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
1. Applica&on rev pipeline, show
comparison to previous version

16. Pla2orm Delivery Pipeline
Serial-tenant
< 10 nodes
Physical, Cloud
Development Pre-Produc&on Produc&on
Test
CDH Cluster
Synthe&c Data
Build
Cloudera Director
Jenkins
Ar&fact Repo
Parcel Repository
Ar&fcatory
Mul&-tenant
> 10 nodes
Physical, Cloud
Non-SLA
Produc&on Data
CDH Cluster
DS Workbench
Mul&-tenant
> 10 nodes
Physical, Cloud
SLA
Produc&on Data
CDH Cluster
Tableau JDBC
Opera&ons
Cloudera BDR
Cloudera Manager
Governance
Cloudera Op&mizer
Cloudera Navigator
Single-tenant
1 Laptop, Desktop
Physical
Worksta&on
Synthe&c Data
CDH Single Node
Eclipse, Maven
Linux, OS-X
Source Repo
Git
Gerrit
1. Pla2orm rev
pipeline, show
comparison to
previous version

17. Logical Con&nuous Delivery Pipeline
Test
Ar&fact Repo
Build
Acceptance Test
Release Ar&fact
Unit Suite Test
Bake Ar&fact
Deploy Pipeline
Dev Ops
Hourly – Daily
15% Bugs
System Smoke Test
Worksta&on
Source Repo
Release Tag
Data Engineers
Hourly
70% Bugs
Light Unit Test
Development
Non-SLA
User Acceptance Test
Backup Data
Disaster Recovery
Data Science
Data Scien&sts
Weekly - Monthly
15% Bugs
Acceptance Test
Opera&ons
Monitor
Provision
Automate
Development Pre-Produc&on Produc&on
Governance
Audit
Security
Lineage
SLA
Analysts, Apps
Weekly - Monthly
0% Bugs
Produc&on Workload

21. Ques&ons?
• Cloudera Framework Example
• https://github.com/ggear/cloudera-framework
• Cloudera Parcel Maven Plugin
• https://github.com/ggear/cloudera-parcel
• Cloudera Manager API
• https://cloudera.github.io/cm_api/apidocs/v12/index.html
• Cloudera Navigator API
• http://cloudera.github.io/navigator/apidocs/v3
• Cloudera Director
• https://director.cloudera.com
• Cloudera Optimizer
• https://optimizer.cloudera.com graham@cloudera.com