Cloud Services have become an integral part of many organizations’ workloads. With the ease of spinning up instances to react to changing or increased business demands, it’s certainly not hard to see why. Many organizations that once relied on internal data centers are even considering migrating to the cloud to take advantage of not just the ability to spin up instances rapidly, but also to leverage all the services that are available with these offerings. Such as Elastic Map Reduce from Amazon and HD Insight from Microsoft. In this talk, we will go over the various methods and strategies you can employ to migrate your data and workloads to the cloud and what all options this opens up for your business and IT.

1. www.clairvoyantsoft.com
Migrating Big Data Workloads
to the Cloud
By: Robert Sanders

2. | 2
Robert Sanders
Director of Big Data Engineering
https://www.linkedin.com/in/robert-sanders-cs/
Robert Sanders is the Director of Big Data Engineering at
Clairvoyant Insight. In his day job, Robert wears multiple
hats and goes between leading members of the Insight
development team to working directly with clients to
Architecting and Engineering large scale Data projects.
Robert has a deep background in enterprise systems,
initially working on full-stack implementations and then
focusing on building Data Management Platforms.

3. | 3
About Clairvoyant
Background Awards & Recognition
Boutique consulting firm centered on building data solutions and
products
All things Web and Data Engineering, Analytics, ML and User
Experience to bring it all together
Support core Hadoop platform, data engineering pipelines and provide
administrative and devops expertise focused on Hadoop

4. | 4
● “The Cloud” and “The Why”
● Cloud Migration Strategies
● Data Migration
● Application Migration
Agenda

5. | 5
Rented Infrastructure and Services
The Cloud

6. | 6
Reasons to Move to the Cloud
● Ability to make use of Cloud Services
● Reduce Cost
● Reduce Time to Spin Up new Instances
● Scalability

7. | 7
● Repurchase (Drop and Shop)
● Rehost (Lift and Shift)
● Replatform (Lift, Tinker and Shift)
● Refactor/Re-Architect
● Retiring
● Retaining
General Migration Strategies - 6 R’s

8. | 8
1. Forklift
2. Hybrid
3. Cloud Native
Big Data Migration Strategies

9. | 9
● Rehost Option
Forklift

10. | 10
● Rehost, Replatform and Refactor/Re-Architect Option
Hybrid

11. | 11
Options with Blob Storage

12. | 12
● Replatform and Refactor/Re-Architect Option
Cloud Native

13. | 13
Forklift Hybrid Cloud Native

14. | 14
$ hadoop distcp hdfs://src_host/path/* hdfs://dest_host/path/
$ hadoop distcp hdfs://src_host/path/* s3a://dest_bucket/
Data Migration Strategies - DistCP

15. | 15
File Transfer Rate
Note: Estimates (assumes - 25% network overhead)
100 Mbps 1 Gbps 10 Gbps
1 TB 30 hrs 3 hrs 18 min
10 TB 12 days 30 hrs 3 hrs
100 TB 124 days 12 days 30 hrs
1 PB 3 years 124 days 12 days
10 PB 34 years 3 years 124 days
Usable Network Bandwidth
Data to Transfer

16. | 16
● Azure ExpressRoute
Data Migration Strategies - DistCP

17. | 17
● Use an External Appliance
○ AWS Snowball (50 or 80 TB of data)
○ AWS Snowmobile (20+ PB Data)
● Loads Data into S3
● Then copy Deltas from On-Prem to Cloud
Data Migration Strategies - External Appliance

18. | 18
● S3 Transfer Acceleration
● AWS Snowball (50 or 80 TB of data)
● AWS Snowmobile (20+ PB Data)
● AWS Direct Connect
● AWS DataSync
● AWS Transfer for SFTP
● Amazon Kinesis Firehose
● More
Data Migration Services
● Azure Data Factory
● Azure Database Migration Service
● Azure ExpressRoute
Amazon Azure

19. | 19
On-Prem Workloads
Time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Daily Reporting
Analytics
Real-Time
MLTraining
Higher Utilization Lower Utilization

20. | 20
Cloud Native Workloads

21. | 21
● Batch
○ Amazon EMR (MapReduce, Hive, Pig, Spark), Amazon Redshift, AWS Glue
● Interactive
○ Amazon Redshift, Amazon Athena, Amazon EMR (Presto, Spark, Hive)
● Stream
○ Amazon EMR (Spark Streaming, Flink, Storm), Amazon Kinesis Analytics
● AI
○ Amazon AI (Lex, Polly, ML, Rekognition), Amazon EMR (Spark ML), Deep Learning)
Workload Migration to be Cloud Native

22. Thank You!
| 22
Questions?
robert.sanders@clairvoyantsoft.co
m