Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)


Published on

When evaluating and planning migrating your data from on premises to the Cloud, you might encounter physical limitations. Amazon offers a suite of tools to help you surmount these limitations by moving data using networks, roads, and technology partners. In this session, we discuss how to move large amounts of data into and out of the Cloud in batches, increments, and streams.

Published in: Technology
  • Be the first to comment

AWS re:Invent 2016: Deep Dive on AWS Cloud Data Migration Services (ENT210)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Marc Trimuschat AWS Storage Services November 2016 AWS Data Transfer Services Data Ingest Strategies into the AWS Cloud ENT210
  2. 2. Storage is the gravity for cloud applications
  3. 3. Batches and Streams Direct Connect Snowball, Snowball Edge, Snowmobile 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose File Amazon EFS Block Amazon EBS (persistent) Object Amazon GlacierAmazon S3 Amazon EC2 Instance Store (ephemeral) Internet/VPN CloudFront
  4. 4. Internet/VPN ingest
  5. 5. What is Internet/VPN? Globally available Default method of ingesting content into Amazon S3 Simple standards-based (HTTP) connection Use your existing internet connection Available in a VPC for VPN connectivity Acceleration through multipart upload Data transfer into AWS is free VPN connections using VPC virtual private gateway •$0.05 per VPN connection-hour •$0.048 per VPN connection-hour for connections to the Tokyo region
  6. 6. How does Internet/VPN ingest work? Accelerate data transfer using multipart upload Ingest data directly into S3 buckets with existing internet connectivity S3 bucket AWS Region and through the console or API customer gateway endpoints VPN connection Internet Internet through VPN + VPC
  7. 7. Amazon S3 Transfer Acceleration
  8. 8. What is Transfer Acceleration? Network- and protocol-based data transfer service Acceleration of data ingress/egress with S3 buckets Typically 50% to 300% faster Feature of S3 enabled at the bucket level Available in all S3 regions worldwide No client/server software required No code changes to your application No firewall exceptions Simple pricing model
  9. 9. Ingest & egress with Transfer Acceleration S3 bucket AWS edge location Uploader Optimized throughput! Uses AWS 59 global edge locations AWS determines best edge location Data transfer optimized between edge and customer, and edge and S3 Data is not stored on the edge cache
  10. 10. Customers:, Hudl, Viocorp Problem Statement: • Needed to accelerate customer content ingest into their respective applications running on AWS • Existing ingest options were proprietary and too expensive Use of AWS: • S3 and S3 transfer acceleration for massively scalable ingest • S3 for storage, CloudFront and S3 transfer acceleration for ingest Business Benefits: • Global highly distributed data transport available on demand • Massive scalability and elasticity • Lower TCO for storage and data transport infrastructure Accelerating media content uploads to their platforms S3 Bucket AWS Edge Location Uploader Optimized Throughput!
  11. 11. Amazon Route 53 Resolve HTTPS PUT/POST HTTP/S PUT/POST “” Service traffic flow Client to S3 bucket example S3 bucket EC2 proxy AWS region AWS edge location Customer client 1 2 3 4 Data is not cached on the AWS edge location Fully managed file transfer acceleration using all AWS edge locations
  12. 12. Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore Time[hrs] 500 GB upload from these edge locations to a bucket in Singapore Public internet How fast is S3 Transfer Acceleration? S3 transfer acceleration
  13. 13. AWS Direct Connect
  14. 14. What is AWS Direct Connect? Dedicated, 1 or 10 GE private pipes into AWS Create private (VPC) or public virtual interfaces to AWS Reduced data-out rates (data-in still free) Consistent network performance At least 1 location to each AWS region Option for redundant connections Uses BGP to exchange routing information over a VLAN
  15. 15. Physical connection • Cross-connect at the location • Single-mode optical fiber - 1000Base-LX or 10GBASE-LR • Potential onward delivery through Direct Connect partner • Customer router
  16. 16. At the Direct Connect location CORP AWS Direct Connect Routers Customer Router Colocation DX Location Customer network ` AWS backbone network Cross- connect Customer router Customer’s network Demarcation
  17. 17. Dedicated port through Direct Connect partner CORP AWS Direct Connect Routers Colocation DX Location Partner network AWS backbone network Cross- connect Customer router Partner network Access circuit Demarcation Partner equipment
  18. 18. Hybrid cloud storage expansion: Amazon EFS through Direct Connect “Bursting” File Workloads Data Migration into EFS Amazon EFSOn-Premises AWS Direct Connect
  19. 19. AWS Storage Gateway
  20. 20. What is AWS Storage Gateway? Works with your existing applications Secure and durable storage in AWS Low latency for frequently used data Scalable and cost-effective on-premises storage - $.01/GB written to AWS + S3/Amazon Glacier storage fees Service connecting an on-premises software appliance with cloud-based storage
  21. 21. Hybrid storage use cases and architectures for AWS Storage Gateway Enabling cloud workloads Move data to AWS storage for Big Data, cloud bursting, or migration Tiered cloud storage Easily add AWS storage to your on-premises environment Backup, archive, and disaster recovery Cost effective storage in AWS with local or cloud restore
  22. 22. Storage Gateway hybrid storage solutions Enables using standard storage protocols to access AWS storage services Customer Premises Storage Gateway Amazon EBS snapshots Amazon S3 Amazon Glacier AWS Identity and Access Management (IAM) AWS Key Management Service (KMS) AWS CloudTrail Amazon CloudWatch Enterprise storage Devices Application servers
  23. 23. Storage gateway – Files, volumes, and tapes File gateway NFS (v3 and v4.1) interface **NEW!** On-premises file storage backed by Amazon S3 objects Volume gateway iSCSI block interface On-premises block storage backed by Amazon S3 with EBS snapshots Tape gateway iSCSI virtual tape library (VTL) interface Virtual tape storage in Amazon S3 and Glacier with VTL management
  24. 24. Detail: AWS File Gateway for S3 NFS Interface ElasticityAmazon S3 Bucket Easy Integration Cloud ScaleCloud Access
  25. 25. AWS File Gateway Bursting Tiered Storage STG213
  26. 26. AWS Snowball
  27. 27. What is AWS Snowball? Petabyte-scale data transport E-ink shipping label Ruggedized case “8.5G impact” All data encrypted end-to-end Rain- and dust- resistant Tamper-resistant case and electronics 80 TB 10 GE network
  28. 28. AWS storage migration expansion: AWS Snowball Transfer Capacity Integration Regional Availability 80TB model HDFS support 3rd party API HIPAA support All EXCEPT: Asia Pacific (Singapore) Asia Pacific (Seoul) China (Beijing)
  29. 29. How it works
  30. 30. How fast is Snowball? • Less than 1 day to transfer 200TB via 3x10G connections with 3 Snowballs, less than 1 week including shipping • Number of days to transfer 200TB via the Internet at typical utilizations Internet Connection Speed Utilization 1Gbps 500Mbps 300Mbps 150Mbps 25% 71 141 236 471 50% 36 71 118 236 75% 24 47 225 157
  31. 31. Customer: Scripps Networks Interactive Problem Statement: • Need storage platform to manage active archive content • Existing content repository too large to migrate via available network-based ingest methods Use of AWS: • S3 and Snowball for massively scalable ingest • S3 for storage, Glacier for content archive • Snowball to securely transport existing media content from on- premises storage and tape vault Business Benefits: • Petabyte-scale data transport without increased network costs • Massive scalability and elasticity • Lower TCO for active archive storage Active archive transport and archival for digital content provider
  32. 32. AWS storage migration expansion: AWS Snowmobile
  33. 33. Hybrid cloud storage expansion: AWS Snowball Edge On-premises Capacity On-premises Integration On-premises Compute Clustered local storage 100TB capacity NFS and S3-compatible endpoint AWS Lambda support
  34. 34. Snowball Edge use cases Offline Staging Local Tiering and Compute IoT Local Transformation
  35. 35. AWS Snowball Edge Integrated Storage and Compute Applications STG214
  36. 36. Storage Ecosystem Partners
  37. 37. Hybrid cloud storage ecosystem Backup AWS Storage Gateway VTL Direct to Amazon S3 File Systems Object Storage Block Storage
  38. 38. Backup to AWS approaches Amazon S3 Amazon Glacier AWS Direct Connect Internet Amazon S3-IA Application servers Cloud gateway Local disk Media server Cloud gateway Application servers Backup SW cloud connector Local disk Media server with cloud connector
  39. 39. Bursting Migration Tiering Accessing EFS via Direct Connect Storage Gateway Snowball Edge 3rd Party Ecosystem AWS hybrid cloud storage choices
  40. 40. Related Sessions • Snowball Edge: STG214, ENT211 • Snowmobile: STG214, ENT211 • Storage Gateway: STG213, ENT211
  41. 41. Thank you!
  42. 42. Remember to complete your evaluations!