2016년 1월 7일 AWS Cloud행사에서 김일호 솔루션즈 아키텍트 께서 발표하신 “데이터 마이그레이션 AWS와 같이하기 “ 발표자료입니다.

1. 데이터 마이그레이션 AWS와 같이 하기
김일호 | Solutions Architect

2. AWS Global Infrastructure yesterday
Region
Edge Location
11 Regions
30 Availability Zones
54 Edge Locations
Need to update

3. AWS Global Infrastructure TODAY
Region
Edge Location
12 Regions
32 Availability Zones
54 Edge Locations
Need to update
Seoul Region launched

4. Design Options on AWS
Multi-AZ Cross Region Hybrid
IT
Single AZ
AZ AZ AZ Region Region

5. Design Options on AWS
Multi-AZ Cross Region Hybrid
IT
Single AZ
AZ AZ AZ Region Region
Compute

6. A example architecture on AWS
RDS DB Instance
Active (Multi-AZ)
Availability Zone
ELB Balancer
Amazon S3
Amazon
CloudFront
Amazon
Route 53
User
Web Instances

7. Amazon DynamoDB Amazon DynamoDB
Ap-southeast-1
Ap-northeast-1
AWS Data Pipeline
AMI
Amazon EBS
AMI
Copy
Copy
Copy
RDS SnapshotAmazon RDS Snapshot
Amazon EBSAmazon EBS SnapshotAmazon EBS Snapshot
Snapshot Restore
EC2 EC2
RestoreCreate
Snapshot Restore
Copy
AWS provided cross-region copies

8. Migrating compute and network – EC2 instance
§ Resources
§ SSH Keys
§ Security Groups
§ AMI
§ EBS Volumes
§ Elastic IP Addresses
§ ELB
§ Launch configurations and AutoScaling group
§ VPC
§ Considerations
§ Reserved Instances
§ Services Limits

9. AMI Copy

10. Migrating compute and network – EC2 instance
1. Run Copy AMI at console
2. Check the status of copy AMI across region
3. Launch a new instance in target region

11. Migrating compute and network – EC2 instance
1. Create EBS volume snapshot à Copy Snapshot
2. Check the status of Snapshot copy cross region
3. Create a new volume in target region

12. Migrating compute and network – EC2 instance
§ Collect security group configuration with AWS CLI
§ $ ec2-describe-group -H --region ap-northeast-1 >> sg.txt

13. VM Import/Export

14. What is VM Import/Export
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UsingVirtualMachinesinAmazonEC2.html

15. VM Import/Export will help
§Migrate existing applications and workloads to Amazon
EC2
§Copy your VM image catalog to Amazon EC2
§Create a disaster recovery repository for VM images

16. VM Import/Export steps
1. Install the AWS CLI. For more information.
2. Prepare the VM for import to Amazon EC2.
3. Export the VM from the virtualization environment.
4. Import the VM into Amazon EC2. For information.
Launch the instance in Amazon EC2.

17. § Microsoft Windows Server 2003 (Standard,
Datacenter, Enterprise) with Service Pack 1
(SP1) or later
§ Microsoft Windows Server 2003 R2
(Standard, Datacenter, Enterprise)
§ Microsoft Windows Server 2008 (Standard,
Datacenter, Enterprise)
§ Microsoft Windows Server 2008 R2
(Standard, Datacenter, Enterprise)
§ Microsoft Windows Server 2012 (Standard,
Datacenter)
§ Microsoft Windows Server 2012 R2
(Standard, Datacenter)
§ Red Hat Enterprise Linux (RHEL) 5.1-
5.11, 6.1-6.6, 7.0-7.1
§ SUSE Linux Enterprise Server 11-12
§ Ubuntu 12.04, 12.10, 13.04, 13.10,
14.04, 14.10, 15.04
§ CentOS 5.1-5.11, 6.1-6.6, 7.0-7.1
§ Debian 6.0.0-6.0.8, 7.0.0-7.8.0, 8.0.0
§ Oracle Enterprise Linux 6.1-6.6, 7.0-
7.1
§ Fedora Server 19-21
OSs supported by VM Import/Export
Note: From 21-Dec-2015 Document
You can import Windows and Linux VMs that use VMware ESX or Workstation, Microsoft Hyper-V,
and Citrix Xen virtualization formats

18. Design Options on AWS
Multi-AZ Cross Region Hybrid
IT
Single AZ
Files and
Objects
AZ AZ AZ Region Region
Compute

19. COPY Files from IDC to S3

20. Copy files from IDC to S3
Corporate Data Center
Amazon S3
Bucket
Storage Servers
Files
File systems

21. Think Parallel
1 File 1 File
2 Seconds
Multipart

22. Think Parallel
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
8 Seconds
Foreach($file in $files) {Write-S3Object -BucketName mybucket -Key $file.filename}

23. Think Parallel
120,000 Files
Nearly 3 Days

24. Think Parallel
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
1 File
2 Seconds
120,000 files @ 15,000 TPS = 8 seconds
Multiple Machines, Multiple Threads, Multiple Parts

25. AWS S3 Multipart Upload
Corporate Data center
Amazon S3
Bucket
Storage Servers
Files
File systems

26. COPY Files from S3 to S3

27. Example:
Source Bucket : distcptest Region: Tokyo Size : 9GB Files : 90
Target Bucket : distcptestsydney Region: Sydney
How to copy objects from S3 bucket to another S3 bucket?
Copy files from S3 to S3 with AWSCLI

28. $aws s3 sync s3://distcptest/moveiclip s3://distcptestsydney/movieclip
* Time: 14 mins
Copy files from S3 to S3 with AWSCLI

29. Distrubuted Copy using Elastic MapReduce Hadoop cluster
Copy files from S3 to S3 with EMR

30. 1. Launch EMR cluster with selecting instance size and the number of instances.
Copy files from S3 to S3 with EMR
$ aws emr add-steps --cluster-id j-XXXXXXXXXX --steps Type=CUSTOM_JAR,Name="S3DistCp step",Jar=
/home/hadoop/lib/emr-s3distcp-1.0.jar,
Args=["--src,s3://distcptest/moveiclip ","--dest,s3://distcptestsydney/movieclip”]
2. Submit s3DistCp Job to the cluster.
(3DistCp is an extension of DistCp that is optimized to work with AWS)

31. EMR Job result log
Copy files from S3 to S3 with EMR
* Time: 3.5 mins
(CLI: 14 mins)
Bigger cluster, Save more time to copy

32. Resizing cluster to more 6 core nodes and test copying
Copy files from S3 to S3 with EMR
* Time: 1.5 mins

33. Design Options on AWS
Multi-AZ Cross Region Hybrid
IT
Single AZ
Databases
Files and
Objects
AZ AZ AZ Region Region
Compute

34. Database Migration – RDS
§ RDS supports cross
region DB snapshot copy
§ Full and incremental
copy
§ Simultaneously multiple
regional copy supported
Primary
Secondary
Secondary

35. Database Migration – RDS
§ A snapshot copied across regions does not include either the parameter group or
option group that was used by the DB instance the snapshot was created from.

36. $ awsrds copy-db-parameter-group --region us-east-1 --source-db-parameter-group-identifier
arn:aws:rds:ap-northeast-1:389941452291:pg:src-parameter --target-db-parameter-group-identifier
dest-test-params--target-db-parameter-group-description TEST
A client error (DBParameterGroupNotFound) occurred when calling the CopyDBParameterGroup
operation: DB ParameterGroupnot found, not allowed to do cross region copy.
$
AWSCLI Copy-DB-Parameter-group
Database Migration – RDS

37. $ python copy_db_parameter_group.py --src=src-test --
src_region=ap-northeast-1 --dest=desttestparam --
dest_region=ap-southeast-1
Database Migration – RDS
§Using a custom python script,
§ https://aws.amazon.com/ko/blogs/korea/copy-db-parameters-
between-regions-using-aws-cli/
§Python, boto3 library, deepdiff library required

38. AWS Database Migration Service

39. Purposes of data migration
One-time data migration
Between on premises and AWS
Between Amazon EC2 and Amazon
RDS
Ongoing Replication
Replicate on premises to AWS
Replicate AWS to on premises
Replicate OLTP to BI
Replicate for query offloading

40. Ways to migrate data
Bulk Load
AWS Database Migration Service
Oracle Import/Export
Oracle Data Pump Network Mode
Oracle SQL*Loader
Oracle Materialized Views
CTAS / INSERT over dblink
Ongoing Replication
AWS Database Migration Service
Oracle Data Pump Network Mode
Oracle Materialized Views
Oracle GoldenGate

41. High-speed database migration prior to AWS DMS
EC2
Instance
Linux
Host
On-Premises AWS Availability Zone
Oracle DB
RDS
Oracle
Tsunami Tsunami
DATA_PUMP_DIR
500GB
175GB
~2.5 hours~2.5 hours
Total Time
~7 hours
~3.5 hours
~4 hours

42. Start your first migration in 10 minutes or less
Keep your apps running during the migration
Replicate within, to or from Amazon EC2 or RDS
Move data to the same or different database engine
Sign up for preview at aws.amazon.com/dms
AWS
Database Migration
Service

43. 10 minutes or less to migration

44. Customer
Premises
Application Users
AWS
Internet
VPN
§ Start a replication instance
§ Connect to source and target
databases
§ Select tables, schemas, or databases
Let AWS Database Migration Service
create tables, load data, and keep
them in sync
Switch applications over to the target
at your convenience
Keep your apps running during the migration
AWS
Database Migration Service

45. After migration, use for replication and data
integration
§Replicate data in on-premises databases to AWS
§Replicate OLTP data to Amazon Redshift
§Integrate tables from third-party software into your
reporting or core OLTP systems
§Hybrid cloud is a stepping stone in migration to AWS

46. Cost-effective and no upfront costs
§ T2 pricing starts at $0.018 per Hour for T2.micro
§ C4 pricing starts at $0.154 per Hour for C4.large
§ 50GB GP2 storage included with T2 instances
§ 100GB GP2 storage included with C4 instances
§
§ Data transfer inbound and within AZ is free
§ Data transfer across AZs starts at $0.01 per GB
Swap
Logs
Cache

47. Migrate and replicate between database
engines

48. Introducing
AWS Schema Conversion Tool

49. Migrate off Oracle and SQL Server
Move your tables, views, stored procedures and D
ML to MySQL, MariaDB, and Amazon Aurora
Know exactly where manual edits are needed
Download at aws.amazon.com/dms
AWS
Schema Conversion
Tool

50. Get help with converting tables, views, and code
Schemas
Tables
Indexes
Views
Packages
Stored Procedures
Functions
Triggers
Sequences
User Defined Types
Synonyms

51. Know exactly where manual edits are needed

52. DynamoDB cross region
replication

53. DynamoDB
Data Pipeline
Region A Region B
DynamoDB cross region replication

54. DynamoDB cross region replication

55. DynamoDB cross region replication

56. Design Options on AWS
Multi-AZ Cross Region Hybrid
IT
Single AZ
Databases
Files and
Objects
AZ AZ AZ Region Region
Compute
Network

57. Transition traffic with Route53

58. Amazon Route 53 weighted routing

59. AWS Data Migration Service
Continuous Data Replication
Passive
Amazon
Route 53
Database Server
Weighted Routing example #1
VM Import/Export
AWS Region
Database
Corporate Data Center
100%
Web/ App Server
0%

60. AWS Data Migration Service
Continuous Data Replication
Passive
Amazon
Route 53
Database Server
Weighted Routing example #2
VM Import/Export
AWS Region
Database
Corporate Data Center
90%
Web/ App Server
10%

61. AWS Data Migration Service
Continuous Data Replication
Passive
Amazon
Route 53
Database Server
Weighted Routing example #3
VM Import/Export
AWS Region
Database
Corporate Data Center
50%
Web/ App Server
50%

62. AWS Data Migration Service
Continuous Data Replication
Passive
Amazon
Route 53
Database Server
Weighted Routing example #4
VM Import/Export
AWS Region
Database
Corporate Data Center
0%
Web/ App Server
100%

63. Thank you!