Migrating to Amazon RDS with Database Migration Service
1. Dhanraj Pondicherry, Solution Architect Manager
Database Migration
Simple, Cross-Engine, and Cross-Platform Migrations with Minimal Downtime
February 21, 2017
2. Agenda
• How does the cloud help?
• How do I get there?
• How does it work?
• What have others done?
• Are there any tricks? Where is the magic?
• Show me!
3. Migration options
If you’re not switching engines and can take downtime:
- SQL Server: bak file import
- MySQL: read replicas
- Oracle SQL Developer, Data Pump, Export/Import
- PostgreSQL: pg_dump
- SAP ASE: bcp
4. Migration used to be cost + complexity + time
Commercial data migration and replication software
Complex to set up and manage
Application downtime
Database-engine-specific application code
5. How can I get to the cloud?
How will my on-premises data migrate to the cloud?
How can I make it transparent to my users?
Afterwards, how will on-premises and cloud data interact?
How can I integrate my data assets within AWS?
Can I get help moving off of commercial databases?
6. What are DMS and SCT?
AWS Database Migration Service (DMS) easily and securely
migrate and/or replicate your databases and data warehouses
to AWS
AWS Schema Conversion Tool (SCT) convert your commercial
database and data warehouse schemas to open-source engines or
AWS-native services, such as Amazon Aurora and Redshift
We migrated over 18,000 unique databases. And counting…
7. When to use DMS and SCT?
Modernize Migrate Replicate
8. When to use DMS and SCT? Modernize
Modernize your Database Tier
• Commercial to open-source
• Commercial to Amazon Aurora
Modernize your Warehouse
• Commercial to Redshift
Amazon
Aurora
Amazon
Redshift
9. When to use DMS and SCT?
Modernize Migrate Replicate
Modernize your database tier –
• Commercial to open-source
• Commercial to Amazon Aurora
Modernize your Data Warehouse –
• Commercial to Redshift
• Migrate business-critical
applications
• Migrate from Classic to VPC
• Migrate data warehouse to
Redshift
• Upgrade to a minor version
• Consolidate shards into Aurora
• Create cross-regions Read Replicas
• Run your analytics in the cloud
• Keep your dev/test and production
environment sync
10. New SCT Data Extractors – released 2/15
Extract Data from your data warehouse* and migrate to Amazon Redshift
• Extracts through local migration agents
• Data is optimized for Redshift and Saved
in local files
• Files are loaded to an Amazon S3 bucket
(through network or Amazon Snowball)
and then to Amazon Redshift
Amazon
Redshift
AWS SCT S3 Bucket
* 1st release supports Oracle v11 and up and Teradata v15 and up
11. Why use DMS and SCT?
Secure
Cost Effective
Remove Barriers
to Entry
Allow DB
Freedom
Keep a Leg in
the Cloud
Easy to Use, but
Sophisticated…
Near-Zero
Downtime
12. Who is saying What about DMS and SCT?
"We migrated hundreds of our clients from our in-house data-center to Amazon RDS
Oracle 12c using the AWS Data Migration Service (DMS). Due to this service, we could
live-replicate the databases between our data-center and RDS before the migration. That
kept the migration down-time to the very minimum. We are very happy with DMS and
are planning to use it for Oracle to MySQL migration next”.
”The SCT Assessment Report was the key enabler to allow us to understand the
scope of effort required to complete an Oracle to PostgreSQL migration. What
was originally thought to be a largely manual task that no one was particularly
excited about having to do became a very straight-forward quick and easy
process."
“We are in the process of migrating some databases to Amazon Aurora. The ease
by which we can do this using the AWS Database Migration Service has
simplified this process for us and enabled us to accelerate our migration
efforts. The ability to closely monitor the process, the detailed logging feature,
and the support we received from AWS have given us a great deal of confidence
in a successful migration.”
14. SCT helps with converting tables, views, and code
Sequences
User-defined types
Synonyms
Packages
Stored procedures
Functions
Triggers
Schemas
Tables
Indexes
Views
Sort and distribution keys
15. Load is table by table
Replication instance
Source Target
16. Change data capture (CDC) and apply
Replication instance
Source Target
Update
t1 t2
t1
t2
Transactions Change
apply
after bulk
load
17. Customer
premises
Application users
AWS
Internet
VPN
Start a replication instance
Connect to source and target
databases
Select tables, schemas, or
databases
Let AWS DMS create tables,
load data, and keep them in
sync
Switch applications over to
the target at your convenience
Keep your apps running during the migration
AWS
DMS
18. Multi-AZ option for high availability
Customer
premises
or AWS
AWS
Internet
VPN
AWS DMS
AWS DMS
23. Sources for AWS Database Migration Service
Customers use the following databases as a source for data migration using AWS DMS:
On-premises and Amazon EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.4–9.5
• SAP ASE 15.7+
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008R2–2014. CDC operations are not supported yet.
• MySQL versions 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.4–9.5. CDC operations are not supported yet.
• Amazon Aurora (MySQL-compatible data source)
24. Targets for AWS Database Migration Service
Customers can use the following databases as a target for data replication using
AWS DMS:
On-premises and EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.5
• SAP ASE 15.7+
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008 R2 - 2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.5
• Amazon Aurora (MySQL-compatible data source)
Amazon Redshift
25. AWS Database Migration service pricing
T2 for developing and periodic data migration tasks
C4 for large databases and minimizing time
T2 pricing starts at $0.018 per hour for T2.micro
C4 pricing starts at $0.154 per hour for C4.large
50 GB GP2 storage included with T2 instances
100 GB GP2 storage included with C4 instances
Data transfer inbound and within AZ is free
Data transfer across AZs starts at $0.01 per GB
27. Heterogeneous Migration
Oracle private DC to RDS PostgreSQL migration
Used the AWS Schema Conversion Tool to
convert their database schema
Used on-going replication (CDC) to keep
databases in sync until they reached the cutover
window
Benefits:
• Improved reliability of the cloud environment
• Savings on Oracle licensing costs
• SCT Assessment Report let them understand
the scope of the migration
28. Scale Up Migration
RDS MySQL to Amazon Aurora Migration
Used DMS with on-going replication (CDC) to
migrate the data
Benefits:
• Aurora handles their larger data storage
requirements. Per regulations they are storing
120 TB of data for 2 years
• Reduced cost and improved performance when
compared to large MySQL instances
29. Homogeneous Migration
RDS MySQL on EC Classic to VPC
Database ran an end-user application so could
not take downtime
70 RDS instances have been migrated
Benefits:
• Leveraging CDC, they could decrease
outage per database to less than 5 mins
• 70 RDS instances have been seamlessly
migrated
30. Split Migration
35 million members on it’s site
Many interdependent applications built over the
last 15 years with unique High IOPS
requirements
Migration from legacy cloud service to AWS
Combination of migration to MySQL on EC2 and
Amazon Aurora
Benefits:
• DMS reduced the time required to migrate our
databases by 40 percent
• Realized 55 percent cost savings by moving
some db’s to Amazon Aurora
31. Resources available to customers—DMS
Getting Started Guide: Review technical
documentation.
Features and benefits: Highlights DMS
features.
Pricing: Prices for replication instances,
storage, and data transfer.
Support: Post your questions to our
Support forum.
Java SDK: Java-based API for creating
and managing data migration tasks.
AWS Command Line Interface: Start
and stop replication tasks with simple
commands.
32. Resources available to customers—
AWS Schema Conversion Tool
User Guide: Review technical docs at
aws.amazon.com/documentation/SchemaConversionTool/
or choose the Download button.
Download area: Get installation files for the
Schema Conversion Tool.
Support forums: Ask questions and review
how-to guides.
https://forums.aws.amazon.com/forum.jspa?forumID=208.
Start with some audience interaction… How many of you have been to the AWS Database Migration website? Has anyone tried using DMS yet?
Databases are special! Crown jewels of the business! They’re also the most common cause for lost sleep and IT escalations.
So, its natural that people want to know,
There are other ways of migrating too. Often the native option is the best option. They come with drawbacks though: downtime and the inability to move between engines
- Pre DMS, costly, license-intensive replication software to move data
- Were not simple to setup or use required expert resources
- Long downtimes the other option
- Customers who wanted to switch engines had to deal with differences in datatypes, table syntax, sp’s, functions, triggers
Responsive to customers
Migrations easier and less intrusive
Migrate will low downtime to any RDS engine type
Post migration, customers asked for ongoing sync
Sync between on prem and also between engines
This mobility addresses one of the big requests: The ability to migrate off of commercial, license-intensive database engines, onto cloud-native, non-commercial engines.
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
The database migration process works like this. first start with the SCT.
Creates tables at target database
Sets up metadata required at target
Populates data from source
Each process loads one entire table
Can use multiple processes
Can be paused
When restarted will continue from where it was stopped
Will reload any tables that were currently in progress
Oracle supplemental logging
MySQL row-level bin logging
SQL Server bulk logged/full recovery
Postgres WAL
No agent
Uses recovery log
Native change data capture API
Let’s explore how it is that DMS can enable near zero downtime migration. The customer starts by launching a DMS instance in their AWS account. (click) Then the customer provides database connection information to connect out to their on-premises database, and in to their AWS database. (click) Next the customer will select which tables, schemas or databases they want to migrate, (click) and DMS will load the data, and keep it in sync on an ongoing basis. (click) Finally, at the time of the customer’s choosing, they can simply change the application to point to the new AWS database, instead of their old on-premises database. (click)
When you either migrate or replicate, you can choose the Multi-AZ option. This option provides DMS in a high availability mode–in case the primary DMS server fails, the secondary one kicks in.
So by now it should be pretty obvious that data can move from A to B, and many have used DMS to do just that. Some customers have been pretty ingenious with DMS and have fanned in, or consolidated systems. DMS will put data where you ask it to, so if you want to merge data from a number of legacy systems into a single repository, DMS can help with this. And remember, DMS can just as easily move data between like databases and unlike databases, so you can take information from a number of different platforms and consolidate it in a single, cloud database.
In addition to “fanning-in” or consolidating, we’ve had a customer who has taken their database and “fanned-out” different portions to different targets. For example, from their ERP system they replicate they customer list out to the CRM system and also take some of the general ledger transactions and funnel them over to a BI system.
We’ve also had a customer who, for legacy reasons, used a single database for three different applications. With DMS you can split this information out to different databases. One could be PostgreSQL, one SQL Server, and one Aurora
Some organisations have many years of historical data, and for those that are willing to make the leap, filtering can be done during a migration so that only newer information is kept. Filtering can be done on any field in the source database–not just dates.
Just in case it wasn’t abundantly clear–DMS can move your data from pretty much anywhere to pretty much anywhere. Sort of like a courier company for the cloud
Mention why no older engines as sources
Where can you learn more?
(CLICK) The User Guide and download option are available from the DMS section of the AWS Console
(CLICK) Yeah, there’s the Download button. Click it and you’ll see something like this.
(CLICK) Information on installation and use
(CLICK) The Schema Conversion Tool has its own forum to ask questions and get help