More Related Content Similar to The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft (20) More from Amazon Web Services (20) The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Databases on AWS
The Right Tool for the Right Job
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration & Streaming Services
Data Movement
Data
Warehousing
Big Data
Processing
Interactive
Query
Operational
Analytics
Real time
Analytics
Serverless
Data processing
Analytics
Infrastructure
Data Catalog
& ETL
Security &
Management
Data Lake Infrastructure & Management Ledger
Blockchain Central Trust
Dashboards Predictive Analytics
Visualization & Machine Learning
Databases
Relational Key-value Document In-memory Graph Time-Series
Broad and deep portfolio, purpose-built for builders
3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Movement
Analytics
Data Lake Infrastructure & Management Ledger
Visualization & Machine Learning
Databases
+ 10 more
QuickSight SageMaker Comprehend Lex Polly Rekognition
The image part
with relationship
ID rId5 was not
found in the file.
Translate Transcribe
Deep Learning
AMIs
Aurora
MySQL,
PostgreSQL
RDS and RDS on VMWARE
MySQL, PostgreSQL,
Oracle, SQL Server
MariaDB
DynamoDB
Key-value
ElastiCache
Redis,
Memcached
Timestream
Time Series
Neptune
Graph
DocumentDB
Documents
Redshift
EMR
Spark,
Hadoop
Athena
Elasticsearch
Service
Kinesis Data
Analytics
Glue
Scala, Python
S3/Glacier Lake
Formation
Glue
Crawl, ETL
Managed Blockchain Quantum Ledger
Database (QLDB)
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Broad and deep portfolio, purpose-built for builders
4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common data categories and use cases
Relational
Referential
integrity, ACID
transactions,
schema-
on-write
Lift and shift, ERP,
CRM, finance
Key-value
High
throughput, low-
latency reads
and writes,
endless scale
Real-time bidding,
shopping cart,
social, product
catalog, customer
preferences
Document
Store
documents and
quickly access
querying on
any attribute
Content
management,
personalization,
mobile
In-memory
Query by key
with
microsecond
latency
Leaderboards,
real-time analytics,
caching
Graph
Quickly and
easily create
and navigate
relationships
between
data
Fraud detection,
social networking,
recommendation
engine
Time-series
Collect, store,
and process
data sequenced
by time
IoT applications,
event tracking
Ledger
Complete,
immutable, and
verifiable history
of all changes to
application data
Systems
of record, supply
chain, health care,
registrations,
financial
5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Database Services
DynamoDB ElastiCache Neptune
Key value | Document Graph
Non-relational
Database Migration Service
Relational
Aurora RDS
6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Two fundamental areas of focus
“Lift and shift” existing
apps to the cloud
Quickly build new
apps in the cloud
7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Lift and shift” existing apps to the cloud
“Lift and shift” existing
apps to the cloud
Quickly build new
apps in the cloud
8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Aurora
MySQL and PostgreSQL-compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10th the cost
Performance
and scalability
Availability
and durability
Highly secure Fully managed
5x throughput of standard
MySQL and 3x of standard
PostgreSQL; scale-out up to
15 read replicas
Fault-tolerant, self-healing
storage; six copies of data
across three Availability Zones;
continuous backup to Amazon S3
Network isolation,
encryption at
rest/transit
Managed by RDS:
No hardware provisioning,
software patching, setup,
configuration, or backups
9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Relational Database Service (RDS)
Managed relational database service with a choice of six popular database engines
Easy to administer Available and durable Highly scalable Fast and secure
No need for infrastructure
provisioning, installing, and
maintaining DB software
Automatic Multi-AZ data
replication; automated backup,
snapshots, failover
Scale database compute
and storage with a few
clicks with no app
downtime
SSD storage and guaranteed
provisioned I/O; data
encryption at rest and in
transit
10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Database Migration Service (AWS DMS)
M I G R A T I N G
D A T A B A S E S
T O A W S
Migrate between on-premises and AWS
Migrate between databases
Automated schema conversion
Data replication for
zero-downtime migration
100,000+
databases migrated
11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Quickly build new apps in the cloud
“Lift and shift” existing
apps to the cloud
Quickly build new
apps in the cloud
12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Modern apps create new requirements
Users: 1 million+
Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request rate: Millions
Access: Web, mobile, IoT, devices
Scale: Up-down, Out-in
Economics: Pay for what you use
Developer access: No assembly requiredSocial mediaRide hailing Media streaming Dating
13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s take a closer look at…
Relational Key-value Graph
14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Relational data
• Divide data among tables
• Highly structured
• Relationships established via
keys enforced by the system
• Data accuracy and consistency
Patient
* Patient ID
First Name
Last Name
Gender
DOB
* Doctor ID
Visit
* Visit ID
* Patient ID
* Hospital ID
Date
* Treatment ID
Medical Treatment
* Treatment ID
Procedure
How Performed
Adverse Outcome
Contraindication
Doctor
* Doctor ID
First Name
Last Name
Medical Specialty
* Hospital Affiliation
Hospital
* Hospital ID
Name
Address
Rating
15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Relational use case
// Doctors affiliated with Mercy
hospital
Patient
* Patient ID
First Name
Last Name
Gender
DOB
* Doctor ID
Visit
* Visit ID
* Patient ID
* Hospital ID
Date
* Treatment ID
Medical Treatment
* Treatment ID
Procedure
How Performed
Adverse Outcome
Contraindication
Doctor
* Doctor ID
First Name
Last Name
Medical Specialty
* Hospital Affiliation
Hospital
* Hospital ID
Name
Address
Rating
SELECT
d.first_name, d.last_name
FROM
doctor as d,
hospital as h
WHERE
d.hospital = h.hospital_id
AND h.name = ‘Mercy';
// Number of patient visits each doctor
completed last week
SELECT
d.first_name, d.last_name, count(*)
FROM
visit as v,
hospital as h,
doctor as d
WHERE
v.hospital_id = h.hospital_id
AND h.hospital_id = d.hospital
AND v.t_date > date_trunc('week’,
CURRENT_TIMESTAMP - interval '1 week')
GROUP BY
d.first_name, d.last_name;
16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key-value data
• Simple key value
pairs
• Partitioned by keys
• Resilient to failure
• High throughput,
low-latency reads
and writes
• Consistent
performance at scale
Gamers
Primary Key Attributes
GamerTag Level Points High Score Plays
Hammer57 21 4050 483610 1722
FluffyDuffy 5 1123 10863 43
Lol777313 14 3075 380500 1307
Jam22Jam 20 3986 478658 1694
ButterZZ_55 7 1530 12547 66
… … … … …
Gamers
Hammer57
21
4050,
483610,
1722
GET {
TableName:"Gamers",
Key: {
"GamerTag":"Hammer57“,
“ProjectionExpression“:”Points”
} }
17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gamers
Primary Key
Attributes
Gamer Tag Type
Hammer57
Rank
Level Points Tier
87 4050 Elite
Status
Health Progress
90 30
Weapon
Class Damage Range
Taser 87% 50
FluffyDuffy
Rank
Level Points Tier
5 1072 Trainee
Status
Health Progress
37 8
Key-value use case
// Status of Hammer57
GET {
TableName:"Gamers",
Key: {
"GamerTag":"Hammer57",
"Type":"Status” } }
// Return all Hammer57
Gamers
GamerTag = :a
:a Hammer57
18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB
Fast and flexible NoSQL database service for any scale
Performance at
scale
Handles millions of requests per
second
Delivers microsecond latency
Automated global replication
Enterprise
ready
ACID transactions
Encryption at rest
On-demand backup and restore
Serverless
Maintenance free
Auto scaling
On-demand capacity mode
19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB
Document or Key-Value Scales to Any WorkloadFully Managed NoSQL
Access Control Event Driven ProgrammingFast and Consistent
20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Graph data
• Relationships are first-class
objects
• Vertices connected by Edges
PURCHASED PURCHASED
FOLLOWS
PURCHASED
KNOWS
PRODUCT
SPORT
FOLLOWS
21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Graph use case
// Product recommendation to a user
gremlin> V().has(‘name’,’sara’).as(‘customer’).out(‘follows’).in(‘follows’).out(‘purchased’)
( (‘customer’)).dedup() (‘name’) ('name')
KNO
W
S
PURCHASED PURCHASED
FOLLOWS
PURCHASED
KNOWS
PRODUCT
SPORT
FOLLOWS
FOLLOWS
// Identify a friend in common and
make a recommendation
gremlin> g.V().has('name','mary').as(‘start’).
both('knows').both('knows’).
where(neq(‘start’)).
dedup().by('name').properties('name')
22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AMAZON Neptune
Fully managed graph database
FAST RELIABLEOPEN
Query billions of
relationships with
millisecond latency
6 replicas of your data
across 3 AZs with full
backup and restore
Build powerful
queries easily with
Gremlin and SPARQL
Supports Apache
TinkerPop & W3C
RDF graph models
EASY
23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Open Source Apache TinkerPop™
Gremlin Traversal Language
W3C Standard
SPARQL Query Language
RESOURCE DESCRIPTION
FRAMEWORK (RDF)
PROPERTY GRAPH
Graph Models and Frameworks
24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common customer use cases
Ledgers with centralized control
Healthcare
Verify and track hospital
equipment inventory
Manufacturers
Track distribution of a
recalled product
HR & Payroll
Track changes to an
individual’s profile
Government
Track vehicle title
history
26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges with building ledgers
Adds unnecessary
complexity
BlockchainRDBMS - audit tables
Difficult to
maintain
Hard to use
and slow
Hard to build
Custom audit functionality using
triggers or stored procedures
Impossible to verify
No way to verify changes made
to data by sys admins
27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ledger database concepts
C | H
J Journal
C | H Current | History
Current | History
Journal
Ledger comprises
J
L
Ledger databaseL
Journal determines Current | History
28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ID Manufacturer Model Year VIN Owner
ID Version Start End Manufacturer Model Year VIN Owner
How it works
ID Manufacturer Model Year VIN Owner
1 Tesla Model S 2012 123456789 Traci Russell
INSERT INTO cars <<
{ 'Manufacturer': 'Tesla',
'Model': 'Model S',
'Year': '2012',
'VIN': '123456789',
'Owner': 'Traci Russel' }
>>
FROM cars WHERE VIN = '123456789' UPDATE owner = 'Ronnie Nash'FROM cars WHERE VIN = '123456789' UPDATE owner = 'Elmer Hubbard'
J
ID Version Start End Manufacturer Model Year VIN Owner
1 1 07/16/2012 NULL Tesla Model S 2012 123456789 Traci Russell
current.cars
C
history.cars
H ID Version Start End Manufacturer Model Year VIN Owner
1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 NULL Tesla Model S 2012 123456789 Ronnie Nash
ID Version Start End Manufacturer Model Year VIN Owner
1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 09/02/2016 Tesla Model S 2012 123456789 Ronnie Nash
1 3 09/02/2016 NULL Tesla Model S 2012 123456789 Elmer Hubbard
ID Manufacturer Model Year VIN Owner
1 Tesla Model S 2012 123456789 Ronnie Nash
ID Manufacturer Model Year VIN Owner
1 Tesla Model S 2012 123456789 Elmer Hubbard
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Traci Russell
Metadata: {
Date:07/16/2012
}
H (x) UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (x) UPDATE cars
ID:1
Owner: Elmer Hubbard
Metadata: {
Date: 09/02/2016
}
H (x)
29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Quantum Ledger Database (QLDB)
Fully managed ledger database
Track and verify history of all changes made to your application’s data
Immutable
Maintains a sequenced record of
all changes to your data, which
cannot be deleted or modified;
you have the ability to query and
analyze the full history
Cryptographically
verifiable
Uses cryptography to
generate a secure output
file of your data’s history
Easy to use
Easy to use, letting you
use familiar database
capabilities like SQL APIs
for querying the data
Highly scalable
Executes 2–3X as many
transactions than ledgers
in common blockchain
frameworks
30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Time-series data
What is time-series
data?
What is special about a
time-series database? A sequence of data points
recorded over a time interval
Time is the
single primary axis
of the data model
t
32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Time-series use case
Application events
IoT Sensor
Readings
DevOps data
Humidity
% WATER VAPOR
91.094.086.093.0
33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Existing time-series databasesRelational databases
Difficult to
maintain high
availability
Difficult to scale Limited data
lifecycle
management
Inefficient
time-series data
processing
Unnatural for
time-series
data
Rigid schema
inflexible for fast
moving time-series
data
Building with time-series data is challenging
34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Timestream
Fast, scalable, fully managed time-series database
1,000x faster and 1/10th the
cost of relational databases
Collect data at the rate of
millions of inserts per
second (10M/second)
Trillions of
daily events
Adaptive query processing
engine maintains steady,
predictable performance
Time-series analytics
Built-in functions for
interpolation, smoothing,
and approximation
Serverless
Automated setup,
configuration, server
provisioning, software patching
35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Purpose-built databases
Relational Key-value Document In-memory Graph Time-series Ledger
DynamoDB NeptuneAmazon RDS
Aurora CommercialCommunity
Timestream QLDBElastiCache
37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Migration
Introduction
38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Challenges
Migration patterns
AWS Database Migration Service
AWS Schema Conversion Tool
Database Migration Overview
Summary
Questions
39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Migration Challenges
Cost
Complexity
Application downtime
Engine specific database code
40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Value Proposition – What determines value?
From an Organizational Perspective…
• Value enables a positive outcome
• Value IS NOT preventing a negative outcome
• Value is a function of Capability, Risk, and Cost.
Value = f(Capability, Risk, Cost)
41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Value Proposition – Operations
DevOps NoOpsSustainment
Value
42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Value Proposition – Hosting
IaaS PaaSOn-Premises
Value
43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Value Proposition – Migration
Re-Host Re-Platform Re-Factor
Value
Linux à Linux
Oracle à Oracle
Solaris à Linux
Oracle à Aurora
Linux à Serverless
Oracle à DynamoDB
44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Patterns
Re-Host (Homogenous)
• “Lift-and-Leverage”
• Same database engine / Same operating system
Re-Platform (Heterogeneous)
• Different operating system
• Different database engine
• Convert the database and/or application code
Re-Factor
• Decompose into smaller, logical components
• Right tool for the workload
45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Database Migration Service
Sources* Targets*
Oracle Oracle
SQL Server SQL Server
Azure SQL PostgreSQL
PostgreSQL MySQL
MySQL Amazon Redshift
SAP ASE SAP ASE
MongoDB Amazon S3
Amazon S3 Amazon DynamoDB
IBM DB2 Amazon Kinesis
Amazon ElasticSearch
• Start your first migration in 10
minutes or less
• Keep your apps running during the
migration
• Replicate from within, to, or from
AWS
• Move data to the same or different
database engine
Consult CHAP_Source.html and CHAP_Target.html pages
for latest DMS sources and targets
46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Keep your application running during migration
On-Premise AWS Cloud
Generic database DB Instance
Start a replication instance
Connect to the source and target
Select tables, schemas, or databases
Let DMS create the target objects
Move data and synchronize objects
Switch applications when ready
47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Migration Options
Load is table by table
• Configurable number of tables in parallel
One time load or Change Data Capture (CDC)
• Read from database log on the source and apply to the target
Filter criteria available for selective loading
• Select only a few tables or a subset of data in your tables
Multiple sources and targets. Mix and match.
• One side of the migration must be in AWS
Ongoing replication support
• Keep your replication going until your application is ready to cutover
48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Schema Conversion Tool
Makes heterogeneous database migrations
predictable by automatically converting the source
database schema and a majority of the database
code objects, including views, stored procedures,
and functions, to a format compatible with the
target database
Features
Database Migration Assessment report for choosing the right target engine
Automatic conversion for eligible database objects and code
Code browser to highlight places where manual edits are required
49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Support for the following conversions
Source* Database Target* Database on AWS
Oracle database Amazon Aurora, MySQL, PostgreSQL, Oracle
Oracle data warehouse Amazon Redshift
Azure SQL Amazon Aurora, MySQL, PostgreSQL
Microsoft SQL Server Amazon Aurora, Amazon Redshift, MySQL PostgreSQL
Teradata Amazon Redshift
IBM Netezza Amazon Redshift
Greenplum Amazon Redshift
HPE Vertica Amazon Redshift
MySQL and Maria DB PostgreSQL
PostgreSQL Amazon Aurora, MySQL
Amazon Aurora PostgreSQL
IBM DB2 LUW Amazon Aurora, MySQL, PostgreSQL
Apache Cassandra Amazon DynamoDB
Consult CHAP_Source.html and CHAP_Target.html pages
for latest DMS sources and targets
50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SCT converts objects and code
51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pricing and platform support
Microsoft Windows
Apple Mac
Fedora Linux (rpm)
Ubuntu Linux (deb)
$0You can download AWS Schema Conversion Tool for your platform of choice
52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Migration Overview
Data
migration
Application
code
Schema
objects
Cutover
Analysis
Testing
53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example migration plan
Analysis
Assessment 2%
Planning 7%
Schema
conversion
TESTING
Database conversion 14%
Procedure, function, and db script conversion 25%
Application
conversion
Application conversion/remediation 5%
Integration with 3rd party applications 2%
Data
migration
Data migration 3%
Functional end to end testing 28%
Performance testing 2%
Cutover
Integration and deployment 5%
Training and knowledge transfer 2%
Documentation and version control 2%
Post production support 3%
54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Resources and skills required
Database expertise is required
• Working knowledge of target database engine
Basic networking knowledge
• Familiarity with AWS VPC concepts
AWS knowledge is required
Software architecture knowledge is an advantage
55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Educate yourself and your team
Getting started
FAQS
Migration best practices
Database migration playbook
56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Plan your time
• Migration projects can take from several weeks to
several months to complete depending on the
complexity of the code conversion
• It may take several iterations and may take longer than
you anticipate
• Good planning will increase the success and reduce the
number of iterations required
57. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Understand your database
Size
• Uncompressed size
• Number schemas and objects
• Relationships between schema
objects
• Number large tables (>500GB)
Datatypes
• LOB column number and size
• Custom or unusual datatypes
• Tables without PKs
Transaction profile
• Transactions per second
• Transaction boundaries
Authentication
• User authentication
• Roles and permissions
Integrations
• ETL jobs, interfaces, applications
58. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Understand your network
Database access
• Firewalls, tunnels, VPN, Direct Connect
VPC
• Identify the target VPC and subnets you will use
IAM permissions
• Security groups and IAM permissions needed for DMS
Do you have enough bandwidth to move all your data?
59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Clarify your requirements
Can you afford downtime?
Do you need the source database to stay alive after migration?
Do you know why you prefer one target engine over another?
What are your high availability requirements?
Does all the data need to move?
Does it need to move to the same place?
Do you understand RDS benefits (backups, HA, etc.)
Do you understand RDS limitations? (storage size, admin user, etc.)
60. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Create your target schema
SCT
• For heterogeneous migrations
• Orchestrate your migration through SCT by configuring your access keys
DMS
• Creates only tables and primary keys
• Uses transformation rules to convert case, filter, or change schema
Native tools for homogeneous migrations
61. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Start your data migration
• Check prerequisites on the source
• Check the logs
• Check CloudWatch metrics
• Proper instance sizing
• Migration can take time, especially if your database is
large or your network connection is small
62. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Things that affect your migration speed
Size of the source
Transaction pressure on the source
• Volume of transactions
• Run length of transactions
Size of the target
DMS task settings (# tasks, # tables, settings)
Bandwidth available
Size of the replication instance
Schema configuration
LOBs in the schema
63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to choose native migration tools
Target supports native replication
Moving ALL the data
No transformations required
Target is a new database
64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When to use backup/restore or import/export
Database is small
Downtime window is large enough
Moving all the data
No transformations required
65. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Larger database migration
What is large?
• Is a relative term
• Moving all the data takes longer than you are willing to wait
What can I do?
• Use AWS Snowball
• Let AWS Professional Services help you
66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lessons Learned
ü Plan, Plan, Plan
• 80+% of the time spent during the migration should be planning
• Each aspect should be well documented, scripted, and automated
ü Test, Test, Test
• Conduct multiple test runs of the migration process until it is muscle memory
• It is the customer’s responsibility to provide a test plan
67. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary
Migrations are complex
AWS SCT & DMS can automate a lot of the work
Know your environment (existing and new)
Bring the right skills
Allocate enough time
Sometimes native tools are the right choice
Many successful migrations
AWS is here to help
68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions?
69. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS
AWS Schema Conversion Tool
John Franklin
Solutions Architect – Database Specialist
70. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
AWS Schema Conversion Tool
• Overview
• Getting started
• Menu navigation
• Settings
71. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overview
72. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Schema Conversion Tool (SCT)
The AWS Schema Conversion Tool helps
automate many database schema and code
conversion tasks when migrating to a new
database engine.
Features
Schema conversion between database engines
Database Migration Assessment report for choosing the best target engine
Code browser that highlights places where manual edits are required
73. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pricing and platform support
Microsoft Windows
Apple Mac
Fedora Linux (rpm)
Ubuntu Linux (deb)
$0You can download AWS Schema Conversion Tool for your platform of choice
74. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting Started
75. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Connecting to your database
1. Create a new project
2. Connect to your source database
3. Enter the connection details
76. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration assessment report
4. Run assessment
report
5. Review executive
summary
6. Review detailed
instructions
77. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Convert schema and code objects
• Functions
• Indexes
• Packages
• Tables
• Triggers
• Schemas
• Sequences
• Stored Procedures
• Synonyms
• User Defined Types
• Views
78. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tree view of objects
• Tree view of
source
• Lazy loaded
objects
79. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Actions at a glance
• Action items to
help with manual
conversion
80. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Target database schema
• Tree view of
target
• Lazy loaded
objects
81. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source object code
• SQL code to
create chosen
source database
object
• Properties of
chosen source
database object
82. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Target object code
• Editable SQL
code to create
chosen object in
the target
database
• Properties of
chosen target
database object
83. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apply to target database
Select Apply to
Database from the
Action menu
or
Right click on the
target schema to
apply
84. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Menu navigation
85. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work with projects
Save your work
Open an existing project from a
list of recent ones
Connect to a source or target
86. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Context sensitive actions
Menu options change when you work with the source versus the target
87. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Switch between views
• Main view shows the working window for code conversion
• Assessment report view shows the summary and action item detail
• Database migration view for orchestrating your migration from within SCT
88. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Convert an application
Use the AWS Schema Conversion Tool to convert your application code
89. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Get help
Update AWS SCT
User Guide
90. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Closer look at Settings
91. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Global settings
Applies to your installation of SCT
Control
• Logging
• Drivers
• Performance and Memory
• Assessment Report
• AWS Service Profiles
• Security
• Notifications
• Tree View
92. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Global settings - logging
• Application activity is
logged and file size is
retained to 100 MB up
to 20 files by default
• Increase the
information in the log
by setting Debug
mode to true
93. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Drivers
• Add or update
drivers for your
databases
• Drivers are used
to connect to
the source and
target
• Use the
recommended
drivers in the
user guide
94. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Service Profiles
Add your AWS
Credentials to initiate
an AWS DMS
migration from within
AWS SCT
Enter your AWS
Access Key and AWS
Secret Key
Test the connection
to check permissions
95. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Control your application and project settings
Default Project Settings
Indicates where your project
will be stored by default
96. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Control your application and project settings
Project Settings
Settings specific to the open
project
Control
• Conversion settings
• Project environment
• SQL Scripting
• Tree View
97. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Modify or add mapping rules
Import or export
transformation rules as JSON
98. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions?
99. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Database Migration Service
100. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Overview
DMS components
Database migration options
Migration use cases
Best practices
Questions
101. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overview
102. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overview
• Simple to use
• Minimal downtime
• Supports widely used
databases
• Low cost
• Fast and easy to setup
• Reliable
103. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simple to use
No drivers or applications to install
No changes to the source database in most cases
Just a few clicks to start a migration from the console
DMS manages the complexities of migration for you
Automatically replicate changes
Can be used for continuous replication
104. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minimal downtime
Source
All changes can be
replicated to the target
Source database stays
fully operational during
the migration
Target
Target database stays
in sync with the source
for as long as you
choose
Switch over when the
target is fully sync’d
and without lag
105. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Supports widely used databases
Sources* Targets**
Oracle Oracle
SQL Server SQL Server
Azure SQL PostgreSQL
PostgreSQL MySQL
MySQL Amazon Redshift
SAP ASE SAP ASE
MongoDB Amazon S3
Amazon S3 Amazon DynamoDB
IBM DB2 Amazon Kinesis
Amazon ElasticSearch
On-premises
database
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html
*
**
106. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Low cost
Compute
resource
Additional
log
storage
Instance
Type
Hourly
Rate
Duration Activities Total
t2.small $0.036 2 weeks Testing $12.096
c4.large $0.154 2 weeks Initial
Load &
CDC Until
Cutover
$51.744
Migrate a 1 TB DB for under $65 ($63.84)
Pricing Example
https://aws.amazon.com/dms/pricing/
107. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fast and easy to setup
Set up a migration task in minutes
Connect to
the source
database
Connect to
the target
database
Create a
replication
instance to
run the
migration
Create a task Run the task
You can use different tasks with different settings for different environments
108. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reliable
• Highly resilient and self–healing
• Continual monitoring
• Source and target databases
• Network connectivity
• Replication instance
• In case of interruption, it automatically restarts the
process and continues the migration from where it was
halted
• Multi-AZ option for high-availability
CloudWatch
109. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS Components
110. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Migration Service components
Replication instance
Endpoint
Task
111. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replication instance
• T2, C4, or R4 instance types
Dedicated EC2 instance
• Private: Source or Target inside VPC / VPN / DC
• Public: Source or Target outside VPC
Public or private IP address
• Moves the data from the source to the target
• Support for multiple tasks
Task execution
Replication
Instance
112. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Endpoint
Connection information to source or target database
• Endpoint type, identifier, engine (source or target)
• Server name, port, SSL mode
• Username, password
Advanced attributes
• Extra connection information
• KMS key
Test with a replication instance
113. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Endpoint sources
Oracle
SQL Server
Azure SQL PostgreSQL
MySQL
SAP ASE
MongoDB Amazon S3
IBM DB2
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html
114. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Endpoint targets
Targets
Oracle
SQL Server
PostgreSQL
MySQL
Amazon Redshift
SAP ASE
Amazon S3
Amazon DynamoDB
Amazon Kinesis
Amazon ElasticSearch
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html
115. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task
Name your
task
No spaces
or
underscores
Connect to
endpoints
Source
Target
Choose
migration type
Existing data
Existing data
and replicate
changes
Replicate
changes
only
Target
preparation
Do nothing
Drop tables
on target
Truncate
Include LOBs
Don't include
LOBs
Full LOB
mode
Limited LOB
mode*✓ Enable validation
✓ Enable logging
116. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task – migration type
Choose
migration type
Existing data
Existing data
and replicate
changes
Replicate
changes
only
Creates files or tables in the target database
Populates the tables with data from the source.
Migrate existing data option in the AWS console and Full Load in the
API.
Captures changes on the source during migration
Once initial migration completes, changes are applied to the target as
units of completed transactions
Migrate existing data and replicate ongoing changes option in the AWS
console and full-load-and-cdc in the API.
Reads the recovery file on the source database
Groups together transactions and applies them to the target. Buffering
as needed
Replicate data changes only option in the AWS console
117. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
In Do nothing mode, AWS DMS assumes target tables are pre-created. In
full load or full load plus CDC, ensure that the target tables are empty
before starting the migration.
In Drop tables on target mode, AWS DMS drops the target tables and
recreates them before starting the migration. This ensures that the
target tables are empty when the migration starts.
In Truncate mode, AWS DMS truncates all target tables before the
migration starts.
Task – Target preparation
Target
preparation
Do nothing
Drop tables
on target
Truncate
118. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LOB columns are excluded from the migration.
Migrate complete LOBs regardless of size. AWS DMS migrates LOBs
piecewise in chunks controlled by the Max LOB size parameter. This
mode is slower than using Limited LOB mode.
Truncate LOBs to the value of the Max LOB size parameter. This mode is
faster than using Full LOB mode.
Task – Include LOBs
Include
LOBs
Don't
include
LOBs
Full LOB
mode
Limited
LOB mode*
119. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
More about replication tasks
• Tasks are the workhorse of the migration
• Tasks run on a replication instance
• Multiple tasks can be run in parallel
• Task settings JSON gives you control over performance
and debugging
120. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration scenarios
121. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Possible migration preparation scenarios
Using DMS alone
• DMS Creates tables and primary keys using Drop tables on target
• Manually add secondary indexes and foreign key objects after initial load has
completed.
Using SCT and DMS together
• Use Schema Conversion Tool to create all objects
• Disable secondary indexes and foreign key objects
• Load using DMS using Do nothing
• Re-enable secondary indexes and foreign key objects
Using Native Tools and DMS
• Use database scripts to create tables, primary keys, views, and sequences
• Load tables with DMS using Truncate
• Manually add secondary indexes and foreign key objects after initial load has
completed.
122. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Steps to migrate - overview
Start the full load
While loading data, also capture changes
Load complete - apply captured changes
Changes reach steady state
Shutdown apps and apply remaining changes
Change your application endpoint
123. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replication
Instance
Source Target
Start full load
124. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source TargetReplication Instance
App
While loading data also capture changes
125. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source TargetReplication Instance
App
Load complete - apply captured changes
126. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changes reach steady state
Source TargetReplication Instance
App
127. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source TargetReplication Instance
App
Shutdown apps and apply remaining changes
128. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Change your application endpoint
Source TargetReplication Instance
App
129. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load is table by table
Replication Instance
Source Target
130. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replication Server
Source Target
App
t
1
t
2
t
1
t
2
Changes are transactional - from db logs
131. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS – change data capture (CDC)
“No Touch” design
• Reads recovery log of source database
• Using the engine’s native change data capture API
• No agent required on the source
• Changes captured as transactions and applied in order
• Activated when load starts
• Changes are applied after initial load is complete
132. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DMS – change data capture (CDC)
Some requirements
• Oracle: Supplemental logging required
• MySQL: Full image row level bin logging required
• SQL Server: Recovery model bulk logged or full
• Postgres: wal_level = logical; max_replication_slots >= 1;
max_wal_Senders >=1; wal_sender_timeout = 0
133. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Replication
Instance
Source Target
Multiple targets
Target
Target
134. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration
Server
Source Target
Multiple Sources
Source
Source
135. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration
Server
Source
Target
Multiple sources and targets
Source
Source
Target
136. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
You don’t have to take everything
Source
L
Target
Replication Instance
137. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Homogenous or heterogeneous
Replication
Instance
SQL Server MySQL
Replication
Instance
Oracle Oracle
Replication
Instance
Oracle PostgreSQL
138. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Size in GB Transaction volume
Size
Transaction Load
DMS limits
Too big
Toomanychanges
139. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Setup and configuration tips
• Enable CloudWatch logs (not enabled by default)
• Choose LOB mode carefully
• Replication instance security group is default for VPC –
change after creation
• Extra connection attributes can alter how the migration
task operates
• Provide transformation rules for changing case
140. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source configuration tips
• Oracle specific settings for CDC
• Enable supplemental logging on tables with no primary key
• If you add a table mid-migration add supplemental logging
• Even if the table has a primary key
• To have DMS automatically configure supplemental logging in
Oracle add an extra connection attribute
• addSupplementalLogging=Y
• Check for unsupported datatypes
• Can use a read replica or standby as a migration source
141. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance tips
• Use larger DMS instance for maximum throughput
• CPU for type conversions
• Check network throughput
• Split load across multiple tasks and/or DMS instances
• Remember transaction boundaries when capturing changes
• Reduce contention on your target
• Turn off logging
• Run in single AZ
Further reading on the AWS Database Blog
142. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions