Deep Dive on Amazon Aurora

AWS re:INVENT
Deep-dive on Amazon Aurora
D i c k s o n Yu e , S o l u t i o n s A r c h i t e c t
300
January 18, 2018

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
>
>
>
Aurora fundamentals
Recent improvements
Coming soon

AURORA FUNDAMENTALS

Amazon Aurora:
A r elational databas e r eimagined for the c loud
 Speed and availability of high-end commercial databases
 Simplicity and cost-effectiveness of open source databases
 Drop-in compatibility with MySQL and PostgreSQL
 Simple pay as you go pricing
Delivered as a managed service

R elational databas es w er e not des igned for the c loud
Monolithic architecture
Large failure blast radius
SQL
TRANSACTIONS
CACHING
LOGGING

Scale-out, distributed architecture
Master Replica Replica Replica
AVAILABILITY
ZONE 1
SHARED STORAGE VOLUME
AVAILABILITY
ZONE 2
AVAILABILITY
ZONE 3
STORAGE NODES WITH SSDS
Logging pushed down to a purpose-built
log-structured distributed storage system
Storage volume is striped across hundreds
of storage nodes distributed across 3
availability zones (AZ)
Six copies of data, two copies in each AZ
SQL
TRANSACTIONS
CACHING
SQL
TRANSACTIONS
CACHING
SQL
TRANSACTIONS
CACHING

Why are 6 copies necessary?
 In a large fleet, always some
failures
 AZ failures have ”shared fate”
 Need to tolerate AZ+1 failures and
still be able to repair
 For 3 AZs, requires 6 copies
AZ 1 AZ 2 AZ 3
Quorum
break on
AZ failure
2/3 read
2/3 write
AZ 1 AZ 2 AZ 3
Quorum
survives
AZ failure
3/6 read
4/6 write

Minimal time to repair
 Small segments shorten repair
 Aurora uses 10GB segments
 Can use fault-tolerance to:
• patch without impact
• balance hot and cold nodes
Segment size

Membership changes without stalls
 Most systems use consensus for
membership changes. Causes jitter
and stalls.
 Aurora uses quorum sets and epochs
 No stalls, AZ+1 fault–tolerance, can
aggressively detect failure
A B C D E F
A B C D E F
A B C D E G
A B C D E F
A B C D E G
Epoch 1: All node healthy
Epoch 2: Node F is in suspect state; second
quorum group is formed with node G; both
quorums are active
Epoch 3: Node F is confirmed unhealthy; new
quorum group with node G is active.

Avoiding quorum reads
 Reads are expensive in most
quorum-based systems
 Aurora knows which nodes are
up to date, latency to each node
 Read quorum only needed for
repairs or crash recovery
LSN
N
LSN
N
LSN
N-1
LSN
N
LSN
N-1
LSN
N
LSN
N
LSN
N
LSN
N-1
LSN
N
LSN
N-1
LSN
N
For each data block, at least 4 nodes in the
quorum group will have the most recent data
Read from any one of these four
nodes will return most recent data

Aurora performance
WRITE PERFORMANCE READ PERFORMANCE
MySQL SysBench results; R4.16XL: 64cores / 488 GB RAM
Aurora read/write throughput compared to MySQL 5.6
based on industry standard benchmarks.
Aurora MySQL 5.6
0
100000
200000
300000
400000
500000
600000
700000
0
50000
100000
150000
200000
250000

Do fewer IOs
Minimize network packets
Cache prior results
Offload the database engine
DO LESS WORK
Process asynchronously
Reduce latency path
Use lock-free data structures
Batch operations together
BE MORE EFFICIENT
How did we achieve this?
DATABASES ARE ALL ABOUT I/O
NETWORK-ATTACHED STORAGE IS ALL ABOUT PACKETS/SECOND
HIGH-THROUGHPUT PROCESSING IS ALL ABOUT CONTEXT SWITCHES

IO traffic in MySQL
BINLOG DATA DOUBLE-WRITELOG FRM FILES
T Y P E O F W R IT E
MYSQL WITH REPLICA
EBS MIRROREBS MIRROR
AZ 1 AZ 2
AMAZON S3
EBS
AMAZON
ELASTIC BLOCK
STORE (EBS)
PRIMARY
INSTANCE
REPLICA
INSTANCE
1
2
3
4
5
Issue write to EBS – EBS issues to mirror, ack when both done
Stage write to standby instance
Issue write to EBS on standby instance
IO FLOW
Steps 1, 3, 4 are sequential and synchronous
This amplifies both latency and jitter
Many types of writes for each user operation
Have to write data blocks twice to avoid torn writes
OBSERVATIONS
780K transactions
7,388K I/Os per million txns (excludes mirroring, standby)
Average 7.4 I/Os per transaction
PERFORMANCE
30 minute SysBench writeonly workload, 100GB dataset, RDS MultiAZ, 30K PIOPS

IO traffic in Aurora
AZ 1 AZ 3
PRIMARY
INSTANCE
Amazon S3
AZ 2
REPLICA
INSTANCE
AMAZON AURORA
ASYNC
4/6 QUORUM
DISTRIBUTED
WRITES
BINLOG DATA DOUBLE-WRITELOG FRM FILES
T Y P E O F W R IT E
IO FLOW
Only write redo log records; all steps asynchronous
No data block writes (checkpoint, cache replacement)
6X more log writes, but 9X less network traffic
Tolerant of network and storage outlier latency
OBSERVATIONS
27,378K transactions 35X MORE
950K I/Os per 1M txns (6X amplification) 7.7X LESS
PERFORMANCE
Boxcar redo log records – fully ordered by LSN
Shuffle to appropriate segments – partially ordered
Boxcar to storage nodes and issue writesREPLICA
INSTANCE

IO traffic in Aurora (Storage Node)
LOG RECORDS
Primary
Instance
INCOMING QUEUE
STORAGE NODE
S3 BACKUP
1
2
3
4
5
6
7
8
UPDATE
QUEUE
ACK
HOT
LOG
DATA
BLOCKS
POINT IN TIME
SNAPSHOT
GC
SCRUB
COALESCE
SORT
GROUP
PEER TO PEER GOSSIPPeer
Storage
Nodes
All steps are asynchronous
Only steps 1 and 2 are in foreground latency path
Input queue is 46X less than MySQL (unamplified, per node)
Favor latency-sensitive operations
Use disk space to buffer against spikes in activity
OBSERVATIONS
IO FLOW
① Receive record and add to in-memory queue
② Persist record and ACK
③ Organize records and identify gaps in log
④ Gossip with peers to fill in holes
⑤ Coalesce log records into new data block versions
⑥ Periodically stage log and new block versions to S3
⑦ Periodically garbage collect old versions
⑧ Periodically validate CRC codes on blocks

TRADITIONAL DATABASE
Have to replay logs since the last
checkpoint
Typically 5 minutes between checkpoints
Single-threaded in MySQL; requires a
large number of disk accesses
AMAZON AURORA
Underlying storage replays redo records
on demand as part of a disk read
Parallel, distributed, asynchronous
No replay for startup
Checkpointed Data Redo Log
Crash at T0 requires
a re-application of the
SQL in the redo log since
last checkpoint
T0 T0
Crash at T0 will result in redo logs being
applied to each segment on demand, in
parallel, asynchronously
Instant crash recovery

RECENT IMPROVEMENTS

Fast database cloning *August 2017*
Clone database without copying data
 Creation of a clone is nearly instantaneous
 Data copy happens only on write – when
original and cloned volume data differ
Example use cases
 Clone a production DB to run tests
 Reorganize a database
 Save a point in time snapshot for analysis
without impacting production system
PRODUCTION DATABASE
CLONE CLONE
CLONE
DEV/TEST
APPLICATIONS
BENCHMARKS
PRODUCTION
APPLICATIONS
PRODUCTION
APPLICATIONS

Backup and restore in Aurora
Take periodic snapshot of each segment in parallel; stream the redo logs to Amazon S3
Backup happens continuously without performance or availability impact
At restore, retrieve the appropriate segment snapshots and log streams to storage nodes
Apply log streams to segment snapshots in parallel and asynchronously
SEGMENT
SNAPSHOT
LOG
RECORDS
RECOVERY
POINT
SEGMENT 1
SEGMENT 2
SEGMENT 3
TIME

Backtrack provides near-instantaneous restores
* C o m i n g s o o n *
Backtrack quickly brings the database to a desired point in time.
No restore from backup. No copying of data. Not destructive – can backtrack many times.
Quickly recover from unintentional DML/DDL operations.
T0 T1 T2
T0 T1
T2
T3 T4
T3
T4
REWIND TO T1
REWIND TO T3
INVISIBLE INVISIBLE

Read replica end-point auto-scaling *Nov. 2017*
Up to 15 promotable read replicas across multiple availability zones
Re-do log based replication leads to low replica lag – typically < 10ms
Reader end-point with load balancing and auto-scaling * NEW *
MASTER
READ
REPLICA
READ
REPLICA
READ
REPLICA
SHARED DISTRIBUTED STORAGE VOLUME
READER END-POINT

Online DDL: Aurora vs. MySQL
 Full Table copy in the background
 Rebuilds all indexes – can take hours or days
 DDL operation impacts DML throughput
 Table lock applied to apply DML changes
INDEX
LEAFLEAFLEAF LEAF
INDEX
ROOT
table name operation column-name time-stamp
Table 1
Table 2
Table 3
add-col
add-col
add-col
column-abc
column-qpr
column-xyz
t1
t2
t3
 Use schema versioning to decode the block
 Modify-on-write primitive to upgrade to latest schema
 Currently support add NULLable column at end of table
 Add column anywhere and with default coming soon
MySQL Amazon Aurora

Online DDL performance
On r3.large
On r3.8xlarge
Aurora MySQL 5.6 MySQL 5.7
10GB table 0.27 sec 3,960 sec 1,600 sec
Aurora MySQL 5.6 MySQL 5.7
10GB table 0.06 sec 900 sec 1,080 sec

Asynchronous key prefetch *Oct. 2017*
Latency improvement factor vs. Batched Key Access (BKA) join
algorithm. Decision support benchmark, R3.8xlarge
14.57
-
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
Query-1
Query-2
Query-3
Query-4
Query-5
Query-6
Query-7
Query-8
Query-9
Query-10
Query-11
Query-12
Query-13
Query-14
Query-15
Query-16
Query-17
Query-18
Query-19
Query-20
Query-21
Query-22
Cold buffer
Works for queries using Batched Key
Access (BKA) join algorithm + Multi-Range
Read (MRR) Optimization
Performs a secondary to primary index
lookup during JOIN evaluation
Uses background thread to asynchronously
load pages into memory
AKP used in queries 2, 5, 8, 9, 11, 13, 17, 18, 19, 20, 21

Batched scans *c oming s oon*
Standard MySQL pprocesses rows one-at-
a-time resulting in high overhead due to:
 Repeated function calls
 Locking and latching
 cursor store and restore
 InnoDB to MySQL format conversion
Amazon Aurora scan tuples from the
InnoDB buffer pool in batches for
 Table full scans
 Index full scans
 Index range scans Latency improvement factor vs. Batched Key Access (BKA)
join algorithm Decision support benchmark, R3.8xlarge
1.78X
-
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
Query-1
Query-2
Query-3
Query-4
Query-5
Query-6
Query-7
Query-8
Query-9
Query-10
Query-11
Query-12
Query-13
Query-14
Query-15
Query-16
Query-17
Query-18
Query-19
Query-20
Query-21
Query-22

Hash Joins *c oming s oon*
Latency improvement factor vs. Batched Key Access (BKA) join algorithm
Decision support benchmark, R3.8xlarge, cold buffer cache (lower improvement if all data cached)
8.22
-
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
Hash join used in queries 2, 3, 5, 7, 8, 9, 10, 11, 12, 15, 16, 17,18, 19, 21

AURORA SERVERLESS

Aurora Serverless
 Starts up on demand, shuts down
when not in use
 Scales up/down automatically
 No application impact when scaling
 Pay per second, 1 minute minimum
WARM POOL
OF INSTANCES
APPLICATION
DATABASE STORAGE
SCALABLE DB CAPACITY
REQUEST ROUTER
DATABASE END-POINT

Database end-point provisioning
When you provision a database, Aurora
Serverless:
 Provisions VPC end-points for the
application connectors
 Initializes request routers to accept
connections
 Creates an Aurora storage volume
A database instance is only provisioned
when the first request arrives
APPLICATION
CUSTOMER VPC
VPC
END-POINTS
VPC
END-POINTS
NETWORK LOAD BALANCER
STORAGE
VOLUME
REQUEST
ROUTERS

Instance provisioning and scaling
 First request triggers instance
provisioning. Usually 1-3 seconds
 Instance auto-scales up and down as
workload changes. Usually 1-3 seconds
 Instances hibernate after user-defined
period of inactivity
 Scaling operations are transparent to the
application – user sessions are not
terminated
 Database storage is persisted until
explicitly deleted by user
DATABASE STORAGE
WARM POOL
APPLICATION
REQUEST
ROUTER
CURRENT
INSTANCE
NEW
INSTANCE

AURORA MULTI-MASTER

Distributed Lock Manager
GLOBAL
RESOURCE
MANAGER
SQL
TRANSACTION
S
CACHING
LOGGING
SQL
TRANSACTION
S
CACHING
LOGGING
SHARED DISK CLUSTER
STORAGE
APPLICATION
LOCKING PROTOCOL MESSAGES
SHARED STORAGE
M1 M2 M3
M1 M1 M1M2 M3 M2
Cons
Heavyweight cache coherency traffic, on per-lock basis
Networking can be expensive
Negative scaling when hot blocks
Pros
All data available to all nodes
Easy to build applications
Similar cache coherency as in multi-processors

Consensus with two phase or Paxos commit
DATA
RANGE #1
DATA
RANGE #2
DATA
RANGE #4
DATA
RANGE #3
DATA
RANGE #5
L
L L
L
L
SHARED NOTHING
SQL
TRANSACTION
S
CACHING
LOGGING
SQL
TRANSACTION
S
CACHING
LOGGING
APPLICATION
STORAGE STORAGE
Cons
Heavyweight commit and membership change protocols
Range partitioning can result in hot partitions, not just hot
blocks. Re-partitioning expensive.
Cross partition operations expensive. Better at small
requests
Pros
Query broken up and sent to data node
Less coherence traffic – just for commits
Can scale to many nodes

Conflict resolution using distributed ledgers
There are many “oases” of
consistency in Aurora
The database nodes know
transaction orders from that node
The storage nodes know transaction
orders applied at that node
Only have conflicts when data
changed at both multiple database
nodes AND multiple storage nodes
Much less coordination required
2 3 4 5 61
BT1 [P1]
BT2 [P1]
BT3 [P1]
BT4 [P1]
BT1
BT2
BT3
BT4
Page1
Quorum
OT1
OT2
OT3
OT4
Page 1 Page 2
2 3 4 5 61
OT1[P2]
OT2[P2]
OT3[P2]
OT4[P2]
PAGE1 PAGE2
MASTER
MASTER
Page 2

Hierarchical conflict resolution
Both masters are writing to two
pages P1 and P2
BLUE master wins the quorum at
page P1; ORANGE master wins
quorum at P2
Both masters recognize the conflict
and have two choices: (1) roll back
the transactions or (2) escalate to
regional resolver
Regional arbitrator decides who
wins the tie breaker
2 3 4 5 61
BT1 [P1]
OT1 [P1]
2 3 4 5 61
OT1[P2]
BT1[P2]
PAGE1 PAGE2
MASTER
MASTER
BT1 OT1
Regional
resolver
Page 1 Page 2 Page 1 Page 2
Quorum
X X

Crash recovery in multi-master
CRASH
MULTI-MASTERSINGLE MASTER
Log records Gaps
Volume Complete
LSN (VCL)
AT CRASH
IMMEDIATELY AFTER RECOVERY
MASTER 1
CRASHES
GAPS
AT CRASH
Consistency
Point LSN(CPL)
VCLVCL
CPL CPL
MASTER 1
MASTER 2
IMMEDIATELY AFTER RECOVERY
Gap filled New LSNs
and Gaps
Master 1
Recovery Point
Consistency
Point LSN(CPL)

Multi-region Multi-Master
Write accepted locally
Optimistic concurrency control – no distributed lock
manager, no chatty lock management protocol
REGION 1 REGION 2
HEAD NODES HEAD NODES
MULTI-AZ STORAGE VOLUME MULTI-AZ STORAGE VOLUME
LOCAL PARTITION LOCAL PARTITIONREMOTE PARTITION REMOTE PARTITION
Conflicts handled hierarchically – at head nodes, at
storage nodes, at AZ and region level arbitrators
Near-linear performance scaling when there is no or
low levels of conflicts

AURORA PARALLEL QUERY

Parallel query processing
Aurora storage has thousands of CPUs
 Presents opportunity to push down and parallelize
query processing using the storage fleet
 Moving processing close to data reduces network traffic
and latency
However, there are significant challenges
 Data stored in storage node is not range partitioned –
require full scans
 Data may be in-flight
 Read views may not allow viewing most recent data
 Not all functions can be pushed down to storage nodes
DATABASE NODE
STORAGE NODES
PUSH DOWN
PREDICATES
AGGREGATE
RESULTS

Processing at head node
STORAGE NODES
OPTIMIZER
EXECUTOR
INNODB
NETWORK STORAGE DRIVER
AGGREGATOR
APPLICATION
PARTIAL
RESULTS
STREAM
RESULTS
IN-FLIGHT
DATA
PQ CONTEXT
PQ PLAN
Query Optimizer produces PQ Plan and
creates PQ context based on leaf page
discovery
PQ request is sent to storage node along
with PQ context
Storage node produces
 Partial results streams with processed
stable rows
 Raw stream of unprocessed rows with
pending undos
Head node aggregates these data
streams to produce final results

Processing at storage node
Each storage node runs up to 16 PQ
processes, each associated with a
parallel query
PQ process receives PQ context
 List of pages to scan
 Read view and projections
 Expression evaluation byte code
PQ process makes two passes through
the page list
 Pass 1: Filter evaluation on
InnoDB formatted raw data
 Pass 2: Expression evaluation
on MySQL formatted data
PQ PROCESS
PQ PROCESS
Up to 16
TO/FROM HEAD NODE
STORAGE
NODE PROCESS
PAGE LISTS

Example: Parallel hash join
Head node sends hash join context to
storage node consisting of:
 List of pages – Page ID and LSNs
 Join expression byte code
 Read view – what rows to include
 Projections – what columns to select
PQ process performs the following steps:
 Scans through the lists of pages
 Builds partial bloom filters; collect stats
 Sends two streams to head node
 Rows with matching hashes
 Rows with pending undo processing
Head node performs the Join function after
merging data streams from different storage
nodes
JOIN CONTEXT
PAGE SCAN
STABLE ROWS
ROWS WITH
PENDING UNDO
BLOOM
FILTER
FILTERED AND PROJECTED
ROW STREAM
UNPROCESSED
ROW STREAM

Parallel query performance
Latency (seconds)
Decision support benchmark, R3.8xlarge, cold buffer cache
Improvement factor
with Parallel Query
24.6x
18.3x
5.0x
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Aggregate + 2-table join
Aggregate query
Point query on non-indexed column
With Parallel Query Without Parallel Query

Deep Dive on Amazon Aurora

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deep Dive on Amazon Aurora

Similar to Deep Dive on Amazon Aurora (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Deep Dive on Amazon Aurora