Workshop on Advanced Design Patterns for Amazon DynamoDB - DAT405 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Advanced Design Patterns for Amazon
DynamoDB
P A D M A M A L L I G A R J U N A N
R E G I S G I M E N I S
R I C K H O U L I H A N
A W S P r o f e s s i o n a l S e r v i c e s
D A T 4 0 5
N o v e m b e r 3 0 , 2 0 1 7
AWS re:INVENT

Environment Setup (15 minute)
1. Open https://tinyurl.com/awsddbwks
2. Under “Preparation,” run Steps 1-8.
3. Stuck on Step1? Raise your hands.
DAT 405 - Workshop on Advanced Design
Patterns for Amazon DynamoDB
Prerequisites:
1. Laptop
2. Basic
understanding
of DynamoDB

AGENDA
• Prerequisites for the workshop
• Key DynamoDB concepts
• Tenets of NoSQL data modeling
• Workshop (2 hours, 7 labs)
• Design patterns - m:n relationships, index
overloading, sparse indexes, locking, adjacency
lists, DynamoDB Streams

AMAZON DYNAMODB - KEY CONCEPTS
Document or Key-Value
Scales to Any Workload
Fully Managed NoSQL
Access Control
Event-Driven Programming
Fast and Consistent
Table
Items
Attributes
Partition
Key
Sort
Key
• Local Secondary Index
• Global Secondary Index

Selecting a Partition Key
• Large number of distinct
values
• Values are uniformly
requested, at random
AMAZON DYNAMODB - KEY CONCEPTS
Selecting a Sort Key
• Model 1:n and m:n
relationships
• Query across entity types
• Efficient/selective queries
• Range queries
Examples:
• Bad: Status, Gender
• Good: CustomerId, DeviceId
Examples:
• Orders and OrderItems per
Customer

• Understand the use case
• Identify the access patterns
• Read/Write workloads
• Query dimensions and
aggregations
• Data-modeling
• Using NoSQL design patterns
• Review -> Repeat -> Review
TENETS OF DYNAMODB DATA MODELING

aggregations
• Data-modeling
• Nature of the data
• OLTP / OLAP / full-text search
• Relationships between the entities
• What does concurrent access look like?
• Time series data
• Archiving needs, etc.

aggregations
• Data-modeling
• Source Data analysis (write workload)
• Reading one item vs multiple items
(read workload)
• Query aggregations and KPIs

aggregations
• Data-modeling
• 1-1, 1-m, n-m relationships
• 1 application = 1 table
• Avoid unnecessary fetches
• Simplify access patterns
• Identify Primary Key
• Partition-Key and Sort-Key
• Query dimensions using LSIs and GSIs

Local Secondary Index Global Secondary Index
CAPACITY PLANNING FOR INDEXES
• Index is local to the partition
• Data size per partition <10GB
• Index across partitions
• Eventually consistent
• Consumes RCUs/WCUs from the
table’s provisioning
• RCUs/WCUs are provisioned separately
for GSIs
If GSIs don’t have enough write capacity, table writes will be throttled!

Throughput
§ Provision any amount of throughput to a table
Size
§ Add any number of items to a table
- Max item size is 400 KB
- LSIs limit the number of range keys due to 10 GB limit
Scaling is achieved through partitioning
SCALING

PARTITIONING MATH
Number of Partitions
By Capacity (Total RCU / 3000) + (Total WCU / 1000)
By Size Total Size / 10 GB
Total Partitions CEILING(MAX (Capacity, Size))
In the future, these details might change…

PARTITIONING EXAMPLE
Table size = 8 GB, RCUs = 5000, WCUs = 500
RCUs per partition = 5000/3 = 1666.67
WCUs per partition = 500/3 = 166.67
Data/partition = 10/3 = 3.33 GB
RCUs and WCUs are uniformly
spread across partitions
Number of Partitions
By Capacity (5000 / 3000) + (500 / 1000) = 2.17
By Size 8 / 10 = 0.8
Total Partitions CEILING(MAX (2.17, 0.8)) = 3

0
400
800
1200
1600
CapacityUnits
Time
Provisioned Consumed
“Save up” unused
capacity
Consume saved-up capacity
Burst: 300 seconds
(1200 × 300 = 360k CU)
BURST CAPACITY IS BUILT-IN

ADAPTIVE THROUGHPUT
Helped over 2800 Amazon tables & indexes on Prime Day!

AWS STAFF IN THE ROOM
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs
• ~15 minutes per Lab
• Raise your hands for questions
• Staying in sync with the group
• Review the Python scripts
WORKSHOP

Exercise #1 - DynamoDB Capacity Units &
Partitioning (20 minutes)
What you’ll learn:
• Tables with lower WCUs
• Write a smaller data set
• Write larger data sets at high velocity
• Burst capacity
• Behavior when dialing up capacity
• Under-provisioned WCUs on a GSI
• Viewing these metrics on Cloudwatch
WORKSHOP
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs
• ~15 minute per Lab

DYNAMODB SCAN OPERATIONS
• Access every item in a table on an index
• Read 1MB data in each operation
• Use LastEvaluatedKey to continue..
• Reads up to the max throughput of a single partition
• Parallel scans vs Sequential scans

PARALLEL SCAN
• Read all the items of a table faster
• Take advantage of the table’s provisioned capacity
• Set TotalSegments = number of application workers, each worker scans a
different segment
All Data items
Segment 0 Segment 1 Segment 2
Worker-0 Worker-1 Worker-2
Application - Main thread
Worker-3
Segment 3

The upcoming lab…
Scenario:
Scan Server logs data for response code = OK
• Sequential Scan
fe = "responsecode <> :f"
eav = {":f": 200}
response = table.scan(
FilterExpression=fe,
ExpressionAttributeValues=eav,
Limit=pageSize
)
• Parallel Scan
fe = "responsecode <> :f"
eav = {":f": 200}
response = table.scan(
FilterExpression=fe,
ExpressionAttributeValues=eav,
Limit=pageSize,
TotalSegments=totalsegments,
Segment=threadsegment
)

• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
Exercise #2 – DynamoDB scans – Sequential &
Parallel table scan (10 minutes)
• Time a sequential (simple) scan vs a parallel
scan.
• Populate a table with a large data set.
• Scan and compare run times.
• Time difference will be significant for
larger data sets.
• Don’t forget to review the Python scripts!
WORKSHOP
WORKSHOP STRUCTURE
• 7 Labs

DATA MODELING: GSI OVERLOADING
• Working with the 5 GSIs limitation
• Overload attribute values based on an item’s context
Example: Query Employee Database by (1) Employee Name (2) Desk
(3) Hire Date (4) Quarterly Sales (5) Current Job role
(6) Employees in a City (7) Warehouse Location (8) Employee ID
…. and many more

Example: Query Employee Database by (1) Employee Name (2) Warehouse
(3) Hire Date (4) Quarterly Sales
Partition Key = “Employee ID”
Sort Key = (Let’s look at final design)
DynamoDB Table design…

Example: Query Employee Database by
(1) Employee Name (2) Desk (3) Hire Date (4) Quarterly Sales

Partition Key = “Employee ID”
Sort Key = Use a generic attribute-name “A”
Value is based on the item type.
Create a GSI on “A”.

In the upcoming lab…

Exercise #3 – GSI Overloading (15 minute)
• Load the Employee data, create an
overloaded-GSI, and query the results!
WORKSHOP
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs

DATA MODELING: SPARSE INDEXES
• Saves WCUs
• Selective querying
Example: Investigators use a GSI to query “Assigned” Cases
A
U
D
I
T
S
Primary Key
Attributes
CaseID Item
CASE1
DETAILS1
Assignee Tags ASIN OrderID
INV1
Assignee Metadata
INV1#AUDIT1
Assignee Status_Date Type TicketID
ASSIGNED_2017-08-08 T1
INV1#AUDIT2
COMPLETED_2017-08-08 T1
INV1#AUDIT3
POST-AUDIT_2017-08-08 T1
INV1#AUDIT4
ARCHIVED_2017-08-08 T1
S31
S3Datapoints

A
U
D
I
T
S
Primary Key
Attributes
CaseID Item
CASE1
DETAILS1
Other attributes
INV1
Other attributes
INV1#AUDIT1
ASSIGNEE STATUS_DATE
INV1#AUDIT2
INV1#AUDIT3
INV1#AUDIT4

A
U
D
I
T
S
Primary Key
Attributes
CaseID Item
CASE1
DETAILS1
Other attributes
INV1
Other attributes
INV1#AUDIT1
ASSIGNEE (GSI-PK) STATUS_DATE (GSI-SK)
INV1#AUDIT2
INV1#AUDIT3
INV1#AUDIT4

A
U
D
I
T
S
Primary Key
Attributes
CaseID Item
CASE1
DETAILS1
Other attributes
INV1
Other attributes
INV1#AUDIT1
ASSIGNEE GSI_STATUS_DATE Status_Date
ASSIGNED_2017-08-08 T1 ASSIGNED_2017-08-08 T1
INV1#AUDIT2
Assignee Status_Date
INV1#AUDIT3
INV1#AUDIT4

DATA MODELING: SPARSE INDEXES –
ANOTHER EXAMPLE
Id
(Partition)
User Game Score Date Award
1 Bob G1 1300 2012-12-23
2 Bob G1 1450 2012-12-23
3 Jay G1 1600 2012-12-24
4 Mary G1 2000 2012-10-24 Champ
5 Ryan G2 123 2012-03-10
6 Jones G2 345 2012-03-20
Game-scores-table
Award
(Partition)
Id User Score
Champ 4 Mary 2000
Award-GSI
Scan sparse GSIs

Exercise #4 – Sparse Indexes (15 minute)
• Create a GSI on an existing table (this step
takes a few minutes)
• Review the use case for using a sparse index
• Take advantage of sparse Indexes!
WORKSHOP
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs

DATA MODELING: COMPOSITE SORT KEYS
• Define hierarchical relationships
• Execute selective queries

Multivalue sorts and filters
Secondary index
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
Bob
Partition key Sort key

Approach 1 : Query Filter
Secondary index
Opponent Date GameId Status Host
Alice 2014-10-02 d9bl3 DONE David
Carol 2014-10-08 o2pnb IN_PROGRESS Bob
Bob 2014-09-30 72f49 PENDING Alice
Bob 2014-10-03 b932s PENDING Carol
Bob 2014-10-03 ef9ca IN_PROGRESS David
SELECT * FROM Game
WHERE Opponent='Bob'
ORDER BY Date DESC
FILTER ON Status='PENDING'
Bob
(filtered out)

Approach 2 : Composite Sort Keys
StatusDate
DONE_2014-10-02
IN_PROGRESS_2014-10-08
IN_PROGRESS_2014-10-03
PENDING_2014-09-30
PENDING_2014-10-03
Status
DONE
IN_PROGRESS
IN_PROGRESS
PENDING
PENDING
Date
2014-10-02
2014-10-08
2014-10-03
2014-10-03
2014-09-30
+ =

Secondary Index
Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Partition key Sort key

Opponent StatusDate GameId Host
Alice DONE_2014-10-02 d9bl3 David
Carol IN_PROGRESS_2014-10-08 o2pnb Bob
Bob IN_PROGRESS_2014-10-03 ef9ca David
Bob PENDING_2014-09-30 72f49 Alice
Bob PENDING_2014-10-03 b932s Carol
Secondary index
Bob
SELECT * FROM Game
WHERE Opponent='Bob'
AND StatusDate BEGINS_WITH 'PENDING'

Example: Query Amazon buildings by Country, State, City, Office Location
Partition Key = “USA”
Sort Key = “TX”CONCAT ( , “AUSTIN”, “AUS-007” )

Example: Query Amazon buildings by State, City, Office Location

• Use composite sort key to define a hierarchy
• Highly selective queries with sort conditions
• Reduce query complexity

Exercise #5 – Composite Sort Keys (15 minute)
• Query Employees data using State, City,
Department.
• Create a GSI on PartitionKey=State;
SortKey=City#Dept.
• If the attribute doesn’t exist, you have to
add it to the table.
• Recap: In the real world, understand all the
access patterns to design your table
WORKSHOP
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs

DATA MODELING: ADJACENCY LISTS
• Easily model many-to-many relationships
• Without excessive data duplication
Example: An invoice contains many bills. A bill can be broken up into
multiple invoices.
Desired access patterns: (1,) Get Bills by InvoiceID
(2,) Get invoices by BillID

Example: An invoice contains many bills. A bill can be broken up into multiple invoices.
Desired access patterns: (1,) Get Bills by InvoiceID (2,) Get invoices by BillID
Primary Key
Attributes
ID Item
INVOICE1
INVOICE1
Other Invoice Attributes
BILL1
Bill-Invoice Attributes
BILL2
BILL1 BILL1
Other Bill Attributes
BILL2 BILL1

Primary Key
Attributes
ID Item (GSI-PK)
INVOICE1
INVOICE1
BILL1
BILL2
BILL1 BILL1
BILL2 BILL2

GSI Primary Key
Projected Attributes
Item
INVOICE1
ID Item Other Invoice Attributes
INVOICE1 INVOICE1
BILL1
ID Item Bill-Invoice Attributes
INVOICE1 BILL1
ID Item Other Bill Attributes
BILL1 BILL1
BILL2
ID Item Bill-Invoice Attributes
INVOICE1 BILL2
ID Item Other Bill Attributes
BILL2 BILL2
Secondary index

DATA MODELING: LOCKING
(3,) Modify an invoice within a transaction
Primary Key
Attributes
ID Item
INVOICE1
INVOICE1
BILL1
BILL2
BILL1 BILL1
BILL2 BILL1

DATA MODELING: LOCKING
(3,) Modify an invoice within a transaction
Primary Key
Attributes
ID Item
INVOICE1
INVOICE1
LOCK Other Invoice Attributes
0/1
BILL1
BILL2
BILL1 BILL1
BILL2 BILL1

DATA MODELING: VERSIONING
(3,) Modify an invoice within a transaction (4,) Versioning (history of Invoice updates)
Primary Key
Attributes
ID Item
INVOICE1
v0-INVOICE1
0/1
v1-INVOICE1

(3,) Modify an invoice within a transaction (4,) Versioning (history of Invoice updates)
Primary Key
Attributes
ID Item
INVOICE1
v0-INVOICE1
0/1
v1-INVOICE1
v2-INVOICE1

Primary Key
Attributes
ID Item
INVOICE1
v0-INVOICE1
0/1
v0-BILL1
v0-BILL2
v1-INVOICE1
v1-BILL1
v2-INVOICE1
v2-BILL1
v2-BILL2
BILL1 v0-BILL1

Exercise #6 – Locking (10 minute)
• Lock an employee record, and prevent dirty
writes.
• After successful obtaining a lock, make the
changes, then release the lock.
• Recap: In the real-world, understand all the
access patterns to design your table
WORKSHOP
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs

DYNAMODB STREAMS AND AWS LAMBDA

DYNAMODB STREAMS AS TRIGGERS
Lambda function
Notify change
Item/Table Level Metrics
Amazon CloudSearch
Kinesis Firehose

Exercise #7 – DynamoDB Streams & Lambda
(10 minutes)
• Build a replication workflow
• Enable Streams on the tlog table
• Create a replication target
• Create a Lambda function (with IAM
policies) to read the streams and write to
the target
WORKSHOP
• Rick Houlihan
• Regis Gimenis
• Sean Shriver
• Edin Zulich
WORKSHOP STRUCTURE
• 7 Labs

D A T 4 0 5 – A d v a n c e d D e s i g n P a t t e r n s f o r A m a z o n D y n a m o D B
P A D M A M A L L I G A R J U N A N * R E G I S G I M E N I S * R I C K H O U L I H A N
thank you!
Please fill out the evaluations :)

Workshop on Advanced Design Patterns for Amazon DynamoDB - DAT405 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Workshop on Advanced Design Patterns for Amazon DynamoDB - DAT405 - re:Invent 2017

Similar to Workshop on Advanced Design Patterns for Amazon DynamoDB - DAT405 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Workshop on Advanced Design Patterns for Amazon DynamoDB - DAT405 - re:Invent 2017