More Related Content Similar to [NEW LAUNCH!] Building modern applications using Amazon DynamoDB transactions (DAT374) - AWS re:Invent 2018 (20) More from Amazon Web Services (20) [NEW LAUNCH!] Building modern applications using Amazon DynamoDB transactions (DAT374) - AWS re:Invent 20182. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building Modern Applications Using
Amazon DynamoDB Transactions
Yossi Levanoni
Senior Manager, Software Development
Amazon DynamoDB
D A T 3 7 4
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
1. Introduction to atomicity, consistency, isolation, and durability (ACID) in
DynamoDB, and the new transactional API
2. Modern application use cases for the transactional API
3. Important things to know when integrating the transactional API into
your application: metering, concurrency control, and interaction with
other features
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What you will learn
• How the new transactional API works
• How you can use the new API in your application through case
studies demonstrating transactional design patterns
• How the new API interacts with DynamoDB features
5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why is this important?
The transactional API can be used to
implement real-world transactional scenarios
The transactional API can be used to simplify
the implementation of cloud application flows
6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prerequisites
You should be familiar with DynamoDB concepts such as tables,
items, partition keys and sort keys, indexes, Time To Live, and
streams. We will only briefly introduce these concepts in this
session before using them in the use cases.
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transactional API at a glance
TransactWriteItems
Transact up to 10 items
Evaluate conditions
If all conditions are simultaneously true, perform write operations
TransactGetItems
Transact up to 10 items
Return a consistent, isolated snapshot of all items
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Atomicity, consistency, isolation, and durability (ACID) in
DynamoDB
• Before: ACID support for single-item operations
• NEW: ACID support for multi-item operations
Nontransactional Transactional
New
TransactGetItems
TransactWriteItems
Existing
BatchGetItem, BatchWriteItem
Query, Scan
GetItem, PutItem, UpdateItem,
DeleteItem
10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transactions at DynamoDB scale
• Apply transactions to items in:
• Same partition key, or across partition keys
• Same table, or across tables
• Same Region only
• Same account only
• DynamoDB tables only
• Transactions scale like the rest of DynamoDB
• Limitless concurrent transactions
• Low latency, high availability
• Scales horizontally
• Your responsibility: Design to avoid hot key
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Path of success with NoSQL transactions
Design choices
• Transact any items in your account
• Service APIs vs. client-side library
• Request/response-based
• Limit of 10 items
How they help you
• Scale your data and relations within it
• Reduces complexity, and improves cost and
performance
• Keeps business logic in your application and
not in the database layer
• Consistent low latency, never deadlock
• Consistent low latency, high availability
• Addresses vast majority of use cases
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User profile management (use case #1)
• Problem statement
• Users identified by user name and optionally email
• No two users can share the same identifier (user name or email)
• Users can change their email
• What is demonstrated
• Enforcing unique constraints
• Strongly consistent global indexing
• Heterogeneous tables
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User profile management
• Solution
• Users table contains two kinds of records
• Profile record, keyed by opaque user name and containing user details
• Alias records, keyed by user email and referencing the profile record
Id UserNameRef Email Phone #
“John123” “john@example.com” “444-555-0110”
“john@example.com” “Joe123”
“jane456” “444-555-0199”
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Insert new profile
data = await dynamoDb.transactWriteItems({ TransactItems: [
{ Put: {
TableName: 'Users',
Item: { Id: { S: 'John123' }, Email: {S: 'john123@example.com'}, ...},
ConditionExpression: 'attribute_not_exists(Id)‘ }},
{ Put: {
TableName: 'Users',
Item: { Id: { S: 'john123@example.com' }, UserNameRef: {S: 'John123'}},
ConditionExpression: 'attribute_not_exists(Id)',}}
] }).promise();
16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Update email address
data = await dynamoDb.transactWriteItems({ TransactItems: [
{ Put: { TableName: 'Users',
Item: { Id: { S: 'John123' }, Email: {S: 'john123@example.org'}, ...},
ConditionExpression: 'attribute_exists(Id) and Email = :old_email',
ExpressionAttributeValues: {':old_email' :{'S': 'john123@example.com'}}}},
{ Put: { TableName: 'Users',
Item: { Id: { S: 'john123@example.org' }, UserNameRef: {S: 'John123'}},
ConditionExpression: 'attribute_not_exists(Id)', }},
{ Delete: { TableName: 'Users',
Item: { Id: { S: 'john123@example.com' }}}).promise();
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management (use case #2)
• Problem statement
• Guests create room reservations
• Guests check in to rooms, fulfilling a reservation
• Guests check out of rooms, closing a reservation
• What is demonstrated
• Ensuring idempotency – operations with side effects happen at most once
• Ensuring multi-item changes are only triggered when valid (consistency)
• Ensuring multi-item changes happen all together (atomicity)
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management data model
Guests
Id (primary key)
Name, Email, etc.
Reservations (set)
OccupiesRooms (set)
Reservations
Id (primary key)
GuestId
FulfilledByRoom
State
Rooms
Id (primary key)
RentedToReservation
State
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management data model
Record Type Id (PK) Attributes
Guest “John”
Reservations : { “500”, “501” }
OccupiesRooms : { “20014” }
Reservation “500”
GuestId: “John”
State: “PENDING”
Reservation “501”
GuestId: “John”
State: “FULFILLED”
FulfilledByRoom: “20014”
Room “20014”
State: “OCCUPIED”
RentedToReservation : “501”
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management: Create a reservation
• Idempotency – how to ensure only a single reservation is created
upon retries?
• Solution
• Client generates a unique reservation ID
• Client generates TransactWriteItems request with the following operations
• Conditional Put for the new reservation
• Update the customer reservations set: Add the new reservation
• Client transaction is now idempotent and can be retried without
double booking
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management: Check-in
Record Type Id (PK) Attributes
Customer “John123”
Reservations : { “567” }
OccupiesRooms : {}
Reservation “567”
CustomerId: “John123”
State: “PENDING”
FulfilledByRoom: null
Room “20014”
State: “FREE”
RentedToReservation : null
Before
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management: Check-in
Record Type Id (PK) Attributes
Customer “John123”
Reservations : { “567” }
OccupiesRooms : { “20014” }
Reservation “567”
CustomerId: “John123”
State: “FULFILLED”
FulfilledByRoom: “20014”
Room “20014”
State: “OCCUPIED”
RentedToReservation : “567”
After
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management: Checkout
Record Type Id (PK) Attributes
Customer “John123”
Reservations : { “567” }
OccupiesRooms : { “20014” }
Reservation “567”
CustomerId: “John123”
State: “FULFILLED”
FulfilledByRoom: “20014”
Room “20014”
State: “OCCUPIED”
RentedToReservation : “567”
Before
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hotel reservation management: Checkout
Record Type Id (PK) Attributes
Customer “John123”
Reservations : { “567” }
OccupiesRooms : { “20014” }
Reservation “567”
CustomerId: “John123”
State: “CLOSED”
FulfilledByRoom: “20014”
Room “20014”
State: “FREE”
RentedToReservation : “567”
After
25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Concurrency control
• Optimistic
• No deadlocks possible
• Low latency
• Optimized for scale-out
• Your responsibility
• Design for scale-out
• Avoid unnecessary conflicts
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Metering transactions
Transactions perform 2x the work of nontransactional
counterparts
Transactions are metered 2x their constituent operations
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why didn’t my transaction succeed?
• Per-item failure reasons
• Precondition failure
• Insufficient capacity
• Transactional conflicts
• Other reasons
• Transaction is still in progress
• Service error
• Malformed request
• Permissions
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Retrying transactions: ClientRequestToken
• Ensures SDK can retry without “replaying” the same intent
multiple times
• ClientRequestToken – idempotency token at the SDK or API level
• Autogenerated, you can override
• Applies for 10 minutes
• Provides information about transaction state and outcome
• Sometimes doesn’t remove the need for application-level idempotency or
intent design
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application-level retries
• When the SDK gives up
• Build a new view of the world
• Retry with new preconditions and desirable end state
• How to build a snapshot of the world? Three options:
• Set ReturnValuesOnConditionCheckFailure = ALL_OLD
• Perform TransactGetItems
• Work from an eventually consistent view (results of queries, etc.)
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Integrating with other DynamoDB features
• General rule about DynamoDB data
• Committed state only
• But might not be isolated
• Examples
• Streams – Committed writes, sharded
• Backups and point-in-time recovery (PITR) – Subset of a committed transaction is
possible
• Global secondary index (GSI) – Committed writes are indexed individually
• Global tables – Transactional API disabled by default; writes are replicated
individually
• Amazon DynamoDB Accelerator (DAX) support is on the roadmap
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Access control
• Individual operations are authorized separately
• (NEW) dynamodb:ConditionCheck
• dynamodb:UpdateItem
• dynamodb:PutItem
• dynamodb:DeleteItem
• No separate permissions for top-level new APIs
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attachment management (use case #3)
• Problem statement
• A social media site allows attaching media to posts
• Posts need to share media efficiently
• Posts can have multiple attachments
• Unreferenced media needs to be deleted
• What is demonstrated
• External resource metadata and lifetime management
• Using Time To Live (TTL) and streams
• Reference counting using transactions
35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attachment management data model
Attachment item
S3Ref (PK) – Reference to an object in Amazon S3
RefCount – Number of posts referencing this attachment
TTL – Time when this attachment is eligible to be deleted
Post item
Id (PK) – Post ID
Attachments (set) – Set of attachment IDs
36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attachment management invariants
1. RefCount = number of posts referencing the attachment
2. If RefCount > 0, then S3Ref refers to a valid Amazon S3 object
3. TTL is set if, and only if, RefCount = 0
37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attachment management: Create attachment and link it
to a post
• Generate unique Amazon S3 object name
• Create attachment record in DynamoDB
• Set RefCount = 0,
• Set TTL = 24 hours from now
• Upload object to Amazon S3
• In a TransactWriteItems Request
• Check that the attachment is not already referenced by the post
• Increase RefCount by 1, remove TTL
• Add attachment reference to post
38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attachment management: Clone attachment reference
• Copy the reference of an attachment from one post to another
• In a TransactWriteItems Request:
• Check that the attachment is not already referenced by the new post
• Increase RefCount by 1, remove TTL
• Add attachment reference to new post
39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Attachment management: Delete unused external
resources
• Attachment records will be deleted a day after their reference
count drops to zero
• In AWS Lambda, process stream records representing items
deleted by TTL
• Delete the corresponding objects from Amazon S3
40. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Yossi Levanoni
Contact us through the AWS forums or
@dynamodb on Twitter
Get the Database Freedom T-shirt in our
swag booth!
41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.