by Rich Cowper, Solutions Architect Manager & Itzik Paz, Solutions Architect, AWS Calling all serverless developers! Wild Rydes (www.wildrydes.com), the world’s leading unicorn transportation startup, needs your help! The company's rydesharing network of unicorns has grown to thousands, fulfilling hundreds of thousands of passenger rydes each day. Your mission is to collect, store, process, and analyze data to track the real-time location and health of our unicorns. In this workshop, learn how to build infrastructure to process data streams in real time using Amazon Kinesis. Build serverless applications using Amazon Kinesis Analytics to aggregate and summarize data, and use AWS Lambda to store aggregated data in Amazon DynamoDB. Finally, use Amazon Kinesis Firehose to build a data lake in Amazon S3, and use Amazon Athena to run ad-hoc queries against it. Requirements: laptop, text editor, AWS account, and AWS Command Line Interface (CLI) installed and configured.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building Serverless Real-time Data
Processing
W O R K S H O P
N o v e m b e r 3 0 , 2 0 1 7
I t z i k P a z , A W S S t a r t u p S o l u t i o n A r c h i t e c t
S R V 3 3 2

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHAT TO EXPECT
This workshop is an interactive exercise that builds infrastructure
to collect, process, and persist data without using servers.
Agenda
• Overview of workshop scenario, modules, and the services we’ll
utilize
• Review of workshop pre-requisites and tools
• Execution of four modules – each about thirty minutes in
length

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HELLO!
I’m Itzik. That’s Ramesh, Ashwini and Gareth.
We’re here to work with you to explore serverless data
processing. Please flag us down any time to help or if
there’s anything we can do.
Work through this workshop with your fellow participants!
Let’s take five minutes to go around the table and
introduce ourselves.

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WORKSHOP SCENARIO

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Wild Rydes is an innovative
transportation service that
helps people get to their
destination faster and hassle-
free via unicorns.
http://www.wildrydes.com

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHALLENGE
Your operations team needs a way to monitor the status
and health of thousands of unicorns in real-time.
Where are they?
How many magic points do they have left?
How fast are they traveling?

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MODULES

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MODULE 1: REAL-TIME STREAMING DATA
Shadowfax Kinesis stream Consumer
Build and demonstrate a stream
for real-time data from the
unicorn fleet
• Create a Kinesis stream
• Use the Kinesis command-line
producer to write simulated sensor
data to the stream every second
• Use the Kinesis command-line
consumer to read the sensor data
from the stream
• Use the Unicorn Dashboard to view
unicorns in real-time on a map

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MODULE 2: STREAMING AGGREGATION
Shadowfax Kinesis stream
Consumer
Create a serverless application to
summarize sensor data every
minute
• Create a Kinesis Analytics application
to output one row per minute per
unicorn to a new stream
• Use the Kinesis command-line
producer to write simulated sensor
data to the stream
• Use the Kinesis command-line
consumer to read the summarized
sensor data from the stream
Kinesis Analytics
application
Kinesis stream

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MODULE 3: STREAM PROCESSING
Shadowfax Kinesis stream
Build a Lambda function to
persist sensor data to a
DynamoDB table
• Create a new DynamoDB table
• Build a Lambda function and
configure it to trigger when data is in
the stream
• Use the Kinesis command-line
producer to write simulated sensor
data to the stream
• Use the DynamoDB console to verify
data is being persisted
Lambda
DynamoDB

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MODULE 4: DATA LAKE
Shadowfax Kinesis stream
Write raw sensor data to S3 via
Kinesis Firehose and run queries
using Athena
• Create an S3 bucket to store raw
sensor data from our unicorns in
JSON format
• Create a Kinesis Firehose delivery
stream to deliver batches from the
Kinesis stream to S3
• Use Amazon Athena to run queries
against the raw data stored in S3
Kinesis Firehose
S3 Bucket
Athena

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TOOLS AND PRE-REQUISITES

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WEB BROWSER
This workshop has been tested
in the latest versions of Mozilla
Firefox and Google Chrome.
Both browsers feature a
Developer Console which you
may need to use to see log
information from the Unicorn
Dashboard.

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB GUIDE
Includes setup
information, instructions
for each module, and
step-by-step instructions
if you get stuck.
Expand ▶ Step-by-step
Instructions for a
detailed walk through in
each section.

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
UNICORN DASHBOARD

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS ACCOUNT
You’ll need an AWS account and access to
administer resources for:
• AWS Identity and Access Management (IAM)
• Amazon Simple Storage Service (S3)
• Amazon DynamoDB
• AWS Lambda
• Amazon Kinesis Streams
• Amazon Kinesis Firehose
• Amazon Kinesis Analytics
• Amazon Athena
The workshop instructions assume only
one participant is using a given AWS
account at a time
It’s best to use a personal account or
register a new account rather than using
your organization’s AWS account
You will receive $25 of AWS credit to
cover any expenses incurred in this
workshop

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS REGION
• Build within a single Region that supports all necessary services
• The Region Table shows what services are supported within each Region
• https://aws.amazon.com/about-aws/global-infrastructure/regional-
product-services/ (search “aws region table”)
• Regions for this workshop:
• US East (N. Virginia)
• US West (Oregon)
• EU (Ireland)

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
KINESIS COMMAND-LINE CLIENTS
CLI client to simulate and display sensor data from the unicorn fleet
Producer
• Simulates unicorn sensor data
• Unicorn name
• Timestamp
• Current position (latitude, longitude)
• Distance traveled in the last second in meters
• Magic points
• Health points
• Data encoded as a JSON object
Consumer
• Reads from the end of the stream and outputs data

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
KINESIS COMMAND-LINE CLIENTS
Written in the Go Programming Language and provided as binaries for Windows, Linux, and macOS or build
from source yourself with the Go tools

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
http://reinvent2017.wildrydes.com

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
http://reinvent2017.wildrydes.com

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!