In this session, you will learn how to designed clickstream analytics application and how you can use the same architecture to build your own and be ready to handle the changing world of clickstream data. Dive into how to perform advanced user retention and cohort analysis to make near–real time product and marketing decisions. Learn how to build infrastructure that is fast, easy, and cost-effective with AWS resources such as Amazon Kinesis, Spark on Amazon EMR, Amazon S3, Amazon Redshift, and Amazon Elasticsearch.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dickson Yue, Solutons Architect
20 May
數位行銷大數據顯學
產業議程 - 電子商務、數位媒體與行銷

2. CRM ERP
DBs
Log file
AWStats
days
MB
2002
Big bang

3. <2005
Hello world
Page/Event
tracking
GA
hours
GB

4. SOLOMO
minutes - hours
TB
<2008
New customer service
New System monitoring
New QA

5. IoT
O2O
seconds – hours
PB
2016
Fast and big
data driven marketing

6. Insights

7. NEO

8. seconds – hours, future
Past
Retrospective report
Present
Dashboard, alert
Future
Prediction, insight

9. How to get started?

10. Collect Process Analyze
Store
Data Answers
Time to Answer (Latency)
Throughput
Cost
Batch Real-time Prediction
Report Alert Future

11. Amazon S3
Amazon Kinesis
Amazon DynamoDB
Amazon RDS (Aurora)
AWS Lambda
KCL Apps
Amazon
EMR
Amazon
Redshift
Amazon Machine
Learning
Collect Process Analyze
Store
Data Collection
and Storage
Data
Processing
Event
Processing
Data
Analysis
Data Answers

12. Tracking
Clickstream, user retention

13. Data source
• Page
• Click event
• Web log
• Thing event
Use case
Answer
• User retention
• High spending customer
navigation pattern
• User segmentation
• UX improvement
• What deal/ad to try
next

14. Requirement
Ingest
• Scalability
• Raw data
• Low running cost
Analyze
• Full visibility
• Data without sampling
• Data from device in any
form factor
• Flexibility
• Join with different
datasets

15. JavaScript
(Snowplow)
AWS SDK
logstach
Fluentd
Ingest Store
@ 30km/s a.k.a 300 rps
HTTP Post
Amazon
S3
Storage

16. @ 100km/s
Ingest Store Process
JavaScript
(Snowplow)
AWS SDK
LOG4J
Flume
Fluentd
HTTP Post
Amazon
Lambda
Amazon
S3
Amazon
Kinesis
API Server Streaming
Buffer
24hrs-7days
Compute Storage
Store
Web
Servers
50k rps

17. @ 100km/s
Ingest Store Process
JavaScript
(Snowplow)
AWS SDK
LOG4J
Flume
Fluentd
HTTP Post
Amazon
Lambda
Amazon
S3
Amazon
Kinesis
API Gateway
API Server Streaming
Buffer
24hrs-7days
Compute Storage
Store
50k rps

18. Amazon
S3
Storage
Store Process
EMR
Redshift
RedshiftEMR
ETL
Visualize
JDBC
ODBC
JDBC
ODBC

19. Many storage layers to choose from
Amazon DynamoDB
EMR-DynamoDB
connector
Amazon RDS
Amazon
Kinesis
Streaming data
connectorsJDBC Data Source
w/ Spark SQL
Elasticsearch
connector
Amazon Redshift
Amazon Redshift Copy
From HDFS
EMR File System
(EMRFS)
Amazon S3
Amazon EMR

20. Amazon S3
Store Process
EMR
Visualize
JDBC
ODBC
RedshiftBasket
CRM ERP
DBs
Log file

21. Day-14 retention over time
User retention and growth
N-day retention

22. User retention and growth
0
1000
2000
3000
4000
5000
6000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
DailyActiveUsers
Product Age (days)
Product A
Product B

23. High churn = wasted ad dollars
$-
$5,000.00
$10,000.00
$15,000.00
$20,000.00
$25,000.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Product age (days)
Product A
Product B

25. BUSINESS MEDIA
operates more than 20 business-to-businesses with significant holdings in the
automotive, electronic, medical and finance industries
MAGAZINES
publishes 20 U.S. titles and close to 300 international editions
BROADCASTING
comprises 31 television and two radio stations
NEWSPAPERS
owns 15 daily and 34 weekly newspapers
Hearst includes over 200 businesses in over
100 countries around the world

26. 1

27. Data Pipeline
Buzzing API
API
Ready
Data
Amazon
Kinesis
S3 Storage
Node.JS
App- Proxy
Users to
Hearst
Properties
Clickstream
Data Science
Application
Amazon Redshift
ETL on EMR
100 seconds
1G/day
30 seconds
5GB/day
5 seconds
1G/day
Milliseconds
100GB/day
LATENCY
THROUGHPUT Models
Agg Data

28. Social listening
Social CRM, Chatbots

29. Data
Brand page activity
Post #hashtag
User profile
Use case
Answer
Campaign performance
Customer service automation
Building Chatbot

30. Logstash
AWS SDK
Ingest Store
Bot AWS SDK
App
Crawlers
AWS SDK
Amazon
Kinesis
Process Store
Amazon S3
Amazon
Lambda
ElasticSearch
Analysts
AWS SDK

32. Motivation for listening to social media
Customer is reporting a possible service issue

33. Motivation for listening to social media
Customer is making a feature request

34. Motivation for listening to social media
Customer is angry or unhappy

35. Motivation for listening to social media
Customer is asking a question

36. Why do we need machine learning for this?
The social media stream is high-volume, and most of the
messages are not CS-actionable

37. Logstash
AWS SDK
Ingest Store
Bot AWS SDK
App
Crawlers
AWS SDK
Amazon
Kinesis
Process
Amazon
Lambda
Analysts
AWS SDK
Machine
learning
Notification
Action
Support
issue
Database
Feature
request
Keep training the ML model with new data
Action
Amazon S3

38. AWS SDK
Ingest Store
Bot AWS SDK
Messenger
Amazon
Kinesis
Process
Amazon
Lambda
Analysts
Machine
learning
Action
Bot
App
Get prediction
Keep training the ML model with new data
Amazon S3

39. OI from Business view
with custom source

40. Operational intelligence (OI)

41. What is the view of my storefront?
What is your campaign performance now?
index=bakery-storefront

43. Refrigerator
POS
Door sensor
Water
Camera
Storefront
Kitchen
Lambda
SQS
AWS IoT
SQSPoller
Http Event Collector
Serverless
Architecture

44. Sushiro – Real-time streaming from IoT & analysis

45. Sushiro – Real-time streaming & analysis
Real-time data ingested by Amazon Kinesis is analyzed in Amazon Redshift
380 stores stream live data from
Sushi plates
Inventory information combined
with consumption information
near real-time
Forecast demand by store,
minimize food waste, and
improve efficiencies
Amazon

46. Source DBs
3rd Party Data
Log Data
Reporting
Analysis
Processing
Data Lake
S3
Source of truth

48. Data
Shop sensors
Pos transactions
Use case
Answer
Cross sell up sell in real time