Success in free-to-play gaming requires knowing what your players love most. The faster you can respond to players' behavior, the better your chances of success. Learn how mobile game company GREE, with over 150 million users worldwide, built a real-time analytics pipeline for their games using Amazon Kinesis, Amazon Redshift, and Amazon DynamoDB. They walk through their analytics architecture, the choices they made, the challenges they overcame, and the benefits they gained. Also hear how GREE migrated to the new system while keeping their games running and collecting metrics.
2. Talk Outline
•Mobile Game Analytics –use case
•Decisions, Mistakes & Challenges
•Deep dive –Analytics Platform using AWS Tech
•Lessons we learned
3. GREE Headquarters
Tokyo, Japan
GREE International, Inc.
San Francisco, CA
GREE Canada
Vancouver, BC
QUICK FACTS
6
Continents playing GREE games
1,882
Employees Worldwide
13
Games made in North America
2004
2011
2013
MILESTONES
GAME STATS -4 titles in top 100 grossing*
Crime City (Studios)
Reached Top 10 Grossing in 140 countries
Top 100 Grossing in 19 countries, over 3 years since launch
*As of Sep. 2014 –Source: App Annie
A Global Gaming Powerhouse
Knights & Dragons (Publishing)
Reached Top 10 Grossing in 41 countries
Top 100 Grossing in 22 countries
4. Success Factors in Mobile Gaming
•Great gameplay & mechanics
•Great content
•Effective engagement & retention
•Generate in-app purchases
….. keep adding new content, features
?
?
?
I know its a great game.
Why is my game not successful anymore?
It had good KPIs initially.
5. As a game developer ...
Why?
•Game not performing well
•Players not spending?
•Retention so poor?
How?
•Optimize game design
•Improve ARPDAU, %Spenders
•Improve Retention
You need –Game Analytics & Insights
6. Analytics @ GREE
Ad Clicks
Downloads
Perf Data
Attribution
Campaign Performance
SC Balance
HC Balance
IAP
Player Targeting
7. Data Collection
•Mobile Devices
•Game Servers
•Ad Networks
•Size of event ~ 1 KB
•500M+ events/day
•500G+/day & growing
•JSON format
Source of Data
Data Size & Growth
8. Database Schema
•Every game –database schema
•Each game event = table (e.g., battle_fight, iap)
•40-50 tables per DB schema
•All game titles ~ 1000 tables in DW
9. Key Requirements
•Data collection & streaming to database
•Zero data loss
•Zero data corruption
•Guaranteed data delivery
11. Gen 1 – Analytics Platform
Analytics DB
Game DB
LAMP
Built on a LAMP Stack
Sharded DBs
Not scalable
Game
Servers
12. Gen2 – Flume/MPP Data
Warehouse
Game DB
Flume MPP Data Warehouse
Collectors
Flume
Master
Consumer
Game
Servers
13. In-house: Relay Engine
Game DB
Replicator
Amazon S3
DW
Relay Pipeline
Senders
Copiers
Game
Servers
Listeners
Cost of maintenance - HIGH
14. Challenges
•Hard to maintain and scale
•Spike in Live Ops events can clog other events
•Difficult to add new sink
•Writes to DW impacted query performance for BI users
•Poor data latency
15. Key Requirements –the list grew
•Data collection & streaming to database
•Zero data loss
•Zero data corruption
•Guaranteed data delivery
•Near real-time data latency
•Real-time ad-hoc analysis
•Ease of adding consumers
•Managed Service
21. Design Choices for Sender
•Single stream VS stream per game
•Batch VS Single Event
•Compressed VS Uncompressed
•PartitionKeyVS ExplicitHashKey
22. Sender Deployment
Elastic Load
Balancing
AMI
Amazon
EC2
Auto Scaling Group
Amazon
EBS Data
Volume
Pending: Wait
EC2 EBS Data
Volume
Amazon
S3
Sender
Scale
Out
Pending: Proceed
EC2 EBS Data
Volume
EC2
Auto Scaling Group
EBS Data
Volume
InService
Update
Terminating: Wait
EC2 EBS Data
Volume
Scale
Checks In
Terminating: Proceed
EC2 EBS Data
Volume
Terminated
24. Consumer –Amazon S3 Store
Kinesis Stream
Shard 1
Shard 2
Shard n
S3
File Metadata DB
Decompress
De-Dupe
Buffer
Transformation
Validation
Target Table
Compress
Size/
Timeout
Record
Consumer
Kinesis Client Library
Record Processor
Record Processor
Consumer
Kinesis Client Library
Record Processor
Auto Scaling Group
36. Lessons Learned
Sender
•Decouple data generation from sending
•Batch and compress
•PutRecordHTTP:5XX can result in duplicates
•Monitor ProvisionedThroughputExceededexception
37. Lessons Learned (Cont.)
Consumer
•Use KCL
•Auto-scale and monitor load
Overall
•Provision enough shards
•Handle shutdown gracefully
•Follow AWS best practices for error retries and exponential back-off
39. Takeaway
Kinesis
•Data available for processing within seconds
•Robust API, KCL, and Connector libraries
AWS
•Managed
•Scalable
•Cost effective
•Quick to get up and running
40. Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals