At the StampedeCon 2013 Big Data conference in St. Louis, Riot Games discussed Using Hadoop to Understand and Improve Player Experience. Riot Games aims to be the most player-focused game company in the world. To fulfill that mission, it’s vital we develop a deep, detailed understanding of players’ experiences. This is particularly challenging since our debut title, League of Legends, is one of the most played video games in the world, with more than 32 million active monthly players across the globe. In this presentation, we’ll discuss several use cases where we sought to understand and improve the player experience, the challenges we faced to solve those use cases, and the big data infrastructure that supports our capability to provide continued insight.
11. INITIAL LAUNCH / SCRAPPY START UP PHASE
‣ Had
a
single,
dedicated
MySQL
instance
for
the
DW
‣ Data
was
ETL’d
from
produc@on
slaves
into
this
instance
‣ Queries
were
run
in
MySQL
‣ Repor@ng
was
done
in
Excel
▾ All
ETLs,
queries
and
repor@ng
were
done
by
one
person
HISTORY
START-‐UP
THIS WORKED GREAT!
12. THEN – CRAZY GROWTH
HISTORY
START-‐UP
@me
#
unique
logins
TOTAL
ACTIVE
PLAYERS
June
2012
CRAZY
GROWTH
13. THE BREAKING POINT
HISTORY
START-‐UP
CRAZY
GROWTH
BREAKING
POINT
‣ Data
warehouse
reached
a
breaking
point
▾ 24
hours
of
data
took
24.5
hours
to
ETL
‣ We
couldn’t
handle…
▾ Mul@ple
environments
in
a
ver@cal
MySQL
instance
▾ A
single
environment
in
a
ver@cal
MySQL
instance
‣ We
needed
to
change
14. INTRODUCTION OF HADOOP
HISTORY
START-‐UP
CRAZY
GROWTH
BREAKING
POINT
‣ Hadoop
has
a
number
of
great
quali@es
▾ Cost
effec@ve
▾ Scalable
▾ Open
source
▾ We
could
execute
quickly
HADOOP
15. HIGH LEVEL ARCHITECTURE – JUNE 2012
Tableau
Hive
Data
Warehouse
Pentaho
+
Custom
ETL
+
Sqoop
MySQL
Pentaho
Analysts
EUROPE
Audit
Plat
LoL
KOREA
Audit
Plat
LoL
NORTH
AMERICA
Audit
Plat
LoL
Business
Analyst
16. BUT, THIS WASN’T GOOD ENOUGH
‣ The
@me
to
arrive
at
insight
was
too
long!
‣ Our
solu@on
required
too
much
data
team
involvement
▾ Schema
changes
▾ ETL
tweaks
▾ Hive
metadata
updates
‣ Hive
is
painful
for
ad-‐hoc
or
interac@ve
analysis
▾ Especially
for
non-‐technical
folks
17. GOALS
‣ Democra@ze
data
access
▾ Enable
Self-‐service
Data
Collec@on
and
Analysis
‣ Create
ac@onable
insights
‣ Increase
speed
to
insight
19. CLIENT FOOTPRINT
‣ Significant
por@on
of
our
soware
runs
directly
on
players’
machines
▾ High
performance
graphics
▾ Responsiveness
‣ There
is
logic
in
these
components
that's
ONLY
exercised
on
the
client-‐side
‣ Understanding
the
performance,
reliability
and
stability
of
these
features
is
paramount
to
improving
the
player
experience
24. CHALLENGE: THE GAME IS ALIVE
The
game
is
a
living,
breathing
service
that’s
always
in
mo@on
‣ New
champions
‣ New
items
‣ New
effects/par@cles
‣ Changes
in
environment
‣ Changes
in
design
and
design
balance
UPDATE
2-3WEEKS
26. CHALLENGE: PC VARIABILITY
‣ Hardware
and
OS
profiles
are
significantly
different
even
within
regions
▾ OS
and
patch
level
▾ CPU
▾ Memory
▾ Video
card
▾ Video
card
memory
▾ Drivers
29. IMPROVING THE PLAYER EXPERIENCE
‣ We
need
to
gather
informa@on
across
all
of
these
dimensions
in
order
to
UNDERSTAND
the
player
experience
‣ We
use
this
info
to:
▾ React
quickly
to
changes
▾ Op@mize
performance
▾ Op@mize
designs
▾ Improve
our
tes@ng
• Like
crea@ng
our
compa@bility
tes@ng
lab
40. HONU CLIENT SDK
Select
avg(f[‘pingAVG’])
from
game_client_stats
group
by
f[‘serverId’];
pingAvg
serverId
system
source
app
@mestamp
1234567890
99.123.456.78
game_client
220.9542
12.345.678.90
Intel64
…
GAME_CLIENT_STATS
41. EVENT COLLECTION
‣ Honu
collector
‣ Online
system
‣ High
availability
–
100%
up@me
‣ Horizontally
scalable
‣ Elas@c
‣ Fault
tolerant
‣ Neulix
OSS
Eureka
discovery
service
42. HONU COLLECTOR
‣ Collect
events
from
mul@ple
clients
(Thri/NIO)
‣ Save
all
events
to
one
compressed
file
locally
‣ Upload
that
file
every
XX
minutes
to
HDFS/S3
‣ Send
a
message
to
Queue/SQS
for
Demux
H
o
n
u
C
o
l
l
e
c
t
o
r
s
S
Q
S
S
3
50. STARTUP TIPS
TEAMS THAT USE SMART PINGS TO ALERT OTHER PLAYERS TO THREATS ARE MORE LIKELY TO WIN GAME
PLAYERS WHO FOLLOW THE SUMMONER'S CODE WIN 27% MORE GAMES
THE TRIBUNAL BANS PLAYERS FOR NEGATIVE BEHAVIOR SUCH AS VERBAL HARASSMENT
PLAYERS WHO COOPERATE WITH THEIR TEAM WIN 31% MORE GAMES
51. HOW WE SOLVED IT – EXTEND HONU
HONU
CLIENT
SDK
HONU
COLLECTORS
HONU
DEMUX
ORGANIZECOLLECTGENERATE
52. HONU TOOLS: DRADIS
‣ Hwp
based
data
collec@on
‣ Large
volume
of
data
from
untrusted
source
‣ C10K
‣ Nginx
+
Newy
‣ 4+
billion
API
calls/day
‣ Peak
100K+
calls/sec
54. HONU TOOLS: ECHO SERVICE
‣ Web
UI
to
easily
and
immediately
visualize
the
data
that
has
been
sent
to
Honu
collectors
‣ Self-‐service
end-‐to-‐end
pipeline
55. HONU TOOLS: ECHO SERVICE
‣ Web
UI
to
easily
and
immediately
visualize
the
data
that
has
been
sent
to
Honu
collectors
‣ Self-‐service
end-‐to-‐end
pipeline
56. HONU TOOLS: ECHO SERVICE
‣ Web
UI
to
easily
and
immediately
visualize
the
data
that
has
been
sent
to
Honu
collectors
‣ Self-‐service
end-‐to-‐end
pipeline
57. HONU TOOLS: METADATA SERVICE
‣ Data
discovery
‣ Schema
management
‣ Counter,
@me
67. SHAMELESS HIRING PLUG
Like most everybody else at this conference… we’re hiring!
PLAYER EXPERIENCE FIRST
CHALLENGE CONVENTION
FOCUS ON TALENT AND TEAM
TAKE PLAY SERIOUSLY
STAY HUNGRY, STAY HUMBLE
THE RIOT MANIFESTO