Cloud gaming is seen as a major driver for future gaming business. However, cloud gaming is also a big challenge regarding the technical aspects. Researchers have worked on the area in the recent years. This presentation provides a tour on the research activities in the area. We make a focus on network latency aspects. We provide all along the presentation some research challenges.
3. About gaming
59% of Americans play video games
39% of gamers are 36+ years old
48% of gamers are female
2 / 36 Gwendal Simon Cloud Gaming
4. About gaming
59% of Americans play video games
39% of gamers are 36+ years old
48% of gamers are female
The global revenue will hit $100 billion by 2018
More than Hollywood movie industry
Gaming is 20% of iPhone downloaded apps
2 / 36 Gwendal Simon Cloud Gaming
5. About gaming
59% of Americans play video games
39% of gamers are 36+ years old
48% of gamers are female
The global revenue will hit $100 billion by 2018
More than Hollywood movie industry
Gaming is 20% of iPhone downloaded apps
147.000 jobs are related to gaming in 2014 in US
2 / 36 Gwendal Simon Cloud Gaming
7. Gaming modular architecture
Gamer Command
Interface
Game Engine
Graphical Output
Game
Server
Console/PC market based on power-
ful hardware. Target hardcore gamers
PC + software
3 / 36 Gwendal Simon Cloud Gaming
8. Gaming modular architecture
Gamer Command
Interface
Game Engine
Graphical Output
Game
Server
Console/PC market based on power-
ful hardware. Target hardcore gamers
Smartphone market based on app
store. Target casual gamers
smartphone + app
3 / 36 Gwendal Simon Cloud Gaming
9. Gaming modular architecture
Gamer Command
Interface
Game Engine
Graphical Output
Console/PC market based on power-
ful hardware. Target hardcore gamers
Smartphone market based on app
store. Target casual gamers
Cloud gaming based on the cloud and
lightweight devices. Target everybody
cloud
home
3 / 36 Gwendal Simon Cloud Gaming
12. Cloud gaming architecture in a nutshell
Data-Center
Internet
user actions
video stream
4 / 36 Gwendal Simon Cloud Gaming
13. Why does it make sense ?
For game developers :
Better piracy control
5 / 36 Gwendal Simon Cloud Gaming
14. Why does it make sense ?
For game developers :
Better piracy control
Support more client platforms
5 / 36 Gwendal Simon Cloud Gaming
15. Why does it make sense ?
For game developers :
Better piracy control
Support more client platforms
Easier to upgrade
5 / 36 Gwendal Simon Cloud Gaming
16. Why does it make sense ?
For game developers :
Better piracy control
Support more client platforms
Easier to upgrade
New business offers (e.g. subscription-based)
5 / 36 Gwendal Simon Cloud Gaming
17. Why does it make sense ?
For game developers :
Better piracy control
Support more client platforms
Easier to upgrade
New business offers (e.g. subscription-based)
For gamers :
No more hardware
5 / 36 Gwendal Simon Cloud Gaming
18. Why does it make sense ?
For game developers :
Better piracy control
Support more client platforms
Easier to upgrade
New business offers (e.g. subscription-based)
For gamers :
No more hardware
Play games on thin clients
5 / 36 Gwendal Simon Cloud Gaming
19. Why does it make sense ?
For game developers :
Better piracy control
Support more client platforms
Easier to upgrade
New business offers (e.g. subscription-based)
For gamers :
No more hardware
Play games on thin clients
Follow-me gaming
5 / 36 Gwendal Simon Cloud Gaming
20. Business status
The pioneers have shut down :
OnLive bankrupted in 2013. Too early
StreamMyGame never took off.
6 / 36 Gwendal Simon Cloud Gaming
21. Business status
The pioneers have shut down :
OnLive bankrupted in 2013. Too early
StreamMyGame never took off.
Last game consoles integrate cloud gaming solutions :
Sony acquired Gaikai (PS3 backward compatibility)
Microsoft launched XBox Live Cloud a year later
6 / 36 Gwendal Simon Cloud Gaming
22. Business status
The pioneers have shut down :
OnLive bankrupted in 2013. Too early
StreamMyGame never took off.
Last game consoles integrate cloud gaming solutions :
Sony acquired Gaikai (PS3 backward compatibility)
Microsoft launched XBox Live Cloud a year later
New actors enter the market :
NVidia released Nvidia Grid and Shield
6 / 36 Gwendal Simon Cloud Gaming
24. The main reasons for latency
Response time is the delay from an
action done at the user and the result of
this action on the display
8 / 36 Gwendal Simon Cloud Gaming
25. The main reasons for latency
gateway
t1
8 / 36 Gwendal Simon Cloud Gaming
26. The main reasons for latency
gateway
t1
Internet
t2
8 / 36 Gwendal Simon Cloud Gaming
27. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3
8 / 36 Gwendal Simon Cloud Gaming
28. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
8 / 36 Gwendal Simon Cloud Gaming
29. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
8 / 36 Gwendal Simon Cloud Gaming
30. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
8 / 36 Gwendal Simon Cloud Gaming
31. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
8 / 36 Gwendal Simon Cloud Gaming
32. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
network
delivery
t1 + t2 + t3
8 / 36 Gwendal Simon Cloud Gaming
33. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
network
delivery
t1 + t2 + t3video
decoding
t8
8 / 36 Gwendal Simon Cloud Gaming
34. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
network
delivery
t1 + t2 + t3video
decoding
t8
latency due to network
8 / 36 Gwendal Simon Cloud Gaming
35. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
network
delivery
t1 + t2 + t3video
decoding
t8
latency due to network
latency due to game engine
8 / 36 Gwendal Simon Cloud Gaming
36. The main reasons for latency
gateway
t1
Internet
t2
DC router
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
network
delivery
t1 + t2 + t3video
decoding
t8
latency due to network
latency due to game engine
latency due to video encoding and decoding
8 / 36 Gwendal Simon Cloud Gaming
46. Genre, latency, and performance
The Claypool law of latency acceptance :
genre first-person third-person omnipresent
lag in ms 100 500 1000
11 / 36 Gwendal Simon Cloud Gaming
47. Genre, latency, and performance
The Claypool law of latency acceptance :
genre first-person third-person omnipresent
lag in ms 100 500 1000
Various other factors to consider :
Limitations in the classification (e.g. StarCraft)
User control games (more or less immersive)
Gamer skill level (from casual to professional)
11 / 36 Gwendal Simon Cloud Gaming
51. Latency target
The shorter the better
The best range is from 100 to 150 ms
13 / 36 Gwendal Simon Cloud Gaming
52. Latency target
The shorter the better
The best range is from 100 to 150 ms
Up to 300 ms is enough for some games
13 / 36 Gwendal Simon Cloud Gaming
53. Latency target
The shorter the better
The best range is from 100 to 150 ms
Up to 300 ms is enough for some games
Action prioritization may be standardized
13 / 36 Gwendal Simon Cloud Gaming
55. Video
Gamers have high expectations :
High-resolution (1080p possible since PS3)
High frame rate (60 fps is common)
Complex, dynamic scenes
15 / 36 Gwendal Simon Cloud Gaming
56. Video
Gamers have high expectations :
High-resolution (1080p possible since PS3)
High frame rate (60 fps is common)
Complex, dynamic scenes
Screencasting technologies are booming :
Capture, encode, and packetize
e.g., Open Broadcast Software
15 / 36 Gwendal Simon Cloud Gaming
58. Video decoding/encoding
In the Rhizome platform :
encoding capture im. convert im. encode frame stream frame total
720p 6.4ms 4.4ms 2.8ms 0.2ms 13.8ms
1080p 11.0ms 4.9ms 3.0ms 0.3ms 19.2ms
17 / 36 Gwendal Simon Cloud Gaming
59. Video decoding/encoding
In the Rhizome platform :
encoding capture im. convert im. encode frame stream frame total
720p 6.4ms 4.4ms 2.8ms 0.2ms 13.8ms
1080p 11.0ms 4.9ms 3.0ms 0.3ms 19.2ms
Use info from the game engine to encode video
Reduce Motion Estimation (ME) using
Game object information
Depth map
17 / 36 Gwendal Simon Cloud Gaming
60. Video decoding/encoding
In the Rhizome platform :
encoding capture im. convert im. encode frame stream frame total
720p 6.4ms 4.4ms 2.8ms 0.2ms 13.8ms
1080p 11.0ms 4.9ms 3.0ms 0.3ms 19.2ms
Use info from the game engine to encode video
Reduce Motion Estimation (ME) using
Game object information
Depth map
Reduce encoding complexity using
Object priority (adaptive object encoding)
17 / 36 Gwendal Simon Cloud Gaming
62. Virtualization on GPU
Cloud management is based on Virtual Machines
Mature technologies to create, shut down, migrate
Hardware consolidation : several VMs per machine
19 / 36 Gwendal Simon Cloud Gaming
63. Virtualization on GPU
Cloud management is based on Virtual Machines
Mature technologies to create, shut down, migrate
Hardware consolidation : several VMs per machine
Poor performances on traditional GPU
Degrades memory transfer (GPU bottleneck)
Prevents Single-Instruction-Multiple-Data
19 / 36 Gwendal Simon Cloud Gaming
64. Virtualization on GPU
Cloud management is based on Virtual Machines
Mature technologies to create, shut down, migrate
Hardware consolidation : several VMs per machine
Poor performances on traditional GPU
Degrades memory transfer (GPU bottleneck)
Prevents Single-Instruction-Multiple-Data
Pass-through GPU enables better performances :
One-to-one mapping between device and VM
New technologies coming (e.g., NVidia Grid)
19 / 36 Gwendal Simon Cloud Gaming
67. Game virtualization : consolidation
nb. of instances Bare-metal (in fps) VM (in fps)
1 30.2 29.2
2 30.0 28.9
3 29.8 28.9
21 / 36 Gwendal Simon Cloud Gaming
68. Game virtualization : consolidation
nb. of instances Bare-metal (in fps) VM (in fps)
1 30.2 29.2
2 30.0 28.9
3 29.8 28.9
1 2 3
0
200
400
600
800
1,000
287
475
662
296
485
704
Nb. of game instances
Watts
Bare-metal
Virtual machines
21 / 36 Gwendal Simon Cloud Gaming
70. Latency : summary
Incompressible latency (without network) :
Video encoding and decoding : 40 ms
22 / 36 Gwendal Simon Cloud Gaming
71. Latency : summary
Incompressible latency (without network) :
Video encoding and decoding : 40 ms
Game engine : from 15 to 50 ms
22 / 36 Gwendal Simon Cloud Gaming
72. Latency : summary
Incompressible latency (without network) :
Video encoding and decoding : 40 ms
Game engine : from 15 to 50 ms
A lot of works for key latency gains here and there :
22 / 36 Gwendal Simon Cloud Gaming
73. Latency : summary
Incompressible latency (without network) :
Video encoding and decoding : 40 ms
Game engine : from 15 to 50 ms
A lot of works for key latency gains here and there :
Adaptive streaming in both resolution and fps
Prioritize actions
Provide metadata and information on game objects
Shared processes for multiple game instances
22 / 36 Gwendal Simon Cloud Gaming
74. To contribute to the research effort
Some open platforms :
GamingAnywhere, the most mature
23 / 36 Gwendal Simon Cloud Gaming
75. To contribute to the research effort
Some open platforms :
GamingAnywhere, the most mature
Uniquitous based on Unity 3D
23 / 36 Gwendal Simon Cloud Gaming
76. To contribute to the research effort
Some open platforms :
GamingAnywhere, the most mature
Uniquitous based on Unity 3D
Rhizome with integration of NVidia Grid
23 / 36 Gwendal Simon Cloud Gaming
78. Our objectives
Can the current cloud support cloud gaming ?
25 / 36 Gwendal Simon Cloud Gaming
79. Our objectives
Can the current cloud support cloud gaming ?
Which percentage of users can play cloud games ?
25 / 36 Gwendal Simon Cloud Gaming
80. Our objectives
Can the current cloud support cloud gaming ?
Which percentage of users can play cloud games ?
1. In excellent conditions (40 ms network latency)
25 / 36 Gwendal Simon Cloud Gaming
81. Our objectives
Can the current cloud support cloud gaming ?
Which percentage of users can play cloud games ?
1. In excellent conditions (40 ms network latency)
2. In good conditions (80 ms network latency)
25 / 36 Gwendal Simon Cloud Gaming
82. Our population sample
A modified BitTorrent crawler in Dec. 2012 :
Collected 2,504 IP addresses in US
Most likely machines for home entertainment
26 / 36 Gwendal Simon Cloud Gaming
83. Our population sample
A modified BitTorrent crawler in Dec. 2012 :
Collected 2,504 IP addresses in US
Most likely machines for home entertainment
26 / 36 Gwendal Simon Cloud Gaming
84. Latency to EC2 : experiment
Amazon datacenters (Portland, Virginia, North Calif.)
27 / 36 Gwendal Simon Cloud Gaming
85. Latency to EC2 : experiment
Amazon datacenters (Portland, Virginia, North Calif.)
For each client (IP address)
1. Measure latency from each DC
27 / 36 Gwendal Simon Cloud Gaming
86. Latency to EC2 : experiment
Amazon datacenters (Portland, Virginia, North Calif.)
For each client (IP address)
1. Measure latency from each DC
2. Pick the shortest latency
27 / 36 Gwendal Simon Cloud Gaming
87. Latency to EC2 : experiment
Amazon datacenters (Portland, Virginia, North Calif.)
For each client (IP address)
1. Measure latency from each DC
2. Pick the shortest latency
3. Do it ten times
27 / 36 Gwendal Simon Cloud Gaming
88. Latency to EC2 : results
0 20 40 60 80 100 120 140 160
0
0.2
0.4
0.6
0.8
1
median latency (in ms)
ratioofcoveredusers
28 / 36 Gwendal Simon Cloud Gaming
89. Latency to EC2 : results
0 20 40 60 80 100 120 140 160
0
0.2
0.4
0.6
0.8
1
median latency (in ms)
ratioofcoveredusers
28 / 36 Gwendal Simon Cloud Gaming
90. Latency to EC2 : results
0 20 40 60 80 100 120 140 160
0
0.2
0.4
0.6
0.8
1
two thirds of gamers in
good conditions
one third of gamers in
excellent conditions
one tenth of gamers
cannot play at all
median latency (in ms)
ratioofcoveredusers
28 / 36 Gwendal Simon Cloud Gaming
91. What if more DCs are used
44 PlanetLabs nodes (all over the US)
29 / 36 Gwendal Simon Cloud Gaming
92. What if more DCs are used
44 PlanetLabs nodes (all over the US)
For each client (IP address) :
1. Measure latency from each PlanetLabs DC
29 / 36 Gwendal Simon Cloud Gaming
93. What if more DCs are used
44 PlanetLabs nodes (all over the US)
For each client (IP address) :
1. Measure latency from each PlanetLabs DC
2. Do it ten times
29 / 36 Gwendal Simon Cloud Gaming
94. What if more DCs are used
44 PlanetLabs nodes (all over the US)
For each client (IP address) :
1. Measure latency from each PlanetLabs DC
2. Do it ten times
For each k-subset of DCs :
29 / 36 Gwendal Simon Cloud Gaming
95. What if more DCs are used
44 PlanetLabs nodes (all over the US)
For each client (IP address) :
1. Measure latency from each PlanetLabs DC
2. Do it ten times
For each k-subset of DCs :
1. Choose for each client the shortest latency
2. Sum all shortest latencies
29 / 36 Gwendal Simon Cloud Gaming
96. What if more DCs are used
44 PlanetLabs nodes (all over the US)
For each client (IP address) :
1. Measure latency from each PlanetLabs DC
2. Do it ten times
For each k-subset of DCs :
1. Choose for each client the shortest latency
2. Sum all shortest latencies
Then pick the best k-subset of DCs
29 / 36 Gwendal Simon Cloud Gaming
97. More DCs : results
5 10 15 20 25
0
0.2
0.4
0.6
0.8
1
EC2 80 ms
EC2 40 ms
number of datacenters
ratioofcoveredusers
30 / 36 Gwendal Simon Cloud Gaming
98. More DCs : results
5 10 15 20 25
0
0.2
0.4
0.6
0.8
1
EC2 80 ms
EC2 40 ms
number of datacenters
ratioofcoveredusers
80 ms latency threshold
40 ms latency threshold
30 / 36 Gwendal Simon Cloud Gaming
99. Toward a hybrid DC-P2P infrastructure
Out of the 2,504 clients :
1,000 are edge servers
1,504 are clients
31 / 36 Gwendal Simon Cloud Gaming
100. Toward a hybrid DC-P2P infrastructure
Out of the 2,504 clients :
1,000 are edge servers
1,504 are clients
For each client :
1. Measure latency to each Amazon DC
31 / 36 Gwendal Simon Cloud Gaming
101. Toward a hybrid DC-P2P infrastructure
Out of the 2,504 clients :
1,000 are edge servers
1,504 are clients
For each client :
1. Measure latency to each Amazon DC
2. Measure latency to all edge-servers
31 / 36 Gwendal Simon Cloud Gaming
102. Toward a hybrid DC-P2P infrastructure
Out of the 2,504 clients :
1,000 are edge servers
1,504 are clients
For each client :
1. Measure latency to each Amazon DC
2. Measure latency to all edge-servers
3. Pick the shortest latency
31 / 36 Gwendal Simon Cloud Gaming
103. Toward a hybrid DC-P2P infrastructure
Out of the 2,504 clients :
1,000 are edge servers
1,504 are clients
For each client :
1. Measure latency to each Amazon DC
2. Measure latency to all edge-servers
3. Pick the shortest latency
4. Do it ten times
31 / 36 Gwendal Simon Cloud Gaming
107. Network latency : summary
Not all users will be served
33 / 36 Gwendal Simon Cloud Gaming
108. Network latency : summary
Not all users will be served
100 ms overall lag needs a lot of servers at the edges
33 / 36 Gwendal Simon Cloud Gaming
109. Network latency : summary
Not all users will be served
100 ms overall lag needs a lot of servers at the edges
Game placement on servers will be key
33 / 36 Gwendal Simon Cloud Gaming
111. Main researchers in the area
Kuan-Ta Chen (Sinica - Taiwan)
35 / 36 Gwendal Simon Cloud Gaming
112. Main researchers in the area
Kuan-Ta Chen (Sinica - Taiwan)
Shervin Shirmohammadi (Ottawa - Canada)
35 / 36 Gwendal Simon Cloud Gaming
113. Main researchers in the area
Kuan-Ta Chen (Sinica - Taiwan)
Shervin Shirmohammadi (Ottawa - Canada)
Mark Claypool (Worcester - UK)
35 / 36 Gwendal Simon Cloud Gaming
114. Main researchers in the area
Kuan-Ta Chen (Sinica - Taiwan)
Shervin Shirmohammadi (Ottawa - Canada)
Mark Claypool (Worcester - UK)
Jiangchuan Liu (Simon Fraser - Canada)
35 / 36 Gwendal Simon Cloud Gaming
115. Main researchers in the area
Kuan-Ta Chen (Sinica - Taiwan)
Shervin Shirmohammadi (Ottawa - Canada)
Mark Claypool (Worcester - UK)
Jiangchuan Liu (Simon Fraser - Canada)
Carsten Griwodz (Simula - Norway)
35 / 36 Gwendal Simon Cloud Gaming
117. Takeaway
Business attractive, but techno challenging
Many research areas to explore :
Delivery network
36 / 36 Gwendal Simon Cloud Gaming
118. Takeaway
Business attractive, but techno challenging
Many research areas to explore :
Delivery network
Game virtualization
36 / 36 Gwendal Simon Cloud Gaming
119. Takeaway
Business attractive, but techno challenging
Many research areas to explore :
Delivery network
Game virtualization
Priority actions and adaptive encoding
36 / 36 Gwendal Simon Cloud Gaming
120. Takeaway
Business attractive, but techno challenging
Many research areas to explore :
Delivery network
Game virtualization
Priority actions and adaptive encoding
... and everything to chase the ms everywhere
36 / 36 Gwendal Simon Cloud Gaming
121. Takeaway
Business attractive, but techno challenging
Many research areas to explore :
Delivery network
Game virtualization
Priority actions and adaptive encoding
... and everything to chase the ms everywhere
And what about multi-player games ?
36 / 36 Gwendal Simon Cloud Gaming