GDC 2015 - Low-latency Multiplayer Gaming with AWS

1
GDC 2015 – Amazon Developer Day
Deploying a Low-Latency Multiplayer Game Globally
Nate Wiger
Principal, Gaming Solutions
@nateware
nateware@amazon.com
James Gwertzman
CEO, PlayFab
@playfabnetwork
http://playfab.com

AWS Gaming Solutions
[Game] Online Problems Continue; Features
Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online
as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving
disconnected players stranded
[Game] has a great online mode – if you
can get online, that is
“This is total bull**** I paid for a preorder and
I can’t even connect WTF [Game Studio]??”

AWS Gaming Solutions http://bit.ly/verizon-latency
http://bit.ly/superdata-latency

100+ms 100+ms
100+ms
100+ms

How To Be Awesome
1.  Game servers near players
2.  Everything else as HTTP APIs
3.  Data replication = bad
4.  Local caches = good
5.  Nobody really “plays everywhere”

Loosely Coupled Pods
Tokyo
Oregon
Frankfurt
Virginia

VPC Subnet
VPC Subnet
Game API Pods
Availability Zone A
Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group
WEB
VPC Subnet
WEB
JOBS
Cognito
SNS Mobile
Push
SES

Region
①  Login via HTTP API
②  Download Game Assets
③  Matchmaking to Game Server
EC2
Game Flow
EC2
EC2

Region
④  Connect to Server
⑤  Hack Apart Your Friends
⑥  Game Over
Game Flow
EC2
EC2

Region
④  Connect to Server
⑤  Hack Apart Your Friends
⑥  Game Over
⑦  Write via HTTP API
Game Flow
EC2
EC2

VPC Private Subnet
VPC Public Subnet
Game Server Pods
Availability Zone A
Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME

Make. Network. Faster.
•  C3/C4 instance family (or R3 for more memory)
•  In a VPC
•  HVM AMI – not PV
•  Enable Enhanced Networking (SR-IOV)
http://bit.ly/ec2-enhanced-net
•  Linux: Tweak Networking Stack
http://bit.ly/linux-tuning

Enhanced Networking (SR-IOV)
Before:
Hypervisor
After:
Hardware

VPC Private Subnet
VPC Private Subnet
RabbitMQ + Elastic Load Balancing
Availability Zone A
Availability Zone B
10.1.0.13 10.2.0.16
TCP 5672
rabbitmq-node1 rabbitmq-node2
TCP 4369
& 25672

RabbitMQ + Elastic Load Balancing
•  Clustering Tutorial
https://www.rabbitmq.com/clustering.html
•  Set Queue HA policy to "all"
https://www.rabbitmq.com/ha.html
•  Create Internal load balancer
–  Listen Port: TCP 5672
•  VPC Security Group
–  For load balancer: TCP 5672
–  Between EC2 nodes: TCP 4369 & 25672
•  Set up Client to Heartbeat

/etc/rabbitmq/rabbitmq.config
5672
rabbitmq-node1
rabbitmq-node2

VPC Private Subnet
VPC Public Subnet
Redis Pub / Sub
Availability Zone A
Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
Auto Scaling group

VPC Private Subnet
VPC Public Subnet
CloudFormation + Chef
Availability Zone A
GAME GAME GAME
Auto Scaling group

Unreal Engine with CloudFormation
UnrealServer
unreal_linux.template.json
true
my-bucket-name
Unreal-123.zip
sshkeyname
1.2.3.4/24
MyPass123

Where My Servers At?
Tokyo
Oregon
Frankfurt
Virginia
?
?

VPC Subnet
Server Registration
Availability Zone A
Availability Zone B
VPC Subnet
Auto Scaling group
WEB WEB
Oregon
Tokyo
VPC Subnet
JOBS

Server Registration & Scaling
•  HTTPS POST /api/servers/register
•  Include an HMAC (RFC 2104)
•  Send Server Status
–  Public IP
–  # Players
–  Game Modes
•  Matchmaking Service
–  Maintains server list
–  Removes servers

VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
S3 Bucket
Amazon CloudWatch Logs
GAME GAME GAME

VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
Amazon 
Kinesis
S3 Bucket
Amazon Redshift
GAME GAME GAME

Mini Pods Everywhere
China
Oregon
Frankfurt
Virginia
China
Leaderboard

PlayFab: Loadout Architecture

What is PlayFab?
PlayFab is a complete live game operations platform.
Focus on the fun – we’ll do the boring bits.
+
Tools for operating &
monetizing live games
Complete set of cloud-based
backend services
Eco-system of pre-integrated
third-party services
.. and
more
+

Some Loadout Stats
•  Launched 1/31/14
•  Over 1 billion API Calls
•  At Peak (Feb 2014):
–  300k DAU
–  120 fixed game servers
–  110 EC2 game servers

PlayFab Core Architecture Tenants
•  No scheduled downtime.
•  Automatic scaling for load.
•  Backend changes should be
invisible to developers.
•  Use buffering to smooth over
spikes or failures.

AWS cloud: PlayFab Web Services
US
EU
Hardware
Game Servers
Hardware
Game Servers
Game Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2
(API handling)
Matchmaker
Instance
Instance
Game Server
Monitor
DynamoDB
Amazon RDS
Amazon S3
Elastic Load Balancing
Amazon Redshift
Reports
service
Instance
Logs
ArchitectureOverview
Matchmaker
(Secondary)
Instance
US
EU
SA
AUS
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers

List of PlayFab Web APIs
Client Server
Authentication Title-Wide Data Management Shared Group Data Authentication
AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket
LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management
LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo
LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification
LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management
LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard
LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser
LoginWithSteam GetUserInventory GetUserData
RegisterPlayFabUser RedeemCoupon GetUserInternalData
SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData
Account Management UnlockContainerItem GetUserStatistics
GetAccountInfo StartPurchase UpdateUserData
GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData
GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData
LinkFacebookAccount PurchaseItem UpdateUserStatistics
LinkGameCenterAccount Friend List Management Title-Wide Data Management
LinkSteamAccount AddFriend GetCatalogItems
UnlinkFacebookAccount GetFriendsList GetTitleData
UnlinkGameCenterAccount RemoveFriend SetTitleData
UnlinkSteamAccount SetFriendTags Player Item Management
UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency
UpdatePassword RegisterForIOSPushNotification GetUserInventory
UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers
Player Data Management Matchmaking APIs SubtractUserVirtualCurrency
GetFriendLeaderboard GetCurrentGames Matchmaking APIs
GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft
GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket
GetUserData StartGame Steam-Specific APIs
GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement
GetUserStatistics AndroidDevicePushNotificationRegistration
UpdateUserData ValidateGooglePlayPurchase
UpdateUserStatistics Analytics

Example of Web API Call
API Request: Response from Web Services:

ELB Logging for Title Tracking
Game Client
Amazon Route 53
(3.playfabapi.com)
API Request:
Elastic Load
Balancing
Logs
Amazon Redshift
Amazon S3  
bucket
Reports
service
Logs
Instance
Logs include title-specific
endpoint for analytics
AWS wildcard DNS
(*.playfabapi.com)
Log event:

Loadout API Usage Report (Oct 2014)
API NAME
TOTAL
CALLS
AVERAGE
BACKEND
TIME (S)
AVERAGE
RESPONSE
TIME (MS)
AVERAGE
REQUEST
TIME (MS)
TOTAL
REQUEST
TIME (M)
AVERAGE
RECIEVED
(BYTES)
AVERAGE
SENT
(BYTES)
TOTAL
ERRORS
adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979
gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290
matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319
matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93
matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400
matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107
gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0
gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2

Instance
Making a Singleton Highly Available
Health
Check
Auto Scaling group
Auto Scaling
service
Instance
Matchmaker
(Primary)
Instance
Amazon Route 53)
Elastic Load
Balancing
Matchmaker
(Secondary)
1. Auto Scaling is set to keep
two instances up.
2. Elastic Beanstalk Health
Check terminates bad
instance.
3. Amazon Route 53 fails
over all traffic to secondary
server.
4. Auto Scaling detects
failure and starts new
instance.

Monitoring Service Health
Amazon EC2
(API handling)
Elastic Load
Balancing
CloudWatch
Health
Check
Email Alerts
On-call Engineer
Mobile Alerts

Handling Spikes Without Losing Data
Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports
service
Local event
logs
Instance
Events:
logins and
purchases
Reports
service
Logs
Customer’s game
management dashboard
Analytics
Server
Logs
RabbitMQ
message
queue
Logs
Instance
Logstash
service
Logstash
Instance
ElasticSearch
Instance
Instance

Storage in DynamoDB, Amazon RDS, Amazon S3,
Amazon Redshift
DynamoDB
Amazon RDS
Amazon S3
Player data:
•  Accounts
•  Profile
•  Inventory
•  Save data
•  Player stats
•  Player currency
values
•  Auditing
•  Log files
•  Game replays
Amazon Redshift
•  Analytics data

Managing Tables in DynamoDB
Amazon
DynamoDB
•  Player data: accounts, profile, inventory, save data
User:
UserID: Hash
TitleID: range key
-  DisplayName
-  Linked accounts
-  …

Custom user data:
UserID: Hash
TitleID: range key
-  Key/value 1
-  Key/value 2
-  …

Inventory:
UserID: Hash
ItemID: range key
-  titleID
-  Catalog Item
-  …

…

Physical Deployment
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
N. California
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
Oregon
AZ 1
PlayFab Web
Services
Beijing
•  Original deployment: us-west-1
•  Expanded to us-west-2 in July
•  Coming soon: cn-north-1
Amazon EC2
Game Server
Singapore
Game Client
Instance

Managing Cross-Region Data
Player accounts are shared
across all regions:
•  Cross-regional wrapper for
user.get sets order of
databases and regions to
check
•  user.get then checks for
player account data
•  High availability, transparent
to customers and players
// Returns the first matching player record, searching DynamoDB and SimpleDB
// in both regions in the specified order.
// Updates the player record in the store and region from which it was loaded.
// Creates a player record in the default store and region.

AWS cloud: PlayFab Web Services
US
EU
Hardware
Game Servers
Hardware
Game ServersGame Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2
(API handling)
Matchmaker
Instance
Instance
Game Server
Monitor
DynamoDB
Amazon RDS
Amazon S3
Elastic Load Balancing
Amazon Redshift
Reports
service
Instance
Logs
ArchitectureOverview
Matchmaker
(Secondary)
Instance
US
EU
SA
AUS
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers

52
Nate Wiger
Principal, Gaming Solutions
@nateware
nateware@amazon.com

GDC 2015 - Low-latency Multiplayer Gaming with AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to GDC 2015 - Low-latency Multiplayer Gaming with AWS

Similar to GDC 2015 - Low-latency Multiplayer Gaming with AWS (20)

Recently uploaded

Recently uploaded (20)

GDC 2015 - Low-latency Multiplayer Gaming with AWS