Short talk about what Wooga learned by implementing and operating two social games at AWS using EC2 and S3.

1. ARCHITECTURE EVOLUTION
AT WOOGA
How does your company learn new things?
Jesper Richter-­‐Reichhelm, @jrirei

3. Our games all look the same
Flash client Backend

4. But the scale is interesFng
14 billion requests / month

5. But the scale is interesFng
14 billion requests / month

6. But the scale is interesFng
14 billion requests / month
>100,000 DB operaFons / second

7. But the scale is interesFng
14 billion requests / month
>100,000 DB operaFons / second
>50,000 DB updates / second

8. Architecture EvoluFon at Wooga
The Start
The Next Step
Best of Two Worlds

9. Oct 2009: Monster World based on Ruby
Good code quality
Easy to understand
Easy to test
Easy to refactor

10. Oct 2009: Monster World based on Ruby
Good code quality
Easy to understand
Easy to test
Easy to refactor

11. Oct 2009: Monster World based on Ruby
Good code quality
Easy to understand
Easy to test
Easy to refactor

12. A basic setup using sharding worked fine
lb
app app app
My My
SQL SQL
slave slave

13. A basic setup using sharding worked fine
lb
app app app app app
My My
SQL SQL
slave slave

14. A basic setup using sharding worked fine
lb
app app app app app app app app app
My My
SQL SQL
slave slave

15. 250K daily users
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
Life was good
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

16. 250K daily users
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
Life was good NO MORE
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

17. Early sharding hell: 8 master and 8 slaves
lb
app app app app app app app app app
app app app app app app app app app
My My
SQL SQL
slave slave

18. Early sharding hell: 8 master and 8 slaves
lb
app app app app app app app app app
app app app app app app app app app
My My My My
SQL SQL SQL SQL
slave slave slave slave

19. Early sharding hell: 8 master and 8 slaves
lb
app app app app app app app app app
app app app app app app app app app
My My My My My My My My
SQL SQL SQL SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave slave

20. At 500K daily users we were at a dead end
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

21. OUCH!
http://www.flickr.com/photos/billue_the_bear/

22. Big and staFc data in MySQL, rest goes to Redis
MySQL

23. Big and staFc data in MySQL, rest goes to Redis
MySQL Redis

24. Big and staFc data in MySQL, rest goes to Redis
MySQL Redis
256 GB data
10% writes

25. Big and staFc data in MySQL, rest goes to Redis
MySQL Redis
256 GB data 60 GB data
10% writes 50% writes

26. Redis saved the day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

27. Redis saved the day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

28. We now have more than 2 million users / day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

29. We now have more than 2 million users / day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
AWS service
#!!$!!!" disrupFon
in Ireland
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

30. 10 single-­‐points-­‐of-­‐failure -­‐ no fun at all!
lb lb
app app app app app app app app app app app app app
app app app app app app app app app app app app app
app app app app app app app app app app app app app
My My
redis redis
SQL SQL
slave slave slave slave

31. 10 single-­‐points-­‐of-­‐failure -­‐ no fun at all!
lb lb
app app app app app app app app app app app app app
app app app app app app app app app app app app app
app app app app app app app app app app app app app
My My My My My
redis redis
SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave

32. 10 single-­‐points-­‐of-­‐failure -­‐ no fun at all!
lb lb
app app app app app app app app app app app app app
app app app app app app app app app app app app app
app app app app app app app app app app app app app
My My My My My
redis redis redis redis redis
SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave slave slave slave

33. http://www.flickr.com/photos/wolfsavard/
OUCH!

34. Architecture EvoluFon at Wooga
The Start: Ruby + AutomaFon
The Next Step
Best of Two Worlds

35. Stateless servers and DBs
Server Database

36. Stateless servers and DBs
Server Database

37. Stateless servers and DBs
Server Database

38. Stateless servers and DBs
Server Database

39. Stateless servers and DBs
Server Database

40. Stateless servers and DBs
Server Database

41. “Stateless applica,on servers
guarantee one thing:

42. “Stateless applica,on servers
guarantee one thing:
The data is never
where you need it!”
Paolo Negri, Developer @ Wooga

43. Stateful servers and DBs
Server Database

44. Stateful servers and DBs
Server Database

45. Stateful servers and DBs
Server Database

46. Stateful servers and DBs
Server Database
One Game Session

47. Stateful servers and DBs
Server Database
One Game Session

48. Oct 2010: Magic Land based on stateful server
If DBs are the problem
Don’t use them
Store state in server
Need to be robust

49. Oct 2010: Magic Land based on stateful server
If DBs are the problem
Don’t use them
Store state in server
Need to be robust

50. Oct 2010: Magic Land based on stateful server
If DBs are the problem
Don’t use them
Store state in server
Need to be robust

51. Stateful servers are not as hard as you think
session

52. Stateful servers are not as hard as you think
session
session
session
session

53. Stateful servers are not as hard as you think
Server
session
session
session
session

54. Stateful servers are not as hard as you think
Server
session
session
session
session
S3

55. Stateful servers are not as hard as you think
Server
session
session
session
session
S3

56. Stateful servers are not as hard as you think
Server
session
session
session
session
S3

57. Stateful servers are not as hard as you think
Server
session
session
session
session
S3

58. Stateful servers are not as hard as you think
Server
session
session
session
session
S3

59. Stateful servers are not as hard as you think
Server
session
session
session
session
S3

60. Stateful servers are not as hard as you think
Server Server Server
session
session session
session session
session
session
session session
session session
session
S3

61. With stateful server the DB is less used
Ruby Stateless Erlang Stateful
30,000
22,500
15,000
7,500
0
database operations / sec

62. With stateful server the DB is less used
Ruby Stateless Erlang Stateful
30,000
22,500
15,000
700
7,500
0
database operations / sec

63. Deploying with a stateful server
In order to bring up a new version

64. Deploying with a stateful server
In order to bring up a new version
Just deploy it
Hot code replacement is great!

65. There are even more advantages
Faster than Ruby (5,000 rps / node)
-­‐ CPU bound

66. There are even more advantages
Faster than Ruby (5,000 rps / node)
-­‐ CPU bound
Very few SPOFs
-­‐ ... and those are easy to recover

67. There are even more advantages
Faster than Ruby (5,000 rps / node)
-­‐ CPU bound
Very few SPOFs
-­‐ ... and those are easy to recover
TransacFonal logic
-­‐ Invariants instead of explicit error handling

68. NICE!
http://www.flickr.com/photos/aigle_dore/

69. Architecture EvoluFon at Wooga
The Start: Ruby + AutomaFon
The Next Step: Erlang + S3
Company Values

70. A good value system
We’ve learned to value

71. A good value system
We’ve learned to value
Small teams over Big teams

72. A good value system
We’ve learned to value
Small teams over Big teams
CollaboraFon over Compe??on

73. A good value system
We’ve learned to value
Small teams over Big teams
CollaboraFon over Compe??on
Generalists over Specialists

74. A good value system
We’ve learned to value
Small teams over Big teams
CollaboraFon over Compe??on
Generalists over Specialists
Effort reducFon over Cost reduc?on

75. A good value system
We’ve learned to value
Small teams over Big teams
CollaboraFon over Compe??on
Generalists over Specialists
Effort reducFon over Cost reduc?on
InnovaFon over Risk mi?ga?on

76. It works!

77. It works!
Be fast, be bold!

78. QuesFons?
Jesper Richter-­‐Reichhelm
@jrirei
slideshare.net/wooga
wooga.com/jobs