More Related Content Similar to Surge 2010 - from disaster to stability - scaling my.opera.com Similar to Surge 2010 - from disaster to stability - scaling my.opera.com (20) More from Cosimo Streppone More from Cosimo Streppone (9) Surge 2010 - from disaster to stability - scaling my.opera.com1. from disaster to stability
the scaling challenges of my.opera.com
Surge 2010 – Version 3
3. 1999
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
4. 2001
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
5. 2004
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
6. 2007
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
7. 2009
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
16. many improvements since then
➔ Efficient filesystem cache
➔ "Dogpile effect" AKA stampeding AKA ...
➔ Persistent db + memcached connections
➔ Soft counters
➔ Profiling, profiling, …
17. code profiling
[DML] time=1237308152, user=,
url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,
name=XWA::User, variable=active, type=module,
elapsed=0.068473, host=my.opera.com
[DML] time=1237308152, user=, url=/community/,
name=XWA::User, variable=, type=module,
elapsed=0.015935, host=my.opera.com
[DML] ...
18. top time-intensive modules
XWA::User::Sidebar 2024.919s (27.2%, 0.28 s/call)
XWA::User 1778.445s (23.9%, 0.09 s/call)
XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)
XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)
XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)
XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)
XWA::Skins 180.486s ( 2.4%, 0.49 s/call)
XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)
XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)
XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)
XWA::Albums 93.740s ( 1.3%, 2.04 s/call)
XWA::Journals 92.390s ( 1.2%, 2.37 s/call)
19. many improvements since then
➔ YSlow?
➔ The Expires header is your friend!
➔ Hot MyISAM tables converted to InnoDB
➔ MySQL Master/Master setup
➔ Jet Profiler
24. Avatars - 2007
75%
/<user-name>/avatar.pl
/<user-name>/avatar.pl?xscale=8192 (!)
25. Avatars wtf!?
my $sql = DBConnect('master');
my %user = $sql->get(
"SELECT a.blob, a.filename,
FROM avatars a, users u
WHERE u.user=? AND u.id=a.user",
$user);
$req->print( $user{'blob'} );
26. Avatars - reloaded
➔ Export to balanced fs (5 formats)
➔ Zero SQL queries
➔ Storage subsystem
➔ static.myopera.com was born
27. resources
(user uploads, binary blobs, ...)
Pools
or single servers
URLs
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg
28. + x
➔ Load ➔ HTTP::DAV
➔ Flexibility ➔ Precomp URLs
➔ Static scales!
36. IT NEEDS
TO BE OUT
TOMORROW
!!!
THERE
WILL BE A
PRESS
RELEASE !
43. + x
➔ Load ➔ Chainsaw!
➔ Flexibility ➔ Purging
➔ Instant scaling
46. + x
➔ Prototype 1 week ➔ Accuracy
➔ Geo-scaling ➔ No DC feedback
➔ Redundant ➔ Monitoring
47. Next steps
➔ Search (Solr?)
➔ Batch activity feed
➔ Real connection pooling
➔ … and on ...
48. Remember!
➔ Team spirit is important
➔ Another level of indirection...
➔ Keep it simple
➔ Keep a log