Slide for a talk I presented internally at Opera in December 2009 about the deployment of varnish in our production environment at my.opera.com, the social network community.

1. My Opera meets Varnish
varnish
high performance web caching
cosimo@opera.com

2. What is Varnish?
varnish
● Caching reverse proxy, like Squid backends
● Delegates memory mgmt to OS cache
● Mainly developed at Linpro in Oslo

3. Two typical Varnish setups
incoming
requests
frontends
varnish
backends

4. VCL - Varnish Config Language
• man vcl
• VCL is compiled to C code
• Injected into the running instance, without restart
• Must define a backend or a director
• VCL gives you several hooks:
vcl_recv()
vcl_hash()
vcl_fetch()
vcl_hit()
vcl_miss()
vcl_deliver()

5. Varnish deployment in My Opera
• In production beginning of October 2009
• 1 old recycled machine, 2 Gb of disk allocated
• Started serving avatars
1M+ requests per day before Unite
http://my.opera.com/<username>/avatar.pl
• Soon after, added Desktop Team RSS (very popular!)
• then user pictures, hundreds of thousands req/day
• then Unite/ASD API requests
- friends of a user
- groups of a user
• In total, 13,25% of all My Opera requests are «varnished»
• Around 7,2M req/day

6. Varnish deployment in My Opera
Problems /1
• Still using Debian Etch?
First Varnish instance was running v1.x from Etch.
several years old, not good
• Experienced VIPs
– ”Very Interesting Problems”
– User X getting User Y's session
– Random users getting admin powers. Nightmare!
• Theory: Varnish was caching response bodies that contained
Set-Cookie: opera_session=<session_id>

7. Varnish deployment in My Opera
Problems /2
• There wasn't any obvious configuration problem.
Same config worked with 2.0.x from Backports.
• v2.0.{4,5} is highly recommended!

8. Varnish deployment in My Opera
Problems /3
• We tried caching the frontpage of My Opera, but had to revert the
change due to too many different custom layouts for Opera Mobile,
Mini, IE, Firefox, etc...
• Maybe using clever vcl_hash() tricks we can achieve that too.

9. My Opera configuration

10. Backends and Directors
• Backend
single backend machine, or load-balanced virtual server
• Director
– simple round-robin or random weighted “balancing” logic
– has basic connection retries mechanism
– has basic backend health check
• If you already have an LVS, define a single Backend
Otherwise, go for the Director

11. Backends and Directors
Define a backend
# Only hit the upload servers
backend myopera {
.host = "upload.my.opera.com";
.port = "80";
}

12. Backends and Directors
Define a director
director myopera round-robin {
.backend {
.host = "b1.opera.com";
.port = "80";
}
.backend {
.host = "b2.opera.com";
.port = "80";
}
...
}

13. Backends and Directors
...and then use them
sub vcl_recv {
...
set req.backend = myopera;
...
}

14. vcl_recv() / 1
sub vcl_recv {
set req.backend = myopera;
set req.grace = 3m;
# URL patterns based cache.
# Avoid possible mixups.
if(req.http.host !~ "^my.opera.com$") {
pass;
}

15. vcl_recv() / 2
if (req.url ~ "^/community/users/avatar.pl/[0-9]+$"
|| req.url ~ "^/.+/avatar.pl$"
|| req.url ~ "^/.+/picture.pl?xscale=100$"
|| req.url ~ "^/desktopteam/xml/atom/blog/?$"
|| req.url ~ "^/desktopteam/xml/rss/blog/?$"
|| req.url ~ "^/community/api/users/friends.pl?user=.+$"
|| req.url ~ "^/community/api/users/groups.pl?user=.+$"
) {
unset req.http.Cookie;
unset req.http.Authorization;
lookup;
}

16. vcl_recv() / 3
...
# Check for cookie only after always-cache URLs
if (req.http.Cookie ~ "(opera_session|opera_persistent_)") {
pass;
}
# DANGER, Will Robinson! Caching the front-page
# At this point, lots of Google Analytics cookies will go in.
# No problem. It's stuff used by Javascript
if (req.url ~ "^/community/$") {
lookup;
}
pass;
}

17. vcl_fetch() / 1
sub vcl_fetch {
set obj.http.X-Varnish-URL = req.url;
set obj.grace = 3m;
if (obj.http.Set-Cookie) {
set obj.http.X-Varnish-Cacheable = "no, set-cookie";
pass;
}
if (req.request != "GET") {
set obj.http.X-Varnish-Cacheable = "no, !GET";
pass;
}

18. vcl_fetch() / 2
if (req.http.host !~ "^my.opera.com$") {
set obj.http.X-Varnish-Cacheable = "no, !my.opera.com";
pass;
}
if (req.url ~ "^/community/users/avatar.pl/[0-9]+$"
|| req.url ~ "^/[A-Za-z0-9]+/avatar.pl$"
|| ... ) {
unset obj.http.Set-Cookie;
set obj.http.X-Varnish-Cacheable = "yes, url";
set obj.ttl = 24h;
deliver;
}

19. vcl_hash()
sub vcl_hash {
# Default Varnish behavior
set req.hash += req.url;
set req.hash += req.http.host;
# Have a different cached frontpage per language
if (req.url ~ "^/community/$") {
set req.http.X-FrontPage-Language = regsub(
req.http.Cookie,
"^.*?language=([^;]*?);*.*$", "1"
);
set req.hash += "lang:";
set req.hash += req.http.X-FrontPage-Language;
}
hash;
}

20. Testing Varnish
how to avoid nightmares...
• Developed a testing tool (varnish-test)
– outputs a TAP stream and some debug info
– works best if varnish is specially tuned
• Can quickly check if a test/production instance is performing
correctly or having problems
• Invoked as a simple script:
va rnis h-tes t --profile=tes ts .url --hos t=b1

21. Testing Varnish
caching test list
# Fro ntpa g e
/ N O _C O O K I E S V A R N I S H _C A C H E D
/ N O _C O O K I E S V A R N I S H _N O T _C A C H E D H o s t: m y.c n.o pera .c o m
/ N O _C O O K I E S V A R N I S H _C A C H E D C o o k ie:la ng ua g e=it
# B lo g s
/des k to ptea m /blo g / N O _C O O K I E S V A R N I S H _N O T _C A C H E D
# A va ta rs
/c o m m unity/us ers /a va ta r/817271 N O _C O O K I E S V A R N I S H _C A C H E D
/c o m m unity/us ers /a va ta r/442 N O _C O O K I E S V A R N I S H _C A C H E D
/g ra phic s /a va ta r.g if N O _C O O K I E S V A R N I S H _N O T _C A C H E D

22. Testing Varnish
caching test list
• We can specify exactly how the varnish instance should behave.
– Production acceptance tests
– Test new varnish versions, new OS distributions
– Fine tune config changes quickly with no impact on production
• Midway through there's a request that logs in as a test user.
From then on, we can verify what resources are cached when
a user is logged in. Some resources should be cached in any case.

23. Testing Varnish
sample run
...
ok 289 - Got response from backend for /community/ (from ...)
ok 290 - Correct status line
# Adding header [Cookie] => [language=it]
# ----------
# GET http://cache01.my.opera.com:6081/community/
# Host: my.opera.com
# ------------
ok 291 - 2nd request: got response from backend for /community/ (from...)
ok 292 - Correct status line
X-Varnish: 1211283813 1211283812
# X-Varnish: 1211283813 1211283812
X-Varnish-Status: hit
# X-Varnish-Status: hit
# X-Varnish-Cacheable: yes, language cookie
X-Varnish-Cacheable: yes, language cookie
# X-Varnish-URL: /community/
X-Varnish-URL: /community/
ok 293 - URL '/community/' was handled correctly by varnish
# cookie_header:
ok 294 - URL '/community/' has correct cookies (or no cookies)
1..294
All tests successful.

24. Monitoring Varnish
built-in tools
• varnishlog
– Reads shared memory log info and displays it
– Full instance log, on My Opera, 1 day is about 15 Gb
– You can get an emulated Apache-style access.log from it
• varnishncsa
– Displays requests to Varnish as Apache access logs
– Can read from an archived log by varnishlog
• varnishstat
– Displays realtime stats (hit ratio, space allocated, connections,...)

25. Monitoring Varnish
external tools
• Munin plugins
– Hit ratio
– Requests rate
– Backend traffic
• Nagios plugins
– Nothing special, TCP connection to port 6081

26. Monitoring Varnish

27. Monitoring Varnish

28. Monitoring Varnish

29. Next steps
• My Opera front page caching
• My Opera files server?
• Working on a prototype thumbnail server

30. References and more information
• Redpill-Linpro website
– http://varnish.projects.linpro.no
– Bug tracking, documentation and community support
– Users and developers mailing lists
• Commercial support and training
– http://www.varnish-cache.com

31. Questions?
• At Opera, there's several teams using Varnish in production
• If you want to know more, contact me: cosimo@opera.com