How we use Varnish at Opera Software, from the beginning (2009) to now.
Presentation hold for the 5th Varnish Users Group meeting (VUG5) held in Paris on March 22nd 2012.
1. Varnish @ Opera
Varnish Users Group Meeting
Paris, 22nd March 2012
Cosimo Streppone <cosimo@opera.com>
2. 1st Varnish deployment: My Opera
• October 2009
• 1 old recycled machine, 2 Gb of disk allocated
• Started serving static pictures (1M+ req/day)
• Then more...
• Even more...
• ...
• ~15% of all My Opera requests were «varnished»
• Around 8M req/day
3. My Opera – The start
• Still using Debian Etch
First Varnish instance was running v1.x from Etch.
several years old, not good
• Experienced VIPs
– ”Very Interesting Problems”
– User X getting User Y's session
– Random users getting admin powers. Nightmare!
• Theory: Varnish was caching response bodies that contained
Set-Cookie: opera_session=<session_id>
5. My Opera – Pass logged in users
...
# Check for cookie only after always-cache URLs
if (req.http.Cookie ~ "(opera_session|opera_persistent_)") {
pass;
}
# DANGER, Will Robinson! Caching the front-page
# At this point, lots of Google Analytics cookies will go in.
# No problem. It's stuff used by Javascript
if (req.url ~ "^/community/$") {
lookup;
}
pass;
}
6. My Opera: testing Varnish setup
...
ok 289 - Got response from backend for /community/ (from ...)
ok 290 - Correct status line
# Adding header [Cookie] => [language=it]
# ----------
# GET http://cache01.my.opera.com:6081/community/
# Host: my.opera.com
# ------------
ok 291 - 2nd request: got response from backend for /community/ (from...)
X-Varnish: 1211283813 1211283812
ok 292 - Correct status line
# X-Varnish: 1211283813 1211283812
X-Varnish-Status: hit
# X-Varnish-Status: hit
# X-Varnish-Cacheable: yes, language cookie
X-Varnish-Cacheable: yes, language cookie
# X-Varnish-URL: /community/
X-Varnish-URL: /community/
ok 293 - URL '/community/' was handled correctly by varnish
# cookie_header:
ok 294 - URL '/community/' has correct cookies (or no cookies)
1..294
All tests successful.
12. Static assets and UGC servers
Problem Solution
• One central location • Decentralized varnish
• SPOF servers in multiple DC
• High latency US -> NO • Talking to 1 backend
• Very long TTL
• Health probes
• Cache invalidation API
• Built our GeoDNS
13. Thumbnail generation and caching
Problem Solution
• Change of Design™ • Switch to on-the-fly
made our millions of generation model
pre-generated • Used mod_dims (AOL)
thumbnails useless • Varnish on :80
• 2 backends
300k objects
95% hit rate avg
800 req/s/backend peak
14. Thumbnail generation and caching
How it works
http://localhost/dims/
crop/472x360/
contrast/+1/
quality/90/
/actual/picture/url.jpg (remote too!)
Using rewrite rules
Http://localhost/tn/small/
/actual/picture/url.jpg
15. Thumbnail generation and caching
●
Recognize mobile/non-mobile
●
Scale thumbnails on the fly
●
Reduce JPEG quality
Ex.: /thumb/small/quality/80/some/path/pic.jpg
16. Shields-up configuration
Problem Solution
• Original setup too • DDoS
specific to My Opera • Varnish in front, rather
• Long tail of non- than after frontends
popular content • Cache most logged out
“unprotected” requests with lower TTL
• Can we find some • Compromise solution,
more generic setup? but generic enough
18. Sitecheck – Malware, fraud protection
• Used by the Opera browser
• Must work! Failure not an option
• ~8k req/s/backend peak
• 2 varnish boxes, 16k req/s, 20k peak
• 85% hit rate
• TTL 10'
20. Country-level ban
• Contract mandates that TV Store shouldn't
be available in specific countries
• Country check in the backend means no
caching is possible
• Implemented with varnish-geoip
21.
22. Country-level ban
sub country_ban_list_check {
# Allow testing of country ban
if (req.http.Cookie ~ "x_geo_ip_forceds*=s*country:..") {
set req.http.X-Geo-IP = regsuball(
req.http.Cookie,
"^.*x_geo_ip_forceds*=s*(country:..).*$", "1"
);
log "Forced X-Geo-IP to '" req.http.X-Geo-IP "'";
}
# Block access to tvstore in these countries
if (req.http.X-Geo-IP &&
req.http.X-Geo-IP ~ "^country:(C1|C2|C3|...)$") {
log "Country ban";
error 750 "tvstore is not available in your country";
}
}
sub vcl_recv {
C{ vcl_geoip_country_set_header_xff(sp); }C
call country_ban_list_check;
}
23. Brand + device TV detection
• Analyze User-Agent header
• Regex the hell out of it
• Send X-Brand, X-Device header to backend
• Fallback Device detection in the backend
(for development, test setups, ...)
25. accept-encoding.vcl
# STD: Deal with different Accept-Encoding formats
sub accept_encoding_normalize {
if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
}
elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
}
else {
unset req.http.Accept-Encoding;
}
}
}
26. accept-language.vcl
C{
/*
* Accept-language header normalization
*
* - Parses client Accept-Language HTTP header
* - Tries to find the best match with the supported languages
* - Writes the best match as req.http.X-Varnish-Accept-Language
*
* http://github.com/cosimo/varnish-accept-language
*/
#include <ctype.h> /* isupper */
#include <stdio.h>
#include <stdlib.h> /* qsort */
#include <string.h>
#define DEFAULT_LANGUAGE "en"
#define SUPPORTED_LANGUAGES ":de:en:es-la:fr:fy:hu:ja:no:pl:pt-
br:ru:sk:sq:sr:tr:uk:vn:xx-lol:zh-tw:"
…
27. maintenance.vcl + {up,down}.sh
include "/etc/varnish/accept-encoding.vcl";
backend oopsy {
.host = "10.20.21.22”;
.port = "80";
}
sub vcl_recv {
set req.backend = oopsy;
# Serve page from within Varnish. See vcl_error()
if (req.url == "/ping.html") {
error 700;
}
call accept_encoding_normalize;
# Collapse URLs, so that we have just one cached object
set req.url = "/maintenance-down";
remove req.http.Cookie;
remove req.http.Authorization;
return (lookup);
}
29. X-forwarded-for.vcl
# See http://www.varnish-cache.org/trac/ticket/540
sub inject_forwarded_for {
# Rename the incoming XFF header to work around a Varnish bug
if (req.http.X-Forwarded-For) {
# Append the client IP
set req.http.X-Real-Forwarded-For =
req.http.X-Forwarded-For ", "
regsub(client.ip, ":.*", "");
}
else {
# Simply use the client IP
set req.http.X-Real-Forwarded-For = regsub(client.ip,
":.*", "");
}
}
Wat!?
31. http-cuke – csrf.test
Feature: TVStore uses cookies to protect against CSRF attacks
In order to protect the users from CSRF attacks
As a TV Store developer
I want to verify that some pages send out a CSRF cookie token to
the browser or device
Scenario: Accessing the Backgammon application URL
Given a "Opera/9.80 (Linux … Opera TV Store)" user agent
When I go to "https://tvstore.server/store/app/backgammon"
Then the final HTTP status code should be "200"
Then the page should contain "A board game for one player"
Then the page should not be cached by varnish
Then the server should send a CSRF token
33. http-cuke – a sample test run
# ============================================================
# FEATURE: TV Store uses cookies to protect against CSRF attacks
# ============================================================
# ------------------------------------------------------------
# SCENARIO: Accessing the Backgammon application URL
# ------------------------------------------------------------
ok 1 - Given a "Opera/9.80 (Linux...)" user agent
ok 2 - When I go to "https://tvstore.server/app/backgammon"
ok 3 - Status code is 200 (expected 200)
ok 4 - Then the final HTTP status code should be "200"
ok 5 - String 'A board game for one player' was found in the page
ok 6 - Then the page should contain "A board game for one player"
ok 7 - X-Varnish header contains only current XID (523289525)
ok 8 - Age of cached resource is zero
ok 9 - Then the page should not be cached by varnish
ok 10 - CSRF token was found (49a0da1b2758bf62a028072e4f7f36dc)
ok 11 - Then the server should send a CSRF token
51. varnishtop -t10s
Collect traffic for 10s and then report
Bonus feature: tags, AKA group-by
varnishtop -i RxURL -g RxHeader.Referer -t 60s
52. Use headers as vars less than ideal
Introduce variables or registers to avoid
set req.http.X-Var = regsuball(
req.http.Some-Header, '…', '1'
);
53. Better Cookies inspection
Avoid regsuball() mess on req.http.Cookie
set req.http.Cookie.SomeName = “xxx”;
set req.http.X-Var1 = req.http.Cookie.sessionid;
and...
set var.SessionID = req.http.Cookie.sessionid;