Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Varnish @ Opera 
v3 / DevOps Norway Meetup 
Oslo, 17th September 2014 
Cosimo Streppone <cosimo@opera.com>
1st Varnish deployment: My Opera 
• October 2009 
• 1 old recycled machine, 2 Gb of disk allocated 
• Started serving stat...
My Opera – The start 
• Still using Debian Etch 
First Varnish instance was running v1.x from Etch. 
several years old, no...
My Opera – The start 
if (req.url ~ "^/community/users/avatar.pl/[0-9]+$" 
|| req.url ~ "^/.+/avatar.pl$" 
|| req.url ~ "^...
My Opera – Pass logged in users 
... 
# Check for cookie only after always-cache URLs 
if (req.http.Cookie ~ "(opera_sessi...
My Opera: testing Varnish setup 
... 
ok 289 - Got response from backend for /community/ (from ...) 
ok 290 - Correct stat...
My Opera – Next steps
My Opera – Next steps 
● Front page caching 
● Static assets and UGC 
● On-the-fly thumbnails 
● “Shields-up” configuratio...
Front page caching 
Problem 
• Very dynamic, i18n 
• Accept-Language 
header variation 
• Vary: Accept- 
Language sub-opti...
Front page caching - Accept-Language 
SUPPORTED_LANGUAGES = “:de:es:it:ru:” 
DEFAULT_LANGUAGE = “en” 
Client sends 
Accept...
Front page caching
Static assets and UGC servers 
Problem 
• One central location 
• SPOF 
• High latency US -> NO 
Solution 
• Decentralized...
Thumbnail generation and caching 
Problem 
• Change of Design™ 
made our millions of 
pre-generated 
thumbnails useless 
S...
Thumbnail generation and caching 
How it works 
http://localhost/dims/ 
crop/472x360/ 
contrast/+1/ 
quality/90/ 
/actual/...
Thumbnail generation and caching 
● Recognize mobile/non-mobile 
● Scale thumbnails on the fly 
● Reduce JPEG quality 
Ex....
Shields-up configuration 
Problem 
• Original setup too 
specific to My Opera 
• Long tail of non-popular 
content 
“unpro...
Other projects
Other projects 
Many since then! 
• Sitecheck 
• Opera.com 
• TV Store 
• Speeddials 
• Discover 
• ...
Opera Discover 
My current project 
80M backend API requests/day 
260M image requests/day
GeoIP country check
Country-level ban 
• Contract mandates that TV Store shouldn't 
be available in specific countries 
• Country check in the...
https://github.com/cosimo/varnish-geoip
Country-level ban 
sub country_ban_list_check { 
# Allow testing of country ban 
if (req.http.Cookie ~ "x_geo_ip_forceds*=...
VCL library
accept-encoding.vcl (now obsolete) 
# STD: Deal with different Accept-Encoding formats 
sub accept_encoding_normalize { 
i...
accept-language.vcl 
C{ 
/* 
* Accept-language header normalization 
* 
* - Parses client Accept-Language HTTP header 
* -...
maintenance.vcl + {up,down}.sh 
include "/etc/varnish/accept-encoding.vcl"; 
backend oopsy { 
.host = "10.20.21.22”; 
.por...
purge.vcl 
acl purge { … } 
sub vcl_recv { 
if (req.request == "PURGE") { 
If (! (client.ip ~ purge)) { 
error 405 "Not al...
X-forwarded-for.vcl 
# See http://www.varnish-cache.org/trac/ticket/540 
sub inject_forwarded_for { 
# Rename the incoming...
Testing VCLs – http-cuke
http-cuke – csrf.test 
Feature: Site uses cookies to protect against CSRF attacks 
In order to protect the users from CSRF...
http-cuke – prove-like output 
$ http-cuke --test ./csrf.test 
$ http-cuke --test-dir ./some-dir
http-cuke – a sample test run 
# ============================================================ 
# FEATURE: Web site uses co...
https://github.com/cosimo/http-cuke
Dumping varnishlog
vlogdump 
$ varnishlog | vlogdump
vlogdump – a sample test run 
$ varnishlog | vlogdump -v only_misses=1 
172.22.0.15 => GET /assets/tn/m/mq/e85ed...6733a48...
vlogdump – demo?
https://github.com/cosimo/vlogdump
vlogdump + rtail 
Remote tailing made easy. 
60 lines of Perl. 
$ rtail --host=h1 --host=h2 --host=h3 ...  
--command varn...
vlogdump + rtail 
No github yet :-)
Puppet module
varnish/manifests/init.pp 
class varnish { 
package { "varnish": ensure => "installed" } 
file { "/etc/init.d/varnish": … ...
Custom init script 
# Lower stack limit demand for every Varnish thread 
# http://projects.linpro.no/pipermail/varnish-mis...
Custom init script 
# Optionally warm up the cache 
# 
# Drop a custom script into this path 
# to have it being picked up...
Custom sysctl settings 
# From http://varnish.projects.linpro.no/wiki/Performance 
# + our own tweaking and tuning 
net.ip...
Purge cache script 
Modeled after Debian vcl-reload script 
$ purge-cache -a 
$ purge-cache -u http://some.url 
$ purge-ca...
Cache warmup script 
Drop-in script in 
/usr/share/varnish/cache-warmup 
Invoked right after startup
varnish/manifests/init.pp – 2 
define varnish::config ( 
$vcl_conf="default.vcl", $listen_address="", 
$listen_port=6081, ...
Example of varnish::config 
varnish::config { 'cache-varnish-config': 
vcl_conf => 'cache.vcl', 
storage_type => 'malloc',...
varnish/manifests/init.pp – 3 
define varnish::vcl ($source) { 
file { "/etc/varnish/${name}.vcl": 
ensure => 'file', 
own...
https://github.com/cosimo/puppet-modules
Migration to Varnish 3
Following Debian stable 
• Wheezy now ships with 3.0.2 
• 2.0 > 2.1 migration was painless 
• 2.1 > 3.0 migration was pain...
Next steps?
Next Steps? 
• Personalized (but cached) content 
• A/B Testing 
• ESI? 
• VMods? 
• Migration to V4
Spørsmål?
Takk!
Upcoming SlideShare
Loading in …5
×

1

Share

Download to read offline

How we use and deploy Varnish at Opera

Download to read offline

This is a new version of a talk I presented at a Varnish Users Group meeting in Paris in 2012. We've added a few useful tools and improved our Puppet module since then.


Presented at the Devops Norway meetup in Oslo on 17th of September 2014.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

How we use and deploy Varnish at Opera

  1. 1. Varnish @ Opera v3 / DevOps Norway Meetup Oslo, 17th September 2014 Cosimo Streppone <cosimo@opera.com>
  2. 2. 1st Varnish deployment: My Opera • October 2009 • 1 old recycled machine, 2 Gb of disk allocated • Started serving static pictures (1M+ req/day) • Then more... • Even more... • ... • ~15% of all My Opera requests were «varnished» • Around 8M req/day
  3. 3. My Opera – The start • Still using Debian Etch First Varnish instance was running v1.x from Etch. several years old, not good • Experienced VIPs – ”Very Interesting Problems” – User X getting User Y's session – Random users getting admin powers. Nightmare! • Theory: Varnish was caching response bodies that contained Set-Cookie: opera_session=<session_id>
  4. 4. My Opera – The start if (req.url ~ "^/community/users/avatar.pl/[0-9]+$" || req.url ~ "^/.+/avatar.pl$" || req.url ~ "^/.+/picture.pl?xscale=100$" || req.url ~ "^/desktopteam/xml/atom/blog/?$" || req.url ~ "^/desktopteam/xml/rss/blog/?$" || req.url ~ "^/community/api/users/friends.pl?user=.+$" || req.url ~ "^/community/api/users/groups.pl?user=.+$" ) { unset req.http.Cookie; unset req.http.Authorization; lookup; }
  5. 5. My Opera – Pass logged in users ... # Check for cookie only after always-cache URLs if (req.http.Cookie ~ "(opera_session|opera_persistent_)") { pass; } # DANGER, Will Robinson! Caching the front-page # At this point, lots of Google Analytics cookies will go in. # No problem. It's stuff used by Javascript if (req.url ~ "^/community/$") { lookup; } pass; }
  6. 6. My Opera: testing Varnish setup ... ok 289 - Got response from backend for /community/ (from ...) ok 290 - Correct status line # Adding header [Cookie] => [language=it] # ---------- # GET http://cache01.my.opera.com:6081/community/ # Host: my.opera.com # ------------ ok 291 - 2nd request: got response from backend for /community/ (from...) ok 292 - Correct status line # X-Varnish: 1211283813 1211283812 # X-Varnish-Status: hit # X-Varnish-Cacheable: yes, language cookie # X-Varnish-URL: /community/ ok 293 - URL '/community/' was handled correctly by varnish # cookie_header: ok 294 - URL '/community/' has correct cookies (or no cookies) 1..294 X-Varnish: 1211283813 1211283812 X-Varnish-Status: hit X-Varnish-Cacheable: yes, language cookie X-Varnish-URL: /community/ All tests successful.
  7. 7. My Opera – Next steps
  8. 8. My Opera – Next steps ● Front page caching ● Static assets and UGC ● On-the-fly thumbnails ● “Shields-up” configuration
  9. 9. Front page caching Problem • Very dynamic, i18n • Accept-Language header variation • Vary: Accept- Language sub-optimal Solution • varnish-accept-language “extension”
  10. 10. Front page caching - Accept-Language SUPPORTED_LANGUAGES = “:de:es:it:ru:” DEFAULT_LANGUAGE = “en” Client sends Accept-Language: ru, uk;q=0.9 Accept-Language: es-ES, es;q=0.8 Accept-Language: fr, it;q=0.7 Accept-Language: fr Backend receives Accept-Language: ru Accept-Language: es Accept-Language: it Accept-Language: ben
  11. 11. Front page caching
  12. 12. Static assets and UGC servers Problem • One central location • SPOF • High latency US -> NO Solution • Decentralized varnish servers in multiple DC • Talking to 1 backend • Very long TTL • Health probes • Cache invalidation API • Built our GeoDNS
  13. 13. Thumbnail generation and caching Problem • Change of Design™ made our millions of pre-generated thumbnails useless Solution • Switch to on-the-fly generation model • Used mod_dims (AOL) • Varnish on :80 • 2 backends 300k objects 95% hit rate avg 800 req/s/backend peak
  14. 14. Thumbnail generation and caching How it works http://localhost/dims/ crop/472x360/ contrast/+1/ quality/90/ /actual/picture/url.jpg (remote too!) Using rewrite rules Http://localhost/tn/small/ /actual/picture/url.jpg
  15. 15. Thumbnail generation and caching ● Recognize mobile/non-mobile ● Scale thumbnails on the fly ● Reduce JPEG quality Ex.: /thumb/small/quality/80/some/path/pic.jpg
  16. 16. Shields-up configuration Problem • Original setup too specific to My Opera • Long tail of non-popular content “unprotected” • Can we find some more generic setup? Solution • DDoS • Varnish in front, rather than after frontends • Cache most logged out requests with lower TTL • Compromise solution, but generic enough
  17. 17. Other projects
  18. 18. Other projects Many since then! • Sitecheck • Opera.com • TV Store • Speeddials • Discover • ...
  19. 19. Opera Discover My current project 80M backend API requests/day 260M image requests/day
  20. 20. GeoIP country check
  21. 21. Country-level ban • Contract mandates that TV Store shouldn't be available in specific countries • Country check in the backend means no caching is possible • Implemented with varnish-geoip https://github.com/cosimo/varnish-geoip
  22. 22. https://github.com/cosimo/varnish-geoip
  23. 23. Country-level ban sub country_ban_list_check { # Allow testing of country ban if (req.http.Cookie ~ "x_geo_ip_forceds*=s*country:..") { set req.http.X-Geo-IP = regsuball( req.http.Cookie, "^.*x_geo_ip_forceds*=s*(country:..).*$", "1" ); log "Forced X-Geo-IP to '" req.http.X-Geo-IP "'"; } # Block access to tvstore in these countries if (req.http.X-Geo-IP && req.http.X-Geo-IP ~ "^country:(C1|C2|C3|...)$") { log "Country ban"; error 750 "tvstore is not available in your country"; } } sub vcl_recv { C{ vcl_geoip_country_set_header_xff(sp); }C call country_ban_list_check; }
  24. 24. VCL library
  25. 25. accept-encoding.vcl (now obsolete) # STD: Deal with different Accept-Encoding formats sub accept_encoding_normalize { if (req.http.Accept-Encoding) { if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } }
  26. 26. accept-language.vcl C{ /* * Accept-language header normalization * * - Parses client Accept-Language HTTP header * - Tries to find the best match with the supported languages * - Writes the best match as req.http.X-Varnish-Accept-Language * * http://github.com/cosimo/varnish-accept-language */ #include <ctype.h> /* isupper */ #include <stdio.h> #include <stdlib.h> /* qsort */ #include <string.h> #define DEFAULT_LANGUAGE "en" #define SUPPORTED_LANGUAGES ":de:en:es-la:fr:fy:hu:ja:no:pl:pt-br: ru:sk:sq:sr:tr:uk:vn:xx-lol:zh-tw:" …
  27. 27. maintenance.vcl + {up,down}.sh include "/etc/varnish/accept-encoding.vcl"; backend oopsy { .host = "10.20.21.22”; .port = "80"; } sub vcl_recv { set req.backend = oopsy; # Serve page from within Varnish. See vcl_error() if (req.url == "/ping.html") { error 700; } call accept_encoding_normalize; # Collapse URLs, so that we have just one cached object set req.url = "/maintenance-down"; remove req.http.Cookie; remove req.http.Authorization; return (lookup); }
  28. 28. purge.vcl acl purge { … } sub vcl_recv { if (req.request == "PURGE") { If (! (client.ip ~ purge)) { error 405 "Not allowed."; } purge("req.url == " req.url); error 200 "Purged."; } else if (req.request == "PURGE_SUFFIX") { set req.http.X-URL = regsuball(req.url, "[|]|[^.$|()*+?{}]", "0") "$"; purge_url(req.http.X-URL); unset req.http.X-URL; error 200 "Purged suffix."; } else if (req.request == "PURGE_PREFIX") { … } } Ugly!
  29. 29. X-forwarded-for.vcl # See http://www.varnish-cache.org/trac/ticket/540 sub inject_forwarded_for { # Rename the incoming XFF header to work around a Varnish bug if (req.http.X-Forwarded-For) { # Append the client IP set req.http.X-Real-Forwarded-For = req.http.X-Forwarded-For ", " regsub(client.ip, ":.*", ""); } else { # Simply use the client IP set req.http.X-Real-Forwarded-For = regsub(client.ip, ":.*", ""); } } Wat!?
  30. 30. Testing VCLs – http-cuke
  31. 31. http-cuke – csrf.test Feature: Site uses cookies to protect against CSRF attacks In order to protect the users from CSRF attacks As a web site developer I want to verify that some pages send out a CSRF cookie token to the browser or device Scenario: Accessing the Backgammon application URL Given a "OPR/24.0.1558.23 (Linux … Opera)" user agent When I go to "https://server/store/app/backgammon" Then the final HTTP status code should be "200" And the page should contain "A board game for one player" And the page should not be cached by varnish And the server should send a CSRF token
  32. 32. http-cuke – prove-like output $ http-cuke --test ./csrf.test $ http-cuke --test-dir ./some-dir
  33. 33. http-cuke – a sample test run # ============================================================ # FEATURE: Web site uses cookies to protect against CSRF attacks # ============================================================ # ------------------------------------------------------------ # SCENARIO: Accessing the Backgammon application URL # ------------------------------------------------------------ ok 1 - Given a "OPR/24... (Linux...)" user agent ok 2 - When I go to "https://server/app/backgammon" ok 3 - Status code is 200 (expected 200) ok 4 - Then the final HTTP status code should be "200" ok 5 - String 'A board game for one player' was found in the page ok 6 - Then the page should contain "A board game for one player" ok 7 - X-Varnish header contains only current XID (523289525) ok 8 - Age of cached resource is zero ok 9 - Then the page should not be cached by varnish ok 10 - CSRF token was found (49a0da1b2758bf62a028072e4f7f36dc) ok 11 - Then the server should send a CSRF token
  34. 34. https://github.com/cosimo/http-cuke
  35. 35. Dumping varnishlog
  36. 36. vlogdump $ varnishlog | vlogdump
  37. 37. vlogdump – a sample test run $ varnishlog | vlogdump -v only_misses=1 172.22.0.15 => GET /assets/tn/m/mq/e85ed...6733a48802 HTTP/1.0 MISS <= 200 OK 172.22.0.18 => GET /assets/icons/.....-technology.png HTTP/1.0 MISS <= 304 Not Modified $ varnishlog | vlogdump -v show_req_headers=1 172.22.0.15 => GET /assets/3a626ed......e168914568080 HTTP/1.0 MISS <= 200 OK 51.483 ms req.http.Host = discovery.opera.com req.http.User-Agent = Amazon CloudFront req.http.X-Forwarded-For = 11.12.34.56 req.http.X-Amz-Cf-Id = ...0AZZaPkt87avA== req.http.Connection = keep-alive ...
  38. 38. vlogdump – demo?
  39. 39. https://github.com/cosimo/vlogdump
  40. 40. vlogdump + rtail Remote tailing made easy. 60 lines of Perl. $ rtail --host=h1 --host=h2 --host=h3 ... --command varnishlog | vlogdump
  41. 41. vlogdump + rtail No github yet :-)
  42. 42. Puppet module
  43. 43. varnish/manifests/init.pp class varnish { package { "varnish": ensure => "installed" } file { "/etc/init.d/varnish": … } file { "/etc/sysctl.conf": … } exec { "update-sysctl": … } file { "/usr/share/varnish/purge-cache": … } service { "varnish": ensure => "running", … } munin::plugin::custom { "varnish_": } munin::plugin { [ "varnish_backend_traffic", "varnish_expunge", … } }
  44. 44. Custom init script # Lower stack limit demand for every Varnish thread # http://projects.linpro.no/pipermail/varnish-misc/2009-August/002977.html # Still relevant for Varnish 3 ?? ulimit -s 256 # Startup with custom cc_command fails # http://stackoverflow.com/a/8333333 # Filed Debian bug #659005 if bash -c "start-stop-daemon --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"; then log_end_msg 0 else …
  45. 45. Custom init script # Optionally warm up the cache # # Drop a custom script into this path # to have it being picked up by the # main init script. if [ -x /usr/share/varnish/cache-warmup ]; then /usr/share/varnish/cache-warmup fi
  46. 46. Custom sysctl settings # From http://varnish.projects.linpro.no/wiki/Performance # + our own tweaking and tuning net.ipv4.ip_local_port_range = 1024 65536 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_fin_timeout = 30 net.core.netdev_max_backlog = 30000 net.ipv4.tcp_no_metrics_save = 1 net.core.somaxconn = 262144 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2
  47. 47. Purge cache script Modeled after Debian vcl-reload script $ purge-cache -a $ purge-cache -u http://some.url $ purge-cache -r '^/(home|user)/'
  48. 48. Cache warmup script Drop-in script in /usr/share/varnish/cache-warmup Invoked right after startup
  49. 49. varnish/manifests/init.pp – 2 define varnish::config ( $vcl_conf="default.vcl", $listen_address="", $listen_port=6081, $thread_min=400, $thread_max=5000, $thread_timeout=30, $storage_type="malloc", $storage_size="12G", $ttl=60, $thread_pools=$processorcount, $sess_workspace=131072, $cc_command="", $sess_timeout=3 ) { file { "/etc/default/varnish": ensure => "present", owner => "root", group => "root", mode => 644, content => template("varnish/debian-defaults.erb"), require => Package["varnish"], notify => Service["varnish"], } }
  50. 50. Example of varnish::config varnish::config { 'cache-varnish-config': vcl_conf => 'cache.vcl', storage_type => 'malloc', storage_size => '20G', listen_port => 80, sess_workspace => 131072, ttl => 86400, thread_pools => 4, thread_min => 800, thread_max => 2000, # Necessary for GeoIP cc_command => 'exec cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o %o %s', }
  51. 51. varnish/manifests/init.pp – 3 define varnish::vcl ($source) { file { "/etc/varnish/${name}.vcl": ensure => 'file', owner => 'root', group => 'root', Mode => '0644', source => $source, require => Package['varnish'], notify => Service['varnish'], } }
  52. 52. https://github.com/cosimo/puppet-modules
  53. 53. Migration to Varnish 3
  54. 54. Following Debian stable • Wheezy now ships with 3.0.2 • 2.0 > 2.1 migration was painless • 2.1 > 3.0 migration was painless too
  55. 55. Next steps?
  56. 56. Next Steps? • Personalized (but cached) content • A/B Testing • ESI? • VMods? • Migration to V4
  57. 57. Spørsmål?
  58. 58. Takk!
  • phongdle5

    Jun. 17, 2015

This is a new version of a talk I presented at a Varnish Users Group meeting in Paris in 2012. We've added a few useful tools and improved our Puppet module since then. Presented at the Devops Norway meetup in Oslo on 17th of September 2014.

Views

Total views

1,280

On Slideshare

0

From embeds

0

Number of embeds

13

Actions

Downloads

2

Shares

0

Comments

0

Likes

1

×