SlideShare a Scribd company logo
1 of 45
Download to read offline
Using ngx_lua in UPYUN 
Monkey Zhang (timebug) 
2014.11 @ Beijing OSC
Hello Nginx 
nginx [engine x] is an HTTP and reverse proxy server, as 
well as a mail proxy server, written by Igor Sysoev.
A simple example 
$ ./configure --prefix=/opt/nginx  
--add-module=/path/to/echo-nginx-module 
http { 
server { 
listen 8080; 
location /hello { 
set $foo "hello"; 
echo $foo; 
set $foo "world"; 
echo $foo; 
} 
} 
} 
$ curl http://localhost:8080/hello 
world 
world
In Fact … 
http { 
server { 
listen 8080; 
location /hello { 
set $foo "hello"; 
set $foo "world"; 
echo $foo; 
echo $foo; 
} 
} 
} 
✦ REWRITE PHASE: set (ngx_http_rewrite_module) 
✦ CONTENT PHASE: echo (echo-nginx-module)
Nginx Internals: 
HTTP request phase handlers 
✦ POST READ PHASE 
✦ SERVER REWRITE PHASE 
✦ FIND CONFIG PHASE 
✦ REWRITE PHASE 
✦ POST REWRITE PHASE 
✦ PRE ACCESS PHASE 
✦ ACCESS PHASE 
✦ POST ACCESS PHASE 
✦ TRY FILES PHASE 
✦ CONTENT PHASE 
✦ LOG PHASE
Why Nginx Module 
Development is Not Easy ?
A true story 
$ ./configure --prefix=/opt/nginx  
--add-module=/path/to/echo-nginx-module  
--add-module=/path/to/base64-nginx-module 
http { 
server { 
listen 8080; 
location /base64 { 
base64 on; 
base64_max_length 10485760; 
echo "hello world"; 
} 
} 
} 
$ curl http://localhost:8080/base64 
aGVsbG8gd29ybGQK
500 lines C
If is Evil 
http://agentzh.blogspot.jp/2011/03/how-nginx-location-if-works.html 
http { 
server { 
listen 8080; 
location /if { 
set $foo 1; 
if ($foo = 1) { 
set $foo 2; 
echo "foo = $foo"; 
} 
set $foo 3; 
proxy_pass http://127.0.0.1:$server_port/$foo; 
} 
location ~ /(d+) { 
echo "bar = $1"; 
} 
} 
} 
$ curl http://localhost:8080/if 
foo = 3
If is Evil: 
How it works 
✦ REWRITE PHASE 
set $foo 1; 
if ($foo = 1) { 
set $foo 2; 
} 
set $foo 3; 
✦ CONTENT PHASE 
if ($foo = 1) { 
echo "foo = $foo"; 
}
If is Evil: 
Break ngx_rewite Directives 
✦ REWRITE PHASE 
set $foo 1; 
if ($foo = 1) { 
set $foo 2; 
break; 
} 
✦ CONTENT PHASE 
if ($foo = 1) { 
echo "foo = $foo"; 
} 
http { 
server { 
listen 8080; 
location /if { 
set $foo 1; 
if ($foo = 1) { 
set $foo 2; 
break; 
echo "foo = $foo"; 
} 
set $foo 3; 
proxy_pass http://127.0.0.1:$server_port/$foo; 
} 
location ~ /(d+) { 
echo "bar = $1"; 
} 
} 
} 
$ curl http://localhost:8080/if 
foo = 2
Hello Lua 
Lua is a powerful, fast, lightweight, embeddable 
scripting language.
$ ./configure --prefix=/opt/nginx  
--add-module=/path/to/lua-nginx-module 
http { 
server { 
listen 8080; 
location /add { 
set $res ''; 
rewrite_by_lua ' 
local a = tonumber(ngx.var.arg_a) or 0 
local b = tonumber(ngx.var.arg_b) or 0 
ngx.var.res = a + b 
'; 
content_by_lua ' 
ngx.say(ngx.var.res) 
'; 
} 
} 
} 
$ curl 'http://localhost:8080/add?a=6&b=7' 
13
LuaJIT 
LuaJIT is a Just-In-Time Compiler (JIT) for the Lua 
programming language.
How it works 
LuaJIT VM embedded into the Nginx
Base64 Filter by Lua 
http { 
lua_package_path “$prefix/app/src/?.lua;;"; 
server { 
listen 8080; 
location /base64 { 
set $b64_en ''; 
set $b64_e0 ''; 
set $b64_e1 ''; 
echo_duplicate 1000 hello; 
header_filter_by_lua ' 
ngx.header.content_length = nil -- ((n + 2) / 3 ) * 4 
ngx.header.content_type = "text/plain" 
ngx.header.content_transfer_encoding = "base64" 
'; 
body_filter_by_lua_file app/src/b64_body_filter.lua; 
} 
} 
}
Base64 Filter by Lua: 
local chunk = ngx.arg[1] Chunk by Chunk 
local e0 = ngx.var.b64_e0 or '' 
local e1 = ngx.var.b64_e1 or '' 
local en = tonumber(ngx.var.b64_en) or 0 
if en == 1 then 
chunk = e0 .. chunk 
elseif en == 2 then 
chunk = e0 .. e1 .. chunk 
end 
if not ngx.arg[2] then 
en = #chunk % 3 
if en == 1 then 
e0 = chunk:sub(-1) 
elseif en == 2 then 
e1 = chunk:sub(-1) 
e0 = chunk:sub(-2, -2) 
end 
chunk = chunk:sub(1, #chunk - en) 
else -- eof 
en = 0 
end 
ngx.var.b64_en = en 
ngx.var.b64_e0 = e0 
ngx.var.b64_e1 = e1 
ngx.arg[1] = ngx.encode_base64(chunk)
30 lines Lua
★ngx_upreferer_module.c ~ 2100 C 
★src/modules/referer.lua ~ 500 Lua 
ngx.md5 ngx.time ngx.re.* ngx.req.* 
ngx.decode_args 
string.sub string.find string.byte 
table.concat
based on Lua coroutines & synchronous & 100% non-blocking 
cosocket API 
ngx.socket.* connect send receive 
sslhandshake close settimeout etc.
A true story: 
Yupoo Referer Redirect (Year 2012) 
eval_escalate off; 
eval_override_content_type text/plain; 
eval $answer { 
set $redis_key "$scheme://<key>"; 
redis_pass redis; 
} 
if ($answer = "101") { 
rewrite ^ http://r.yupoo.com/101.gif redirect; 
break; 
} 
if ($answer = "102") { 
rewrite ^ http://r.yupoo.com/102.gif redirect; 
break; 
} 
if ($answer ~ "^http://") { 
rewrite ^ $answer redirect; 
break; 
}
WTF! 
✦Fork ngx_http_redis to support 
ypacl command 
✦vkholodkov/nginx-eval-module 
last commit on Nov 26, 2010
when upgrade Nginx to the 
latest version 
Coredump
Yupoo Referer Redirect by Lua 
lua-resty-redis (based on the cosocket API) 
ngx.redirect rewrite_by_lua_file 
Local redis = require "resty.redis" 
local red = redis:new() 
redis.add_commands("ypacl") 
-- set_timeout and connect 
local res, err = red:ypacl(key) 
if res == "101" then 
return ngx.redirect("http://r.yupoo.com/101.gif") 
else 
-- do something else 
end
LuaJIT FFI
lua-resty-uuid: 
Based on LuaJIT FFI 
-- modified version of original pull request by smallfish 
-- https://github.com/openresty/lua-resty-string/pull/7 
local ffi = require "ffi" 
local new = ffi.new 
local string = ffi.string 
local _M = {} 
ffi.cdef[[ 
typedef unsigned char uuid_t[16]; 
void uuid_generate(uuid_t out); 
void uuid_unparse(const uuid_t uu, char *out); 
]] 
local libuuid = ffi.load("libuuid") 
function _M.generate() 
if libuuid then 
local uuid = new("uuid_t") 
local result = new("char[36]") 
libuuid.uuid_generate(uuid) 
libuuid.uuid_unparse(uuid, result) 
return string(result) 
end 
end 
return _M
Openresty 
Yichun "agentzh" Zhang (章亦春) agentzh@gmail.com, CloudFlare Inc.
ONEPIECE
UPYUN CDN
UPYUN CDN: 
Metadata Cache 
✦CJSON, MessagePack 
✦ngx.shared.DICT 
✦lua-resty-lock 
✦lua-resty-shcache 
✦lua-resty-lrucache (*)
Metadata Cache: 
The original version (Year 2012) 
Issues 
local metadata = ngx.shared.metadata 
-- local key, bucket = ... 
local value = metadata:get(key) 
if value ~= nil then 
if value == "404" then 
return -- HIT_NEGATIVE 
else 
return value -- HIT 
end 
end 
local rds = redis:new() 
local ok, err = rds:connect("127.0.0.1", 6379) 
if not ok then 
metadata:set(key, "404", 120) -- expires 2 minutes 
return -- NO_DATA 
end 
res, err = rds:hget("upyun:" .. bucket, ":something") 
if not res or res == ngx.null then 
metadata:set(key, "404", 120) 
return -- NO_DATA 
end 
metadata:set(key, res, 300) -- expires 5 minutes 
rds:set_keepalive() 
return res -- MISS 
✦ "dog-pile effect" 
✦ NO_DATA when redis crash 
✦ code inflexible
Metadata Cache: 
lua-resty-shcache 
✦ cache locks (based on lua-resty-lock) 
✦ serialization / de-serialization 
✦ external lookup via Lua closure 
✦ MISS, HIT, HIT_NEGATIVE, STALE, NET_ERR 
-- app/src/modules/metadata.lua 
local shcache = require "resty.shcache" 
function _M.get_metadata(bucket) 
local lookup_metadata = function () 
-- fetch from redis 
return res 
end 
local cache_data = shcache:new( 
ngx.shared.metadata, 
{ external_lookup = lookup_metadata, 
encode = cmsgpack.pack, 
decode = cmsgpack.unpack, 
}, 
{ positive_ttl = cache_positive_ttl, 
negative_ttl = cache_negative_ttl, 
name = "metadata", 
}) 
-- local key = ... 
local data, _ = cache_data:load(key) 
if not data then 
return 
end 
return data 
end
Metadata Cache: 
MessagePack 
Bucket JSON MessagePack 
huab*** 6035 bytes 4135 bytes - 69%
Metadata Cache: 
lua-resty-lrucache 
✦ based on LuaJIT FFI 
✦ HIT_LRU 
✦ avoid serialization / de-serialization
UPYUN CDN: 
Upstream Health Check 
✦lua-resty-checkups (*) 
✦lua-upstream-nginx-module
Upstream Health Check: 
lua-resty-checkups 
-- app/etc/config.lua 
_M.global = { 
checkup_timer_interval = 5, 
checkup_timer_overtime = 60, 
} 
_M.api = { 
timeout = 2, 
typ = "general", -- http, redis, mysql etc. 
cluster = { 
{ -- level 1 
try = 2, 
servers = { 
{ host = "127.0.0.1", port = 12354 }, 
{ host = "127.0.0.1", port = 12355 }, 
{ host = "127.0.0.1", port = 12356 }, 
} 
}, 
{ -- level 2 
servers = { 
{ host = "127.0.0.1", port = 12360 }, 
{ host = "127.0.0.1", port = 12361 }, 
} 
}, 
}, 
}
Upstream Health Check: 
checkups with nginx.conf 
-- app/etc/config.lua 
_M.global = { 
checkup_timer_interval = 5, 
checkup_timer_overtime = 60, 
ups_status_sync_enable = true, 
ups_status_timer_interval = 2, 
} 
_M.api = { 
cluster = { 
{ -- level 1 
try = 2, 
upstream = "api.com", 
}, 
{ -- level 2 
upstream = "api.com", 
upstream_only_backup = true, 
}, 
}, 
} 
# nginx/conf/upstream.conf 
upstream api.com { 
server 127.0.0.1:12354; 
server 127.0.0.1:12355; 
server 127.0.0.1:12356; 
server 127.0.0.1:12360 backup; 
server 127.0.0.1:12361 backup; 
} 
type status timer interval 
checkup shm_zone global 5s 
upstream 
per 
worker 
per 
worker 
2s
nginx.conf service 
server_name *.b0.upaiyun.com Custom Domain Binding 
valid_referers, rewrite, allow, deny Custom Antileech Rules and Redirect: 
ip, user-agent, referer, token etc. 
expires 7d Custom Expires Time: 
support specific URI rules etc. 
ssl_certificate* Custom SSL Certificates Load 
upstream { server 127.0.0.1 } 
Custom CDN Source: 
support multi-network routing etc. 
max_fails=3 fail_timeout=30s 
health_check (*) 
Custom Health Check Strategy: 
passive, active 
round-robin, ip_hash, hash (1.7.2+) Custom Load Balancing Strategy 
… …
nginx.conf as a service 
powered by ngx_lua
UPYUN DevOps 
conf hash + project version + upyun.cfg 
Ansible Playbook 
✦ rsync code and binary 
✦ conf template instantiate 
✦ kill -HUP `cat /var/run/nginx.pid` (*)
Lua CDN 
Lua WAF 
Lua SSL
Join our team 
©2012-2014 Trybiane
A Systems Engineer at UPYUN 
★Email: timebug.info@gmail.com 
★Github: https://github.com/timebug
Nginx ngx_lua agentzh Lua 
blog.cloudflare.com Openresty LuaJIT Github 
Maxim Dounin Igor Sysoev chaoslawful 
https://groups.google.com/forum/#!forum/OpenResty 
Ansible Michael Pall Open source 
Thanks 
…
Q & A

More Related Content

What's hot

Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteBram Vogelaar
 
Redis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationRedis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationrjsmelo
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queueBrandon Lamb
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in VaultGlynnForrest
 
Securing Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp VaultSecuring Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp VaultBram Vogelaar
 
Observability with Consul Connect
Observability with Consul ConnectObservability with Consul Connect
Observability with Consul ConnectBram Vogelaar
 
Puppet and the HashiStack
Puppet and the HashiStackPuppet and the HashiStack
Puppet and the HashiStackBram Vogelaar
 
Testing your infrastructure with litmus
Testing your infrastructure with litmusTesting your infrastructure with litmus
Testing your infrastructure with litmusBram Vogelaar
 
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
Using Node.js to  Build Great  Streaming Services - HTML5 Dev ConfUsing Node.js to  Build Great  Streaming Services - HTML5 Dev Conf
Using Node.js to Build Great Streaming Services - HTML5 Dev ConfTom Croucher
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxyIsmael Celis
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stackBram Vogelaar
 
Новый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныНовый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныTimur Safin
 
glance replicator
glance replicatorglance replicator
glance replicatoririx_jp
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyTim Bunce
 
Autoscaling with hashi_corp_nomad
Autoscaling with hashi_corp_nomadAutoscaling with hashi_corp_nomad
Autoscaling with hashi_corp_nomadBram Vogelaar
 
Facebook的缓存系统
Facebook的缓存系统Facebook的缓存系统
Facebook的缓存系统yiditushe
 
Failsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageFailsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageKit Chan
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLitecharsbar
 

What's hot (20)

Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
 
Redis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationRedis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your application
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queue
 
Static Typing in Vault
Static Typing in VaultStatic Typing in Vault
Static Typing in Vault
 
Securing Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp VaultSecuring Prometheus exporters using HashiCorp Vault
Securing Prometheus exporters using HashiCorp Vault
 
Observability with Consul Connect
Observability with Consul ConnectObservability with Consul Connect
Observability with Consul Connect
 
Puppet and the HashiStack
Puppet and the HashiStackPuppet and the HashiStack
Puppet and the HashiStack
 
Testing your infrastructure with litmus
Testing your infrastructure with litmusTesting your infrastructure with litmus
Testing your infrastructure with litmus
 
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
Using Node.js to  Build Great  Streaming Services - HTML5 Dev ConfUsing Node.js to  Build Great  Streaming Services - HTML5 Dev Conf
Using Node.js to Build Great Streaming Services - HTML5 Dev Conf
 
Node.js streaming csv downloads proxy
Node.js streaming csv downloads proxyNode.js streaming csv downloads proxy
Node.js streaming csv downloads proxy
 
Bootstrapping multidc observability stack
Bootstrapping multidc observability stackBootstrapping multidc observability stack
Bootstrapping multidc observability stack
 
Top Node.js Metrics to Watch
Top Node.js Metrics to WatchTop Node.js Metrics to Watch
Top Node.js Metrics to Watch
 
Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)
 
Новый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныНовый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоны
 
glance replicator
glance replicatorglance replicator
glance replicator
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
 
Autoscaling with hashi_corp_nomad
Autoscaling with hashi_corp_nomadAutoscaling with hashi_corp_nomad
Autoscaling with hashi_corp_nomad
 
Facebook的缓存系统
Facebook的缓存系统Facebook的缓存系统
Facebook的缓存系统
 
Failsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo HomepageFailsafe Mechanism for Yahoo Homepage
Failsafe Mechanism for Yahoo Homepage
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLite
 

Viewers also liked

lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用hugo
 
Infinit demo day pitch at Techstars NYC'14
Infinit demo day pitch at Techstars NYC'14Infinit demo day pitch at Techstars NYC'14
Infinit demo day pitch at Techstars NYC'14Infinit
 
基于OpenResty的百万级长连接推送
基于OpenResty的百万级长连接推送基于OpenResty的百万级长连接推送
基于OpenResty的百万级长连接推送OpenRestyCon
 
Httpool_WeTransfer_2016_ANG
Httpool_WeTransfer_2016_ANGHttpool_WeTransfer_2016_ANG
Httpool_WeTransfer_2016_ANGNastja Breg
 
Nginx Scripting - Extending Nginx Functionalities with Lua
Nginx Scripting - Extending Nginx Functionalities with LuaNginx Scripting - Extending Nginx Functionalities with Lua
Nginx Scripting - Extending Nginx Functionalities with LuaTony Fabeen
 
AWS Summit Benelux 2013 - Media and Online Advertising on AWS
AWS Summit Benelux 2013 - Media and Online Advertising on AWSAWS Summit Benelux 2013 - Media and Online Advertising on AWS
AWS Summit Benelux 2013 - Media and Online Advertising on AWSAmazon Web Services
 
Lua: the world's most infuriating language
Lua: the world's most infuriating languageLua: the world's most infuriating language
Lua: the world's most infuriating languagejgrahamc
 
Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...
Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...
Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...Ontico
 
Using ngx_lua / lua-nginx-module in pixiv
Using ngx_lua / lua-nginx-module in pixivUsing ngx_lua / lua-nginx-module in pixiv
Using ngx_lua / lua-nginx-module in pixivShunsuke Michii
 
MySQL Replication: What’s New in MySQL 5.7 and Beyond
MySQL Replication: What’s New in MySQL 5.7 and BeyondMySQL Replication: What’s New in MySQL 5.7 and Beyond
MySQL Replication: What’s New in MySQL 5.7 and BeyondAndrew Morgan
 
Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBleesjensen
 
텐서플로 걸음마 (TensorFlow Tutorial)
텐서플로 걸음마 (TensorFlow Tutorial)텐서플로 걸음마 (TensorFlow Tutorial)
텐서플로 걸음마 (TensorFlow Tutorial)Taejun Kim
 
Introduction to Zabbix - Company, Product, Services and Use Cases
Introduction to Zabbix - Company, Product, Services and Use CasesIntroduction to Zabbix - Company, Product, Services and Use Cases
Introduction to Zabbix - Company, Product, Services and Use CasesZabbix
 
Introduction to Machine Learning and Deep Learning
Introduction to Machine Learning and Deep LearningIntroduction to Machine Learning and Deep Learning
Introduction to Machine Learning and Deep LearningTerry Taewoong Um
 
The Barcelona Tech Startup Guide
The Barcelona Tech Startup GuideThe Barcelona Tech Startup Guide
The Barcelona Tech Startup GuideStartup Tour
 
The Amsterdam Tech Startup Guide
The Amsterdam Tech Startup GuideThe Amsterdam Tech Startup Guide
The Amsterdam Tech Startup GuideStartup Tour
 
OpenResty/Lua Practical Experience
OpenResty/Lua Practical ExperienceOpenResty/Lua Practical Experience
OpenResty/Lua Practical ExperienceHo Kim
 

Viewers also liked (20)

lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用
 
Infinit demo day pitch at Techstars NYC'14
Infinit demo day pitch at Techstars NYC'14Infinit demo day pitch at Techstars NYC'14
Infinit demo day pitch at Techstars NYC'14
 
Nginx lua
Nginx luaNginx lua
Nginx lua
 
基于OpenResty的百万级长连接推送
基于OpenResty的百万级长连接推送基于OpenResty的百万级长连接推送
基于OpenResty的百万级长连接推送
 
We transfer
We transferWe transfer
We transfer
 
Httpool_WeTransfer_2016_ANG
Httpool_WeTransfer_2016_ANGHttpool_WeTransfer_2016_ANG
Httpool_WeTransfer_2016_ANG
 
Nginx Scripting - Extending Nginx Functionalities with Lua
Nginx Scripting - Extending Nginx Functionalities with LuaNginx Scripting - Extending Nginx Functionalities with Lua
Nginx Scripting - Extending Nginx Functionalities with Lua
 
Practical ngx_mruby
Practical ngx_mrubyPractical ngx_mruby
Practical ngx_mruby
 
AWS Summit Benelux 2013 - Media and Online Advertising on AWS
AWS Summit Benelux 2013 - Media and Online Advertising on AWSAWS Summit Benelux 2013 - Media and Online Advertising on AWS
AWS Summit Benelux 2013 - Media and Online Advertising on AWS
 
Lua: the world's most infuriating language
Lua: the world's most infuriating languageLua: the world's most infuriating language
Lua: the world's most infuriating language
 
Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...
Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...
Как и зачем создавать NginX-модуль - теория, практика, профит / Василий Сошни...
 
Using ngx_lua / lua-nginx-module in pixiv
Using ngx_lua / lua-nginx-module in pixivUsing ngx_lua / lua-nginx-module in pixiv
Using ngx_lua / lua-nginx-module in pixiv
 
MySQL Replication: What’s New in MySQL 5.7 and Beyond
MySQL Replication: What’s New in MySQL 5.7 and BeyondMySQL Replication: What’s New in MySQL 5.7 and Beyond
MySQL Replication: What’s New in MySQL 5.7 and Beyond
 
Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDB
 
텐서플로 걸음마 (TensorFlow Tutorial)
텐서플로 걸음마 (TensorFlow Tutorial)텐서플로 걸음마 (TensorFlow Tutorial)
텐서플로 걸음마 (TensorFlow Tutorial)
 
Introduction to Zabbix - Company, Product, Services and Use Cases
Introduction to Zabbix - Company, Product, Services and Use CasesIntroduction to Zabbix - Company, Product, Services and Use Cases
Introduction to Zabbix - Company, Product, Services and Use Cases
 
Introduction to Machine Learning and Deep Learning
Introduction to Machine Learning and Deep LearningIntroduction to Machine Learning and Deep Learning
Introduction to Machine Learning and Deep Learning
 
The Barcelona Tech Startup Guide
The Barcelona Tech Startup GuideThe Barcelona Tech Startup Guide
The Barcelona Tech Startup Guide
 
The Amsterdam Tech Startup Guide
The Amsterdam Tech Startup GuideThe Amsterdam Tech Startup Guide
The Amsterdam Tech Startup Guide
 
OpenResty/Lua Practical Experience
OpenResty/Lua Practical ExperienceOpenResty/Lua Practical Experience
OpenResty/Lua Practical Experience
 

Similar to Using ngx_lua in UPYUN

Debugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 VersionDebugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 VersionIan Barber
 
Debugging: Rules & Tools
Debugging: Rules & ToolsDebugging: Rules & Tools
Debugging: Rules & ToolsIan Barber
 
Vagrant, Ansible, and OpenStack on your laptop
Vagrant, Ansible, and OpenStack on your laptopVagrant, Ansible, and OpenStack on your laptop
Vagrant, Ansible, and OpenStack on your laptopLorin Hochstein
 
X64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 newX64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 newYiwei Ma
 
Burn down the silos! Helping dev and ops gel on high availability websites
Burn down the silos! Helping dev and ops gel on high availability websitesBurn down the silos! Helping dev and ops gel on high availability websites
Burn down the silos! Helping dev and ops gel on high availability websitesLindsay Holmwood
 
Spring data iii
Spring data iiiSpring data iii
Spring data iii명철 강
 
Gazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapmGazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapmMasahiro Nagano
 
Railsconf2011 deployment tips_for_slideshare
Railsconf2011 deployment tips_for_slideshareRailsconf2011 deployment tips_for_slideshare
Railsconf2011 deployment tips_for_slidesharetomcopeland
 
Kansai.pm 10周年記念 Plack/PSGI 入門
Kansai.pm 10周年記念 Plack/PSGI 入門Kansai.pm 10周年記念 Plack/PSGI 入門
Kansai.pm 10周年記念 Plack/PSGI 入門lestrrat
 
When symfony met promises
When symfony met promises When symfony met promises
When symfony met promises Marc Morera
 
Introduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 TokyoIntroduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 TokyoMasahiro Nagano
 
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)PROIDEA
 
Practical Chef and Capistrano for Your Rails App
Practical Chef and Capistrano for Your Rails AppPractical Chef and Capistrano for Your Rails App
Practical Chef and Capistrano for Your Rails AppSmartLogic
 
Marrow: A Meta-Framework for Python 2.6+ and 3.1+
Marrow: A Meta-Framework for Python 2.6+ and 3.1+Marrow: A Meta-Framework for Python 2.6+ and 3.1+
Marrow: A Meta-Framework for Python 2.6+ and 3.1+ConFoo
 

Similar to Using ngx_lua in UPYUN (20)

Debugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 VersionDebugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 Version
 
Debugging: Rules & Tools
Debugging: Rules & ToolsDebugging: Rules & Tools
Debugging: Rules & Tools
 
EC2
EC2EC2
EC2
 
Vagrant, Ansible, and OpenStack on your laptop
Vagrant, Ansible, and OpenStack on your laptopVagrant, Ansible, and OpenStack on your laptop
Vagrant, Ansible, and OpenStack on your laptop
 
X64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 newX64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 new
 
Burn down the silos! Helping dev and ops gel on high availability websites
Burn down the silos! Helping dev and ops gel on high availability websitesBurn down the silos! Helping dev and ops gel on high availability websites
Burn down the silos! Helping dev and ops gel on high availability websites
 
Spring data iii
Spring data iiiSpring data iii
Spring data iii
 
Ubic
UbicUbic
Ubic
 
Ubic-public
Ubic-publicUbic-public
Ubic-public
 
Puppet @ Seat
Puppet @ SeatPuppet @ Seat
Puppet @ Seat
 
Gazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapmGazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapm
 
Railsconf2011 deployment tips_for_slideshare
Railsconf2011 deployment tips_for_slideshareRailsconf2011 deployment tips_for_slideshare
Railsconf2011 deployment tips_for_slideshare
 
Kansai.pm 10周年記念 Plack/PSGI 入門
Kansai.pm 10周年記念 Plack/PSGI 入門Kansai.pm 10周年記念 Plack/PSGI 入門
Kansai.pm 10周年記念 Plack/PSGI 入門
 
Puppet Camp 2012
Puppet Camp 2012Puppet Camp 2012
Puppet Camp 2012
 
When symfony met promises
When symfony met promises When symfony met promises
When symfony met promises
 
Introduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 TokyoIntroduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 Tokyo
 
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
JDD 2017: Nginx + Lua = OpenResty (Marcin Stożek)
 
Puppet
PuppetPuppet
Puppet
 
Practical Chef and Capistrano for Your Rails App
Practical Chef and Capistrano for Your Rails AppPractical Chef and Capistrano for Your Rails App
Practical Chef and Capistrano for Your Rails App
 
Marrow: A Meta-Framework for Python 2.6+ and 3.1+
Marrow: A Meta-Framework for Python 2.6+ and 3.1+Marrow: A Meta-Framework for Python 2.6+ and 3.1+
Marrow: A Meta-Framework for Python 2.6+ and 3.1+
 

Recently uploaded

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimizationarrow10202532yuvraj
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 

Recently uploaded (20)

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
UiPath Clipboard AI: "A TIME Magazine Best Invention of 2023 Unveiled"
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 

Using ngx_lua in UPYUN

  • 1. Using ngx_lua in UPYUN Monkey Zhang (timebug) 2014.11 @ Beijing OSC
  • 2. Hello Nginx nginx [engine x] is an HTTP and reverse proxy server, as well as a mail proxy server, written by Igor Sysoev.
  • 3. A simple example $ ./configure --prefix=/opt/nginx --add-module=/path/to/echo-nginx-module http { server { listen 8080; location /hello { set $foo "hello"; echo $foo; set $foo "world"; echo $foo; } } } $ curl http://localhost:8080/hello world world
  • 4. In Fact … http { server { listen 8080; location /hello { set $foo "hello"; set $foo "world"; echo $foo; echo $foo; } } } ✦ REWRITE PHASE: set (ngx_http_rewrite_module) ✦ CONTENT PHASE: echo (echo-nginx-module)
  • 5. Nginx Internals: HTTP request phase handlers ✦ POST READ PHASE ✦ SERVER REWRITE PHASE ✦ FIND CONFIG PHASE ✦ REWRITE PHASE ✦ POST REWRITE PHASE ✦ PRE ACCESS PHASE ✦ ACCESS PHASE ✦ POST ACCESS PHASE ✦ TRY FILES PHASE ✦ CONTENT PHASE ✦ LOG PHASE
  • 6. Why Nginx Module Development is Not Easy ?
  • 7. A true story $ ./configure --prefix=/opt/nginx --add-module=/path/to/echo-nginx-module --add-module=/path/to/base64-nginx-module http { server { listen 8080; location /base64 { base64 on; base64_max_length 10485760; echo "hello world"; } } } $ curl http://localhost:8080/base64 aGVsbG8gd29ybGQK
  • 9. If is Evil http://agentzh.blogspot.jp/2011/03/how-nginx-location-if-works.html http { server { listen 8080; location /if { set $foo 1; if ($foo = 1) { set $foo 2; echo "foo = $foo"; } set $foo 3; proxy_pass http://127.0.0.1:$server_port/$foo; } location ~ /(d+) { echo "bar = $1"; } } } $ curl http://localhost:8080/if foo = 3
  • 10. If is Evil: How it works ✦ REWRITE PHASE set $foo 1; if ($foo = 1) { set $foo 2; } set $foo 3; ✦ CONTENT PHASE if ($foo = 1) { echo "foo = $foo"; }
  • 11. If is Evil: Break ngx_rewite Directives ✦ REWRITE PHASE set $foo 1; if ($foo = 1) { set $foo 2; break; } ✦ CONTENT PHASE if ($foo = 1) { echo "foo = $foo"; } http { server { listen 8080; location /if { set $foo 1; if ($foo = 1) { set $foo 2; break; echo "foo = $foo"; } set $foo 3; proxy_pass http://127.0.0.1:$server_port/$foo; } location ~ /(d+) { echo "bar = $1"; } } } $ curl http://localhost:8080/if foo = 2
  • 12. Hello Lua Lua is a powerful, fast, lightweight, embeddable scripting language.
  • 13. $ ./configure --prefix=/opt/nginx --add-module=/path/to/lua-nginx-module http { server { listen 8080; location /add { set $res ''; rewrite_by_lua ' local a = tonumber(ngx.var.arg_a) or 0 local b = tonumber(ngx.var.arg_b) or 0 ngx.var.res = a + b '; content_by_lua ' ngx.say(ngx.var.res) '; } } } $ curl 'http://localhost:8080/add?a=6&b=7' 13
  • 14. LuaJIT LuaJIT is a Just-In-Time Compiler (JIT) for the Lua programming language.
  • 15. How it works LuaJIT VM embedded into the Nginx
  • 16. Base64 Filter by Lua http { lua_package_path “$prefix/app/src/?.lua;;"; server { listen 8080; location /base64 { set $b64_en ''; set $b64_e0 ''; set $b64_e1 ''; echo_duplicate 1000 hello; header_filter_by_lua ' ngx.header.content_length = nil -- ((n + 2) / 3 ) * 4 ngx.header.content_type = "text/plain" ngx.header.content_transfer_encoding = "base64" '; body_filter_by_lua_file app/src/b64_body_filter.lua; } } }
  • 17. Base64 Filter by Lua: local chunk = ngx.arg[1] Chunk by Chunk local e0 = ngx.var.b64_e0 or '' local e1 = ngx.var.b64_e1 or '' local en = tonumber(ngx.var.b64_en) or 0 if en == 1 then chunk = e0 .. chunk elseif en == 2 then chunk = e0 .. e1 .. chunk end if not ngx.arg[2] then en = #chunk % 3 if en == 1 then e0 = chunk:sub(-1) elseif en == 2 then e1 = chunk:sub(-1) e0 = chunk:sub(-2, -2) end chunk = chunk:sub(1, #chunk - en) else -- eof en = 0 end ngx.var.b64_en = en ngx.var.b64_e0 = e0 ngx.var.b64_e1 = e1 ngx.arg[1] = ngx.encode_base64(chunk)
  • 19. ★ngx_upreferer_module.c ~ 2100 C ★src/modules/referer.lua ~ 500 Lua ngx.md5 ngx.time ngx.re.* ngx.req.* ngx.decode_args string.sub string.find string.byte table.concat
  • 20. based on Lua coroutines & synchronous & 100% non-blocking cosocket API ngx.socket.* connect send receive sslhandshake close settimeout etc.
  • 21. A true story: Yupoo Referer Redirect (Year 2012) eval_escalate off; eval_override_content_type text/plain; eval $answer { set $redis_key "$scheme://<key>"; redis_pass redis; } if ($answer = "101") { rewrite ^ http://r.yupoo.com/101.gif redirect; break; } if ($answer = "102") { rewrite ^ http://r.yupoo.com/102.gif redirect; break; } if ($answer ~ "^http://") { rewrite ^ $answer redirect; break; }
  • 22. WTF! ✦Fork ngx_http_redis to support ypacl command ✦vkholodkov/nginx-eval-module last commit on Nov 26, 2010
  • 23. when upgrade Nginx to the latest version Coredump
  • 24. Yupoo Referer Redirect by Lua lua-resty-redis (based on the cosocket API) ngx.redirect rewrite_by_lua_file Local redis = require "resty.redis" local red = redis:new() redis.add_commands("ypacl") -- set_timeout and connect local res, err = red:ypacl(key) if res == "101" then return ngx.redirect("http://r.yupoo.com/101.gif") else -- do something else end
  • 26. lua-resty-uuid: Based on LuaJIT FFI -- modified version of original pull request by smallfish -- https://github.com/openresty/lua-resty-string/pull/7 local ffi = require "ffi" local new = ffi.new local string = ffi.string local _M = {} ffi.cdef[[ typedef unsigned char uuid_t[16]; void uuid_generate(uuid_t out); void uuid_unparse(const uuid_t uu, char *out); ]] local libuuid = ffi.load("libuuid") function _M.generate() if libuuid then local uuid = new("uuid_t") local result = new("char[36]") libuuid.uuid_generate(uuid) libuuid.uuid_unparse(uuid, result) return string(result) end end return _M
  • 27. Openresty Yichun "agentzh" Zhang (章亦春) agentzh@gmail.com, CloudFlare Inc.
  • 30. UPYUN CDN: Metadata Cache ✦CJSON, MessagePack ✦ngx.shared.DICT ✦lua-resty-lock ✦lua-resty-shcache ✦lua-resty-lrucache (*)
  • 31. Metadata Cache: The original version (Year 2012) Issues local metadata = ngx.shared.metadata -- local key, bucket = ... local value = metadata:get(key) if value ~= nil then if value == "404" then return -- HIT_NEGATIVE else return value -- HIT end end local rds = redis:new() local ok, err = rds:connect("127.0.0.1", 6379) if not ok then metadata:set(key, "404", 120) -- expires 2 minutes return -- NO_DATA end res, err = rds:hget("upyun:" .. bucket, ":something") if not res or res == ngx.null then metadata:set(key, "404", 120) return -- NO_DATA end metadata:set(key, res, 300) -- expires 5 minutes rds:set_keepalive() return res -- MISS ✦ "dog-pile effect" ✦ NO_DATA when redis crash ✦ code inflexible
  • 32. Metadata Cache: lua-resty-shcache ✦ cache locks (based on lua-resty-lock) ✦ serialization / de-serialization ✦ external lookup via Lua closure ✦ MISS, HIT, HIT_NEGATIVE, STALE, NET_ERR -- app/src/modules/metadata.lua local shcache = require "resty.shcache" function _M.get_metadata(bucket) local lookup_metadata = function () -- fetch from redis return res end local cache_data = shcache:new( ngx.shared.metadata, { external_lookup = lookup_metadata, encode = cmsgpack.pack, decode = cmsgpack.unpack, }, { positive_ttl = cache_positive_ttl, negative_ttl = cache_negative_ttl, name = "metadata", }) -- local key = ... local data, _ = cache_data:load(key) if not data then return end return data end
  • 33. Metadata Cache: MessagePack Bucket JSON MessagePack huab*** 6035 bytes 4135 bytes - 69%
  • 34. Metadata Cache: lua-resty-lrucache ✦ based on LuaJIT FFI ✦ HIT_LRU ✦ avoid serialization / de-serialization
  • 35. UPYUN CDN: Upstream Health Check ✦lua-resty-checkups (*) ✦lua-upstream-nginx-module
  • 36. Upstream Health Check: lua-resty-checkups -- app/etc/config.lua _M.global = { checkup_timer_interval = 5, checkup_timer_overtime = 60, } _M.api = { timeout = 2, typ = "general", -- http, redis, mysql etc. cluster = { { -- level 1 try = 2, servers = { { host = "127.0.0.1", port = 12354 }, { host = "127.0.0.1", port = 12355 }, { host = "127.0.0.1", port = 12356 }, } }, { -- level 2 servers = { { host = "127.0.0.1", port = 12360 }, { host = "127.0.0.1", port = 12361 }, } }, }, }
  • 37. Upstream Health Check: checkups with nginx.conf -- app/etc/config.lua _M.global = { checkup_timer_interval = 5, checkup_timer_overtime = 60, ups_status_sync_enable = true, ups_status_timer_interval = 2, } _M.api = { cluster = { { -- level 1 try = 2, upstream = "api.com", }, { -- level 2 upstream = "api.com", upstream_only_backup = true, }, }, } # nginx/conf/upstream.conf upstream api.com { server 127.0.0.1:12354; server 127.0.0.1:12355; server 127.0.0.1:12356; server 127.0.0.1:12360 backup; server 127.0.0.1:12361 backup; } type status timer interval checkup shm_zone global 5s upstream per worker per worker 2s
  • 38. nginx.conf service server_name *.b0.upaiyun.com Custom Domain Binding valid_referers, rewrite, allow, deny Custom Antileech Rules and Redirect: ip, user-agent, referer, token etc. expires 7d Custom Expires Time: support specific URI rules etc. ssl_certificate* Custom SSL Certificates Load upstream { server 127.0.0.1 } Custom CDN Source: support multi-network routing etc. max_fails=3 fail_timeout=30s health_check (*) Custom Health Check Strategy: passive, active round-robin, ip_hash, hash (1.7.2+) Custom Load Balancing Strategy … …
  • 39. nginx.conf as a service powered by ngx_lua
  • 40. UPYUN DevOps conf hash + project version + upyun.cfg Ansible Playbook ✦ rsync code and binary ✦ conf template instantiate ✦ kill -HUP `cat /var/run/nginx.pid` (*)
  • 41. Lua CDN Lua WAF Lua SSL
  • 42. Join our team ©2012-2014 Trybiane
  • 43. A Systems Engineer at UPYUN ★Email: timebug.info@gmail.com ★Github: https://github.com/timebug
  • 44. Nginx ngx_lua agentzh Lua blog.cloudflare.com Openresty LuaJIT Github Maxim Dounin Igor Sysoev chaoslawful https://groups.google.com/forum/#!forum/OpenResty Ansible Michael Pall Open source Thanks …
  • 45. Q & A