1. Inside Sqale’s Backend
Sapporo Ruby Kaigi 2012
Gosuke Miyashita
paperboy&co. Inc.

2. A little bit about me

4. Technical Manager
at
paperboy&co.

5. https://github.com/mizzy
http://mizzy.org/
@gosukenator

6. Inside Sqale’s Backend

7. http://www.facebook.com/sqalejp

8. WARNING
There are little topics
about Ruby in this talk

9. What is Sqale?

11. Cloud Application
Platform like Heroku

12. Architecture Overview

13. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

14. Containers

15. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

16. Virtual Environments
Assigned To Users

17. Similar to Dynos of
Heroku

18. Containers made by
LXC (Linux Containers)

19. EC2 Instance (1 Virtual Machine)
Container Container Container Container Container
for for for for for
user A user A user B user B user B
Container Container Container Container Container
for for for for for
user C user D user D user E user E
Container Container Container Container Container
for for for for for
user E user F user F user F user F

20. Nginx
Unicorn
sshd
supervisrod
on each container

21. Amazon Linux
+
Patched kernel(3.2.16)

22. grsecurity kernel patch
for various restrictions

23. original kernel patches
to restrict tcp port
bind and fork bomb

26. Anti fork bomb patch
makes some changes to
cgroup and fork process

27. See
paperboy-sqale/sqale-patches
on GitHub

28. Web Proxy

29. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

30. HTTP/HTTPS
ELB
nginx nginx
Container Container Container Container Container Container
for for for for for for
user A user B user B user C user C user C

31. nginx
lua-nginx-module
redis2-nginx-module

32. http://lokka-mizzy.sqale.jp/
Which containers?
Redis nginx
host001:8083, host001:8084
or
host001
nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084
Container Container Container Container
for for for for
i4pc-mizzy i4pc-mizzy lokka-mizzy lokka-mizzy

33. nginx.conf (excerpt)
location / {
set $container "";
set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;
proxy_pass http://$container;
}

34. dynamic-proxy.lua (excerpt)
local reply = ngx.location.capture("/redis")
if reply.status ~= ngx.HTTP_OK then
ngx.exit(503)
end
local containers, type =
parser.parse_reply(reply.body)

35. dynamic-proxy.lua (excerpt)
while #containers > 0 do
tmp = table.remove(
containers,
math.random(#containers))
if ngx.shared.downed_containers:get(tmp) then
ngx.log(ngx.DEBUG, tmp .. " is down")
else
container = tmp
break
end
end

36. dynamic-proxy.lua (excerpt)
ngx.var.container = container
ngx.var.next_containers
= luabins.save(containers)

37. nginx.conf (again)
location / {
set $container "";
set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;
proxy_pass http://$container;
}

38. nginx.conf (excerpt)
location @failover {
error_page 502 = @failover;
rewrite_by_lua_file failover.lua;
proxy_pass http://$container;
}

39. failover.lua (excerpt)
local downed_container = ngx.var.container
if downed_container then
ngx.shared.downed_containers:set(
downed_container,
1,
sqale.NEGATIVE_CACHE_SECONDS
)
end

40. failover.lua (excerpt)
while #containers > 0 do
tmp = table.remove(
containers,
math.random(#containers))
if ngx.shared.downed_containers:get(tmp) then
ngx.log(ngx.DEBUG, tmp .. " is down")
else
container = tmp
break
end
end

41. failover.lua (excerpt)
if not container then
ngx.exit(503)
end
ngx.var.container = container
ngx.var.next_containers
= luabins.save(containers)

42. nginx.conf (agin)
location @failover {
error_page 502 = @failover;
rewrite_by_lua_file failover.lua;
proxy_pass http://$container;
}

43. See
http://bit.ly/UHbHIb
by @hiboma

44. SSH Router

45. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

46. Git SFTP SSH Login
SSH Router
File File
Repositories Repositories Containers
(Git Server) (File Server)

47. How implement this
routing?

48. OpenSSH with script
authentication patch

50. See
mizzy/openssh-script-auth
on GitHub

51. Change routes by
SSH_ORIGNAL_COMMAND

53. In case of
SSH_ORIGINAL_COMMAND
is “git-*”

54. git push
(ssh sqale@gateway.sqale.jp git-recieve-pack
‘/mizzy/lokka.git’)
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s git
server
command=“ssh sqale@git001.sqale.lan
git-recieve-pack
File ‘/var/repos/mizzy/lokka.git’”
Repository
(Git Server)

55. In case of
SSH_ORIGINAL_COMMAND
is “sftp-server”

56. sftp sqale@gateway.sqale.jp
(ssh sqale@gateway.sqale.jp sftp-server)
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s file
server
command=“ssh sqale@file001.sqale.lan
sftp-server”
File git push File
Repository Repository
(File Server) (Git Server)

57. In case of
SSH_ORIGINAL_COMMAND
is empty

58. ssh sqale@gateway.sqale.jp
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s
cotainers list
Display the user’s containers list and
wait the user’s selection
command=“ssh sqale@
Container users001.sqale.lan -p 8081”

60. Deploy Servers

61. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

62. Please ask to
@kyanny

63. Other

64. About Sqale’s Server
Build Automation
http://bit.ly/NBbj9F
by @lamanotrama

65. Thanks