Babel Compiler - Transforming JavaScript for All Browsers.pptx
Kauli SSPにおけるVyOSの導入事例
1. Case studies of VyOS
in Kauli SSP
Flandre Scarlet favorite Platform Engineer
Kazuhito Ohkawa
at
Kauli, Inc.
2. Agenda
- Self‐Introduction
- About Kauli SSP
- Case studies in Kauli SSP of VyOS
- Tuning tips
- About microburst traffic(digress)
3. Self‐Introduction
- おおかわ かずひと
Kazuhito Ohkawa
(twitter@SatchanP)
- Aug 2012 Joined Kauli, Inc.
Platform Engineer
- My Lover
THE IDOLM@STER : Yayoi, Mami
Touhou Project : Flandre, Sakuya
- Private Rallyist
This is a my co-driver and
three-dimensional parking of impreza.
6. About SSP
A supply-side platform or sell-side platform (SSP) is a technology platform with the single
mission of enabling publishers to manage their advertising impression inventory and
maximize revenue from digital media. As such, they offer an efficient, automated and secure
way to tap into the different sources of advertising income that are available, and provide
insight into the various revenue streams and audiences. Many of the larger web publishers of
the world use a supply-side platform to automate and optimize the selling of their online
media space.[1]
A supply-side platform on the publisher side interfaces to an ad exchange, which in turn
interfaces to a demand-side platform (DSP) on the advertiser side.
This system allows advertisers to put online advertising before a selected target audience.[2]
Often, real-time bidding (RTB) is used to complete DSP transactions.[3]。
http://en.wikipedia.org/wiki/Supply-side_platform
7. About RTB
Audience
Media
AD
Select the DSP in conditions.
Request in parallel.
Request for SSP
Browse
Kauli connected DSPs
Bid winner is DSP B
Displayed DSP B's AD
8. Many connections for Ad delivery.
Up to 400 million Ad per day.
All traffic via the VyOS.
9. Agony of SSP Platform Engineer
Very very very many many many traffics...
As well internal and external...
Various traffics, cookie sync, banner,
flash and movies, JS tags...etc...
About 80 % traffic is short packet...
Claim for delay of Ad...
SSP isn't profitable! Many media rewards!
13. Mainly running on a physical server
Gen-1
Intel Core i7 870
RAM 16G
Intel 82574L x2
M/B ASUS
HDD
Gen-2
Intel Xeon E3-1280 v3
RAM 32G
Intel I350/I210
M/B Supermicro
SSD
14. Using at the Default Gateway for all servers
Internet
L3 Core
LVS
DR
Real Server
nginx
VyOS
DMZ
Default GW IP Masquarede
LAN
RTB Requests
SSP Server
16. Logic of LVS-DR
SRC : 8.8.4.4
DST : 8.8.8.8
LVS
VIP : 8.8.8.8
Client A
IP : 8.8.4.4
Internet
Real Server
IP : 10.1.1.2
SRC : 8.8.4.4
DST : 8.8.8.8
MAC : 0000.0000.0000
lo : 8.8.8.8
MAC : 0000.0000.0000
VyOS
Default GW
IP : 10.1.1.1
SRC : 8.8.8.8
DST : 8.8.4.4
Source address is
Solved by MAC Address LVS VIP
Make possible by loopback
SRC : 8.8.8.8
DST : 8.8.4.4
LAN
FP Filter off
17. Router is unnecessary, If server have global IPs
SRC : 8.8.4.4
DST : 8.8.8.8
LVS
VIP : 8.8.8.8
Client A
IP : 8.8.4.4
SRC : 8.8.8.8
Internet
DMZ DST : 8.8.4.4
Real Server
IP : 8.8.8.9
SRC : 8.8.4.4
DST : 8.8.8.9
MAC : 0000.0000.0000
lo : 8.8.8.8
MAC : 0000.0000.0000
18. Scaling VyOS router by OSPF/ECMP after replacement
L3 Core
LVS
DR
VyOS VyOS VyOS
Real Server L3 Switch
Default GW
Internet
OSPF ECMP
Other Vlan
Real Server
LVS
DR
19. Checking new data center application by Cloud
Bridge
Vyatta Vyatta
Internet
SSP Server LVS-DR
DB
KVS
Index
Cloud Bridge
SSP Server
New Data Center Old Data Center
DB KVS Index
Internet
20. Sakura cloud between VPN
Internet
Internet
Data Center Sakura Cloud
VyOS VyOS
API Server
IPSec
Crawler Crawler
22. NUMA I/O
NAPI
circular buffer
CPU Affinity
conntrack
23. Use a uni-processor server (NUMA I/O)
PCI Express controller is integrated into the CPU in the sandy
bridge.
High access costs between processors.
or using memory mirroring...
RAM CPU1 CPU2 RAM
PCI Express
NIC
QPI
25. Reconsider the polling of buffer (NAPI)
Buffer overflows even Intel's I350.(Amazing!)
It is set the maximum value at 4096.
Confirmed with ifconfig and ethtool -S.
ifconfig:
RX packets:1215382409979 errors:0 dropped:9836789
overruns:9836789 frame:0
ethtool -S:
rx_no_buffer_count: 220474
26. Change the NAPI kernel parameters
- net.core.netdev_budget
Increase the processing queue.
- net.core.dev_weight
Shorten the polling sensation.
However CPU usage rises.
27. circular buffer
igb is not set to the maximum value.
And too large buffer will cause a delay.
Consider the balance to CPU by NAPI and circular buffers.
# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256
# ethtool -G eth0 rx 4096 tx 4096
28. CPU Affinity
Case of multi-queue, specific cpu core only high load.
Adjust these manually.
$ cat /proc/interrupts | egrep 'eth|CPU'
CPU0 CPU1 CPU2 CPU3
50: 1406514518 0 0 0 PCI-MSI-edge eth0-rx-0
51: 84923776 383727140 0 0 PCI-MSI-edge eth0-tx-0
52: 2951 0 0 0 PCI-MSI-edge eth0
53: 2 31961537 1787069187 0 PCI-MSI-edge eth1-rx-0
54: 1 6218033 0 510452860 PCI-MSI-edge eth1-tx-0
55: 115 0 0 0 PCI-MSI-edge eth1
$ sudo cat /proc/irq/5[0-1,3-4]/smp_affinity
0001
0002
0004
0008
29. conntrack tuning
Here is the essential part in the IP Masquarede.
Maybe 10G-40G class of IP Masquarede also possible.
Established time is very short.
The high cost of connection open and close processing.
Setting value depends on the memory.
30. conntrack parameter
- hash-size
conntrack table hashes.
Processed faster conntracks scan by hashed.
Hash algorithm is chaining scheme.
- table-size
Raw conntrack tables.
- expect-table-size
Use FTP, SIP, H.323...
http://conntrack-tools.netfilter.org/conntrack.html
32. Setting conntrack tables and hash size
- table-size
CONNTRACK_MAX = RAMSIZE (bytes) / 16384 / (x / 32)
x = 32bit or 64bit
- hash-size
tablesize / 8
- expect-table-size
In preference
33. True upper limit of conntrack
Focus on the status of the conntrack table.
[ASSURED] is not dropping from conntrack tables.
Comparison with the [ASSURED] total value and the maximum
value.
Sample:
tcp 6 23 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43708 dport=80 packets=6 bytes=663
src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43708 packets=4 bytes=542 [ASSURED] mark=0 secmark=0
use=2
34. Shorten the timeout of conntrack table
conntrack table is supposed to be used recursively.
But our traffic has very many hosts.
Unable to keep conntrack table.
Short set a time-out so it not overflow conntrack table.
timeout {
icmp 3
other 600
tcp {
close 10
close-wait 1
established 10
fin-wait 10
last-ack 30
syn-recv 60
syn-sent 5
time-wait 3
}
udp {
other 30
stream 10
}
}
36. About microburst traffic
Microburst is not visible, but our network have it.
Can be confirmed by various phenomena.
One example is a packet discard of switchs.
37. Read the signs of microburst
Expand the graph in a narrow range.
Spikes confirm.
38. Read the signs of microburst
This is a poll of 1 minute sensation.
Ave 85 Packets discard/sec = 85Packets * 60 = 5160
5160 packets lost in a moment.
I have prepared a movie today.