SlideShare a Scribd company logo
1 of 39
Download to read offline
Case studies of VyOS 
in Kauli SSP 
Flandre Scarlet favorite Platform Engineer 
Kazuhito Ohkawa 
at 
Kauli, Inc.
Agenda 
- Self‐Introduction 
- About Kauli SSP 
- Case studies in Kauli SSP of VyOS 
- Tuning tips 
- About microburst traffic(digress)
Self‐Introduction 
- おおかわ かずひと 
Kazuhito Ohkawa 
(twitter@SatchanP) 
- Aug 2012 Joined Kauli, Inc. 
Platform Engineer 
- My Lover 
THE IDOLM@STER : Yayoi, Mami 
Touhou Project : Flandre, Sakuya 
- Private Rallyist 
This is a my co-driver and 
three-dimensional parking of impreza.
About Kauli SSP
SSPとは 
SSPとは、「Supply Side Platform」(サプライサイドプラット 
フォーム)の略で、オンライン広告において、広告枠を提供している 
メディア(Webサービス、アプリデベロッパー)など媒体社の広告枠 
販売や広告収益最大化などを支援するツールのこと。 主に、広告の 
インプレッションが発生するごとに最適な広告を自動的に選択し、収 
益性の向上を図るという仕組みが提供されるが、アドネットワーク、 
アドエクスチェンジの一元的管理、リアルタイム入札(RTB)への対 
応など、具体的な提供機能はサービスによって異なる。 
DSP、SSP - SMMLab(ソーシャルメディアマーケティングラボ) 
smmlab.jp/?p=30268
About SSP 
A supply-side platform or sell-side platform (SSP) is a technology platform with the single 
mission of enabling publishers to manage their advertising impression inventory and 
maximize revenue from digital media. As such, they offer an efficient, automated and secure 
way to tap into the different sources of advertising income that are available, and provide 
insight into the various revenue streams and audiences. Many of the larger web publishers of 
the world use a supply-side platform to automate and optimize the selling of their online 
media space.[1] 
A supply-side platform on the publisher side interfaces to an ad exchange, which in turn 
interfaces to a demand-side platform (DSP) on the advertiser side. 
This system allows advertisers to put online advertising before a selected target audience.[2] 
Often, real-time bidding (RTB) is used to complete DSP transactions.[3]。 
http://en.wikipedia.org/wiki/Supply-side_platform
About RTB 
Audience 
Media 
AD 
Select the DSP in conditions. 
Request in parallel. 
Request for SSP 
Browse 
Kauli connected DSPs 
Bid winner is DSP B 
Displayed DSP B's AD
Many connections for Ad delivery. 
Up to 400 million Ad per day. 
All traffic via the VyOS.
Agony of SSP Platform Engineer 
Very very very many many many traffics... 
As well internal and external... 
Various traffics, cookie sync, banner, 
flash and movies, JS tags...etc... 
About 80 % traffic is short packet... 
Claim for delay of Ad... 
SSP isn't profitable! Many media rewards!
SSP Handmade Servers
Infrastructure engineers of SSP. 
I can not recommend!
Case studies in Kauli SSP of 
VyOS
Mainly running on a physical server 
Gen-1 
Intel Core i7 870 
RAM 16G 
Intel 82574L x2 
M/B ASUS 
HDD 
Gen-2 
Intel Xeon E3-1280 v3 
RAM 32G 
Intel I350/I210 
M/B Supermicro 
SSD
Using at the Default Gateway for all servers 
Internet 
L3 Core 
LVS 
DR 
Real Server 
nginx 
VyOS 
DMZ 
Default GW IP Masquarede 
LAN 
RTB Requests 
SSP Server
Peak traffic graphs of Default Gateway
Logic of LVS-DR 
SRC : 8.8.4.4 
DST : 8.8.8.8 
LVS 
VIP : 8.8.8.8 
Client A 
IP : 8.8.4.4 
Internet 
Real Server 
IP : 10.1.1.2 
SRC : 8.8.4.4 
DST : 8.8.8.8 
MAC : 0000.0000.0000 
lo : 8.8.8.8 
MAC : 0000.0000.0000 
VyOS 
Default GW 
IP : 10.1.1.1 
SRC : 8.8.8.8 
DST : 8.8.4.4 
Source address is 
Solved by MAC Address LVS VIP 
Make possible by loopback 
SRC : 8.8.8.8 
DST : 8.8.4.4 
LAN 
FP Filter off
Router is unnecessary, If server have global IPs 
SRC : 8.8.4.4 
DST : 8.8.8.8 
LVS 
VIP : 8.8.8.8 
Client A 
IP : 8.8.4.4 
SRC : 8.8.8.8 
Internet 
DMZ DST : 8.8.4.4 
Real Server 
IP : 8.8.8.9 
SRC : 8.8.4.4 
DST : 8.8.8.9 
MAC : 0000.0000.0000 
lo : 8.8.8.8 
MAC : 0000.0000.0000
Scaling VyOS router by OSPF/ECMP after replacement 
L3 Core 
LVS 
DR 
VyOS VyOS VyOS 
Real Server L3 Switch 
Default GW 
Internet 
OSPF ECMP 
Other Vlan 
Real Server 
LVS 
DR
Checking new data center application by Cloud 
Bridge 
Vyatta Vyatta 
Internet 
SSP Server LVS-DR 
DB 
KVS 
Index 
Cloud Bridge 
SSP Server 
New Data Center Old Data Center 
DB KVS Index 
Internet
Sakura cloud between VPN 
Internet 
Internet 
Data Center Sakura Cloud 
VyOS VyOS 
API Server 
IPSec 
Crawler Crawler
Tuning Tips
NUMA I/O 
NAPI 
circular buffer 
CPU Affinity 
conntrack
Use a uni-processor server (NUMA I/O) 
PCI Express controller is integrated into the CPU in the sandy 
bridge. 
High access costs between processors. 
or using memory mirroring... 
RAM CPU1 CPU2 RAM 
PCI Express 
NIC 
QPI
It is printed on motherbord
Reconsider the polling of buffer (NAPI) 
Buffer overflows even Intel's I350.(Amazing!) 
It is set the maximum value at 4096. 
Confirmed with ifconfig and ethtool -S. 
ifconfig: 
RX packets:1215382409979 errors:0 dropped:9836789 
overruns:9836789 frame:0 
ethtool -S: 
rx_no_buffer_count: 220474
Change the NAPI kernel parameters 
- net.core.netdev_budget 
Increase the processing queue. 
- net.core.dev_weight 
Shorten the polling sensation. 
However CPU usage rises.
circular buffer 
igb is not set to the maximum value. 
And too large buffer will cause a delay. 
Consider the balance to CPU by NAPI and circular buffers. 
# ethtool -g eth0 
Ring parameters for eth0: 
Pre-set maximums: 
RX: 4096 
RX Mini: 0 
RX Jumbo: 0 
TX: 4096 
Current hardware settings: 
RX: 256 
RX Mini: 0 
RX Jumbo: 0 
TX: 256 
# ethtool -G eth0 rx 4096 tx 4096
CPU Affinity 
Case of multi-queue, specific cpu core only high load. 
Adjust these manually. 
$ cat /proc/interrupts | egrep 'eth|CPU' 
CPU0 CPU1 CPU2 CPU3 
50: 1406514518 0 0 0 PCI-MSI-edge eth0-rx-0 
51: 84923776 383727140 0 0 PCI-MSI-edge eth0-tx-0 
52: 2951 0 0 0 PCI-MSI-edge eth0 
53: 2 31961537 1787069187 0 PCI-MSI-edge eth1-rx-0 
54: 1 6218033 0 510452860 PCI-MSI-edge eth1-tx-0 
55: 115 0 0 0 PCI-MSI-edge eth1 
$ sudo cat /proc/irq/5[0-1,3-4]/smp_affinity 
0001 
0002 
0004 
0008
conntrack tuning 
Here is the essential part in the IP Masquarede. 
Maybe 10G-40G class of IP Masquarede also possible. 
Established time is very short. 
The high cost of connection open and close processing. 
Setting value depends on the memory.
conntrack parameter 
- hash-size 
conntrack table hashes. 
Processed faster conntracks scan by hashed. 
Hash algorithm is chaining scheme. 
- table-size 
Raw conntrack tables. 
- expect-table-size 
Use FTP, SIP, H.323... 
http://conntrack-tools.netfilter.org/conntrack.html
Raw conntrack table samples 
tcp 6 128 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43860 dport=80 packets=6 
bytes=698 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43860 packets=4 bytes=419 
[ASSURED] mark=0 secmark=0 use=2
Setting conntrack tables and hash size 
- table-size 
CONNTRACK_MAX = RAMSIZE (bytes) / 16384 / (x / 32) 
x = 32bit or 64bit 
- hash-size 
tablesize / 8 
- expect-table-size 
In preference
True upper limit of conntrack 
Focus on the status of the conntrack table. 
[ASSURED] is not dropping from conntrack tables. 
Comparison with the [ASSURED] total value and the maximum 
value. 
Sample: 
tcp 6 23 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43708 dport=80 packets=6 bytes=663 
src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43708 packets=4 bytes=542 [ASSURED] mark=0 secmark=0 
use=2
Shorten the timeout of conntrack table 
conntrack table is supposed to be used recursively. 
But our traffic has very many hosts. 
Unable to keep conntrack table. 
Short set a time-out so it not overflow conntrack table. 
timeout { 
icmp 3 
other 600 
tcp { 
close 10 
close-wait 1 
established 10 
fin-wait 10 
last-ack 30 
syn-recv 60 
syn-sent 5 
time-wait 3 
} 
udp { 
other 30 
stream 10 
} 
}
Microburst traffic 
(digress)
About microburst traffic 
Microburst is not visible, but our network have it. 
Can be confirmed by various phenomena. 
One example is a packet discard of switchs.
Read the signs of microburst 
Expand the graph in a narrow range. 
Spikes confirm.
Read the signs of microburst 
This is a poll of 1 minute sensation. 
Ave 85 Packets discard/sec = 85Packets * 60 = 5160 
5160 packets lost in a moment. 
I have prepared a movie today.
Thank you for your attention!

More Related Content

What's hot

(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per SecondAmazon Web Services
 
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...DataStax
 
Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...
Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...
Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...DataStax Academy
 
Seastar @ SF/BA C++UG
Seastar @ SF/BA C++UGSeastar @ SF/BA C++UG
Seastar @ SF/BA C++UGAvi Kivity
 
Seastar at Linux Foundation Collaboration Summit
Seastar at Linux Foundation Collaboration SummitSeastar at Linux Foundation Collaboration Summit
Seastar at Linux Foundation Collaboration SummitDon Marti
 
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...DataStax
 
Build an affordable Cloud Stroage
Build an affordable Cloud StroageBuild an affordable Cloud Stroage
Build an affordable Cloud StroageAlex Lau
 
(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014
(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014
(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014Amazon Web Services
 
Couch to OpenStack: Nova - July, 30, 2013
Couch to OpenStack: Nova - July, 30, 2013Couch to OpenStack: Nova - July, 30, 2013
Couch to OpenStack: Nova - July, 30, 2013Trevor Roberts Jr.
 
On MongoDB backup
On MongoDB backupOn MongoDB backup
On MongoDB backupWilliam Yeh
 
Solr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the UglySolr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the UglySematext Group, Inc.
 
OOPs, OOMs, oh my! Containerizing JVM apps
OOPs, OOMs, oh my! Containerizing JVM appsOOPs, OOMs, oh my! Containerizing JVM apps
OOPs, OOMs, oh my! Containerizing JVM appsSematext Group, Inc.
 
[233] level 2 network programming using packet ngin rtos
[233] level 2 network programming using packet ngin rtos[233] level 2 network programming using packet ngin rtos
[233] level 2 network programming using packet ngin rtosNAVER D2
 
Container Orchestration with Amazon ECS
Container Orchestration with Amazon ECSContainer Orchestration with Amazon ECS
Container Orchestration with Amazon ECSAmazon Web Services
 
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...Ontico
 
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)Ontico
 
openSUSE storage workshop 2016
openSUSE storage workshop 2016openSUSE storage workshop 2016
openSUSE storage workshop 2016Alex Lau
 
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico
 
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixXPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixThe Linux Foundation
 
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화OpenStack Korea Community
 

What's hot (20)

(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second
 
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
 
Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...
Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...
Cassandra Summit 2014: Down with Tweaking! Removing Tunable Complexity for Ca...
 
Seastar @ SF/BA C++UG
Seastar @ SF/BA C++UGSeastar @ SF/BA C++UG
Seastar @ SF/BA C++UG
 
Seastar at Linux Foundation Collaboration Summit
Seastar at Linux Foundation Collaboration SummitSeastar at Linux Foundation Collaboration Summit
Seastar at Linux Foundation Collaboration Summit
 
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
 
Build an affordable Cloud Stroage
Build an affordable Cloud StroageBuild an affordable Cloud Stroage
Build an affordable Cloud Stroage
 
(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014
(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014
(SDD404) Amazon RDS for Microsoft SQL Server Deep Dive | AWS re:Invent 2014
 
Couch to OpenStack: Nova - July, 30, 2013
Couch to OpenStack: Nova - July, 30, 2013Couch to OpenStack: Nova - July, 30, 2013
Couch to OpenStack: Nova - July, 30, 2013
 
On MongoDB backup
On MongoDB backupOn MongoDB backup
On MongoDB backup
 
Solr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the UglySolr on Docker - the Good, the Bad and the Ugly
Solr on Docker - the Good, the Bad and the Ugly
 
OOPs, OOMs, oh my! Containerizing JVM apps
OOPs, OOMs, oh my! Containerizing JVM appsOOPs, OOMs, oh my! Containerizing JVM apps
OOPs, OOMs, oh my! Containerizing JVM apps
 
[233] level 2 network programming using packet ngin rtos
[233] level 2 network programming using packet ngin rtos[233] level 2 network programming using packet ngin rtos
[233] level 2 network programming using packet ngin rtos
 
Container Orchestration with Amazon ECS
Container Orchestration with Amazon ECSContainer Orchestration with Amazon ECS
Container Orchestration with Amazon ECS
 
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
 
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
 
openSUSE storage workshop 2016
openSUSE storage workshop 2016openSUSE storage workshop 2016
openSUSE storage workshop 2016
 
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
 
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixXPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
 
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
 

Viewers also liked

VMware ESXi トラブルシューティング
VMware ESXi トラブルシューティングVMware ESXi トラブルシューティング
VMware ESXi トラブルシューティングKazuhito Ohkawa
 
モバイルネットワークと広告配信
モバイルネットワークと広告配信モバイルネットワークと広告配信
モバイルネットワークと広告配信Kazuhito Ohkawa
 
密かに話題のBufferbloat
密かに話題のBufferbloat密かに話題のBufferbloat
密かに話題のBufferbloatKazuhito Ohkawa
 
Nutanix@Open Source Conference 2015 Tokyo/Fall
Nutanix@Open Source Conference 2015 Tokyo/FallNutanix@Open Source Conference 2015 Tokyo/Fall
Nutanix@Open Source Conference 2015 Tokyo/FallSatoshi Shimazaki
 
AHVでみるCVM Autopathの仕組み
AHVでみるCVM Autopathの仕組みAHVでみるCVM Autopathの仕組み
AHVでみるCVM Autopathの仕組みKazuhito Ohkawa
 
Nutanixを導入してみて思ったこと(仮)
Nutanixを導入してみて思ったこと(仮)Nutanixを導入してみて思ったこと(仮)
Nutanixを導入してみて思ったこと(仮)Kazuhito Ohkawa
 
netfilterを利用したDSP監視
netfilterを利用したDSP監視netfilterを利用したDSP監視
netfilterを利用したDSP監視Kazuhito Ohkawa
 
インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形
インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形
インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形Satoshi Shimazaki
 

Viewers also liked (10)

VMware ESXi トラブルシューティング
VMware ESXi トラブルシューティングVMware ESXi トラブルシューティング
VMware ESXi トラブルシューティング
 
モバイルネットワークと広告配信
モバイルネットワークと広告配信モバイルネットワークと広告配信
モバイルネットワークと広告配信
 
密かに話題のBufferbloat
密かに話題のBufferbloat密かに話題のBufferbloat
密かに話題のBufferbloat
 
Nutanix@Open Source Conference 2015 Tokyo/Fall
Nutanix@Open Source Conference 2015 Tokyo/FallNutanix@Open Source Conference 2015 Tokyo/Fall
Nutanix@Open Source Conference 2015 Tokyo/Fall
 
AHVでみるCVM Autopathの仕組み
AHVでみるCVM Autopathの仕組みAHVでみるCVM Autopathの仕組み
AHVでみるCVM Autopathの仕組み
 
Nutanixってナニ?
Nutanixってナニ?Nutanixってナニ?
Nutanixってナニ?
 
Nutanixを導入してみて思ったこと(仮)
Nutanixを導入してみて思ったこと(仮)Nutanixを導入してみて思ったこと(仮)
Nutanixを導入してみて思ったこと(仮)
 
netfilterを利用したDSP監視
netfilterを利用したDSP監視netfilterを利用したDSP監視
netfilterを利用したDSP監視
 
インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形
インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形
インフラエンジニアなら知っておきたい 仮想化環境とストレージの新しい形
 
Nutanix 概要紹介
Nutanix 概要紹介Nutanix 概要紹介
Nutanix 概要紹介
 

Similar to Kauli SSPにおけるVyOSの導入事例

Advanced RAC troubleshooting: Network
Advanced RAC troubleshooting: NetworkAdvanced RAC troubleshooting: Network
Advanced RAC troubleshooting: NetworkRiyaj Shamsudeen
 
PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...
PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...
PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...PROIDEA
 
Super scaling singleton inserts
Super scaling singleton insertsSuper scaling singleton inserts
Super scaling singleton insertsChris Adkin
 
Packaging Strategy for Community Openstack and Implementation Reference | Hoj...
Packaging Strategy for Community Openstack and Implementation Reference | Hoj...Packaging Strategy for Community Openstack and Implementation Reference | Hoj...
Packaging Strategy for Community Openstack and Implementation Reference | Hoj...Vietnam Open Infrastructure User Group
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network ProcessingRyousei Takano
 
Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021
Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021
Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021Deepak Shankar
 
IBM SAN Volume Controller Performance Analysis
IBM SAN Volume Controller Performance AnalysisIBM SAN Volume Controller Performance Analysis
IBM SAN Volume Controller Performance Analysisbrettallison
 
JetStor X Storage Products 2017! New HOT products!
JetStor X Storage Products 2017! New HOT products!JetStor X Storage Products 2017! New HOT products!
JetStor X Storage Products 2017! New HOT products!Gene Leyzarovich
 
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28Amazon Web Services
 
Cracking the nut, solving edge ai with apache tools and frameworks
Cracking the nut, solving edge ai with apache tools and frameworksCracking the nut, solving edge ai with apache tools and frameworks
Cracking the nut, solving edge ai with apache tools and frameworksTimothy Spann
 
Deep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceDeep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceAmazon Web Services
 
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.ioFast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.ioOPNFV
 
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation CenterDUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation CenterAndrey Kudryavtsev
 
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...Amazon Web Services
 
Deep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceDeep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceAmazon Web Services
 
Ceph Performance Profiling and Reporting
Ceph Performance Profiling and ReportingCeph Performance Profiling and Reporting
Ceph Performance Profiling and ReportingCeph Community
 
Splunk app for stream
Splunk app for stream Splunk app for stream
Splunk app for stream csching
 

Similar to Kauli SSPにおけるVyOSの導入事例 (20)

Advanced RAC troubleshooting: Network
Advanced RAC troubleshooting: NetworkAdvanced RAC troubleshooting: Network
Advanced RAC troubleshooting: Network
 
Stress your DUT
Stress your DUTStress your DUT
Stress your DUT
 
PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...
PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...
PLNOG20 - Paweł Małachowski - Stress your DUT–wykorzystanie narzędzi open sou...
 
Super scaling singleton inserts
Super scaling singleton insertsSuper scaling singleton inserts
Super scaling singleton inserts
 
Vrf Design
Vrf DesignVrf Design
Vrf Design
 
Packaging Strategy for Community Openstack and Implementation Reference | Hoj...
Packaging Strategy for Community Openstack and Implementation Reference | Hoj...Packaging Strategy for Community Openstack and Implementation Reference | Hoj...
Packaging Strategy for Community Openstack and Implementation Reference | Hoj...
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network Processing
 
Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021
Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021
Compare Performance-power of Arm Cortex vs RISC-V for AI applications_oct_2021
 
IBM SAN Volume Controller Performance Analysis
IBM SAN Volume Controller Performance AnalysisIBM SAN Volume Controller Performance Analysis
IBM SAN Volume Controller Performance Analysis
 
JetStor X Storage Products 2017! New HOT products!
JetStor X Storage Products 2017! New HOT products!JetStor X Storage Products 2017! New HOT products!
JetStor X Storage Products 2017! New HOT products!
 
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
Amazon EC2 deepdive and a sprinkel of AWS Compute | AWS Floor28
 
xstream_network
xstream_networkxstream_network
xstream_network
 
Cracking the nut, solving edge ai with apache tools and frameworks
Cracking the nut, solving edge ai with apache tools and frameworksCracking the nut, solving edge ai with apache tools and frameworks
Cracking the nut, solving edge ai with apache tools and frameworks
 
Deep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceDeep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance Performance
 
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.ioFast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
 
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation CenterDUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
DUG'20: 12 - DAOS in Lenovo’s HPC Innovation Center
 
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...
Choosing the Right EC2 Instance and Applicable Use Cases - AWS June 2016 Webi...
 
Deep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance PerformanceDeep Dive on Delivering Amazon EC2 Instance Performance
Deep Dive on Delivering Amazon EC2 Instance Performance
 
Ceph Performance Profiling and Reporting
Ceph Performance Profiling and ReportingCeph Performance Profiling and Reporting
Ceph Performance Profiling and Reporting
 
Splunk app for stream
Splunk app for stream Splunk app for stream
Splunk app for stream
 

Recently uploaded

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 

Recently uploaded (20)

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 

Kauli SSPにおけるVyOSの導入事例

  • 1. Case studies of VyOS in Kauli SSP Flandre Scarlet favorite Platform Engineer Kazuhito Ohkawa at Kauli, Inc.
  • 2. Agenda - Self‐Introduction - About Kauli SSP - Case studies in Kauli SSP of VyOS - Tuning tips - About microburst traffic(digress)
  • 3. Self‐Introduction - おおかわ かずひと Kazuhito Ohkawa (twitter@SatchanP) - Aug 2012 Joined Kauli, Inc. Platform Engineer - My Lover THE IDOLM@STER : Yayoi, Mami Touhou Project : Flandre, Sakuya - Private Rallyist This is a my co-driver and three-dimensional parking of impreza.
  • 5. SSPとは SSPとは、「Supply Side Platform」(サプライサイドプラット フォーム)の略で、オンライン広告において、広告枠を提供している メディア(Webサービス、アプリデベロッパー)など媒体社の広告枠 販売や広告収益最大化などを支援するツールのこと。 主に、広告の インプレッションが発生するごとに最適な広告を自動的に選択し、収 益性の向上を図るという仕組みが提供されるが、アドネットワーク、 アドエクスチェンジの一元的管理、リアルタイム入札(RTB)への対 応など、具体的な提供機能はサービスによって異なる。 DSP、SSP - SMMLab(ソーシャルメディアマーケティングラボ) smmlab.jp/?p=30268
  • 6. About SSP A supply-side platform or sell-side platform (SSP) is a technology platform with the single mission of enabling publishers to manage their advertising impression inventory and maximize revenue from digital media. As such, they offer an efficient, automated and secure way to tap into the different sources of advertising income that are available, and provide insight into the various revenue streams and audiences. Many of the larger web publishers of the world use a supply-side platform to automate and optimize the selling of their online media space.[1] A supply-side platform on the publisher side interfaces to an ad exchange, which in turn interfaces to a demand-side platform (DSP) on the advertiser side. This system allows advertisers to put online advertising before a selected target audience.[2] Often, real-time bidding (RTB) is used to complete DSP transactions.[3]。 http://en.wikipedia.org/wiki/Supply-side_platform
  • 7. About RTB Audience Media AD Select the DSP in conditions. Request in parallel. Request for SSP Browse Kauli connected DSPs Bid winner is DSP B Displayed DSP B's AD
  • 8. Many connections for Ad delivery. Up to 400 million Ad per day. All traffic via the VyOS.
  • 9. Agony of SSP Platform Engineer Very very very many many many traffics... As well internal and external... Various traffics, cookie sync, banner, flash and movies, JS tags...etc... About 80 % traffic is short packet... Claim for delay of Ad... SSP isn't profitable! Many media rewards!
  • 11. Infrastructure engineers of SSP. I can not recommend!
  • 12. Case studies in Kauli SSP of VyOS
  • 13. Mainly running on a physical server Gen-1 Intel Core i7 870 RAM 16G Intel 82574L x2 M/B ASUS HDD Gen-2 Intel Xeon E3-1280 v3 RAM 32G Intel I350/I210 M/B Supermicro SSD
  • 14. Using at the Default Gateway for all servers Internet L3 Core LVS DR Real Server nginx VyOS DMZ Default GW IP Masquarede LAN RTB Requests SSP Server
  • 15. Peak traffic graphs of Default Gateway
  • 16. Logic of LVS-DR SRC : 8.8.4.4 DST : 8.8.8.8 LVS VIP : 8.8.8.8 Client A IP : 8.8.4.4 Internet Real Server IP : 10.1.1.2 SRC : 8.8.4.4 DST : 8.8.8.8 MAC : 0000.0000.0000 lo : 8.8.8.8 MAC : 0000.0000.0000 VyOS Default GW IP : 10.1.1.1 SRC : 8.8.8.8 DST : 8.8.4.4 Source address is Solved by MAC Address LVS VIP Make possible by loopback SRC : 8.8.8.8 DST : 8.8.4.4 LAN FP Filter off
  • 17. Router is unnecessary, If server have global IPs SRC : 8.8.4.4 DST : 8.8.8.8 LVS VIP : 8.8.8.8 Client A IP : 8.8.4.4 SRC : 8.8.8.8 Internet DMZ DST : 8.8.4.4 Real Server IP : 8.8.8.9 SRC : 8.8.4.4 DST : 8.8.8.9 MAC : 0000.0000.0000 lo : 8.8.8.8 MAC : 0000.0000.0000
  • 18. Scaling VyOS router by OSPF/ECMP after replacement L3 Core LVS DR VyOS VyOS VyOS Real Server L3 Switch Default GW Internet OSPF ECMP Other Vlan Real Server LVS DR
  • 19. Checking new data center application by Cloud Bridge Vyatta Vyatta Internet SSP Server LVS-DR DB KVS Index Cloud Bridge SSP Server New Data Center Old Data Center DB KVS Index Internet
  • 20. Sakura cloud between VPN Internet Internet Data Center Sakura Cloud VyOS VyOS API Server IPSec Crawler Crawler
  • 22. NUMA I/O NAPI circular buffer CPU Affinity conntrack
  • 23. Use a uni-processor server (NUMA I/O) PCI Express controller is integrated into the CPU in the sandy bridge. High access costs between processors. or using memory mirroring... RAM CPU1 CPU2 RAM PCI Express NIC QPI
  • 24. It is printed on motherbord
  • 25. Reconsider the polling of buffer (NAPI) Buffer overflows even Intel's I350.(Amazing!) It is set the maximum value at 4096. Confirmed with ifconfig and ethtool -S. ifconfig: RX packets:1215382409979 errors:0 dropped:9836789 overruns:9836789 frame:0 ethtool -S: rx_no_buffer_count: 220474
  • 26. Change the NAPI kernel parameters - net.core.netdev_budget Increase the processing queue. - net.core.dev_weight Shorten the polling sensation. However CPU usage rises.
  • 27. circular buffer igb is not set to the maximum value. And too large buffer will cause a delay. Consider the balance to CPU by NAPI and circular buffers. # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 256 RX Mini: 0 RX Jumbo: 0 TX: 256 # ethtool -G eth0 rx 4096 tx 4096
  • 28. CPU Affinity Case of multi-queue, specific cpu core only high load. Adjust these manually. $ cat /proc/interrupts | egrep 'eth|CPU' CPU0 CPU1 CPU2 CPU3 50: 1406514518 0 0 0 PCI-MSI-edge eth0-rx-0 51: 84923776 383727140 0 0 PCI-MSI-edge eth0-tx-0 52: 2951 0 0 0 PCI-MSI-edge eth0 53: 2 31961537 1787069187 0 PCI-MSI-edge eth1-rx-0 54: 1 6218033 0 510452860 PCI-MSI-edge eth1-tx-0 55: 115 0 0 0 PCI-MSI-edge eth1 $ sudo cat /proc/irq/5[0-1,3-4]/smp_affinity 0001 0002 0004 0008
  • 29. conntrack tuning Here is the essential part in the IP Masquarede. Maybe 10G-40G class of IP Masquarede also possible. Established time is very short. The high cost of connection open and close processing. Setting value depends on the memory.
  • 30. conntrack parameter - hash-size conntrack table hashes. Processed faster conntracks scan by hashed. Hash algorithm is chaining scheme. - table-size Raw conntrack tables. - expect-table-size Use FTP, SIP, H.323... http://conntrack-tools.netfilter.org/conntrack.html
  • 31. Raw conntrack table samples tcp 6 128 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43860 dport=80 packets=6 bytes=698 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43860 packets=4 bytes=419 [ASSURED] mark=0 secmark=0 use=2
  • 32. Setting conntrack tables and hash size - table-size CONNTRACK_MAX = RAMSIZE (bytes) / 16384 / (x / 32) x = 32bit or 64bit - hash-size tablesize / 8 - expect-table-size In preference
  • 33. True upper limit of conntrack Focus on the status of the conntrack table. [ASSURED] is not dropping from conntrack tables. Comparison with the [ASSURED] total value and the maximum value. Sample: tcp 6 23 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43708 dport=80 packets=6 bytes=663 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43708 packets=4 bytes=542 [ASSURED] mark=0 secmark=0 use=2
  • 34. Shorten the timeout of conntrack table conntrack table is supposed to be used recursively. But our traffic has very many hosts. Unable to keep conntrack table. Short set a time-out so it not overflow conntrack table. timeout { icmp 3 other 600 tcp { close 10 close-wait 1 established 10 fin-wait 10 last-ack 30 syn-recv 60 syn-sent 5 time-wait 3 } udp { other 30 stream 10 } }
  • 36. About microburst traffic Microburst is not visible, but our network have it. Can be confirmed by various phenomena. One example is a packet discard of switchs.
  • 37. Read the signs of microburst Expand the graph in a narrow range. Spikes confirm.
  • 38. Read the signs of microburst This is a poll of 1 minute sensation. Ave 85 Packets discard/sec = 85Packets * 60 = 5160 5160 packets lost in a moment. I have prepared a movie today.
  • 39. Thank you for your attention!