[RECAP] Blackholes and Wormholes: Understand and Troubleshoot the “Magic” of Kubernetes Networking

Copyright©2018 NTT corp. All Rights Reserved. 1
[RECAP] Blackholes and Wormholes:
Understand and Troubleshoot the “Magic”
of Kubernetes Networking
日本電信電話株式会社
ソフトウェアイノベーションセンタ
市原裕史

2Copyright©2018 NTT corp. All Rights Reserved.
自己紹介
市原裕史
• 所属
– NTT SIC 分散処理基盤技術プロジェクト
• OpenStack Developer
– Neutron Core Reviewer、その他プロジェクトへの貢献
• 技術: SDN/NFV中心
– パブリッククラウドの開発
– LinuxconでDPDKの性能の発表
– Neutronへの機能提案
– Docker Network Plugin開発

発表情報
https://youtu.be/knIJEzTd3kc

発表概要
• Kubernetes ネットワークの事例紹介
– conntrack による古いセッションによる通信障害
– ホストマシンの負荷増大によるNW設定初期化
– iptables ルール重複による意図しないルーティング
• 事例からの教訓
– OS/Kernel レベルで原因究明とコンフィグチェック
– iptables ルールは明解で優先順位に非依存を意識
• ベストプラクティス
– iptables チェック
– conntrack チェック
– tcpdump, tcpdump, tcpdump

なぜこのセッションが重要なのか
1. 従来のベアマシン、VMにおける物理ネットワーク、仮想
ネットワークと勝手が違う
2. Private Kubernetes だけではなく GKE などの KaaS
においてもネットワークの問題から逃れられない
3. デバッグ領域が広範囲で幅広い知識と経験が必要

ベアメタルから仮想ネットワーク
ファイアウォール
ロードバランサ
L2スイッチ
サーバ
ルータ
VM
Compute node
vSwitch
VM
vRouter
vFW
Network node
vLB
vSwitch
VMVM
FWルール管理
NAT管理
FWルール管理
NAT管理

コンテナネットワーク
Worker
Container Container Container
Container Container Container
Router + FW + LB + Switch
FWルール管理、NAT管理、ルーティング管理、バラン
シング管理、MACアドレス管理、セッション管理

コンテナネットワーク技術
Router + FW + LB + Switch
• Linux bridge, veth, routing
• iptables, ipvs, ipset, conntrack
• OpenVSwitch, VPP, cilium, Tungsten Fabric

VM IPアドレスライフサイクル
VM 1
192.168.100.10
Compute node
VM 2
192.168.100.11
Compute node
VM 1
192.168.100.10
Compute node
VM 2
192.168.100.11
同じIPアドレスで復帰

コンテナ IPアドレスライフサイクル
Worker
Container 1
192.168.100.10
Worker
Container 2
192.168.101.11
Container 1
192.168.101.12
Worker
Container 2
192.168.101.11
異なるIPアドレスで復帰

NWプラグインによるふるまいの違い
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
flannelが追加するiptables filterルール
- A I N P U T - m c o m m e n t - - c o m m e n t " c a l i : C z _ u 1 I Q i X I M m K D 4 c " - j c a l i - I N P U T
- A F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : w U H h o i A Y h p h O 9 M s o " - j c a l i - F O R W A R D
- A O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : t V n H k v A o 1 5 H u i P y 0 " - j c a l i - O U T P U T
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : J V 9 - B R W x j z 8 H e 5 I b " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x e 0 0 0 0 0 0
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : p 3 d I A e G s C a b F 0 C U T " - m m a r k - - m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0 - j c a l i - f r o m - h e p - f o r w a r d
- A c a l i - F O R W A R D - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : D e N l x b 0 s U e v j _ P l t " - j c a l i - f r o m - w l - d i s p a t c h
- A c a l i - F O R W A R D - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : B 8 1 F O a Q N Z y m b X 9 H 8 " - j c a l i - t o - w l - d i s p a t c h
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : b B - I 9 T 0 Y R A Y M A S x 0 " - j c a l i - t o - h e p - f o r w a r d
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : I 1 K i 7 a N g Q s J F z E p G " - m c o m m e n t - - c o m m e n t " P o l i c y e x p l i c i t l y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : i 7 o k J Z p S 8 V x a J B 3 n " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - I N P U T - p i p e n c a p - m c o m m e n t - - c o m m e n t " c a l i : k L J d l 8 - 9 M p S K c c l h " - m c o m m e n t - - c o m m e n t " A l l o w I P I P p a c k e t s f r o m C a l i c o h o s t s " - m s e t - - m a t c h - s e t c a l i 4 - a l l - h o s t s s r c - m a d d r t y p e - - d s t - t y p e L O C A L - j A C C E P T
- A c a l i - I N P U T - p i p e n c a p - m c o m m e n t - - c o m m e n t " c a l i : J h f Q U F F J 2 v 0 j b i p F " - m c o m m e n t - - c o m m e n t " D r o p I P I P p a c k e t s f r o m n o n - C a l i c o h o s t s " - j D R O P
- A c a l i - I N P U T - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : l C c y v g f 8 V e D M 1 u 1 - " - g c a l i - w l - t o - h o s t
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : G l r N b O _ E U W Y W R C a O " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x f 0 0 0 0 0 0
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : K - V 6 z S 0 u X r Z M y a M Z " - j c a l i - f r o m - h o s t - e n d p o i n t
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : L x V l C g v 5 v g F Y 0 h I t " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : Y Q S S J I s R c H j F b X a I " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - O U T P U T - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : N 8 8 2 D x H Z f e d r B 2 1 M " - m i p v s - - i p v s - j c a l i - t o - w l - d i s p a t c h
- A c a l i - O U T P U T - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : 3 D M c C m S o d O 9 P v Z S Q " - j R E T U R N
- A c a l i - O U T P U T - p i p e n c a p - m c o m m e n t - - c o m m e n t " c a l i : z 0 o S s u D E D 7 5 M E j 0 R " - m c o m m e n t - - c o m m e n t " A l l o w I P I P p a c k e t s t o o t h e r C a l i c o h o s t s " - m s e t - - m a t c h - s e t c a l i 4 - a l l - h o s t s d s t - m a d d r t y p e - - s r c - t y p e L O C A L - j
A C C E P T
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : Q N n J d g w P t O b q b U O D " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x f 0 0 0 0 0 0
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : B 2 n j 6 q 0 b l o Z N B I i - " - j c a l i - t o - h o s t - e n d p o i n t
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : E x h 0 j T s M 6 8 P O x M g M " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : w W F Q M 4 3 t J U 7 w w n F Z " - m m u l t i p o r t - - d p o r t s 2 2 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p u d p - m c o m m e n t - - c o m m e n t " c a l i : L w N V - - R 8 M j e U Y a c w " - m m u l t i p o r t - - d p o r t s 6 8 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : Q O O 5 N U O q O S S 1 _ I w 0 " - m m u l t i p o r t - - d p o r t s 1 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : c w Z W o B S w V e I A Z m V N " - m m u l t i p o r t - - d p o r t s 2 3 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : 7 F b N X T 9 1 k u g E _ u p R " - m m u l t i p o r t - - d p o r t s 2 3 8 0 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : y w E 9 W Y U B E p v e 7 0 W T " - m m u l t i p o r t - - d p o r t s 6 6 6 6 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : l - W Q S V B f _ l y g P R 0 J " - m m u l t i p o r t - - d p o r t s 6 6 6 7 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p u d p - m c o m m e n t - - c o m m e n t " c a l i : 8 2 h j f j i - w C h F h A q L " - m m u l t i p o r t - - d p o r t s 5 3 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p u d p - m c o m m e n t - - c o m m e n t " c a l i : T N M 3 R f E j b N r 7 2 h g H " - m m u l t i p o r t - - d p o r t s 6 7 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : y c x K i t I l 4 u 3 d K 0 H R " - m m u l t i p o r t - - d p o r t s 1 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : h x j E W y x d k X X k d v u t " - m m u l t i p o r t - - d p o r t s 2 3 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : c A _ G L t r u u v G 8 8 K i O " - m m u l t i p o r t - - d p o r t s 2 3 8 0 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : S b 1 h k L Y F M r K S 6 r 0 1 " - m m u l t i p o r t - - d p o r t s 6 6 6 6 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : U w L S e b G O N J U G 4 y G - " - m m u l t i p o r t - - d p o r t s 6 6 6 7 - j A C C E P T
- A c a l i - f r o m - w l - d i s p a t c h - m c o m m e n t - - c o m m e n t " c a l i : z T j 6 P 0 T I g Y v g z - m d " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - t o - w l - d i s p a t c h - m c o m m e n t - - c o m m e n t " c a l i : 7 K N p h B 1 n N H w 8 0 n I O " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - w l - t o - h o s t - m c o m m e n t - - c o m m e n t " c a l i : E e 9 S b o 1 0 I p V u j d I Y " - j c a l i - f r o m - w l - d i s p a t c h
- A c a l i - w l - t o - h o s t - m c o m m e n t - - c o m m e n t " c a l i : n S Z b c O o G 1 x P O N x b 8 " - m c o m m e n t - - c o m m e n t " C o n f i g u r e d D e f a u l t E n d p o i n t T o H o s t A c t i o n " - j A C C E P T
calicoが追加するiptables filterルール

calico iptables filter ルール
- A I N P U T - m c o m m e n t - - c o m m e n t " c a l i : C z _ u 1 I Q i X I M m K D 4 c " - j c a l i - I N P U T
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : J V 9 - B R W x j z 8 H e 5 I b " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x e 0 0 0 0 0 0
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : p 3 d I A e G s C a b F 0 C U T " - m m a r k - - m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0 - j c a l i - f r o m - h e p - f o r w a r d
- A c a l i - F O R W A R D - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : D e N l x b 0 s U e v j _ P l t " - j c a l i - f r o m - w l - d i s p a t c h
- A c a l i - F O R W A R D - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : B 8 1 F O a Q N Z y m b X 9 H 8 " - j c a l i - t o - w l - d i s p a t c h
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : b B - I 9 T 0 Y R A Y M A S x 0 " - j c a l i - t o - h e p - f o r w a r d
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : I 1 K i 7 a N g Q s J F z E p G " - m c o m m e n t - - c o m m e n t " P o l i c y e x p l i c i t l y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - I N P U T - p i p e n c a p - m c o m m e n t - - c o m m e n t " c a l i : k L J d l 8 - 9 M p S K c c l h " - m c o m m e n t - - c o m m e n t " A l l o w I P I P p a c k e t s f r o m C a l i c o h o s t s " - m s e t - - m a t c h - s e t c a l i 4 - a l l - h o s t s s r c - m a d d r t y p e - - d s t - t y p e
L O C A L - j A C C E P T
- A c a l i - I N P U T - p i p e n c a p - m c o m m e n t - - c o m m e n t " c a l i : J h f Q U F F J 2 v 0 j b i p F " - m c o m m e n t - - c o m m e n t " D r o p I P I P p a c k e t s f r o m n o n - C a l i c o h o s t s " - j D R O P
- A c a l i - I N P U T - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : l C c y v g f 8 V e D M 1 u 1 - " - g c a l i - w l - t o - h o s t
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : G l r N b O _ E U W Y W R C a O " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x f 0 0 0 0 0 0
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : K - V 6 z S 0 u X r Z M y a M Z " - j c a l i - f r o m - h o s t - e n d p o i n t
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : L x V l C g v 5 v g F Y 0 h I t " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - O U T P U T - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : N 8 8 2 D x H Z f e d r B 2 1 M " - m i p v s - - i p v s - j c a l i - t o - w l - d i s p a t c h
- A c a l i - O U T P U T - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : 3 D M c C m S o d O 9 P v Z S Q " - j R E T U R N
- A c a l i - O U T P U T - p i p e n c a p - m c o m m e n t - - c o m m e n t " c a l i : z 0 o S s u D E D 7 5 M E j 0 R " - m c o m m e n t - - c o m m e n t " A l l o w I P I P p a c k e t s t o o t h e r C a l i c o h o s t s " - m s e t - - m a t c h - s e t c a l i 4 - a l l - h o s t s d s t - m a d d r t y p e - - s r c -
t y p e L O C A L - j A C C E P T
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : Q N n J d g w P t O b q b U O D " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x f 0 0 0 0 0 0
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : B 2 n j 6 q 0 b l o Z N B I i - " - j c a l i - t o - h o s t - e n d p o i n t
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : E x h 0 j T s M 6 8 P O x M g M " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : Q O O 5 N U O q O S S 1 _ I w 0 " - m m u l t i p o r t - - d p o r t s 1 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : c w Z W o B S w V e I A Z m V N " - m m u l t i p o r t - - d p o r t s 2 3 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : 7 F b N X T 9 1 k u g E _ u p R " - m m u l t i p o r t - - d p o r t s 2 3 8 0 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : y w E 9 W Y U B E p v e 7 0 W T " - m m u l t i p o r t - - d p o r t s 6 6 6 6 - j A C C E P T
- A c a l i - f a i l s a f e - i n - p t c p - m c o m m e n t - - c o m m e n t " c a l i : l - W Q S V B f _ l y g P R 0 J " - m m u l t i p o r t - - d p o r t s 6 6 6 7 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p u d p - m c o m m e n t - - c o m m e n t " c a l i : 8 2 h j f j i - w C h F h A q L " - m m u l t i p o r t - - d p o r t s 5 3 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p u d p - m c o m m e n t - - c o m m e n t " c a l i : T N M 3 R f E j b N r 7 2 h g H " - m m u l t i p o r t - - d p o r t s 6 7 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : y c x K i t I l 4 u 3 d K 0 H R " - m m u l t i p o r t - - d p o r t s 1 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : h x j E W y x d k X X k d v u t " - m m u l t i p o r t - - d p o r t s 2 3 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : c A _ G L t r u u v G 8 8 K i O " - m m u l t i p o r t - - d p o r t s 2 3 8 0 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : S b 1 h k L Y F M r K S 6 r 0 1 " - m m u l t i p o r t - - d p o r t s 6 6 6 6 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : U w L S e b G O N J U G 4 y G - " - m m u l t i p o r t - - d p o r t s 6 6 6 7 - j A C C E P T
- A c a l i - f r o m - w l - d i s p a t c h - m c o m m e n t - - c o m m e n t " c a l i : z T j 6 P 0 T I g Y v g z - m d " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - t o - w l - d i s p a t c h - m c o m m e n t - - c o m m e n t " c a l i : 7 K N p h B 1 n N H w 8 0 n I O " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P

GKE
-A FORWARD -p tcp -j ACCEPT
-A FORWARD -p udp -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
iptables filterルール

GKE Network Policy 適用後- A I N P U T - m c o m m e n t - - c o m m e n t " c a l i : C z _ u 1 I Q i X I M m K D 4 c " - j c a l i - I N P U T
- A c a l i - F O R W A R D - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : X 3 v B 2 l G c B r f k Y q u C " - j c a l i - f r o m - w l - d i s p a t c h
- A c a l i - F O R W A R D - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : U t J 9 F n h B n F b y Q M v U " - j c a l i - t o - w l - d i s p a t c h
- A c a l i - F O R W A R D - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : T t 1 9 H c S d A 5 Y I G S s w " - j A C C E P T
- A c a l i - F O R W A R D - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : 9 L z f F C v n p C 5 _ M Y X m " - j A C C E P T
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : 7 A o f L L O q C M 5 j 3 6 r M " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x e 0 0 0 0 0 0
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : Q M 1 _ j o S l 7 t L 7 6 A z 7 " - m m a r k - - m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0 - j c a l i - f r o m - h o s t - e n d p o i n t
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : C 1 Q S o g 3 b k 0 A y k j A O " - j c a l i - t o - h o s t - e n d p o i n t
- A c a l i - F O R W A R D - m c o m m e n t - - c o m m e n t " c a l i : D m F i P A m z c i s q Z c v o " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - I N P U T - i c a l i + - m c o m m e n t - - c o m m e n t " c a l i : J a o D b 6 C L d c G w 8 g 0 Y " - g c a l i - w l - t o - h o s t
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : c 5 e K V W 2 V d K Q _ L i S M " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x f 0 0 0 0 0 0
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : h w Q K Y S l S C k p E _ 9 u N " - j c a l i - f r o m - h o s t - e n d p o i n t
- A c a l i - I N P U T - m c o m m e n t - - c o m m e n t " c a l i : t t p 8 - s e r z K C P - b K Z " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - O U T P U T - o c a l i + - m c o m m e n t - - c o m m e n t " c a l i : K R j B s K s B c F B Y K C E w " - j R E T U R N
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : 3 V K A Q B c y U U W 5 k S _ j " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x f 0 0 0 0 0 0
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : Z 1 m B C S H 1 X H M 6 q q 0 k " - j c a l i - t o - h o s t - e n d p o i n t
- A c a l i - O U T P U T - m c o m m e n t - - c o m m e n t " c a l i : N 0 j y W t 2 R f B e d K w 3 L " - m c o m m e n t - - c o m m e n t " H o s t e n d p o i n t p o l i c y a c c e p t e d p a c k e t . " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : 7 3 b Z K o y D f O p F w C 2 T " - m m u l t i p o r t - - d p o r t s 2 3 7 9 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : Q M F u W o 6 o - d 9 y O p N m " - m m u l t i p o r t - - d p o r t s 2 3 8 0 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : K u p 7 Q k r s d m f G X 0 u L " - m m u l t i p o r t - - d p o r t s 4 0 0 1 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p t c p - m c o m m e n t - - c o m m e n t " c a l i : x Y Y r 5 P E q D f _ P q f k v " - m m u l t i p o r t - - d p o r t s 7 0 0 1 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p u d p - m c o m m e n t - - c o m m e n t " c a l i : n b W B v u 4 O t u d V Y 6 0 Q " - m m u l t i p o r t - - d p o r t s 5 3 - j A C C E P T
- A c a l i - f a i l s a f e - o u t - p u d p - m c o m m e n t - - c o m m e n t " c a l i : U x F u 5 c D K 5 E n 6 d T 3 Y " - m m u l t i p o r t - - d p o r t s 6 7 - j A C C E P T
- A c a l i - f r o m - w l - d i s p a t c h - i c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : S f L j y E i f R Q p i B f H 6 " - g c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7
- A c a l i - f r o m - w l - d i s p a t c h - i c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : A Y J j f k n 0 5 h U d C t t B " - g c a l i - f w - c a l i 8 5 c c a b 1 f f d f
- A c a l i - f r o m - w l - d i s p a t c h - i c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : i Y z N C 2 0 O K x - g Y L - L " - g c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8
- A c a l i - f r o m - w l - d i s p a t c h - i c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : D o G v j 2 v G v 6 r h 2 O v m " - g c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1
- A c a l i - f r o m - w l - d i s p a t c h - i c a l i f + - m c o m m e n t - - c o m m e n t " c a l i : m N R d 3 l 2 j Y F I g n r K t " - g c a l i - f r o m - w l - d i s p a t c h - f
- A c a l i - f r o m - w l - d i s p a t c h - m c o m m e n t - - c o m m e n t " c a l i : _ w 6 B 1 _ B w w k h o J E q 1 " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - f r o m - w l - d i s p a t c h - f - i c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : v Q G x a - G r V x m 2 M p H H " - g c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2
- A c a l i - f r o m - w l - d i s p a t c h - f - i c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : r 9 m v 7 8 n w 2 t I N O n 7 Y " - g c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1
- A c a l i - f r o m - w l - d i s p a t c h - f - m c o m m e n t - - c o m m e n t " c a l i : E 6 P B I e F G B _ W h e X z 0 " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : v r T v G E C L F e s Q W D K R " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : h U H R S 8 U s D W g a _ i M t " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : W r I D C i S 4 3 k j T Z Q G _ " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : 3 d i o z 3 p I Y 0 5 6 d 1 D U " - j c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : 0 c g f 5 4 p p X n 1 N I 1 h p " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - f w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : g L G H g d P A R O q J X G d l " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - f w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : r 8 f y F m G S _ f c v q k a m " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - f w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : j z f J r F t T G O I i d b p e " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - f w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : Y O 2 G V k q t H 1 s n N i o z " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - f w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : C 8 _ H H Y l 0 W i 1 R w s I C " - j c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - f w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : m r L Q W d Y g h - B O D d Y G " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - f w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : G 6 D n x T J P S m X H L g 6 v " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : E y i o 0 9 j w h w y z k B 1 C " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : M e 6 O x O B g v A D d z Z T G " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : z x W v M 3 1 9 x _ - b h t 2 n " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : Q 5 8 3 e 6 w 0 8 Q m d 2 d P z " - j c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : E G x c t Y g H w V M u j N e n " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - f w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : r A Q x L - J r s A 7 w H b O a " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : m c _ j 8 q U 7 v z 3 y X e V g " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : p U v I P a C Q k M M B 9 z c n " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : s S - T a g w P v H c 1 O b P f " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : F 8 z 8 1 C k U 4 6 f a Y G V g " - j c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : R f 2 P 8 0 7 M b 4 f 7 A 1 L v " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - f w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : 5 u L l G D F P D f h U a 1 n l " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : f s l r e 4 k B y b g c h h z V " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : l 4 b W v D V g _ u T M D 1 j n " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : I d q b G 9 1 p o F B j g M r o " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : 9 i D z A q 2 s 3 C 1 E W u h w " - j c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : h V f k Y E C q J P L n J B C A " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - f w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : O I 9 A k A p i 9 i P p C Y z J " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : s H p l t l C 6 F U 2 d J a x u " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : r i w c 7 C b R t m c G T w 1 D " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : N N K x C - a R y e y g C b v U " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : h Y _ - D P p P R X b p R m C j " - j c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : q 5 f F k f k N z k n n B X k 0 " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - f w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : I p P G e 4 m Z 5 7 2 l c W h p " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m - m c o m m e n t - - c o m m e n t " c a l i : p l M T f 6 G G o 5 F L t - z w " - j M A R K - - s e t - x m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m - m c o m m e n t - - c o m m e n t " c a l i : d _ y p s H p l 3 J 9 6 o O p x " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m - m c o m m e n t - - c o m m e n t " c a l i : l D Q G D Z g 5 U A N F 5 w I K " - j M A R K - - s e t - x m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - p r o - k 8 s _ n s . k u b e - s y s t e m - m c o m m e n t - - c o m m e n t " c a l i : w n _ d n W - P 0 C O W n h h y " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t o - w l - d i s p a t c h - o c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : 5 2 B W r U C k s O c 8 r l 8 o " - g c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7
- A c a l i - t o - w l - d i s p a t c h - o c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : y n 8 R R k 2 Z a 2 R B g y E S " - g c a l i - t w - c a l i 8 5 c c a b 1 f f d f
- A c a l i - t o - w l - d i s p a t c h - o c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : J o R 8 Z T h v w E 7 4 O g r k " - g c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8
- A c a l i - t o - w l - d i s p a t c h - o c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : x J a M c X d W _ p r e a u _ e " - g c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1
- A c a l i - t o - w l - d i s p a t c h - o c a l i f + - m c o m m e n t - - c o m m e n t " c a l i : r X e 5 j H c D M o t a w 8 L m " - g c a l i - t o - w l - d i s p a t c h - f
- A c a l i - t o - w l - d i s p a t c h - m c o m m e n t - - c o m m e n t " c a l i : 3 n K e P e F S T P x d 6 L 9 Y " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - t o - w l - d i s p a t c h - f - o c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : L e r h W a 8 _ L 5 y t 9 O M 8 " - g c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2
- A c a l i - t o - w l - d i s p a t c h - f - o c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : t U q f I p b t A R _ t A Q N J " - g c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1
- A c a l i - t o - w l - d i s p a t c h - f - m c o m m e n t - - c o m m e n t " c a l i : m L U t - f n 6 - J W 7 r i R 5 " - m c o m m e n t - - c o m m e n t " U n k n o w n i n t e r f a c e " - j D R O P
- A c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : x H I z Z P i s 1 k v r 1 e o f " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : 9 Y Z e z t O Z C z i t 2 D M 9 " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : Z i n x z 2 i Q 6 h I v _ 4 c c " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : f 7 1 5 H d V v 1 x 0 R t p l y " - j c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : F U y z w P _ T x n Y S a Q o 5 " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t w - c a l i 4 c 6 7 d 7 4 b b b 7 - m c o m m e n t - - c o m m e n t " c a l i : 4 f - Y N r e e s P v 5 7 U 8 S " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - t w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : v v w T a y 7 B U z m J O i Y J " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - t w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : P 1 8 H 2 o m x l l X 3 x u t k " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - t w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : l R Y n o i B Z K r 8 p B B D P " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - t w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : v r b M B v m 8 k j Z B j 8 - I " - j c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - t w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : P N p y e s l y 0 8 I C A r z t " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t w - c a l i 8 5 c c a b 1 f f d f - m c o m m e n t - - c o m m e n t " c a l i : 8 y 0 N M n M 0 Z K 4 s 6 e y b " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : L q m W _ Z R w c Q I P r x P 6 " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : M F _ - z q h r L l E R d Z 8 j " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : t N 2 9 0 E P Y H v 6 l I t M X " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : W N v j 8 f Q P W a M 1 g g F N " - j c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : F u U h j b o s c t s w T U p E " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t w - c a l i a 8 f a 6 5 0 9 e 8 8 - m c o m m e n t - - c o m m e n t " c a l i : C Z T p 5 G F 1 A 5 V L S L N Z " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : o - L D G j c Q K t 8 O 8 W H S " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : d Z 8 a V e R c 0 G z m L 1 C f " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : w P o u d f 1 T q l T F j Y 8 H " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : 5 M L 8 i u L 5 4 m 4 Z p _ G r " - j c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : E s s X C o U o v Y s K F X B X " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t w - c a l i c 3 4 9 b 7 6 f 9 a 1 - m c o m m e n t - - c o m m e n t " c a l i : - G U V o b 8 j 1 u q c P 4 O j " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : C 0 8 d N M f I 9 G S p c x 8 A " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : k v x Q E T W D L i F l d b t e " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : Y u z i 9 5 p 7 _ Q Q F l n J v " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : 5 l W V A I 8 E U u L Q B K N a " - j c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : F S W j F - I 5 f L i c L L Q R " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t w - c a l i f 7 0 f f d 5 9 3 f 2 - m c o m m e n t - - c o m m e n t " c a l i : m I u H 1 e 0 C Y z s C c w u 9 " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
- A c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : W J q 0 E 9 W K F z A v e W T u " - m c o n n t r a c k - - c t s t a t e R E L A T E D , E S T A B L I S H E D - j A C C E P T
- A c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : S 1 w a y I U r U e X 1 F l F 5 " - m c o n n t r a c k - - c t s t a t e I N V A L I D - j D R O P
- A c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : m A V Q y T t x d 5 A s 9 F I t " - j M A R K - - s e t - x m a r k 0 x 0 / 0 x 1 0 0 0 0 0 0
- A c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : t S d r f e Z 6 o J 6 K s P A O " - j c a l i - p r i - k 8 s _ n s . k u b e - s y s t e m
- A c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : W U U - L E t R 0 H i V M u j m " - m c o m m e n t - - c o m m e n t " R e t u r n i f p r o f i l e a c c e p t e d " - m m a r k - - m a r k 0 x 1 0 0 0 0 0 0 / 0 x 1 0 0 0 0 0 0 - j R E T U R N
- A c a l i - t w - c a l i f 9 5 e 2 4 7 1 1 b 1 - m c o m m e n t - - c o m m e n t " c a l i : 1 h m Q U O V H 5 s V u E Q v r " - m c o m m e n t - - c o m m e n t " D r o p i f n o p r o f i l e s m a t c h e d " - j D R O P
Calico の 3倍

ベアメタル、VM、コンテナ
ベアメタル
VM
Pod
Pod
VM
Pod
ベアメタル
ベアメタル
ToR
ベアメタル
ToR
ベアメタル
ベアメタル
ベアメタル
ベアメタル
ToR
ベアメタル
ベアメタル
ベアメタル
ベアメタル
vSwitch
iptables
veth
bridge
…
vlan,…
routing, bgp, …

vSwitch
iptables
veth
bridge
…
vlan,…
routing, bgp, …
GKE利用時
VM
Pod
Pod
VM
Pod
この範囲のみが対象

まとめ
 コンテナのIPアドレスのライフサイクルが非常に短いことに起因
して色々な障害が起こる
2. Private Kubernetes だけではなく GEK などの KaaS
 使用するCNIによってネットワークの実装はバラバラ、GKEにお
いてもネットワークの問題を切り分けるのは苦労する
 通信障害を特定するためには、ベアマシン、VM、コンテナの通信
を様々なプロトコルを横断して見る必要がある

付録
• タイトル
– Blackholes and Wormholes: Understand and Troubleshoot
the “Magic” of Kubernetes Networking
• 発表動画
– https://youtu.be/knIJEzTd3kc
• スライド
– https://schd.ws/hosted_files/kccnceu18/e7/Blackholes%20
%26%20Wormholes%20-
%20KubeCon%202018%20%281%29.pdf

[RECAP] Blackholes and Wormholes: Understand and Troubleshoot the “Magic” of Kubernetes Networking

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to [RECAP] Blackholes and Wormholes: Understand and Troubleshoot the “Magic” of Kubernetes Networking

Similar to [RECAP] Blackholes and Wormholes: Understand and Troubleshoot the “Magic” of Kubernetes Networking (20)

More from Hirofumi Ichihara

More from Hirofumi Ichihara (10)

Recently uploaded

Recently uploaded (20)

[RECAP] Blackholes and Wormholes: Understand and Troubleshoot the “Magic” of Kubernetes Networking