Using open source tools for network device dataplane testing. Our experiences from redGuardian DDoS mitigation scrubber testing. Presented at PLNOG 20 (2018).

1. Stress your DUT
Using open source tools for network device dataplane testing.
Paweł Małachowski, 2018.03.20
@pawmal80

2. Agenda
 introduction
 software packet generators overview
 Packet Of Death story
 redGuardian regression testing internals

3. Basics
 RFC 2544 „Benchmarking Methodology for Network Interconnect Devices”
Frame sizes to be used on Ethernet 64, 128, 256, 512, 1024, 1280, 1518
 RFC 6815 „Use on Production Networks Considered Harmful”
 RFC2889 „Benchmarking Methodology for LAN Switching Devices”
 IMIX concept

4. Example metrics
 throughput (bps, pps)
 zero packet loss
 latency
 connection/flow setup rate

5. Example: packet switching/routing
 „Mellanox Spectrum vs. Broadcom StrataXGS Tomahawk 25GbE&100GbE
Performance Evaluation” (Tolly, 2016)
http://www.mellanox.com/related-docs/products/tolly-report-
performance-evaluation-2016-march.pdf
 „Cisco Integrated Services Routers—Performance Overview” (2010)
https://supportforums.cisco.com/legacyfs/online/legacy/7/2/6/134627-
white_paper_c11_595485.pdf

6. Example: routing lookup performance on Linux
 https://vincent.bernat.im/
en/blog/2017-ipv4-route-
lookup-linux
 https://vincent.bernat.im/
en/blog/2017-
performance-progression-
ipv4-route-lookup-linux
 great in-depth analysis

7. Example factors
 random dst IP
 MSS clamping?
 FIB size (Trie vs. DIR-m-n)
 port interaction
 VM interaction (NFV)
 do you have baseline?

8. Software packet generation: TX methods
API comment
socket (RAW, DGRAM)
pcap classic, mature, available everywhere
in-kernel Linux pktgen
AF_PACKET (V4+ZC) XDP cooperation
kernel bypass (DPDK,
Netmap, PF_RING ZC, Snabb)
fastest possible, available only on some platforms,
frequently require additional compiling/patching

9. Classic generators

10. Classic generators
 available OOTB or easy to install
 mature, well documented
 pcap(3) based
 „fast enough” in some cases

11. hping3
 command-line oriented TCP/IP packet assembler and analyzer
 notable options: flood, spoofing, addres/port randomization
 1 Mpps easy to achieve
 similar tools: nping (nmap)

12. hping3
% hping3 --syn 127.0.0.1 --destport ++31337
HPING 127.0.0.1 (lo 127.0.0.1): S set, 40 headers + 0 data bytes
len=40 ip=127.0.0.1 ttl=64 DF id=46879 sport=31337 flags=RA seq=0 win=0 rtt=7.7 ms
len=40 ip=127.0.0.1 ttl=64 DF id=46992 sport=31338 flags=RA seq=1 win=0 rtt=3.5 ms
len=40 ip=127.0.0.1 ttl=64 DF id=47120 sport=31339 flags=RA seq=2 win=0 rtt=3.3 ms
^C
--- 127.0.0.1 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 3.3/4.8/7.7 ms
% hping3 --syn 127.0.0.1 --destport ++31337 --flood

13. tcpreplay
 tools for editing and replaying network traffic
 pcap(3) based, Netmap support
 idea: record your UDP traffic, replay it against tested service
 https://github.com/appneta/tcpreplay

14. tcpreplay
% tcpreplay -i eth2 -K -t -l 100000 ./test/test.pcap
File Cache is enabled
Actual: 14100000 packets (6270400000 bytes) sent in 9.07 seconds.
Rated: 645928366.4 Bps, 5167.42 Mbps, 1452473.52 pps
Flows: 37 flows, 3.81 fps, 14000000 flow packets, 100000 non-flow
Statistics for network device: eth2
Attempted packets: 14100000
Successful packets: 14100000
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0

15. Packet crafting

16. Scapy
 Python-based packet manipulation program/library
 packet (and layers) are Python objects, e.g. Ether(), IP()
 read/write pcap file/interface
 slow but useful for automated packet crafting

17. Scapy
$ sudo scapy
>>> ICMP()
<ICMP |>
>>> ICMP(seq=0xda).show()
###[ ICMP ]###
type= echo-request
code= 0
chksum= None
id= 0x0
seq= 0xda
>>> pkts=Ether() / IP(src="10.0.0.0/24", dst="192.168.1.1") / ICMP()
>>> pkts
<Ether type=0x800 |<IP frag=0 proto=icmp src=Net('10.0.0.0/24') dst=192.168.1.1 |<ICMP |>>>
>>> sendp(pkts, iface="eth1")
Sent 256 packets.

18. Ostinato
 „Wireshark in Reverse”
 craft packets with GUI
 send at specific rate
 DPDK acceleration
(prototype only, 2014)
 https://ostinato.org/

19. Ostinato
 click & learn
 available as GNS3 and
VIRL image

20. Accelerated generators

21. Accelerated generators
 kernel bypassing for maximum performance
 rough edges
 non–trivial to install, tune (core to RX/TX queue mapping,
driver and PCIe tuning) and operate
 poor man’s alternative for hardware packet generators

22. Snabb packetblaster
 Snabb (LuaJIT) based
 able to push 20x10Gbps with little CPU usage
 tight TX loop over preloaded packets
 https://github.com/snabbco/snabb/tree/master/src/program/packetblaster
% packetblaster replay myfile.cap 0000:01:00.0

23. DPDK pktgen
 classic DPDK-based packet generator
 CLI-oriented, multiple „pages”
 good for „naive” packet flooding
 http://pktgen-dpdk.readthedocs.io/,
http://dpdk.org/browse/apps/pktgen-dpdk/refs/

24. DPDK pktgen
 load pcap and randomize fields
 Lua scripts (API/control only)
% pktgen --socket-mem=128,16384 -w
0000:82:00.0,txq_inline=128 -l 0,12-23 -n 2 -- -N
-T -m "[12:13-23].0"

25. MoonGen
 „Scriptable High-Speed Packet Generator”
 DPDK + LuaJIT based
 craft your packets in Lua!
 nice, scientific approach
 https://github.com/emmericp/MoonGen
% moongen-simple start udp-simple:0:0:rate=1000mbit/s,ratePattern=poisson

26. T-Rex
 „generates L4-7 traffic based on pre-processing and smart replay of
real traffic templates”
 DPDK-based
 feature rich
 https://trex-tgn.cisco.com/,
https://github.com/cisco-system-traffic-generator

27. T-Rex
 IMIX floods
 application traffic
 TCP connections
% t-rex-64 -f cap2/imix_64.yaml
-c 4 -m 1 -d 100 -l 1000

28. WARP17
 „Lightweight solution for generating high volumes of session-based traffic
with high setup rates”
 DPDK-based
 http://warp17.net/,
https://github.com/Juniper/warp17
% warp17 -l 0-7 -n 2 -m 16384 -w 0000:01:00.1 -- --qmap-default max-q --
cmd-file=atds.cfg

29. WARP17
 server &
client
 TCP sessions
setup
 DUT as
middlebox

30. Packet Of Death

31. Packet Of Death
 repeatable fails of nightly regression
tests
 network card hangs, only machine
reboot helps
 WTF?

32. Packet of Death
 git log tests/
 facepalm
 vendor releases firmware upgrade

33. redGuardian testing framework
Goals
 functional regression
 local/pcap-based
 remote (real NICs)
 performance regression
Tools
 Python
 pytest
 Scapy
 DPDK pktgen

34. redGuardian testing framework

35. Example testcase
def _conf(cfg):
cfg.add_target_v4('0.0.0.0/0')
@with_config(_conf)
class TestForward:
def test_ipv4_fragment_beyond_end(self, tester):
"""Fragmented packets with sum of fragment offset and IP length exceeding 65535 are dropped"""
src = '1.2.3.4'
dst = '10.0.0.1'
tester.run(PASS << RAND_ETH << [
PASS << IP(src=src, dst=dst, proto='udp', id=1, frag=8189) / Payload(3), # =65535
DROP << IP(src=src, dst=dst, proto='udp', id=1, frag=8189) / Payload(4), # =65536
])

36. „Expected” packet actions
Action Expected behaviour
DROP silent drop
PASS passthrough
DEC_TTL decrement TTL
DELAY wait (e.g. for state expiration)
FRAGMENT perform IP fragmentation
STRIP_VLAN remove 802.1Q tag
TRUNCATE truncate packet (packet sampling)
…

37. Example performance testcase
TTL1 = RAND_ETH / IP(src=RandIP(), dst=RandIP(), ttl=1) / TCP()
class TestPerfInvalid(TestPerfBase):
def test_ttl1_drop(self, perf_tester, cfg):
cfg.add_target_v4('0.0.0.0/0')
perf_tester.perf_pcap = TTL1
self.run(perf_tester, cfg, 'ttl1_drop')
$ tests/run.sh --perf tests/perf/test_perf_invalid.py::TestPerfInvalid::test_ttl1_drop
platform linux -- Python 3.5.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0 --
/home/pawmal/git/dpapp/dataplane/tests/venv/bin/python
perf_cfg: pktgen_default.cfg
rootdir: /home/pawmal/git/dpapp/dataplane, inifile:
plugins: xdist-1.22.0, forked-0.2
collected 1 item
tests/perf/test_perf_invalid.py::TestPerfInvalid::test_ttl1_drop PASSED [100%]
PERF ttl1_drop rx_median 14.730; tx_median 0.000; rx_mean 14.620; tx_mean 0.000; gen_rx 0.000; gen_tx
14.813; gen_rx_Mbps 0; gen_tx_Mbps 9954

38. Performance regression tracking
 multiple
performance
scenarios
 compare Mpps
between nightly
builds
 1 core E5-2695 v4
2.10GHz, random
tuples

39. Additional materials

40. NIC benchmarking
 https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_perfo
rmance_report.pdf
 https://fast.dpdk.org/doc/perf/DPDK_17_11_Mellanox_NIC_
performance_report.pdf
 consider these „theoretical maximum”

41. Testing frameworks
 DPDK Test Suite: https://dpdk.org/doc/dts/gsg/index.html
 fd.io/VPP Continuous System Integration and Testing (CSIT):
https://docs.fd.io/csit/master/doc/

42. Summary
 theoretical vs. real life vs. IMIX
 understand your DUT internals
 networking product development without automated testing
is impossible

43. ^D
@redguardianeu