Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Stress your DUT Slide 1

YouTube videos are no longer supported on SlideShare

View original on YouTube

Stress your DUT Slide 3 Stress your DUT Slide 4 Stress your DUT Slide 5 Stress your DUT Slide 6 Stress your DUT Slide 7 Stress your DUT Slide 8 Stress your DUT Slide 9 Stress your DUT Slide 10 Stress your DUT Slide 11 Stress your DUT Slide 12 Stress your DUT Slide 13 Stress your DUT Slide 14 Stress your DUT Slide 15 Stress your DUT Slide 16 Stress your DUT Slide 17 Stress your DUT Slide 18 Stress your DUT Slide 19 Stress your DUT Slide 20 Stress your DUT Slide 21 Stress your DUT Slide 22 Stress your DUT Slide 23 Stress your DUT Slide 24 Stress your DUT Slide 25 Stress your DUT Slide 26 Stress your DUT Slide 27 Stress your DUT Slide 28 Stress your DUT Slide 29 Stress your DUT Slide 30 Stress your DUT Slide 31 Stress your DUT Slide 32 Stress your DUT Slide 33 Stress your DUT Slide 34 Stress your DUT Slide 35 Stress your DUT Slide 36 Stress your DUT Slide 37 Stress your DUT Slide 38 Stress your DUT Slide 39 Stress your DUT Slide 40 Stress your DUT Slide 41 Stress your DUT Slide 42 Stress your DUT Slide 43 Stress your DUT Slide 44
Upcoming SlideShare
What to Upload to SlideShare
Next

6

Share

Stress your DUT

Using open source tools for network device dataplane testing.
Our experiences from redGuardian DDoS mitigation scrubber testing.
Presented at PLNOG 20 (2018).

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Stress your DUT

  1. 1. Stress your DUT Using open source tools for network device dataplane testing. Paweł Małachowski, 2018.03.20 @pawmal80
  2. 2. Agenda  introduction  software packet generators overview  Packet Of Death story  redGuardian regression testing internals
  3. 3. Basics  RFC 2544 „Benchmarking Methodology for Network Interconnect Devices” Frame sizes to be used on Ethernet 64, 128, 256, 512, 1024, 1280, 1518  RFC 6815 „Use on Production Networks Considered Harmful”  RFC2889 „Benchmarking Methodology for LAN Switching Devices”  IMIX concept
  4. 4. Example metrics  throughput (bps, pps)  zero packet loss  latency  connection/flow setup rate
  5. 5. Example: packet switching/routing  „Mellanox Spectrum vs. Broadcom StrataXGS Tomahawk 25GbE&100GbE Performance Evaluation” (Tolly, 2016) http://www.mellanox.com/related-docs/products/tolly-report- performance-evaluation-2016-march.pdf  „Cisco Integrated Services Routers—Performance Overview” (2010) https://supportforums.cisco.com/legacyfs/online/legacy/7/2/6/134627- white_paper_c11_595485.pdf
  6. 6. Example: routing lookup performance on Linux  https://vincent.bernat.im/ en/blog/2017-ipv4-route- lookup-linux  https://vincent.bernat.im/ en/blog/2017- performance-progression- ipv4-route-lookup-linux  great in-depth analysis
  7. 7. Example factors  random dst IP  MSS clamping?  FIB size (Trie vs. DIR-m-n)  port interaction  VM interaction (NFV)  do you have baseline?
  8. 8. Software packet generation: TX methods API comment socket (RAW, DGRAM) pcap classic, mature, available everywhere in-kernel Linux pktgen AF_PACKET (V4+ZC) XDP cooperation kernel bypass (DPDK, Netmap, PF_RING ZC, Snabb) fastest possible, available only on some platforms, frequently require additional compiling/patching
  9. 9. Classic generators
  10. 10. Classic generators  available OOTB or easy to install  mature, well documented  pcap(3) based  „fast enough” in some cases
  11. 11. hping3  command-line oriented TCP/IP packet assembler and analyzer  notable options: flood, spoofing, addres/port randomization  1 Mpps easy to achieve  similar tools: nping (nmap)
  12. 12. hping3 % hping3 --syn 127.0.0.1 --destport ++31337 HPING 127.0.0.1 (lo 127.0.0.1): S set, 40 headers + 0 data bytes len=40 ip=127.0.0.1 ttl=64 DF id=46879 sport=31337 flags=RA seq=0 win=0 rtt=7.7 ms len=40 ip=127.0.0.1 ttl=64 DF id=46992 sport=31338 flags=RA seq=1 win=0 rtt=3.5 ms len=40 ip=127.0.0.1 ttl=64 DF id=47120 sport=31339 flags=RA seq=2 win=0 rtt=3.3 ms ^C --- 127.0.0.1 hping statistic --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 3.3/4.8/7.7 ms % hping3 --syn 127.0.0.1 --destport ++31337 --flood
  13. 13. tcpreplay  tools for editing and replaying network traffic  pcap(3) based, Netmap support  idea: record your UDP traffic, replay it against tested service  https://github.com/appneta/tcpreplay
  14. 14. tcpreplay % tcpreplay -i eth2 -K -t -l 100000 ./test/test.pcap File Cache is enabled Actual: 14100000 packets (6270400000 bytes) sent in 9.07 seconds. Rated: 645928366.4 Bps, 5167.42 Mbps, 1452473.52 pps Flows: 37 flows, 3.81 fps, 14000000 flow packets, 100000 non-flow Statistics for network device: eth2 Attempted packets: 14100000 Successful packets: 14100000 Failed packets: 0 Truncated packets: 0 Retried packets (ENOBUFS): 0 Retried packets (EAGAIN): 0
  15. 15. Packet crafting
  16. 16. Scapy  Python-based packet manipulation program/library  packet (and layers) are Python objects, e.g. Ether(), IP()  read/write pcap file/interface  slow but useful for automated packet crafting
  17. 17. Scapy $ sudo scapy >>> ICMP() <ICMP |> >>> ICMP(seq=0xda).show() ###[ ICMP ]### type= echo-request code= 0 chksum= None id= 0x0 seq= 0xda >>> pkts=Ether() / IP(src="10.0.0.0/24", dst="192.168.1.1") / ICMP() >>> pkts <Ether type=0x800 |<IP frag=0 proto=icmp src=Net('10.0.0.0/24') dst=192.168.1.1 |<ICMP |>>> >>> sendp(pkts, iface="eth1") Sent 256 packets.
  18. 18. Ostinato  „Wireshark in Reverse”  craft packets with GUI  send at specific rate  DPDK acceleration (prototype only, 2014)  https://ostinato.org/
  19. 19. Ostinato  click & learn  available as GNS3 and VIRL image
  20. 20. Accelerated generators
  21. 21. Accelerated generators  kernel bypassing for maximum performance  rough edges  non–trivial to install, tune (core to RX/TX queue mapping, driver and PCIe tuning) and operate  poor man’s alternative for hardware packet generators
  22. 22. Snabb packetblaster  Snabb (LuaJIT) based  able to push 20x10Gbps with little CPU usage  tight TX loop over preloaded packets  https://github.com/snabbco/snabb/tree/master/src/program/packetblaster % packetblaster replay myfile.cap 0000:01:00.0
  23. 23. DPDK pktgen  classic DPDK-based packet generator  CLI-oriented, multiple „pages”  good for „naive” packet flooding  http://pktgen-dpdk.readthedocs.io/, http://dpdk.org/browse/apps/pktgen-dpdk/refs/
  24. 24. DPDK pktgen  load pcap and randomize fields  Lua scripts (API/control only) % pktgen --socket-mem=128,16384 -w 0000:82:00.0,txq_inline=128 -l 0,12-23 -n 2 -- -N -T -m "[12:13-23].0"
  25. 25. MoonGen  „Scriptable High-Speed Packet Generator”  DPDK + LuaJIT based  craft your packets in Lua!  nice, scientific approach  https://github.com/emmericp/MoonGen % moongen-simple start udp-simple:0:0:rate=1000mbit/s,ratePattern=poisson
  26. 26. T-Rex  „generates L4-7 traffic based on pre-processing and smart replay of real traffic templates”  DPDK-based  feature rich  https://trex-tgn.cisco.com/, https://github.com/cisco-system-traffic-generator
  27. 27. T-Rex  IMIX floods  application traffic  TCP connections % t-rex-64 -f cap2/imix_64.yaml -c 4 -m 1 -d 100 -l 1000
  28. 28. WARP17  „Lightweight solution for generating high volumes of session-based traffic with high setup rates”  DPDK-based  http://warp17.net/, https://github.com/Juniper/warp17 % warp17 -l 0-7 -n 2 -m 16384 -w 0000:01:00.1 -- --qmap-default max-q -- cmd-file=atds.cfg
  29. 29. WARP17  server & client  TCP sessions setup  DUT as middlebox
  30. 30. Packet Of Death
  31. 31. Packet Of Death  repeatable fails of nightly regression tests  network card hangs, only machine reboot helps  WTF?
  32. 32. Packet of Death  git log tests/  facepalm  vendor releases firmware upgrade
  33. 33. redGuardian testing framework Goals  functional regression  local/pcap-based  remote (real NICs)  performance regression Tools  Python  pytest  Scapy  DPDK pktgen
  34. 34. redGuardian testing framework
  35. 35. Example testcase def _conf(cfg): cfg.add_target_v4('0.0.0.0/0') @with_config(_conf) class TestForward: def test_ipv4_fragment_beyond_end(self, tester): """Fragmented packets with sum of fragment offset and IP length exceeding 65535 are dropped""" src = '1.2.3.4' dst = '10.0.0.1' tester.run(PASS << RAND_ETH << [ PASS << IP(src=src, dst=dst, proto='udp', id=1, frag=8189) / Payload(3), # =65535 DROP << IP(src=src, dst=dst, proto='udp', id=1, frag=8189) / Payload(4), # =65536 ])
  36. 36. „Expected” packet actions Action Expected behaviour DROP silent drop PASS passthrough DEC_TTL decrement TTL DELAY wait (e.g. for state expiration) FRAGMENT perform IP fragmentation STRIP_VLAN remove 802.1Q tag TRUNCATE truncate packet (packet sampling) …
  37. 37. Example performance testcase TTL1 = RAND_ETH / IP(src=RandIP(), dst=RandIP(), ttl=1) / TCP() class TestPerfInvalid(TestPerfBase): def test_ttl1_drop(self, perf_tester, cfg): cfg.add_target_v4('0.0.0.0/0') perf_tester.perf_pcap = TTL1 self.run(perf_tester, cfg, 'ttl1_drop') $ tests/run.sh --perf tests/perf/test_perf_invalid.py::TestPerfInvalid::test_ttl1_drop platform linux -- Python 3.5.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0 -- /home/pawmal/git/dpapp/dataplane/tests/venv/bin/python perf_cfg: pktgen_default.cfg rootdir: /home/pawmal/git/dpapp/dataplane, inifile: plugins: xdist-1.22.0, forked-0.2 collected 1 item tests/perf/test_perf_invalid.py::TestPerfInvalid::test_ttl1_drop PASSED [100%] PERF ttl1_drop rx_median 14.730; tx_median 0.000; rx_mean 14.620; tx_mean 0.000; gen_rx 0.000; gen_tx 14.813; gen_rx_Mbps 0; gen_tx_Mbps 9954
  38. 38. Performance regression tracking  multiple performance scenarios  compare Mpps between nightly builds  1 core E5-2695 v4 2.10GHz, random tuples
  39. 39. Additional materials
  40. 40. NIC benchmarking  https://fast.dpdk.org/doc/perf/DPDK_17_11_Intel_NIC_perfo rmance_report.pdf  https://fast.dpdk.org/doc/perf/DPDK_17_11_Mellanox_NIC_ performance_report.pdf  consider these „theoretical maximum”
  41. 41. Testing frameworks  DPDK Test Suite: https://dpdk.org/doc/dts/gsg/index.html  fd.io/VPP Continuous System Integration and Testing (CSIT): https://docs.fd.io/csit/master/doc/
  42. 42. Summary  theoretical vs. real life vs. IMIX  understand your DUT internals  networking product development without automated testing is impossible
  43. 43. ^D @redguardianeu
  • fairchilddb

    May. 12, 2020
  • ViktorBuchenko1

    May. 8, 2020
  • MahdiSoleimani6

    Dec. 16, 2019
  • soskaykakehi

    Mar. 13, 2019
  • MatthewEckl

    Jan. 16, 2019
  • ssuser52ede6

    May. 6, 2018

Using open source tools for network device dataplane testing. Our experiences from redGuardian DDoS mitigation scrubber testing. Presented at PLNOG 20 (2018).

Views

Total views

2,704

On Slideshare

0

From embeds

0

Number of embeds

31

Actions

Downloads

0

Shares

0

Comments

0

Likes

6

×