DeiC DDoS Prevention System - DDPS by Niels Thomas Haugård, DeiC. See https://www.linkedin.com/in/uninth and https://www.deic.dk

1. 29 May 2017 1
DeiC DDoS Prevention System - DDPS
Niels Thomas Haugård, DeiC
Consultant since 1995 in the security and network field
See https://www.linkedin.com/in/uninth
and https://www.deic.dk

2. 29 May 2017 2
> About DeiC
> Danish NREN
> DeiC – Danish e-Infrastructure Cooperation has the stated aim of supporting the
development of Denmark as an eScience nation through the delivery of e-infrastructure
(computing, data warehousing, network connections and auxiliary services), guidance, and
initiatives at national level. DeiC is a virtual unit under the Danish Ministry of Higher
Education and Science and the result of an agreement concluded between the eight Danish
universities and the Danish Agency for Science and Higher Education. For details, see
www.deic.dk

3. 29 May 2017 3
> Why present it
> Benefit for others
> Exchange of ideas never made something worse
> Open source and on github:https://github.com/deic-dk/DDPS-documentation

4. 29 May 2017 4
> The next 42 minutes
> Why we made DDPS - A short description of DDoS attacks
> How we designed it and why
> What it looks like, how to handle failures etc.
> Lessons learned
> Current Status
> Who made DDPS
> Questions ?

5. 29 May 2017 5
> Why do it - 20 years of DDoS history

6. 29 May 2017 6
> akamai’s state of the internet / security Q2 2017
2017, Q1

7. 29 May 2017 7
> State of Forskningsnet
> Early warnings only
> One small incident in the summer of 2016
> Attacking student dormitories in Ålborg (World of Warcraft) and Ålborg University
> Not visible on traffic graphs
> Our infrastructure attacked
> Hosting Itslearning AS with national tests
> Regular attacked

8. 29 May 2017 8
> DDoS characteristic
> Stateless connections (UDP, TCP syn, GRE, ICMP echo request)
> Amplification attack with spoofed source addresses
> Attack on kernel and network layer (Ping of Death, Teardrop osv.)
> Application layer attack (Slowloris, R-U-Dead-Yet osv.)
> Massive legitimate connections
ISP focus
area

9. 29 May 2017 9
> DDoS mitigation challenges for ISP’s
> Volumetric only - will / cannot take responsibility for application layer attacks
> Attacker has more bandwidth than the target
> Manual blocking is slow, labor intensive and error prone
> The blocking usually takes place to late

10. 29 May 2017 10
> DDoS mitigation ISP - customer expectations
> Very short responce time
> Mitigation must be implemented in uplink and peers
> (as close as possible on the source)
> ISP’s should cooperate

11. 29 May 2017 11
> Mitigation methods
> BGP based Blackhole routing: scarify attacked addresses (først RFC3882 fra 2004)
> Predefined route communities (all, ISP, … )
> Described in rfc5635 og rfc7999
> BGP flowspec (RFC 5575 fra 2009) match/action firewall rules in edge routers
> Attacked addresses not scarified: filtering on 12 network parameters
> Implanted ca 2013 in Juniper routers, Cisco ca 2014
> See Nokia/Alcatel, Cisco, Huawei og Juniper: BGP Flow Specification: Multi Vendor and
Inter AS Interoperability January 2, 2017 (https://www.nextlayer.at/flowspec-
paper.pdf) - a sad and long story, but the most general option

12. 29 May 2017 12
> Other methods
> Loadshare
> Cloudflare and others
> Rewrite your application and hand over your SSL keys - not an option for all
> Assumes service provider have more bandwidth than the attacker (usually they do)
> Any cast
> Multiple hosts one IP address - Service configured as a cluster
> Our MX960’ers must filter differently on traffic from NORDunet and forskningsnet
> Assumes attach only through NORDunet - wrong assumption

13. 29 May 2017 13
> No matter the solution
> Your investment in DDoS protection is worth nothing
while you are not under attack
> Defending is more expensive than attacking
> Attacks never worsens over time
> You cannot predict what kind of attack will be used
next time
> IPv6 not really supported see
https://tools.ietf.org/html/draft-ietf-idr-flow-spec-
v6-03 (2012)

14. 29 May 2017 14
> Existing solutions
> Onsite: Arbor Network, F5 m.fl.:
> Expensive, proprietary, and will they fit our infrastructure and customers?
> Cloud: Cloud Flare, incapsula m.fl.
> Expensive cloud based load share, requires your SSL keys
> Open source alternatives
> FastNetMon
> Firewall on Demand (grnet / GÉANT)

15. 29 May 2017 15
> Firewall on Demand (grnet / GÉANT)
> No longer maintained at the time we started the project
> Configuration based on netconf - not used here
> requires shibboleth for authentication while we would use WAYF — the digital recycling
centre
> Lack of automatic detection
> A good source for inspiration

16. 29 May 2017 16
> Mitigation Opportunities
> Do not change existing customer infrastructure nor scarify hosts for the network
> black hole
> BGP flowspec
> load share
> multicast
> Price, Performance (and maintainable code)
> Commercial
> Open source
> Home grown

17. 29 May 2017 17
> What we would like …
> Existing services should not be changed to match a solution
> BGP flowspec benefits:
> May be implemented in existing (aka nor new) network equipment
> No changes to services needed (unlike e.g. load share)
> Builds on top of existing BGP peering
> Requires
> Manual changes implemented ahead of the attack
> Automatic detection and mitigation

18. 29 May 2017 18
> BGP flowspec as a security tool
> eBGP: announce what network you would like to receive traffic for from a given peer
> only as long as the BGP session is active (TCP)
> BGP flowspec :Add discrimination to your eBGP announcement
> Will receive traffic for our network
> except traffic matching some criteria
> Think of BGP flowspec as a transport protocol for firewall rules
> Rules are implemented in the edge routers
> Rules: closest match wins (cleanup not really required)
> Add start and end time on top

19. 29 May 2017 19
> Simple static filtering - not opt-in
announce flow route default-discard-udp-fragments { match { ; protocol udp; fragment [ is-fragment first-fragment last-fragment ]; } then { discard } }
announce flow route default-discard-ntp-amplification { match { ; protocol udp; source-port =123; packet-length =468; } then { discard } }
announce flow route default-discard-dns-amplification { match { ; protocol udp; source-port =53; packet-length =512; } then { discard } }
announce flow route default-discard-dns-amplification { match { ; protocol udp; source-port =19; } then { discard } }
announce flow route default-discard-chargen { match { ; protocol udp; source-port =19; } then { discard } }
announce flow route default-discard-chargen { match { ; protocol tcp; source-port =19; } then { discard } }
announce flow route default-discard-QOTD { match { ; protocol udp; source-port =17; } then { discard } }
announce flow route default-discard-QOTD { match { ; protocol tcp; source-port =17; } then { discard } }
announce flow route default-discard-gre { match { ; protocol =47; source-port =17; } then { discard } }
announce flow route default-ratelimit-SSDP { match { ; protocol udp; source-port =1900 } then { rate-limit 9600; } }
announce flow route default-ratelimit-snmp { match { ; protocol udp; source-port =161&=162; } then { rate-limit 9600; } }

20. 29 May 2017 20
> A better solution
> Prefer BGP flowspec: integrates seamlessly with existing infrastructure and services
> Opt-in: use on-site probe (FastNetMon)
> Use WAYF for Web-UI authorization
> Non commercial version of FastNetMon, home grown Web-UI, daemon etc
> GUI and decentralized probes to inject rules in a central database
> No direct database access, no fancy interface, use sftp and file upload
> Access only granted through OpenVPN - with few exceptions (protecting the GUI)
> Short-lived blocking only: Not first line firewall with permeant rules

21. 29 May 2017 21
> Design principles
> Precautionary principle: Opt-in
> Both Self-service and Automatic mitigation
> Least privilege / least authority: restrict rule creation to specific network(s)
> Fail safe: do nothing on e.g data failure
> Automatic remove access for administrators who are no longer employed
> Automatic mitigation
> Time critical, mitigation start time/end time, logging and status
> Modular building blocks: software components should be replaceable
> Limit hardware requirements
> install on-site: On-site is 10Gb, core is 100Gb
> 2 x interfaces: one for monitoring (mirror port) one for uplink (assume behind NAT)

22. 29 May 2017 22
> Design
> Redundancy where technical and economical feasible
> Redundant functionality or fast recovery procedure elsewhere
> Decentral probes based on FastNetMon
> Central rule database with Web-UI etc.
> ExaBGP hosts, firewalls, routers etc.

23. 29 May 2017 23
> Components
> FastNetMon host
> Debian8, ixgbe and ixgbe drivers
> 2 connections: mirror port and uplink: (out of band: ADSL/LTE/3G … - behind NAT)
> FastNetMon, influxdb, notify_script, OpenVPN, ssh, configuration
> Two way connections: VPN required for access to influxdb
> DDPS host
> OS, Ubuntu 16.04
> PostgresDB, daemon,
> Web-UI, cli tools
> tool to create ISO images for unattended installation (Debian and Ubuntu)
> ExaBGP, routers etc.

24. 29 May 2017 24
> FastNetMon
> Hardware: Supermicro X10SLL-F with Intel NIC 82599
> Debian 8, looking at Ubuntu 16.04 / FreeBSD 11.1 / Debian 9
> Pktgen-DPDK
> The latest version is located on dpdk.org at http://dpdk.org/download
> Traffic generator powered by Intel's DPDK at 10Gbit wire rate traffic with 64 byte
frames.
> pkt-gen -i eth1 -z -f tx -n 500111222 -l 60 -d a.b.c.d:123
> Handles up to 14 million 64 bit UDP packets
> Tested with FastNetMon / package generator host back-to-back on - Debian 8

25. 29 May 2017 25
> FastNetMon
> Commercial: all the nice options but license requires routable IANA address
> Community edition: perl script, which execute git, cmake etc.
> A new version each time some changes something on git
> Looking at FreeBSD - is in ports together with netmap drivers
> We use OpenBSD, Ubuntu and Debian (and GAiA and SPLAT)
> Working towards compile once - use everywhere on similar OS
> statical linking and cmake (not funny)
> statifier fails (Kernel unble to load executable where phrs took more then
PAGE_SIZE)
> More work here

26. 29 May 2017 26
> FastNetMon on Debian 9
> echo 'deb http://ftp.de.debian.org/debian sid main' > /etc/apt/sources.list.d/fastnetmon.list
> apt-get -y update; apt-get -y upgrade; apt-get -y dist-upgrade
> apt-get -y install fastnetmon
> but for now drivers ixgbe and igb must be compiled from source each time the kernel is
upgraded
> Currently lack hardware to test:
Comparison of performance
and options
Debian 9 Native Intel Pavel Odintsov’s patched
igb ? ? ?
ixgbe ? ? ?

27. 29 May 2017 27
> Connections
> FastNetMon mirror port,
> FastNetMon alerts to database,
> FastNetMon default gateway may be
behind NAT / cellular uplink may have
> Download limitations
> Keep influxdb data with
FastNetMon
> Web-UI to influxdb
> LAN VPN required

28. 29 May 2017 28
> Automatic mitigation walk through
> FastNetMon connected to mirror port detects abnormality
> FastNetMon runs alert script
> Alert script reads TCP dump from STDIN, creates an rulefile and uploads it to the database
host with sftp through OpenVPN tunnel
> Daemon on DDPS host reads rulefile and reduces it to one rule which is inserted in the
database
> Daemon on DDPS reads database and activate new rules and deactivate expired rules
> ExaBGP announces rule changes to infrastructure

29. 29 May 2017 29
> Automatic mitigation walk through - drawing

30. 29 May 2017 30
> Implementation and security
> Limit attack surface
> Do not expose Web-UI unrestricted to the public
> Place behind VPN to limit attach surface
> Limit access to FastNetMon: no console / customer access
> All (relevant) parameters available to customer through Web-UI

31. 29 May 2017 31
> Security and Data validation
> Hierarchy of customers, their administrator right and networks
> FastNetMon:
> Only upload valid data (rule files based on TCP dump output)
> Web-UI: select from predefined choices
> validate data in Web-UI
> CLI:
> Require SSH access, implements full flowspec, minimal sanity check
> DDPS daemon
> Validate data before inserting in database
> Validate data from database before sending to ExaBGP
> Last check: destination addresses must be ours

32. 29 May 2017 32
> Rule file format
> Originally based on tcpdump output from
FastNetMon
> Upload with sftp
> Read literately or optimize before insert in
database
> format:
> Header:
> Lines ..
> last-line

33. 29 May 2017 33
> Rule optimization
> Optimize output from FastNetMon (TCP dump): reduce to one rule
> One stream one rule file
> Find closest match for all non-null fields:
> Don’t match on stateless spoofed sources
> Do match on state full source (e.g established tcp connections)
> Source and destination ports or port ranges
> Package lengths: same or in a range
> Match on fragment?
> Match on TCP flags
> Etc

34. 29 May 2017 34
> Rule optimization cont.
Attack type Mitigation Match on
syn_flood rate-limit tcp option (syn) protocol, destination port, tcp flags, size, (ttl would be nice but
is still in draft), size, and source any
udp_flood rate-limit protocol and destination, size, host and port
icmp flood discard protocol and destination
ip_fragmentation_flood rate-limit protocol size, and destination
DNS amplification rate-limit protocol, size, port and destination
NTP amplification rate-limit protocol, size, port and destination
SSDP amplification discard protocol, size, port 1900, source any
SNMP amplification discard protocol, size, port, destination

35. 29 May 2017 35
> How to handle failure and errors
> Phone
> Panic button: restart 2 x ExaBGP
> Add more specific accept rule (cli only)
> Expire rule(s) now (cli only)

36. 29 May 2017 36
> ddpsrules (cli)
/opt/db2dps/bin/ddpsrules [-v] add [-h] ... | del ... | active | log
active:
Print active rules with rule id's from database
del:
Set expire time to now for rule matching (list of) rule id(s)
add:
--blocktime|b minutes
--dst|D destination: one cidr only (database type limitation)
--src|S source: one cidr only (database type limitation)
--protocol|P protocol:
--dport|d destination port
--sport|s source port
--icmp_type|t icmp type
--icmp_code|c icmp code
--tcpflags|T TCP tcpflags
--length|l package length
--dscp|C DSCP flags
--frag|f fragments
--action|a action: accept discard or 'rate-limit 9600'
flowspec syntax (exabgp) is accepted for all parameters but IP addresses
e.g.
Specify http and https only
-P '=80 =443'
Specify length: 3 specific all more than 300 or less than 302
-l '=205 =206 =207 >=300&<=302'
Specify fragments and TCP tcpflags
-f '[not-a-fragment dont-fragment is-fragment first-fragment last-fragment]'
-T '[fin syn rst push ack urgent]'

37. 29 May 2017 37
> Web-UI
> All Rules Web-UI -
connections etc.

38. 29 May 2017 38
> Web-UI — see https://github.com/deic-dk/gossamer
> Dashboard with the
most relevant network
information

39. 29 May 2017 39
> Firewalls and VPN
> OpenBSD cluster and OpenVPN in cluster
> OpenVPN:
> HMAC in first UDP packet
> Re-establish a client session, after a disconnection
> OpenVPN clients (FastNetMon) served DNS, info, internal authoritative domain
> OpenVPN users: no DNS, split VPN
> Unbound & nsd - recursive, and caching and authoritative DNS.
> pf: force DNS and NTP to predefined servers (localhost), without the clients knowledge
> Same time for all system components

40. 29 May 2017 40
> ExaBGP, routers etc
> Part of our network infrastructure
> Configuration is memory only: deliberately designed not to survive restart

41. 29 May 2017 41
> Deployment and development
> GitHub
> Unattended installation from ISO
> FastNetMon, 10Gb drivers etc. customer specific OpenVPN configurations
> DDPS database server, Web-UI and tools
> Tool to modify ISO images

42. 29 May 2017 42
> Status
> Rough documentation and not everything on github
> No code review yet: probably full of errors
> GUI has missing parts
> Web-UI / Web-server and database not yet cluster aware
> (Probably something else I’ve forgot)

43. 29 May 2017 43
> FastNetMon detection limitation
> Only certain volumetric attacks
> Not attacks masked as legitimate traffic
> Not attacks on the application layer (expensive API calls, Slow Loris based attacks (sending a
partial request filling the web-servers maximum concurrent connection pool) - one source
many requests, small amount of data

44. 29 May 2017 44
> Who made DDPS
> Anders Mundt Due
> Ashokaditya Mohanty
> Kasper Sort
> Nicolai Ernst
> Niels Thomas Haugård
> Tangui Coulouarn
> Who made DDPS possible
> Pavel Odintsov - FastNetMon - and a long list of the people who made postgres, languages,
drivers and operating system

45. S29 May 2017 45
> Questions