The document discusses using network flow data for applied detection and analysis. It describes how to collect flow data using tools like SiLK, analyze the data using rwfilter, rwstats and other SiLK tools, and use the results for detection via visualizations with FlowPlotter and intelligence gathering. Flow data provides benefits like small data footprint and scalability compared to full packet capture.

1. Applied Detection and Analysis
Using Network Flow Data
Chris Sanders & Jason Smith
Security Onion Conference 2014

2. Chris Sanders
• Christian & Husband
• Kentuckian and South
Carolinian
• MS, GSE, et al.
• Non-Profit Director
• BBQ Pit Master

3. Jason Smith
• Kentuckian
• Car Aficionado
• Raspberry Pi Enthusiast
• Junkyard Engineer

4. Applied Network Security Monitoring
“This book should be
required reading for all
intrusion analysts and those
looking to develop a security
monitoring program.”
“Written by analysts, for
analysts.”
- Amazon Reviewers

5. Agenda
Flow Data!
•Why it’s important
•How you can collect it
•What you can do with it
•Tool that’s can help
“[Why, How] to extend Security Onion with Flow Analysis.“

6. The NSM Cycle
Collection
Analysis Detection

7. Evolution of NSM Emphasis
Past
•Detection Era
Present
• Collection Era
Future •Analysis Era

8. NSM Challenges of the Present
We All Want Full PCAP…
•Collection
– Easy to Capture / Filter Stream Data
•Detection
– Major Detection Tools are PCAP Oriented
•Analysis
– Gives us Who, Where, When, and What

9. NSM Challenges of the Present
But, It’s not Feasible for Every Goal…
•Collection
– Not Scalable for Extended Retention
•Detection
– Not Ideal for Hunting / Rapid Pivoting
•Analysis
– Not a Great Starting Point

10. Enter Flow Data
• Often Called Flow / Session / NetFlow
• Summary of Network Communications
• Aggregated Record of Packets
• Gives Us Who, Where, When
• Based on the 5-tuple + Timing/Data Stats
Source IP Source Port Dest IP Dest Port Protocol
192.168.5.1 48293 8.8.8.8 53 UDP

11. Building Flow Records
• Records are Defined
by Unique 5-tuples
• Data is added to the
5-tuple Record until a
termination
condition is met.

12. Flow Record Termination Conditions
• Natural Timeout
– End of communication per protocol (ex. RST/FIN)
• Idle Timeout
– No data received for 30 seconds
• Active Timeout
– Thirty minute max timeout (configurable)

13. Full PCAP vs. Flow Data
PCAP Data Flow Data
Level of Context

14. Full PCAP vs. Flow Data
PCAP Data Flow Data
Storage Requirements

15. Flow Data Benefits
• Most Network Devices Generate it Natively
• Collectors are Easy to Setup
• Data Footprint is Incredibly Small
• Easy for Orgs to Keep Years of Flow Data
• Useful for Detection and Analysis

16. COLLECTION WITH FLOW DATA

17. Generating Flow Data
• Generation
– Routers
– Sensors
• Fprobe
• YAF
• Multiple Types:
– NetFlow (v5,v9)
– IPFIX
– jFlow
– More…

18. Collecting Flow Data
• Popular Platforms
– Argus
+ Reliable & Fast Collection
- Not Well Supported/Documented
– NFDump
+ Easy to Setup and Use
- Not in Wide Use
– SiLK
+ Exceptional Analysis Tools
- More Involved Setup

19. SiLK
• The System for Internet-Level Knowledge
• CERT NetSA Team
• Two Major Components:
– Packing Suite
• Collection and parsing of flow data
– Analysis Suite
• Filter, display, sort, count, group, mate, and more
• Excellent Documentation & Community
– https://tools.netsa.cert.org/silk/docs.html

20. SiLK Collection Architecture

21. SiLK – What You Need
 Flow Sources
− Hardware: Routers, Switches
− Software: YAF, fprobe
 SiLK Server
− Rwflowpack
− Will also have SiLK analysis suite installed
 Analyst Workstation
− Access SiLK server directly
− Locally mirrored database

22. SiLK – Packing Suite Config
 rwflowpack – Listens and sorts incoming flows, preparing them
for the analysis suite.
− --sensor-configuration
 Defines listener options
 Defines ipblocks
 Defines sensor probes
− --site-config-file
 Matches sensor probes with a naming convention
 Defines class and type relationships
− --root-directory
 Location where all binary flat files are stored
 Indexed: Type>Year>Month>Day>Hour

23. SiLK – Analysis Suite
 rwfilter - Filters through data based on conditions.
 rwcut - Converts flow binary data to a human readable format.
 rwstats - Generates statistics from flow data
 rwcount - Summarizes total network traffic over time

24. SiLK Analysis – rwfilter / rwcut (1)
 Display all records from the beginning the current day until the
current time:
rwfilter --type=all --proto=0-255 --pass=stdout | rwcut

25. SiLK Analysis – rwfilter / rwcut (2)
 Display all records of communication to or from Chinese IP
addresses over a specific week to one local CIDR range:
rwfilter --type=all --start-date=2014/08/01 --end-date=
2014/08/07 --any-address=192.168.1.0/24 --any-cc=cn
--pass=stdout | rwcut --
fields=stime,sip,dip,sport,dport,type

26. SiLK Analysis – rwstats (1)
 Display statistics for the total amount of bytes transferred by protocol (top
10):
rwfilter --type=all --proto=0-255 --pass=stdout |
rwstats --top --count=10 --fields=proto --
value=bytes

27. SiLK Analysis – rwstats (2)
 Show the top 10 sip,dip pairs for valid conversations (top 10)
rwfilter --type=all --proto=0-255 --packets=4, --
pass=stdout | rwstats --top --count=10 --
fields=sip,dip --value=bytes

28. SiLK Analysis – rwstats (3)
 Show the top 10 outbound destination country codes by
records:
rwfilter --type=out,outweb --proto=0-255 --
pass=stdout | rwstats --top --count=10 --fields=dcc

29. SiLK Analysis – Zero Access Example (1)
 Rwstats to discover potential victims
rwfilter --type=all --dport=16464,16465,16470,16471 --
pass=stdout | rwstats --top --fields=sip --
value=distinct:dcc --threshold=3
 Filter down to only the potential victim machine
rwfilter --type=all --start-date=2014/08/02 --end-date=
2014/08/03 --saddress=192.168.106.131 --
pass=ZA1.rwf

30. SiLK Analysis – Zero Access Example (2)
 Analyze the data per 10 minute buckets over the course of 24 hours to look
for abnormal user data at bizarre times.
rwfilter ZA1.rwf --type=all --proto=0-255 --active-time=
2014/08/02:00-2014/08/03:00 --pass=stdout | rwcut
--bin-size=600

31. Collecting Intelligence Data
• Friendly Intelligence Gathering
• Identify Services on the Network
• Identify Normal Behaviors of Hosts
• Identify “Friends and Family”
– Friends: Who a host communicates with outside
the network
– Family: Who a host communicates with inside the
network

32. Identifying Services
• Identify SSH Servers
rwfilter --type=out --protocol=6 --
packets=4- --ack-flag=1 --sport=22 -
-pass=stdout | rwcut --fields=sip
• Identify Web Servers
rwfilter --type=outweb --protocol=6
--packets=4- --ack-flag=1 --
sport=80,443,8080 --pass=stdout |
rwcut --fields=sip

33. Identifying Friends and Family
• Identify Friends
rwfilter --type=out,outweb --
saddress=192.168.1.1 --pass=stdout |
rwfilter --input-pipe=stdin --
dcidr=192.168.0.0/24 --fail=stdout
• Identify Family
rwfilter --type=out,outweb --
saddress=192.168.1.1 --pass=stdout |
rwfilter --input-pipe=stdin --
dipset=local --fail=stdout

34. DETECTION WITH FLOW DATA

35. Flow for Detection
Signature-
Based
Reputation-
Based
Behavior-
Based
Statistics-
Based

36. FlowPlotter
• Generates Visualizations from Output of Flow
Tools
• Useful for Detection-Oriented Statistics
• Written in BASH – Flexible/Tweakable
• Maintained in GitHub
• Browser Independent

37. FlowPlotter - GeoMap
rwfilter ../Sampledata/sample.rw --dcc=us,cn,-- --fail=stdout |
./flowplotter.sh geomap dcc bytes > geomap.html

38. FlowPlotter – Line Chart
rwfilter --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh linechart
600 bytes > linechart.html

39. FlowPlotter - TreeMap
rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --proto=0-
--type=all --pass=stdout | ./flowplotter.sh treemap dip records >
treemap.html

40. FlowPlotter - PieChart
rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --proto=0-
--type=all --pass=stdout | ./flowplotter.sh piechart dip bytes >
piechart.html

41. FlowPlotter - Barchart/ColumnChart
rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --proto=0-
--type=all --pass=stdout | ./flowplotter.sh columnchart dip bytes >
columnchart.html

42. FlowPlotter - BubbleChart
rwfilter ../Sampledata/sample.rw --type=all --proto=0-255 --pass=stdout
| ./flowplotter.sh bubblechart sip > bubblechart.html

43. FlowPlotter - Timeline
rwfilter ../Sampledata/sample.rw --proto=0- --dcc=us,-- --fail=stdout |
./flowplotter.sh timeline sip dip > timeline.html

44. FlowPlotter - Force Directed
rwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all --
pass=stdout | ./flowplotter.sh forceopacity sip dip distinct:dport 100 >
forcetest.html

45. FlowPlotter – Asset Discovery
rwfilter ../Sampledata/sample.rw --proto=0- --type=all --pass=stdout |
./flowplotter.sh assetdiscovery > assettest.html

46. ANALYSIS WITH FLOW DATA

47. Flow in Analysis – PCAP Only
Validate Signature TP < 1%
Scoping Relevant Time Range
Find Related Events in Time Range
Reduce Data Set
Analyze / Make Decisions
5%
10%
35%
~ 50%
* Based on the First Hour of Analysis

48. Flow in Analysis – w/ Flow Data
Validate Signature TP < 1%
Scoping Relevant Time Range
Find Related Events in Time Range
Expand Data Set as Needed
Analyze / Make Decisions
5%
10%
5%
~ 80%
* Based on the First Hour of Analysis

49. Flow – Barriers to Entry
• Be Prepared to Look at a LOT of Line-Based
Data
• Very Command Line Oriented
• Not Welcoming to Junior-Level Analysts
• Hard to Display/Interpret Data Visually

50. SiLK Data Output

52. • Flow Basic Analysis Tool
• Graphical Front-End to SiLK
• Easy Two-Step Install on SiLK Capable Box
– Install Locally to SiLK Box
– Install Remotely and Interact via SSH w/ Keys
• Rapid Pivoting Between Data
• Graphing Ability

53. Conclusion
• Flow Data is Underused and Underrated
• Easy to Collect, Enhances Detection & Analysis
• Minimal Barriers to Entry
– SiLK (Easy to Install on SO)
– Argus (Already Installed on SO)
– Bro (Already Installed on SO)

54. Thanks Folks!
• Questions?
– Chris Sanders: chris@chrissanders.org
– Jason Smith: jason.smith.webmail@gmail.com
• Blog/Book
– http://www.appliednsm.com
• FlowPlotter
– http://www.github.com/automayt/FlowPlotter
• FlowBAT – Release in October!