SQL Ginsu (SANS @Night, SANSFIRE 2011)

SQL Ginsu
Better Living (And Data Reduction)
Through Databases

Normalize and reduce lots of data.
Provide quick and decisive insight.

Jul 21, 2011 SANSFIRE

©2011 Lewes Technology Consulting, LLC

Phil Hagen
8yrs Contract InfoSec/
Forensic work with DoD,
IC, LE, commercial
5yrs USAF InfoSec/Comm
BS: CompSci, USAFA
Contacts:
gplus.to/philhagen
@PhilHagen
stuffphilwrites.com


Why “SQL Ginsu”?

Photos:
lifeandtimesincleveland.blogspot.com/
©2011 Lewes Technology Consulting, LLC 2011/06/michael.html

↓ Time spent manually
reviewing data

Photos:

reviewing data

↓ Complexity of data for
clearer presentation

Photos:

reviewing data


↑ YOUR value to the
investigation!

Photos:

reviewing data


investigation!
Cuts a lead pipe and still
cuts wafer-thin tomatoes

Photos:

reviewing data


investigation!
Cuts a lead pipe and still
cuts wafer-thin tomatoes
Infomercials are funny
Photos:

Background
Data continues to grow exponentially
1995: 1GB @ $600; 2011: 1TB @ $90
[6,826x ↓ unit cost!]
Data sources increasingly diverse
3+ browsers, many databases, countless mobile
apps, tools, sites and services, cross-platform
inconsistencies
Core questions remain consistent:
who/what/where/why/when/how


Background
Value of the “all in one” forensic tool is not in question,
but...
...can no longer be the ONLY tool used in an examination
Staying flexible is key!
Cyclical nature of the collect/analyze/report processes
New leads generated - from the exam or elsewhere
Change in legal strategy
New personalities with different perspectives


Foundational Skill Sets


Foundational Skill Sets
In computer forensics and incident response, I contend
two foundational skills are:
Data management and data reduction
Many ways to accomplish, but we’ll focus on using SQL
SQL is very scalable, relatively universal, extremely
powerful and repeatable
Excel, text manipulation (sed, awk, cut, grep), etc are
also perfectly good tools for this


SQL: It’s Not that Hard

CREATE TABLE `tblname` (`col1` integer,
`col2` varchar(10));
INSERT INTO `tblname` (`col1`, `col2`)
VALUES (‘1’, ‘val2’);
SELECT `col2` FROM `tblname` WHERE
`col1` = 1;


OK, That’s Not Quite Fair
SQL can be extremely powerful, but not prohibitively complicated
JOINs: Combine data from different tables into one result set
Sub-SELECTs: Nested queries can simplify logic and
decrease need to master JOINs
UNIONs: Craft similarly-structured queries against different
data sets and present as one result set
INDEXes: Can speed queries and JOINs when created on
commonly-used columns
As with anything worthwhile, it’s an ongoing educational process


SQL Super Functions
Do analysis within SQL statements
SUM(), AVE(), MAX(), MIN(), COUNT(), STD()
Date/time addition/subtraction/slicing
Pull date or clock time from DATETIME values
Math, number formatting
Use most efficient data types
INET_ATON(), INET_NTOA()
Start writing the report with SQL: GROUP_CONCAT()


Step 1: Schema Design
Schema consists of column names and type definitions
Your data might already be databased somewhere!
log2timeline, SGUIL, Splunk
Adapt to developing requirements: ALTER TABLE
(or fix what you messed up when you started)
Doing this efficiently takes practice
Ranges of integers
“Normal Forms”


Step 1: Schema Tricks/Tips
Use an ‘interesting’ column

‘y’, ‘n’, ‘-’ values for later data reduction
Use 32-bit unsigned integers for IP addresses
Might want to use dotted quads too, if you expect on-the-
fly subnet-style queries (add/remove as needed)
Use indexes where sensible
Too many indexes decrease performance, increase storage
Focus on commonly-queried columns


Step 2: Data Load
Lots of data available - how do we normalize it for a
database? By any means possible!
Shell commands/scripts (CLKF!): awk, sed, cut, grep
Scripting languages: Python, Perl, PHP
Office apps: Excel, OOCalc
By any means possible!
Separate individual data items to craft SQL INSERTs

©2011 Lewes Technology Consulting, LLC http://blog.commandlinekungfu.com

Step 3: Data Reduction

“Load less data” or “classify after load”?
Load less? Might not be able to go back and get it!
Needs less disk space, simplifies queries
Load and classify? Might need lots of disk space
Queries can be slower and more complicated
Me? Load everything until resources are an issue


Step 4: Analyze! (Profit?)


Step 4: Analysis Tricks/Tips
Sub-SELECTs and UNIONs can drastically affect
speed
Sometimes a script or two-stage querysaves hours!
Save SQL statements with report for re-generation in
the future - especially with new data
Partially-tested, unverified, embarrassing but functional
scripts at stuffphilwrites.com/2011/07/sql-ginsu
Seriously, don’t laugh. Please.


Example 1: Login Records


Example 1, Step 1: Schema
CREATE TABLE `logins` (
ìd` int(11) unsigned NOT NULL
auto_increment,
ùserid` varchar(20) NOT NULL,
`systemname` varchar(20) NOT NULL
default '',
`src_ip` int unsigned NOT NULL,
`logintime` datetime NOT NULL,
Integer!
`logouttime` datetime NOT NULL,
ìnteresting` enum('y','n','-')
NOT NULL
default '-',

PRIMARY KEY (ìd`),
KEY `src_ip` (`src_ip`),
KEY ùserid` (ùserid`) Indexes
);


Example 1, Step 2: Load
Output of Linux “last -i” command:
phil pts/0 1.2.3.4 Thu May 5 16:39 - 22:02 (10+05:22)

“phil” = username, “pts/0” = terminal
“1.2.3.4” = source IP, “Thu May 5 16:39” = login date and time
“22:02” = logout (time only!), “10+05:22” = duration
INSERT INTO `logins` (`userid`, `src_ip`, `logintime`,
`logouttime`) VALUES (‘phil’, INET_ATON(‘1.2.3.4’),
‘2010-05-05 16:39:00’,
TIMEADD(‘2010-05-05 16:39’, ’10 05:22:00’));

Python script for Linux at stuffphilwrites.com/2011/07/sql-ginsu


Example 1, Step 3: Reduce

Eliminate known-good IPs, system accounts, date
ranges, etc
What pitfalls could this induce?


Example 1, Step 4: Analyze!
Session info for most frequently used source IPs
SELECT systemname, INET_NTOA(src_ip), COUNT(*) AS count
FROM logins
WHERE interesting != 'n'
GROUP BY systemname, src_ip
ORDER BY count DESC, src_ip;
Sessions with login duration > 1 day
SELECT *, INET_NTOA(src_ip),
TIMEDIFF(logouttime, logintime) AS duration
FROM logins
HAVING duration > '24:00:00'
ORDER BY duration DESC;


Daily login window per user
SELECT userid, systemname, COUNT(*),
MIN(TIME(logintime)), MAX(TIME(logintime))
FROM logins
GROUP BY userid,systemname;
IPs per username per host
SELECT userid, systemname, COUNT(*),
GROUP_CONCAT(DISTINCT INET_NTOA(src_ip)
SEPARATOR ', ')
FROM logins
GROUP BY userid, systemname
ORDER BY systemname, userid;


Example 2: Network Flows


Example 2, Step 1: Schema
CREATE TABLE `traffic` (
ìnteresting` enum
('y','n','-') default '-',
`sancp_id` bigint unsigned,
`start_time_gmt` datetime,
`stop_time_gmt` datetime,
èth_proto` smallint
unsigned,
ìp_proto` tinyint unsigned,
`src_ip` int unsigned, Smaller Integers
`src_port` smallint unsigned,
`dst_ip` int unsigned,
`dst_port` smallint unsigned,
`duration` int unsigned,
`src_pkts` bigint unsigned,
`dst_pkts` bigint unsigned,
`src_bytes` bigint unsigned,
`dst_bytes` bigint unsigned,
);


Example 2, Step 2: Load
Distill pcap (or live) network data to sessions with SANCP
Creates pipe-separated list of 50 fields per session
Use script to extract relevant fields
INSERT INTO `traffic` (`sancp_id`, `start_time_gmt`,
`stop_time_gmt`, `eth_proto`, `ip_proto`, `src_ip`,
`src_port`, `dst_ip`, `dst_port`, `duration`,
`src_pkts`, `dst_pkts`, `src_bytes`, `dst_bytes`)
VALUES (5613951026752893809, ‘2011-06-03 11:17:11’,
‘2011-06-03 11:17:25’, 8, 6, INET_NTOA(‘0.2.246.190’),
3306, INET_NTOA(‘0.3.162.24’), 14, 82, 43, 15866, 0);
Python script for pcaps at stuffphilwrites.com/2011/07/sql-ginsu

©2011 Lewes Technology Consulting, LLC http://metre.net/sancp.html

Example 2, Step 3: Reduce
Brute force SSH logins? Do one and observe network traffic
<5 sec && <45 packets && <2500 bytes

UPDATE traffic SET interesting='n'
WHERE (src_port=22 OR dst_port=22)
AND duration < 5
AND src_pkts+dst_pkts < 45
AND src_bytes+dst_bytes < 2500;
DNS, NTP can sometimes be ruled out

UPDATE traffic SET interesting='n'
WHERE eth_proto=8 AND ip_proto=17
AND (dst_port=53 OR dst_port=123);



SSH/SCP/SFTP sources
SELECT INET_NTOA(src_ip), duration,
INET_NTOA(dst_ip),
((src_bytes+dst_bytes)/1024/1024)
AS MB
FROM traffic
WHERE (src_port=22 or dst_port=22)
AND interesting != 'n';


Inbound FTP: multiple sessions used - two-stage query
SELECT GROUP_CONCAT(src_ip SEPARATOR ', ')
FROM traffic
WHERE (dst_port=21 AND dst_bytes>0);
SELECT INET_NTOA(src_ip), duration,
(src_bytes/1024/1024) AS src_MB,
(dst_bytes/1024/1024) AS dst_MB,
start_time_gmt, stop_time_gmt
FROM traffic
WHERE (src_port>1024 AND dst_port>1024)
AND src_ip IN (<list from prev query>) AND
interesting != 'n';


Bot beaconing
Add column

ALTER TABLE `traffic`
ADD COLUMN `elapsed` TIME DEFAULT NULL
AFTER `dst_bytes`;
During data load,“time last seen” = “srcIP:dstIP:dstport”
If you’ve seen the IP+IP+port before, insert time delta:

elapsed = TIMEDIFF(start_time_gmt,
<time_last_seen>);
Set “time last seen” in loader script to new
start_time_gmt

Wrap-up
Mo’ data, mo’ problems
Forensicators need foundational skills of:
data management and data reduction
Use SQL to do this:
1: Schema design
2: Data load
3: Reduce
4: Analyze!


Questions?


SQL Ginsu (SANS @Night, SANSFIRE 2011)

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

SQL Ginsu (SANS @Night, SANSFIRE 2011)

Editor's Notes