An ingite talk given at DataGotham 2012 about how we extract security related events and alerts from our logs. I repeated the same talk at DevOpsDays NYC 2013.

1. Safely Drinking From The Fire Hose @jschauma
Jan Schaumann
Señor Network Security Engineer
jschauma@etsy.com
B60D A9F7 0D89 544A 7995
7D25 5A5B 4375 275F 0BB5

2. I <3 logs! @jschauma
web logs
mail logs
system logs
vpn logs
2
08/28/12

3. Log Bongzilla, aka Splunk @jschauma
Logs go in…
Is this how Octocat
came to be?
ts
come
ler out
ri ty a
secu
2
08/28/12

4. Splunk Alerts FTW! @jschauma
YO DAWG, I HERD YOU LIKE LOGS
SO I PUT SOME LOGS IN YOUR LOGS
SO YOU CAN SPLUNK WHILE YOU SPLUNK
2
08/28/12

5. sudo make me a sandwich @jschauma
2
08/28/12

6. Know your patterns. @jschauma
VPN Connections
July 4th was a Wednesday
People making up for
People slacking off early
last week?
on a Friday, eh?
5
08/28/12

7. That was unexpected… @jschauma

8. XSS detection @jschauma
Announcement of
Bug Bounty program:
http://is.gd/UTZ5wD
code push to address
reported vulnerabilities
6
08/28/12

9. Geolocate all the things! @jschauma
3
08/28/12

10. XSS detection @jschauma
IP : 79.182.16.1 - bzq-79-182-16-1.red.bezeqint.net
Geolocation : Even Yehuda, 02, IL
Whois : *SE4-DRP*, RIPE, BEZEQINT-BROADBAND
Requests : 146
Method : GET
URL : /suggest_username.php?first-name=test&last-name=
onerror%3Dalert(0)%3E&email=shai%40exploit.co.il
13 minutes after we announced
our security bug bounty program
http://is.gd/UTZ5wD
Method : POST
URL : /your/profile
Data : u'fb_avatar_url=&gender=female&city3=&new_city=
"><img src=x onerror=prompt(1);>&new_region=&new_countrycode=
&new_latlon=,&city3_dup="><img src=x’
[…]
6
08/28/12

11. SQLi detection @jschauma
IP : 216.185.114.219 – unknown
Geolocation : Jurong East, 00, SG
Whois : ThePlanet.com Internet Services, Inc., ARIN, NET216
Requests : 20
Method : GET
URL : /listing/102946830/womens-shirt-beige-tunic-womens-
blouse?ref=999999.9%27+union+all+select+0x313032353438303035
36%2C0x31303235343830303536%2C0x31303235343830303536
%2C0x31303235343830303536%2C0x31303235343830303536%2
C0x31303235343830303536%2C0x31303235343830303536%2C0
x31303235343830303536%2C0x31303235343830303536%2C0x31
830303536%2C0x31303235343830303536%2C0x31303235343830
303536%2C0x31303235343830303536%2C0x31303235343830303
536+and+%27x%27%3D%27x
Method : GET
URL : /category/furniture?page=499999%27%20union%20
select%20unhex(hex(version()))%20
[…]
6
08/28/12

12. Know when people can’t log in… @jschauma
2
08/28/12

13. High number of failed logins @jschauma
Admin : <username> (<internal login>, <site login>)
IP : 64.124.192.210 - 64.124.192.210.t01419-07.above.net
Geolocation : Brooklyn, NY, US
Whois : ETSY Inc, ARIN, NET64
# of failed logins : 13
doesn’t know what he’s doing;
do not trust!
Admin : jschauma (jschauma, jschauma)
IP : 207.38.139.33 - 207-38-139-33.c3-0.avec-ubr2.nyr-
avec.ny.cable.rcn.com
Geolocation : New York, United States
Whois : RCN Corporation, ARIN, NET207
# of failed logins : 16
6
08/28/12

14. Geolocate all the things! @jschauma
4
08/28/12

15. “Unexpected” login detection @jschauma
Admin : <username> (<internal login>, <site login>)
IP : 83.160.48.31 - a83-160-48-31.adsl.xs4all.nl
Geolocation : Rotterdam, 11, NL
Whois : XS4ALL Internet BV, RIPE, DEMON-NL-DSL
Admin : <username> (<internal login>, <site login>)
IP : 217.192.56.102 – unknown
Geolocation : Zurich, 25, CH
Whois : The Hub Zuerich Assoc., RIPE, THE-HUB-ZUERICH-NET
Admin : <username> (<internal login>, <site login>)
IP : 24.231.49.240 - unknown
Geolocation : Nassau, 23, BS
Whois : Cable Bahamas, ARIN, CABLEBAHAMAS-NET
Admin : <username> (<internal login>, <site login>)
IP : 200.49.191.120 - map120.network49.191.tigo.net.gt
Geolocation : Guatemala City, 07, GT
Whois : COMCEL GUATEMALA S.A., LACNIC
6
08/28/12

16. I said: “Please insert girder!” @jschauma

17. Identify scrapers. @jschauma
Admin : <username> (<internal login>, <site login>)
IP : 50.17.73.70 - ec2-50-17-73-70.compute-1.amazonaws.com
Geolocation : Ashburn, VA, US
Whois : Amazon.com, Inc., ARIN, NET50
Provider : Amazon AWS
Count :7
Admin : <username> (<internal login>, <site login>)
IP : 207.228.237.110 – unknown
Geolocation : New York, NY, US
Whois : HopOne Internet Corporation, ARIN, NET207
Provider : HopOne
Count :1
6
08/28/12

18. Re-re-re-re-re-CAPTCHA @jschauma
source=”info.log" reCAPTCHA status="incorrect" | transaction ip |
where eventcount > 50 | table ip,eventcount | sort -eventcount
6
08/28/12

19. Of Liars and Outliers (good book, btw) @jschauma
wtf happened
here?
Ooh, right… this:
http://is.gd/fognju
http://is.gd/0hRDLY
http://is.gd/WxcA0r
6
08/28/12

20. This talk was too long! @jschauma
Log it now, log it all.
Geolocate all the things.
Build profiles. (Creepy, I know.)
Reduce false positives. (Whitelists!)
Have defined reactions to all alerts.
Notice the outliers.
Explain them.
That’s all, folks! Thanks!
2
08/28/12