Log Management with Elastaticsearch

1. Surasak Sanguanpong
Surasak.S@ku.ac.th
Applied Network Research Lab
Department of Computer Engineering
Faculty of Engineering, Kasetsart University
Software
Freedom
Day
2016
– Sept
17
Bangkok
Experiences
in
ELK
with
D3.js
for
Large
Log
Analysis
and
Visualization
U-Bahn Station Candid Plazt, Munich,Germany

2. In This Talk
2
Real
Time
Visualization
with
D3.js
Search
Platform
with
ELK
About
Traffic
Log
(KU
case
study)
Lessons
Learnt

3. Chapter I
Network Traffic Log Structure and Sizing
KU Case Study
3

4. Why keeping Log
• Legal compliant
• Troubleshooting
• Security analysis
• Statistics/Analytics
4

5. Log Monitoring
Collecting
Processing
Analysing
Visualising
5
Source:
https://www.flickr.com/photos/sbeebe/4772418919

6. Searching in Log
Find relevant stuff
Find it fast
Make our lives easier
6

7. Traffic Logging Solution
Splunk?
Great, but.. Commercial, Proprietary
Graylog?
Excellence, but too automatic
Elasticsearch, Logtash, Kibana (ELK), D3
That is!, open source, fun to play
7

8. KU Logging
2008-2015 2015-
8
Raw Log
MySQL
Simple Web GUI
Raw Log
Elasticsearch
Web GUI/Kibana/D3
• On the fly text based log to
MySQL converter
• Simple but slow
• Much faster!

9. KU Logging Structure
9
Network
Login
Portal
Search GUI
Logging
Engine
Packet
Capture
Socket
Login Log
Web Log
Packet Log
Multicore x86
with 10 GbE
Raw Log

10. Raw Login Log Format
• Real-time logging, one file per day
10
Date Time Action IP UserName LogServer
Jul 1 10:04:57 login 158.108.X.X XXXXX@ku.ac.th 192.168.1.1
Jul 1 10:04:58 logout 158.108.X.X YYYYY@ku.ac.th 192.168.1.2
Jul 1 10:04:59 timeout 158.108.X.X ZZZZZ@ku.ac.th 192.168.1.2

11. Raw Web Log Format
• Real-time logging, one file per minute
11
UnixTime SrcIPv4 SrcIPv6 DstIPv4 DstIPv6 SrcPort DstPort URL Referer/HTTPS
20151103010000 192.55.X.X - 158.108.X.X - 17490 80
mirror1.ku.ac.th/fedora-epel/6/i386/jday-devel-2.4-5.el6.i686.rpm
http://mirror1.ku.ac.th/fedora-epel/6/i386/
20151103010000 - 2406:3100:1018:1::XX - 2600:1417:a::174c:XX 61154 443
fbcdn-photos-g-a.akamaihd.net HTTPS
20151103010000 - 2406:3100:1018:1::XX - 2a03:2880:f002:105:fa:b0:0:YYXX
59960 443 edge-mqtt.facebook.com HTTPS

12. Raw Packet Log Format (Header Log)
• Real-time logging, one file per minute
12
TimeStamp SrcIP DstIP SrcPort Proto Size DstPort SrcPort [Flag]
2009-07-16 17:53:59.999206 208.117.8.X 158.108.234.X 1514 TCP 80 1371 0x10
2009-07-16 17:53:59.999209 158.108.2.X 202.143.136.X 90 UDP 123 123
TimeStamp SrcIP DstIP Proto Code
2009-07-16 17:53:59.999210 158.108.184.X 218.164.54.X ICMP 168

13. Time based Hierarchical Folder
13
Minutely
FileHourDayMonthYear
2015
01
01
00
201501010000.txt
201501010001.txt
:
201501010059.txt01
:
:
23
201501012300.txt
201501012301.txt
:
201501012359.txt
02
:
30
02
:
12

14. At What Scale?
Quite Large..
14Source:
http://www.24hourcampfire.com/ubbthreads/ubbthreads.php/topics/5976731/all/That_s_a_loa
d_of_logs
SPEED
400,000 req/s peak
STRUCTURE
Text/binary
SIZE
30 TB
3.2 trillion docs

15. Facts about KU
Accounts
113,XXX
4 Campuses
BKN, KPS, SRC, SKN
158.108.0.0/16
192.102.83.0/24
10.0.0.0/8
2406:3100::/32
2001:3c8:1303::/48
2001:f00:2003::/48
IPv4
IPv6
50,XXX
Concurrent Active IP Address
(25,XXX:Wifi)
Registered Devices
210,XXX
1,4XX
Access Points

16. System Structure
Internal
network
Internet
IPv4/IPv6 Parallel Firewalls with Load Balancers
Gateway
Router
Core
Router
Session Manager Login Servers
Quota Manager Traffic Logger
Case
Study
5x1 Gbps
1x10 Gbps

17. Sample Minutely HTTP Request Rate
17
11
days
(11x
24x60=
15,640
data
points)

18. Request Rate and Log Sizing
18
3.1
req/s
27
MB/d
2,100
req/s
33
GB/d
380,000
req/s
330
GB/d

19. Accumulated Log Request and Size
19
#Files
:
120
20M
2.04
GB
14.1B
2.57
TB
#Files
:
172,800
28.03
TB
3.27T
#Files
:
172,800

20. New Logging Architecture
20
Network
Login
Portal
Logging
Engine
Packet
Capture
Socket
Login Log
Web Log
Flow Log
Multicore x86
with 10 GbE
Raw Log
DHCP,
RADIUS
Session
Tracking &
Accounting
Elasticsearch
Real time
Indexing
GUI/
Analytics

21. Chapter II
ELK Stack Testbed
21

22. What is the Elasticsearch?
22
Real-time
Search
Engine SW
Document-
Oriented
JSON based
REST API
JAVA/
Lucene
based
Open
Source
Apache 2
License
REST: Representational State Transfer
JSON: JavaScript Object Notation

23. Elasticsearch and Database
• Roughly layout comparison
23
Relational Database
Database
Table
Row
Column
Schema
Elasticsearch
Index
Type
Document
Field
Mapping

24. Elasticsearch Logical Layout
24
Index: social Index: blog
Elasticsearch Node
Type: story Type: user Type: posts
1 2 1 2 1 2 3 4
Index Application
Search Application
• Using any HTTP client to talk to
Elasticsearch at localhost port 9200
• RESTful : Interact through common
HTTP methods (GET, POST, PUT,
DELETE)
• Not maintain state information
• Each request is independent and
resources are returned in JSON
text formats

25. How the world is using Elasticsearch?
25
Analytics solution on 40 million
documents per day to deliver
real-time visibility
Providing search across
GitHub's code
Full-text search to find related
questions and answers
Full-text search with highlighted
search snippets

26. Elasticsearch and Big Data
ES-Hadoop: Connectivity of Hadoop's big data analytics and the real-time search of Elasticsearch.
26
Source:
https://www.elastic.co/products/hadoop

27. What does Elasticsearch offer?
27
Full Text
Search
Very Fast
Fault
Tolerance
High
Availability
Distributed Scalable
Plugin
Architecture

28. Node and Cluster
28
CLUSTER
P0
Node 1
P1 R2 R0
Node 2
R1 P2
CLUSTER
P0
Node 1
P1 P2
Px: Primary Shard : Chunk of index
Rx: Replica Shard: Copy of Shard
A Node : A running Elasticsearch process
CLUSTER
P0
Node 1 Node 2
R0
Node 3
R1 R2P1 P2R2 R2 R2
(3 Shards/1 Replica)

29. When a Node Fails (3 Shards/1 Replica)
29
CLUSTER
P0
Node 1 Node 2
R0
Node 3
R1 R2P1 P2R2 R0 R1
CLUSTER
P0
Node 1 Node 2
R0
Node 3
P1 R2P1 P2R2 R2 R1

30. When a Node Fails (3 Shards/2 Replicas)
30
CLUSTER
P0
Node 1 Node 2
R0
Node 3
R1 R2P1 P2R2 R0 R1
CLUSTER
P0
Node 1 Node 2
R0
Node 3
P1 R2P1 P2R2 R2 R1

31. Elasticsearch documents
• Document : Basic unit of user data in JSON representation
• Sample Document :
31
{
"user" : ”Chris"
"gender" : ”M",
"birthdate" : "1980-12-11"
}

32. URI of a document
32
http://localhost:9200/sample_index/sample_type/1
Protocol used:
supported HTTP
Port to connect to
Using 9200 by default
Host name of the
Elasticsearch node
Index name
Type name
Document ID

33. HTTP based CRUD operation
Create
curl -XPUT “http://localhost:9200/<index>/<type>/<id>”
Read
curl -XGET "http://localhost:9200/<index>/<type>/<id>"
Update
curl -XPOST "http://localhost:9200/<index>/<type>/<id>"
Delete
curl -XDELETE "http://localhost:9200/<index>/<type>/<id>"
33

34. ELK stack from Elastic
34
Elasticsearch: High-
performance
scalable search engine
Logstash: Log transport
and processing daemon
(Log Shipper)
ELK Stack
Kibana: Visualisation
dashboard

35. Logstash
35
Log aggregator and parser
Transferring parsed data
to Elasticsearch
Configuration file for
specifying input, filtering
(parsing) and output
input
{
stdin {
}
}
filter
{
grok {
match
=>
{
"message"
=>
"%{COMBINEDAPACHELOG}"
}
}
date
{
match
=>
[
"timestamp"
,"dd/MM/yyyy:HH:mm:ss"
]
}
}
output
{
Elasticsearch {
hosts
=>
["localhost:9200"]
}
stdout {
codec
=>
rubydebug }}
Sample
Configuration
Source:
https://www.elastic.co/guide/en/logstash/current/config-­‐examples.html

36. Kibana
36
General purpose query UI
Includes many widgets
Query Elasticsearch without
coding

37. Alternative Stack
37
ELK
EFK

38. Indexing Performance (Weblog)
38
35
36
37
38
39
40
41
42
43
44
45
0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
THOUSANDS
MILLIONS
Daily
Performance
Indexing
#Records Records/s
• Dell R230
• Xeon E3-1271v3 3.6
Ghz 4C/8T
• Hyper-threading off
• 32 GB RAM
• 2x6 TB NLSAS
• Elasticsearch
2.3.2
• 10 Shards/0 Replica
Indexing
Performance
on
single
machine

39. Search Performance
Search keyword:
“ face” against each
daily log
Not yet Optimization
39
2.01
2.33
1.99
2.13
2.67
2.00
1.33
1.02
3.00
2.33
2.00
2.67
3.00
2.67
2.43
3.33
2.67
2.14
3.33
17,551
22,816
16,346
18,218
16,240
7,958
5,622
1,886
23,559
9,127
8,221
12,343
28,259
25,405
22,092
33,528
17,683
12,951
18,054
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
SEARCH
TIME
(MS)
Search
Performance
and
Hits
Search
Time
(ms) Hits

40. Sample Search Performance
40

41. Kibana: Main Dashboard
41

42. Kibana: Login Profile
42

43. Kibana: Concurrent Login View
43

44. Chapter III
Playing with D3.js
44

45. Real Time Visualization with D3.js
• Data-Driven
Documents (D3)
• JavaScript library for
manipulating
documents based on
data
• Developed by Mike
Bostock
45
https://d3js.org/

46. D3 Architecture
§ Input data to build
visualizations (JSON,
CSV,…)
§ Data manipulation of HTML
elements dynamically with
JavaScript
46
node.js
socket.io

47. Sample Gallery
47

48. Real-time makes impression
48
http://map.norsecorp.com/#/Norse
Live
Attack
Map

49. Sample Concurrent Login
49

50. Sample IP Matrix Occupied
50

51. Sample Tree Map Web Access
51

52. Sample Traffic Connectivity
52

53. Chapter IV
Lessons Learned
53

54. Lessons Learned
Elasticsearch offers a very fast full-text search services
Indexing size may 3x to 5x bigger than source data
Use Elasticsearch for search services, not for data archiving
More cores or Faster clock? : Choose CORES
64 GB of RAM is the ideal
Go with SSD if possible
54

55. Lessons Learned
Designed to work in a trusted environment
No built in security
No authentication or authorization, no concept of a user
Anyone that can send a request to the cluster is a super user
Easy to erase all the data:
curl –XDELETE http://<server>:9200/_all
55

56. Lessons Learned
Shield from Elasticsearch: A comprehensive security solution,
including encrypted communications, RBAC, AD/LDAP
integration and auditing
Use with a proxy : Authentication
and request filtering with nginx or others
56

57. Lessons Learned
Logstash : A powerful tool to manipulate log
Kibana : Simple and useful for visualize data
57

58. Lessons Learned
D3 pros:
Flexible, Fascinating Visualization
D3 cons:
Low Level, Steep Learning Curve, CPU intensive
58

59. Thank you for your attention
Q & A Time
Q&A…
59
Kasom Koth-Arsa
Core Log Design and Development
Jautuporn Chuchuay
Peerapol Boonthaganon
Web GUI Development
Sataporn Techaaramwong
Web/Elasticsearch Development
Peerapong Thongpubeth
Jiradech Sirijantadilok
Kibana Development
Poomipat Thongudom
Nichapat Nattee
D3 Development
Surachai Chitpinijyol
Project Coordinator
Surasak Sanguanpong
Project Director
Special Thanks to Kasetsart Office of Computer
Services for supporting traffic dataSunset at Narita Airport
Kasom Koth-Arsa
Core Log Design and Development
Jautuporn Chuchuay
Peerapol Boonthaganon
Web GUI Development
Sataporn Techaaramwong
Web/Elasticsearch Development
Peerapong Thongpubeth
Jiradech Sirijantadilok
Kibana Development
Poomipat Thongudom
Nichapat Nattee
D3 Development
Surachai Chitpinijyol
Project Coordinator
Surasak Sanguanpong
Project Director
Special Thanks to Kasetsart Office of Computer
Services for supporting traffic data