- Log Overview : What, Where, Why, How? - Log in OpenStack : In imagination vs in fact ! - Graylog for OpenStack : To infinity and beyond !

1. Graylog for OpenStack :
3 steps to know WHY

2. MediTech JSC
https://meditech.vn
Private Cloud
Storage
Monitor
Logging
Managed Services
About me
Dinh Van Manh
● System Integration Department in MediTechJSC
● Member of Hocchudong
● Interested in OpenStack, Linux, Monitoring, Logging
and new technology
● Habbit : “tra da + thuoc lao” with friends

3. Agenda
1. Log Overview
1.1. Logs : What & Where?
1.2. Why look at Logs
1.3. How to use Logs effectively
2. Log in OpenStack
2.1. OpenStack log statistics
2.2. OpenStack Log Management : in imagionation & in fact
3. Graylog for OpenStack
3.1. Introduce about Graylog
3.2. Key features
3.3. Architecture/Mechanism/Model of Graylog
3.4. Graylog for OpenStack: 3 steps to know WHY?
4. Demo + Q.A

4. Log Overview
What? Where? Why? How?

5. 1.1. Logs : What & Where
What logs? (from the view of system administrator)
● System event diary
● System status records
● User activities
● Incident notify
Log format

6. 1.1. Logs : What & Where
Log come from WHERE?
● Storage devices
● Application in Linux/Windows
● Cloud Services : OpenStack
● Servers
● Firewalls
● Routers, switches

7. 1.2. Why look at Logs?
Basically :
Incident response
higher
Tracking system event
higher
Measuring security : metrics, trends…
higher and higher
Situational awareness
New threat discovery
Estimating about user habit, trends...

8. 1.3. How to use Logs effectively
Level 1 : Just SSH and view !
● Understanding log location
● Command to view log : tail, more, grep
● Filtering by keyword
Level 2 : Use Syslog
● Collect syslog from client
● Store in log server
Level 3 : Log management Software
● Collect everything
● Retain most everything
● Analyze enough
● Summarize and report
● Advance features : visualize, alert, share...

9. 1.3. How to use Logs effectively
● Facility
○ Application Logs
○ Event Logs
○ Service Logs
○ System Logs
Log Keywords
● Severity
○ 0 - emerg
○ 1 - alert
○ 2 - crit
○ 3 - error
○ 4 - warn
○ 5 - notice
○ 6 - info
○ 7 - debug
● Rotention
○ Time to rotate log
● Retention
○ Delete, archive...log
● Syslog
○ protocol to transfer log

10. Log in OpenStack
Which level is appropriate?

11. 2.1. OpenStack log statistics
OpenStack System : 3 Controller + 30 Compute
node
● Controller Node
○ 6 log folder per OpenStack service
○ system log : auth, dmesg, kernel…
○ application log : apache, haproxy,
pacemaker…
● Compute Node
○ 2 log folder per OpenStack service
○ system log : auth, dmes, kernel…
○ application log : libvirt
○ log of instances
=> Total :
● ~ 220 log file
● 10 GB log = 30 million messages / day

12. 2.2. OpenStack log management : in imagionation & in fact
Communication think Colleagues think In fact
When i said : My job is OpenStack log management !
So Waste !!! What should we do?

13. Graylog for OpenStack:
To infinity & beyond !

14. 3.1. Graylog Introduce
● Log centralized management software
● Released in 2010 by Lenart Koopman with name is Graylog2
● In 1/2015 release Graylog v1., Graylog Inc was established
● Big change from Graylog version 2.0
● Newest version is Graylog 2.3.1, stable version is Graylog 2.3.0

15. 3.2. Key features
Various Input & Output Analyze & Search
Visualize metricAlert & Trigger User management

16. 3.3. Architecture/Mechanism/Model of Graylog
Overall architecture
● Server
○ Graylog
● Client
○ Client host
○ Graylog sidecar
○ Nxlog/Filebeat
Filebeat
Graylog Sidecar : Break the
old path
● Configuration management
system
● Config in client host only
ONCE !
● All in Web
● Secure with SSL/TLS

17. 3.3. Architecture/Mechanism/Model of Graylog
Sidecar Work-flow : Easy config in 3 steps
Step 1 : Config in client
● install sidecar
● declare : graylog ip, client hostname, tags
● start service
Step 2 : Config in Graylog Web
● add tags
● chose what logs you want to collect
Step 3 : Checking
● Check colleted log

18. 3.3. Architecture/Mechanism/Model of Graylog
Deep dive in architecture
Graylog Server
● receive log message
● execute log
● communicate with other components
Elasticsearch
● store log message
● search engine
MongoDB
● store meta infomation
● store config data

19. 3.3. Architecture/Mechanism/Model of Graylog
Log execute processing
Step 1 :
● Spooling & store in disk temporarily
● Prepare for buffer process
Step 2 :
● Messages from disk go in to Input Buffer
● Mission : Filter, classify messages
Step 3 :
● Messages go in to Output Buffer
● Onward to Elasticsearch or user defined
output

20. 3.3. Architecture/Mechanism/Model of Graylog
Elasticsearch & Graylog
● Clustering
● Use API to communicate
● Use unicast-discovery to recogize other nodes
● Graylog as a Master Node
MongoDB & Graylog
● Client - Server mechanism
● Graylog use driver to communicate with MongoDB
Internal Graylog components mechanisms

21. 3.3. Architecture/Mechanism/Model of Graylog
None HA - Small production HA - Bigger Production

22. Code show you HOW !
Log show you WHY !

23. 3.4. Graylog for OpenStack : 3 steps to know WHY?
Just 3 steps to exploiting log in OpenStack

24. 3.4. Graylog for OpenStack : 3 steps to know WHY?
What should i do when instance spawning fail
A. Try to spawn again B. Blame for customer
D. Bug again! I’m quit ! C. Take a search in Graylog
Incident Response
Problem appear ! What should we do?

25. 3.4. Graylog for OpenStack : 3 steps to know WHY?
Step 1 : Collect logs
Take log from :
● nova log
● neutron log
● cinder log
● glance log
● keystone
Step 2 : Analyze
Make a search in Graylog :
Syntax : instance id + ERROR
Step 3 : Now you know WHY
Just solve the problem & Go to sleep !

26. 3.4. Graylog for OpenStack : 3 steps to know WHY?
Tracking a event
My instances was rebooted last night ??? When?

27. 3.4. Graylog for OpenStack : 3 steps to know WHY?
Measuring metric

28. DEMO & Q.A

29. Bonus : Graylog vs ELK
Graylog is coming the closest to the Splunk architecture !
VS

30. Thank you !
ManhDV
manh.dinhvan@meditech.vn
https://meditech.vn/
https://github.com/hocchudong