This talk covers the basics of centralizing logs in Elasticsearch and all the strategies that make it scale with billions of documents in production. Topics include: - Time-based indices and index templates to efficiently slice your data - Different node tiers to de-couple reading from writing, heavy traffic from low traffic - Tuning various Elasticsearch and OS settings to maximize throughput and search performance - Configuring tools such as logstash and rsyslog to maximize throughput and minimize overhead

1. From zero to production hero:
Log analysis with Elasticsearch
Rafał Kuć
Radu Gheorghe

2. Who are we?
RaduRafał

3. Our Company → Sematext
HQ: NYC + Globally Distributed Team
Search & Big Data Consulting
Production Support for Solr & Elasticsearch
Training for Solr & Elasticsearch (online and
onsite)
Training in NYC
next week!
Oct 19 & 20

4. Our Company → Sematext

5. Agenda
Kibana
Elasticsearch
essentials, tuning and scaling
Logstash
rsyslog
Logstash + rsyslog
Commands & Configs:
https://github.com/sematext/velocity

6. Lucene Essentials
{"verb": "GET"}
document

7. Lucene Essentials
{"verb": "GET"}
1)GETdocument
stored

8. Lucene Essentials
GET 1,3,5
PUT 2,4
{"verb": "GET"}
1)GETdocument
stored
indexed

9. Analysis
(Macintosh; Intel Mac OSX; en)
["Macintosh", "Intel", "Mac", "OSX", "en"]
["macintosh", "intel", "mac", "osx", "en"]
standard tokenizer
lowercase token filter

10. Field data
GET 1,2
PUT 2,3

11. Field data
GET 1,2
PUT 2,3

12. Field data
GET 1,2
PUT 2,3
1) GET
2) GET,PUT
3) PUT

13. Field data
GET 1,2
PUT 2,3
1) GET
2) GET,PUT
3) PUT
expensive

14. Field data
GET 1,2
PUT 2,3
1) GET
2) GET,PUT
3) PUT
expensive
heap

15. Field data
GET 1,2
PUT 2,3
1) GET
2) GET,PUT
3) PUT
expensive
heap
http://bio-img.s3.amazonaws.com/bds/formhdr-cvr-5-memory-killing-foods-v2.png

16. DocValues
GET 1,2
PUT 2,3
1) GET
2) GET,PUT
3) PUT
at index time;
on disk
https://www.lorextechnology.com/images/products/HDD250GB/900x600/security-certified-HDD250GB-L1.png

17. DocValues
GET 1,2
PUT 2,3
1) GET
2) GET,PUT
3) PUT
no uninverting!
at index time;
on disk
https://www.lorextechnology.com/images/products/HDD250GB/900x600/security-certified-HDD250GB-L1.png
OS caches
instead of heap

18. Logstash
/var/log/apache.log
GET /index.html
grok
{
"verb": "GET",
"path": "/index.html"
}
- w $numberOfWorkers
workers => 2
filter
output
input
Elasticsearch

19. rsyslog
/var/log/apache.log
GET /index.html
mmnormalize
{
"verb": "GET",
"path": "/index.html"
}
queue.workerThreads
queue.dequeueBatchSize
omelasticsearch
imfile input
module
Elasticsearch
main queue (RAM+Disk)
queue.type
queue.size
...

20. mmnormalize parse tree
sys
tem log
d -ng
=> scales very well with # of rules
(performance depends more on log length)

21. rsyslog + Redis via Kafka
rsyslog Apache Kafka Logstash Elasticsearch
file input
mmnormalize
omkafka +
JSON template
Kafka input +
JSON codec Elasticsearch
output

22. Free eBooks @ sematext.com
We are hiring too
http://sematext.com/about/jobs.html

24. Thank you!
Rafał Kuć
@kucrafal
rafal.kuc@sematext.com
Radu Gheorghe
@radu0gheorghe
radu.gheorghe@sematext.com
Sematext
@sematext
http://sematext.com