Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
LogStash in Action
What’s In It for You?
Data ingestion workhorse
Events enrichment and transformation
Extensible plugin ecosystem
Pluggable pipeline architecture
...
Installation and Configuration
The Pre-requisites
Installation and Configuration
Prerequisites
requires Java 7 or higher
Installation steps
Download from...
The Service Commands
Installation and Configuration
Video Demonstration
Installing LogStash
Play Video
The Service Commands
Installation and Configuration
Start or stop service commands :
sudo /etc/init.d/logstash start
sudo ...
Simple Pipeline
Installation and Configuration
Verify LogStash installation
with a simple pipeline
Will take input from co...
bin/logstash -e 'input { stdin { } } output { stdout {} }'
Simple Pipeline
Installation and Configuration
4
5 # Simple LogStash configuration
6
7 Inputs {
8 Stdin { }
9 }
10
11 Output {
12 Stdout { }
13 }
14
Simple Pipeline
Inst...
Video Demonstration
Configuring a Simple Pipeline
Play Video
Advanced Pipeline
Installation and Configuration
Real world pipelines contain one or
more input, filter and outputs
Is gen...
Skeleton LogStash Configuration
Installation and Configuration
4 # The # character at the beginning of
5 # a line indicate...
LogStash Plugins
Installation and Configuration
LogStash Instance
Data Source ElasticSearch
Filter
Plugin
Output
Plugin
In...
Input
Plugin
elasticsearch
file
imap
jdbc
stdin
s3
syslog
tcp
twitter
udp
Filter
Plugin
csv
date
drop
grok
mutate
range
sleep
translate
Output
Plugin
csv
elasticsearch
email
file
mongodb
stdout
s3
syslog
tcp
udp
4 # ElasticSearch input plugin
5
6 Input {
7
8 # Read all documents from ElasticSearch
9 # matching the given query
10
11 ...
3
4 # File input
5
6 Inputs {
7
8 # Read events from file or folder
9
10 file {
11 path => “/var/log/ * ”
12 exclude => “ ...
4 # JDBC input
5
6 Input {
7
8 # Read all records from mySQL
9 # database
10
11 jdbc {
12
13 jdbc_driver_library => “/opt/...
3 # AWS S3 input
4
5 Input {
6
7 # Read all documents from AWS S3
8
9 s3 {
10
11 bucket => “my-bucket “
12 credentials => ...
4 # TCP input
5
6 Input {
7
8 # Read all events over TCP socket
9
10 tcp {
11 port => “ 5000 “
12 type => “ syslog “
13 }
...
4 # UDP input
5
6 input {
7
8 # Read all events over UDP port
9
10 udp {
11 port => “5001”
12 type => “netflow”
13 }
14
15...
Filter-csv
Installation and Configuration
2
3 # CSV filter
4 filter {
5
6 csv {
7
8 # List of columns as they appear in cs...
2
3 # date filter
4
5 filter {
6
7
8 date {
9
10 match => [ “logdate” , “MMM dd HH:mm:ss” ]
11 # Default for target is @ti...
Filter-drop
Installation and Configuration
2
3 # drop filter
4
5 filter {
6
7 # drop the events of their loglevel is debug...
Filter-range
Installation and Configuration
2
3 # range filter
4
5 filter {
6 range {
7 ranges => [“request_time” , 0, 10,...
Filter-grok
Installation and Configuration
Grok is one of the most
widely used plugin
It is instrumental in parsing
arbitr...
Filter-grok
Installation and Configuration
2
3 # grok filter
4
5 input {
6 file {
7 path => “/ var/log/http.log”
8
9 # sam...
Filter-grok
Installation and Configuration
2
3 # grok filter
4
5 input {
6 file {
7 path => “/ var/log/http.log”
8
9 # sam...
Filter-grok
Installation and Configuration
43 # ( ? < field_name> the pattern here
44
45
46 ( ? < message_id> [0-9A-F] {10...
24
25 # grok filter
26
27 filter {
28
29 grok {
30
31 patterns_dir => [ “~/patterns” ]
32 match => { “message” => “% {SYSL...
Filter-mutate
Installation and Configuration
Filter-sleep
Installation and Configuration
Filter-translate
Installation and Configuration
Output-csv
Installation and Configuration
Output-file
Installation and Configuration
Output-stdout
Installation and Configuration
Output-elasticsearch
Installation and Configuration
Output-email
Installation and Configuration
Output-s3
Installation and Configuration
Output-tcp
Installation and Configuration
LogStash in action
LogStash in action
LogStash in action
Upcoming SlideShare
Loading in …5
×

LogStash in action

Installation and configuration of Logstash. https://goo.gl/oZIrwS

  • Login to see the comments

LogStash in action

  1. 1. LogStash in Action
  2. 2. What’s In It for You?
  3. 3. Data ingestion workhorse Events enrichment and transformation Extensible plugin ecosystem Pluggable pipeline architecture Horizontally scalable data processing pipeline Strong Elasticsearch and Kibana synergy Handles data of all shapes and sizes Key Features of LogStash What’s In It for You?
  4. 4. Installation and Configuration
  5. 5. The Pre-requisites Installation and Configuration Prerequisites requires Java 7 or higher Installation steps Download from elastic.co web site Use Linux package manager to install LogStash Install LogStash as a service
  6. 6. The Service Commands Installation and Configuration
  7. 7. Video Demonstration Installing LogStash
  8. 8. Play Video
  9. 9. The Service Commands Installation and Configuration Start or stop service commands : sudo /etc/init.d/logstash start sudo /etc/init.d/logstash stop sudo /etc/init.d/logstash restart sudo /etc/init.d/logstash status
  10. 10. Simple Pipeline Installation and Configuration Verify LogStash installation with a simple pipeline Will take input from command line and output it back to the command line Pipeline configuration information is passed as text on command line Takes input from standard input “stdin” Outputs to standard output “stdout” in a structured format
  11. 11. bin/logstash -e 'input { stdin { } } output { stdout {} }' Simple Pipeline Installation and Configuration
  12. 12. 4 5 # Simple LogStash configuration 6 7 Inputs { 8 Stdin { } 9 } 10 11 Output { 12 Stdout { } 13 } 14 Simple Pipeline Installation and Configuration
  13. 13. Video Demonstration Configuring a Simple Pipeline
  14. 14. Play Video
  15. 15. Advanced Pipeline Installation and Configuration Real world pipelines contain one or more input, filter and outputs Is generally provided in a configuration file rather than command line Supplied to LogStash with –f command line argument Test the configuration using -- configtest argument
  16. 16. Skeleton LogStash Configuration Installation and Configuration 4 # The # character at the beginning of 5 # a line indicates comment 6 # Use the comments to describe your configuration 7 input { 8 input1 { } 9 input2 { } 10 } 11 12 # The filter part of this file is 13 # commented out to indicate that 14 # it is optional. 15 16 # filter { 17 # } 18 19 output { 20 output1 { } 21 output2 { } 22 }
  17. 17. LogStash Plugins Installation and Configuration LogStash Instance Data Source ElasticSearch Filter Plugin Output Plugin Input Plugin
  18. 18. Input Plugin elasticsearch file imap jdbc stdin s3 syslog tcp twitter udp
  19. 19. Filter Plugin csv date drop grok mutate range sleep translate
  20. 20. Output Plugin csv elasticsearch email file mongodb stdout s3 syslog tcp udp
  21. 21. 4 # ElasticSearch input plugin 5 6 Input { 7 8 # Read all documents from ElasticSearch 9 # matching the given query 10 11 elasticsearch { 12 hosts => “localhost” 13 index => “blogs” 14 query => ‘{ “localhost” : { “match_all” : { } } }’ 15 type => “my-data-elasticsearch” 16 } 17 } 18 Input-ElasticSearch Installation and Configuration
  22. 22. 3 4 # File input 5 6 Inputs { 7 8 # Read events from file or folder 9 10 file { 11 path => “/var/log/ * ” 12 exclude => “ * .gz ” 13 sincedb_path => “ /dev/null ” 14 start_position => “ beginning “ 15 type => “ my-data-csv” 16 } 17 } 18 Input-File Installation and Configuration
  23. 23. 4 # JDBC input 5 6 Input { 7 8 # Read all records from mySQL 9 # database 10 11 jdbc { 12 13 jdbc_driver_library => “/opt/logstash/lib/mysql-connector-java-5.1.6-bin.jar “ 14 jdbc_driver_class => “com.mysql.jdbc.Driver ” 15 jdbc_connection_string => “jdbc : mysql : // localhost : 3306 / mydb ” 16 jdbc_user => “root ” 17 jdbc_password => “password ” 18 statement => “SELECT * from users “ 19 20 } 21 } 22 Input-jbdc Installation and Configuration
  24. 24. 3 # AWS S3 input 4 5 Input { 6 7 # Read all documents from AWS S3 8 9 s3 { 10 11 bucket => “my-bucket “ 12 credentials => [ “my-aws-key “ , “my-aws-token “ ] 13 region_endpoint => “us-east-1 “ 14 codec => “json “ 15 16 } 17 } 18 Input-s3 Installation and Configuration
  25. 25. 4 # TCP input 5 6 Input { 7 8 # Read all events over TCP socket 9 10 tcp { 11 port => “ 5000 “ 12 type => “ syslog “ 13 } 14 15 } Input-tcp Installation and Configuration
  26. 26. 4 # UDP input 5 6 input { 7 8 # Read all events over UDP port 9 10 udp { 11 port => “5001” 12 type => “netflow” 13 } 14 15 } 16 Input-udp Installation and Configuration
  27. 27. Filter-csv Installation and Configuration 2 3 # CSV filter 4 filter { 5 6 csv { 7 8 # List of columns as they appear in csv 9 column => [ “column_1” , “column_2” ] 10 column => { “column_3” => “integer” , “column_4” => “boolean” } 11 type => “syslog” 12 13 } 14 15 } 16
  28. 28. 2 3 # date filter 4 5 filter { 6 7 8 date { 9 10 match => [ “logdate” , “MMM dd HH:mm:ss” ] 11 # Default for target is @timestamp 12 target => “logdate_modified” 13 14 } 15 } Filter-date Installation and Configuration • Used for parsing dates and use as LogStash event timestamp in ISO8601 format • For example “Jan 01 10:40:01” can be parsed using the pattern “MMM dd HH:mm:ss”
  29. 29. Filter-drop Installation and Configuration 2 3 # drop filter 4 5 filter { 6 7 # drop the events of their loglevel is debug 8 drop { 9 if [ loglevel ] = = “debug” { 10 drop { } 11 } 12 } 13 } 14 15
  30. 30. Filter-range Installation and Configuration 2 3 # range filter 4 5 filter { 6 range { 7 ranges => [“request_time” , 0, 10, “tag: short” , 8 “request_time” , 11, 100, “tag: medium”, 9 “request_time” , 101, 1000, “tag: long”, 10 “request_time” , 1001, 100000, “drop”, 11 “request_length” , 0, 100, “field: size: small”, 12 “request_length” , 101, 200, “field: size: normal”, 13 “request_length” , 201, 1000, “field: size: big”, 14 “request_length” , 1001, 100000, “field: size: hugel”, 15 “number_of_requests” , 0, 10, “tag: request_from_%{host}” ] 16 } 17 }
  31. 31. Filter-grok Installation and Configuration Grok is one of the most widely used plugin It is instrumental in parsing arbitrary and unstructured text into structed and queryable data field It is widely used to parse syslog, apache logs, mySQL logs, custom application logs, postfix logs etc. Grok works based on patterns Syntax for grok pattern is %{SYNTAX:SEMANTIC} Custom patterns can be added
  32. 32. Filter-grok Installation and Configuration 2 3 # grok filter 4 5 input { 6 file { 7 path => “/ var/log/http.log” 8 9 # sample log entry 10 # 55.11.55.11 GET/ index.html 453 12 11 } 12 } 13 14 filter { 15 # parse http log 16 17 grok { 18 19 match => { “message” => “% { IP: client} %{WORD: method} %{URIPATHPARAM: request} %{NUMBER: duration}” } 20 21 } 22 } 23
  33. 33. Filter-grok Installation and Configuration 2 3 # grok filter 4 5 input { 6 file { 7 path => “/ var/log/http.log” 8 9 # sample log entry 10 # 55.11.55.11 GET/ index.html 453 12 11 } 12 } 13 14 filter { 15 # parse http log 16 17 grok { 18 19 match => { “message” => “% { IP: client} %{WORD: method} %{URIPATHPARAM: request} %{NUMBER: duration}” } 20 21 } 22 } 23 Grok supports custom patterns • inline custom pattern using Oniguruma syntax • file based custom patterns
  34. 34. Filter-grok Installation and Configuration 43 # ( ? < field_name> the pattern here 44 45 46 ( ? < message_id> [0-9A-F] {10, 11}
  35. 35. 24 25 # grok filter 26 27 filter { 28 29 grok { 30 31 patterns_dir => [ “~/patterns” ] 32 match => { “message” => “% {SYSLOGBASE} %{POSTFIXQUEUEID: queue_id}: %{GREEDYDATA:syslog_message}” } 33 34 } 35 } Filter-grok Installation and Configuration
  36. 36. Filter-mutate Installation and Configuration
  37. 37. Filter-sleep Installation and Configuration
  38. 38. Filter-translate Installation and Configuration
  39. 39. Output-csv Installation and Configuration
  40. 40. Output-file Installation and Configuration
  41. 41. Output-stdout Installation and Configuration
  42. 42. Output-elasticsearch Installation and Configuration
  43. 43. Output-email Installation and Configuration
  44. 44. Output-s3 Installation and Configuration
  45. 45. Output-tcp Installation and Configuration

×