This document discusses using Logstash to collect, process, and store application logs. It begins by describing different types of logs that are generated by applications and services. It then introduces the ELK stack, consisting of Elasticsearch, Logstash, and Kibana, to centralize, index, and visualize log data. Specific examples are provided on using the Monolog PHP logging library to instrument applications and leverage Logstash's processing pipeline to parse, enrich, and output logs to Elasticsearch.

1. Application Logging
With Logstash

2. Ben Waine
• Worked With
PHP For 5Years
• Software
Engineer -
Sainsbury’s
• Dabbles in
devops

3. https://joind.in/talk/view/13369

4. System Logs

5. Application Log
Debug Information - Errors (connections,
uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls,
Event Triggers
Business Events - Purchases, Logins,
Registrations, Unsubscribes

6. Keeping Track Of All This....
ssh webserver@mydomain.net
tail -f /var/log/nginx/my-site.access.log
tail -f /var/log/my.application.log
ssh data@mydomain.net
tail -f /var/log/mysql/mysql.log
ssh q@mydomain.net
tail -f /var/log/rabbitmq/nodename.log

7. The Elk Stack

8. Visualizing Log Data

9. PHP Logging Tools
1) Monolog
2) Everything else....

10. Basic Logging Examples
1) Monolog: Loggers And Handlers
2) Monolog:Tags & Formatters
3) Logging business events

11. use MonologLogger;
use MonologHandlerFingersCrossedHandler;
use MonologHandlerStreamHandler;
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);
$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers

12. // Set A Log Level
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger
$appLog = new Logger('AppLog');

13. $strHandler
= new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler
= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');
$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging

14. EG 2:Tagging Formatting
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");

15. // Create A Logger
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger

16. $id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push

17. Log Levels
2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages
https://tools.ietf.org/html/rfc5424

18. Log Levels
2013 - PSR03 - PHP Logging Interface Standard
Phrase / Severity
emergency Emergency: system is unusable
alert Alert: action must be taken immediately
critical Critical: critical conditions
error Error: error conditions
warning Warning: warning conditions
notice Notice: normal but significant condition
info Informational: informational messages
debug Debug: debug-level messages
http://www.php-fig.org/psr/psr-3/

19. http://imgs.xkcd.com/comics/standards.png

20. EG 3: Event Logging
use MonologLogger;
use SymfonyComponentEventDispatcherEventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener(
"business.registration.post",
function () use ($busLog) {
$busLog−>info("Customer registered");
}
);
$dispatcher−>dispatch("business.registration.post");

21. Logstash Architecture
1. Logstash Shipper ships logs to
logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic
Search
4. Kibana exposes a web interface
to Elastic Search data

22. Logstash Architecture

23. https://joind.in/talk/view/13369
Why not rate the talk now BEFORE
the demo?

24. Logstash Demo

25. https://github.com
/LoveSoftware/
application-logging-with-logstash

31. Logstash Config

32. Logstash Collecting
{
"network": {
"servers": [ "logs.logstashdemo.com:5000" ],
"timeout": 15,
"ssl ca":
"/etc/pki/tls/certs/logstash−forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/nginx/helloapp.access.log"
],
"fields": { "type": "nginx−access" }
}
]
}

33. Logstash Processing
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate =>
"/etc/pki/tls/certs/logstash−forwarder.crt"
ssl_key =>
"/etc/pki/tls/private/logstash−forwarder.key"
}
}
Input

34. Logstash Processing
Filtering
filter {
if [type] == "nginx−access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}

35. Logstash Processing
Output
output {
elasticsearch { host => localhost }
}

36. Groking
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}
%{WORD:method}
%{URIPATHPARAM:request}
%{NUMBER:bytes}
%{NUMBER:duration}

37. Logging Ideas
Release Marker
Error rates of various applications over time
Latency in various percentiles of each application tier
HTTP Responses: 400 series responses
HTTP Responses: 500 series responses
Auto git blame production errors
Auth and Syslogs

38. Go Forth And Log....
BUT
Remember log rotation
Beware running out of space
Beware file logging on NFS

39. Questions?

40. https://joind.in/talk/view/13369