Security Information and Event Management with Kafka, Kafka Connect, KSQL and Logstash

https://digitalis.io
info@digitalis.io
Security Information and
Event Management with Kafka,
Kafka Connect, KSQL and Logstash

2
Jason Bell
ABOUT
Working with Kafka since 2014, in
development, support and now DevOps.
Author of Machine Learning: Hands on
for Developers and Technical
Professionals, published by Wiley.
Kafka DevOps Engineer

©2020 digitalis.io Ltd. Do not distribute without
consent.
What is SIEM?

consent.
SIEM adoption originally driven from Payment Card
Industry Data Security Standard (PCI DSS).

consent.
Data can come from various sources such as
firewalls, anti-virus, login information and intrusion
prevention systems.

consent.
For example: A user does 20 failed login attempts.
Has the user actually forgotten? Let’s class this as
a low priority event. The user may have just
forgotten their password and retried.

consent.
A user does 140 failed login attempts in five
minutes. This is more than likely a brute force
attack and needs investigating.

consent.
Enterprise SIEM Problems

consent.
● Large Volumes of Data.

consent.
● Variety of log formats - RFC5424, RFC3164, Windows Events and other
bespoke log formats from network devices.

consent.
● Regulatory compliance.

consent.
“Virtually every regulatory compliance regime or
standard such as GDPR, ISO 27001, PCI DSS,
HIPAA, FERPA, Sarbanes-Oxley (SOX), FISMA,
and SOC 2 have some requirements of log
management to preserve audit trails of activity that
addresses the CIA (Confidentiality, Integrity, and
Availability) triad.”
https://digitalis.io/blog/kafka/apache-kafka-and-regulatory-compliance/

consent.
● High Availability Requirements

consent.
● Downstream sometimes cannot keep up at peak times – 9am, DDoS events

consent.
● Downstream sometimes cannot keep up at peak times – 9am, DDoS events
● Multiple consumers of data and connectivity to them
○ routing, transforming, filtering

consent.
Why use Kafka?

consent.
Why Kafka?
● High Availability

consent.
Why Kafka?
● Scalable

consent.
Why Kafka?
● Scalable
● High Throughput

consent.
Why Kafka?
● Scalable
● High Throughput
● Rich Ecosystem

consent.
Why Kafka?
● Scalable
● High Throughput
● Rich Ecosystem
● ksqlDB for Implementing Logic for Routing/Filtering/Transforming

consent.
Why Kafka?
● Scalable
● High Throughput
● Rich Ecosystem
● ksqlDB for Implementing Logic for Routing/Filtering/Transforming
● Buffering of data during high peak volumes – a shock absorber.

consent.
Kafka SIEM Architecture

consent.

consent.
Data Flows and Components

consent.
Topic and Outbound Data Flows

consent.
Data Ingestion

consent.
Data Ingestion
● Non-repudiation - fingerprinting source logs
● Transformation to JSON
● Non-standard syslog formats - bespoke grokking

consent.
Logstash - Input

consent.
TODO: Insert Logstash In->Filter-Out diagram

consent.
Logstash Input – All Types input {
udp {
host => "0.0.0.0"
port => 5140
type => rfc5424
tags => ["rfc5424"]
}
tcp {
host => "0.0.0.0"
port => 5140
type => rfc5424
tags => ["rfc5424"]
}
syslog {
port => 5150
type => rfc3164
tags => ["rfc3164"]
}
}

consent.
Logstash - Filtering

consent.
Logstash Filter – RFC3164
filter {
if [type] == "rfc3164" {
# rename and remove fields
mutate {
remove_field => [ "@version", "@timestamp" ]
rename => { "host" => "client_addr" }
rename => { "logsource" => "host" }
rename => { "severity_label" => "severity" }
rename => { "facility_label" => "facility" }
}
}
}
}

consent.
Logstash Filter – RFC5424
filter {
if [type] == "rfc5424" {
# parse RFC5424 log
grok {
patterns_dir => "/etc/logstash/patterns"
match => [ "message", "%{SYSLOG}" ]
tag_on_failure => [ "_grokparsefailure_syslog" ]
}
# rename fields and remove unneeded ones
mutate {
rename => { "syslog_facility" => "facility" }
rename => { "syslog_severity" => "severity" }
# message_syslog contains message content +
extra data
replace => { "message" => "%{message_syslog}" }
remove_field => [ "@version", "facility_label",
"@timestamp", "message_content", "message_syslog" ]
rename => { "program" => "ident" }
rename => { "timestamp_source" => "timestamp"}
rename => { "host" => "client_addr" }
rename => { "host_source" => "host" }
}

consent.
Logstash Filter – RFC JSON

consent.
{
"host":“testhost",
"ident":"info",
"message":"01070417:6: AUDIT - user admin - RAW: httpd(pam_audit): User=admin tty=(unknown) host=10.234.254.90 failed to login after 1 attempt….",
"priority":"info",
"facility":"local0",
"client_addr":"10.234.254.90",
"bucket":"2019042913",
"evt_id":"33a3a040-6a7f-11e9-a8be-0050568115fd",
"extradata":"[ ]",
"fingerprint ":"73dd765f55a1791b667bd6160235e3f6 ",
"rawdata ":"..... ",
"pid":"-",
"msgid":"-",
"timestamp":"2019-04-29T14:03:37.000000Z"
}

consent.
Logstash - Output

consent.
output {
if "syslog_rfc5424" in [tags] {
kafka {
codec => json
topic_id => "syslog_rfc5424"
bootstrap_servers => "{{ confluent_ksql_bootstrap_servers }}"
security_protocol => SSL
ssl_key_password => "{{ logstash_ssl_key_password }}"
ssl_keystore_location => "/etc/logstash/logstash.keystore.jks"
ssl_keystore_password => "{{ logstash_ssl_keystore_password }}"
ssl_truststore_location => "/etc/logstash/logstash.truststore.jks"
ssl_truststore_password => "{{ logstash_ssl_truststore_password }}"
compression_type => "snappy"
acks => "1"
retries => "3"
retry_backoff_ms => "500"
request_timeout_ms => "2000"
batch_size => "32768"
ssl_endpoint_identification_algorithm => "https"
ssl_keystore_type => jks
}
}
}

consent.
Topic Filtering and Routing

consent.
Filter / Routing
● Some downstream systems are not interested in INFO -
too much data

consent.
Filter / Routing
● Some downstream systems are not interested in INFO -
too much data
● Some are only interested in Windows events for
example.

consent.
create stream syslog_rfc3164 (client_addr varchar, host varchar, timestamp varchar, severity varchar,
message varchar, facility varchar, type varchar, priority varchar) with (KAFKA_TOPIC='syslog_rfc3164',
VALUE_FORMAT='JSON’);
create stream auth_rfc3164 with (KAFKA_TOPIC='syslog_auth', VALUE_FORMAT='JSON') AS SELECT * FROM
syslog_rfc3164 WHERE message LIKE '%password check failed for user%' OR message LIKE '%An account
failed to log on.%' OR message LIKE '%%0xC000006D’;
create stream syslog_rfc5424 (facility varchar, message varchar, pid varchar, type varchar, timestamp
varchar, ident varchar, client_addr varchar, host varchar, msgid varchar, extradata varchar, priority
varchar) with (KAFKA_TOPIC='syslog_rfc5424', VALUE_FORMAT='JSON’);
create stream auth_rfc5424 with (KAFKA_TOPIC='syslog_auth', VALUE_FORMAT='JSON') AS SELECT * FROM
syslog_rfc5424 WHERE message LIKE '%password check failed%' OR extradata LIKE '%|309|%' OR message
LIKE '%An account failed to log on.%' OR message LIKE '%%0xC000006D';

consent.
Destinations and Sinks

consent.
Destinations and Sink
● Use existing connectors

consent.
Destinations and Sink
● Use existing connectors
● Build your own connectors

consent.
Splunk HTTP Sink in
Kafka Connect

consent.
{
"name": "syslog-sink-splunk",
"config": {
"connector.class": "SplunkHECSinkConnector",
"tasks.max": "{{ tasks_max }}",
"topics": "{{ topics }}",
"splunk.endpoint.uri": "{{ splunk_endpoint_uri }}",
"splunk.hec.token": "{{ splunk_hec_token }}",
"splunk.index": "{{ splunk_index }}",
"splunk.channelid": "{{ splunk_channelid }}",
"splunk.sourcetype": "{{ splunk_sourcetype }}",
"splunk.http.loglevel": "{{ splunk_http_loglevel }}",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "{{ splunk_value_converter_schemas_enable }}",
"errors.tolerance": "{{ splunk_errors_tolerance }}",
"errors.deadletterqueue.topic.name":"{{ errors_deadletterqueue_topic_name }}",
"errors.deadletterqueue.topic.replication.factor": "{{ errors_deadletterqueue_topic_replication_factor }}"
}
}

consent.
Testing

consent.
Testing
● Process 3TB/day data volumes.

consent.
Testing
● Process 3TB/day data volumes.
● Prove the solution can scale horizontally.

consent.
Testing: Process 2.3TB/day data volumes.
● 3TB/day = 33MB/second

consent.
• 400 threads were set up in the Thread Group to simulate
400 servers sending the logs.

consent.
• 6 load injectors were setup, totalling 2400 threads
(simulated servers), in order to generate between
20MB/second to 40 MB/second load against the endpoint
from the injectors.

consent.
• The load was injected over 5 days period at a sustained
rate to ascertain the performance characteristics of each
component over a prolonged duration.

consent.
Carry on the conversation:
• Website: https://digitalis.io
• Reddit: https://reddit.com/users/digitalis_io
• Twitter: @digitalis_io

consent.
Any Questions?

Security Information and Event Management with Kafka, Kafka Connect, KSQL and Logstash

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Security Information and Event Management with Kafka, Kafka Connect, KSQL and Logstash

Similar to Security Information and Event Management with Kafka, Kafka Connect, KSQL and Logstash (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Security Information and Event Management with Kafka, Kafka Connect, KSQL and Logstash