Brett Randall, Confluent, Solutions Engineer Data anomalies occur in many kinds of event streams, and detecting them in a timely manner can reduce risks and costs associated with invalid data, or undesired and sometimes criminal activities. Financial institutions can use stream-processing to detect potentially fraudulent transactions and activities, and SIEM teams can curate and detect patterns in security and event-log data. In this talk we’ll look at how stream-processing can be used to perform real-time anomaly-detection in streaming event data. https://www.meetup.com/KafkaMelbourne/events/274420345/

1. Detecting data anomalies
with Apache Kafka®,
stream-processing and ksqlDB
Brett Randall, Solutions Engineer - Confluent, November 2020

2. Confluent Cloud Voucher
CC60COMM

3. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Links
3
• https://github.com/confluentinc/examples/tree/6.0.0-post/cp-quickstart
• https://github.com/confluentinc/examples/tree/6.0.0-post/ccloud/ccloud-stack
• https://kafka-tutorials.confluent.io/anomaly-detection/ksql.html
• https://www.confluent.io/blog/build-a-intrusion-detection-using-ksqldb/#aggregation
s-streams-tables
• https://www.confluent.io/blog/how-real-time-stream-processing-works-with-ksqldb/
• https://www.confluent.io/blog/how-real-time-materialized-views-work-with-ksqldb/
• https://www.confluent.io/blog/build-a-intrusion-detection-using-ksqldb/#aggregation
s-streams-tables

4. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
ksqlDB commands and queries
4
• ccloud ksql app configure-acls lksqlc-gjn3m stock_trades --cluster lkc-10j6v
• CREATE STREAM stock_trades WITH (kafka_topic='stock_trades', value_format='AVRO');
SELECT *, QUANTITY * PRICE AS VALUE
FROM STOCK_TRADES
WHERE QUANTITY * PRICE > 99000 AND QUANTITY * PRICE < 100000
EMIT CHANGES;
• SELECT WINDOWSTART, WINDOWEND, SYMBOL, COUNT(*) AS COUNT
FROM STOCK_TRADES
WINDOW TUMBLING (SIZE 1 SECOND)
GROUP BY SYMBOL
HAVING COUNT(*) > 5
EMIT CHANGES;

5. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
ksqlDB queries
5
• commit.interval.ms = 5000
• CREATE STREAM "STOCK_BUYS" AS
SELECT *
FROM STOCK_TRADES
WHERE SIDE = 'BUY'
EMIT CHANGES;
• CREATE STREAM "STOCK_SELLS" AS
SELECT *
FROM STOCK_TRADES
WHERE SIDE = 'SELL'
EMIT CHANGES;
• SELECT *
FROM STOCK_BUYS
LEFT JOIN STOCK_SELLS WITHIN 10 SECONDS
ON STOCK_BUYS.SYMBOL = STOCK_SELLS.SYMBOL
WHERE STOCK_BUYS.PRICE > 2 * STOCK_SELLS.PRICE
EMIT CHANGES;

6. cnfl.io/meetups cnfl.io/slackcnfl.io/blog
Thank you!
@javabrett
brett.randall@confluent.io