This document discusses real-time streaming of data to Greenplum using Kafka. It defines real-time, near real-time, and batch data. It provides examples of using real-time analytics for tasks like fraud detection and machine failure monitoring. The document outlines an analytics pipeline using Kafka for data generation, processing, and persistence, with Greenplum for analytics and visualization. It describes using the Greenplum-Kafka connector to provide parallel, reliable transfer of data between Kafka and Greenplum in mini-batches. Configuration details and a demo architecture are also presented.

2. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Sharath Punreddy and Niranjan Sarvi
Pivotal Data
Greenplum and Kafka: Real-time
Streaming to Greenplum

3. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.© Copyright 2019 Pivotal Software, Inc. All rights Reserved.
“We want answers NOW”

4. Definitions of Real/Near Real/Batch
https://ell.stackexchange.com/questions/8762/what-is-the-opposite-of-real-time

5. Continuous Surveillance Monitoring
● Failure-to-Rescue:
○ Decreased from 3.4 to 1.2
● ICU Transfers:
○ Decreased from 5.6 to 2.9
http://anesthesiology.pubs.asahq.org/article.aspx?articleid=1932627

6. Credit Card Fraud Monitoring
https://www.statista.com/statistics/419628/payment-card-fraud-losses-usa-by-type/

7. Other examples of Real-time Analytics
● Recommendation Engines
● Driver Assist
● Supply Chain
● Trading
● Machine Failure Detection
● Customer Experience

8. Real-time Analytics Pipeline
Real
time
Analytics
Over
time
Analytics
Data
Generators
Data
Processor
Action/Alerts
Data
Persistence
Data
Visualization

9. Kafka
High Concurrent
High Available
High Throughput
Reliable Delivery
Real-time processing
Horizontal Scalability
Distributed

10. Greenplum
Multiple-Cloud deployment
Open-Source Innovation Massively Parallel
Integrated Analytics
Next-Generation Data Platform

11. Real time Process Flow
Parallel Load
Data Consolidation,
Aggregation, and Modeling
Massively Parallel Processing

12. Greenplum-Kafka Connector
Parallel Transfer Cloud Native
Scale Up Reliable
High Speed Extensible

13. ● Flexible Column mapping and
Transformation.
● Strong Consistency guaranteed.
● High speed transfer
● Mini batch loading and history
tracking
● Certified by Confluent
● Supports various data formats
Greenplum-Kafka Connector : Features
CSV, Custom delimiter & Binary
Custom formatter

14. Greenplum-Kafka Connector : Architecture
Partition 1
Topic A
Partition 2
Partition n
Reader 1
Reader 2
Reader n
GP-Kafka
Greenplum
Data
Seg 1
Seg 2
Seg m

15. Configurations
GP Table
DATABASE: tradedb
USER: tradeuser
PASSWORD: tradepwd
HOST: mdw
PORT: 5432
KAFKA:
INPUT:
SOURCE:
BROKERS: kafka.c.pde-nsarvi.internal:9092
TOPIC: trade-input
COLUMNS:
- NAME: id
TYPE: varchar
- NAME: __IGNORED__
TYPE: int
- NAME: units
TYPE: decimal(9,2)
FORMAT: csv
ERROR_LIMIT: 125
LOCAL_HOSTNAME: pivotal-greenplum-byol-1-mdw.c.pde-nsarvi.internal
OUTPUT:
TABLE: trade
MAPPING:
- NAME: id
EXPRESSION: id
- NAME: acountId
EXPRESSION: acountId
- NAME: tax-due
EXPRESSION: expenses * 0.035
- NAME: units
CREATE TABLE trade
(
id VARCHAR(100),
acountid INT8,
tradeid INT8,
cusip VARCHAR(10),
units INT4,
tradedate DATE,
action VARCHAR(10),
amount INT8,
tax-due FLOAT8
) distributed by (id);
YAML file

16. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.© Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Demo

17. Demo - Architecture
GP-Kafka
Trade Generators
Trade table
Google Cloud

18. References
● Demo scripts
https://github.com/Pivotal-Data-Engineering/realtime-streaming-gpdb
● Greenplum Kafka Connector
https://gpdb.docs.pivotal.io/5170/greenplum-kafka/intro.html
● Continuous Surveillance Monitoring
http://anesthesiology.pubs.asahq.org/article.aspx?articleid=1932627
● Credit Card Fraud Monitoring
https://www.statista.com/statistics/419628/payment-card-fraud-losses-usa-by-type

19. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.© Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Thank You
Q & A