Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. Coronavirus is just a piece of the challenge.
This presentation explores use cases, architectures, and references for Apache Kafka as event streaming technology in the aviation industry, including airline, airports, global distribution systems (GDS), aircraft manufacturers, and more.
Examples include Lufthansa, Singapore Airlines, Air France Hop, Amadeus, and more. Technologies include Kafka, Kafka Connect, Kafka Streams, ksqlDB, Machine Learning, Cloud, and more.
Large Language Models for Test Case Evolution and Repair
Event Streaming Powers Aviation Industry Transformation
1. @KaiWaehner - www.kai-waehner.de
Event Streaming with Apache Kafka
in the Aviation Industry
Airlines, Airports, Aerospace, Manufacturing, GDS, Flight Safety, and Retail
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner
2. @KaiWaehner - www.kai-waehner.de
The Aviation Industry is Changing…
Aviation and travel
are notoriously vulnerable to social,
economic, and political events.
Digital modernization is coming
not just due to the Corona Virus,
but because of the ever-changing
expectations of consumers.
4. @KaiWaehner - www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data
17. Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases
18. @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
18
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
21. ‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka
22. Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
23. @KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
25. @KaiWaehner - www.kai-waehner.de
Confluent pioneered Event Streaming
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
Confluent founders are
original creators of Kafka
Confluent team wrote
80% of Kafka software
commits, has over 1M hours
technical experience with
Kafka, and operates 5,000+
clusters
Confluent Cloud
is the only multi-cloud, fully
managed, pay-as-you-go
event streaming service in
the world
Confluent Platform
completes Apache Kafka
and turns it into a secure,
enterprise-ready platform
29. @KaiWaehner - www.kai-waehner.de
Amadeus IT Group
29
https://kafka.apache.org/powered-by
Major European IT provider for the global travel and tourism industry
Global Distribution System (GDS) provides search, pricing, booking,
ticketing and other processing services in real-time
Customers include airlines, hotels, tour operators, insurers, car
rental and railway companies
31. Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams
32. Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect
33. Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0
34. Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
35. Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB
36. Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS
37. ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDB
DB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”
39. @KaiWaehner - www.kai-waehner.de
Event Streaming with Kafka and ksqlDB at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/
42. @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(Data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
Bookings
Team
Fraud
Team
…
Mobile
Team
…
43. Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
44. @KaiWaehner - www.kai-waehner.de
Multi-Region Cluster
Kafka Clusters
can Stretch over Regions
No Downtime + Zero Data loss
e.g. Stretched over US East + Mid + West
Automated Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without Custom Code
45. 45
Cluster Linking requires no additional
infrastructure and preserves offsets:
Global Central
Nervous System
Cluster Linking
simplifies hybrid-cloud
and multi-cloud
deployments for Kafka
47. @KaiWaehner - www.kai-waehner.de
Streaming Aviation Data Exchange
MirrorMaker 2 /
Confluent Replicator
Airport
Server: Latest version
Clients: 0.11, 2.0
Streaming replication
between stakeholders
Version compatibility
between different
clients and servers
Airline
Server: AK 2.3
Clients: 2.0, 2.5
GDS
Server: CP 54 / AK 2.4
Clients: 0.11, 2.4
Confluent
Cluster Linking
48. @KaiWaehner - www.kai-waehner.de
ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming
49. Highly Scalable
Persistent
ETL/Data Integration Messaging
ETL/Data Integration Messaging
Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-time
Highly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming
50. @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design for your Integration Layer
Legacy
Integration
Custom
Application
ESB Connector
Java / Python /
ksqlDB / etc.
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
51. Kafka Cluster
Domain-Driven Design for your Event Steaming Platform
Schema
Registry
Event Streaming Platform
Big Data Cluster Big Data Cluster
KSQL
App
KSQL
App
KSQL
App
52. Use Cases for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of
Historical Data
54. @KaiWaehner - www.kai-waehner.de
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
54
Model X
(at a later time)
55. @KaiWaehner - www.kai-waehner.de
Event Streaming Is The Future Of Data
55
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
57. I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
57
58. @KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
58
59. @KaiWaehner - www.kai-waehner.de
Confluent Platform
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success
60. @KaiWaehner - www.kai-waehner.de
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!