Cloud and on-premises integration with Event-driven

1. CloudandOn-premisesApplications
Integration
UsingEvent-driven
Rich Lee
Taipei, November 2020

2. About Me
Rich Lee
• Integration Architect at Cathay Financial Holdings
• Microservices
• Event-driven
• Cloud Native App Development
• Kafka & Event-driven consultant
rich04230@gmail.com RICH0423 @rich04230

3. Agenda
• FaaS
– Google Cloud Functions
• Kafka Connect
• Demo: Cloud and on-premises integration with Event-driven
– Github repo

4. FaaS

5. Backend as a service approaches
Application
Runtime
Containers
OS
Hardware
Functions
IaaS
Data
CaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
PaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
FaaS
Application
Runtime
Containers
OS
Hardware
Functions
Data
Developer
Cloud
Provider

6. Function-as-a-Service (FaaS)
• FaaS is focused on the event-driven computing paradigm wherein
application code, or containers, only run in response to events or
requests.
Serverless Event-driven
computing
FaaS

7. Function-as-a-Service (FaaS)
container-based
AWS Lambda GCP Cloud Functions Azure Functions
Azure Fargate GCP Cloud Run

8. Google Cloud Functions
Cloud Functions is a serverless execution environment for building and
connecting cloud services.
• Function types:
– Http functions
– Background functions
• Triggers:
– HTTP
– Cloud Sotrage
– Cloud Pub/Sub

10. Kafka Connect

11. Why Kafka Connect
• Simplify and improve getting data in and out of Kafka
– Fault Tolerance
– Exactly Once
– Distribution
– Ordering

12. Kafka Connect
• Kafka Connect is a tool for scalable and reliable streaming data
between Apache Kafka and other systems.

13. Kafka Connect
• The Kafka Connect API is an interface that simplifies and automates the
integration of a new data source or sink to your Kafka cluster.

14. Confluent HUB Discover and share Connectors

15. Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!

16. Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
• Connectors + Configurations --> Tasks
– A task is linked to a connector configuration
– A job configuration may spawn multiple tasks

17. Kafka Connect Concepts
• Kafka Connect Cluster has multiple loaded Connectors
– Each connector is a re usable piece of code (java jars)
– Many connectors exist in the open source world, leverage them!
• Connectors + Configurations --> Tasks
– A task is linked to a connector configuration
– A job configuration may spawn multiple tasks
• Tasks are executed by Kafka Connect Workers (servers)
– A worker is a single java process
– A worker can be standalone or in a cluster (distributed)

18. Kafka Connect Concepts
Distributed Mode

19. Kafka Connect Concepts
Distributed Mode

20. Kafka Connect Concepts
Distributed Mode

21. Kafka Connect Concepts
• Kafka Connect Workers
– Standalone
– Distributed
Standalone Distributed
Parallelism A single process runs your connectors
and tasks
Multiple workers run your connectors
and tasks
Configuration bundled with your process submitted using a REST API
Environment for development and testing for production deployment of
connectors
Reliability /
Scalability
Not fault tolerant, no scalability Easy to scale, and fault tolerant

22. Kafka Connectors
• REST API
– connector
– connector-plugins
Resource
Type
Endpoint Method Description
connector /connectors GET Get a list of active connectors
/connectors POST Create a new connector
/connectors/{name} GET Get information about the connector
/connectors/{name}/config GET Get the configuration for the connector
/connectors/{name}/config PUT Create or update the configuration for an
existing connector
/connectors/{name}/status GET Get current status of the connector
/connectors/{name}/restart POST Restart the connector and its tasks
/connectors/{name}/ DELETE Delete a connector

24. push event
poll event
Kafka Connect

25. Kafka Connectors
• GCP CloudPubSubConnector
– sink connector
– source connector

26. GCP CloudPubSubConnector
• Configure sink connector
{
"name": "cps-sink-connector1",
"config": {
"connector.class":
"com.google.pubsub.kafka.sink.CloudPubSubSinkConnector",
"tasks.max": "1",
"topics": "iot-topic",
"cps.topic": "cps-iot-input",
"cps.project": "app-project-294009",
"value.converter":
"org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false
}
}
Cloud Pub/Sub configs

27. GCP CloudPubSubConnector
• Configure source connector
{
"name": "cps-source-connector1",
"config": {
"connector.class":
"com.google.pubsub.kafka.source.CloudPubSubSourceConnector",
"tasks.max": "1",
"kafka.topic": "iot-processed-topic",
"cps.subscription": "subscription2",
"cps.project": "app-project-294009",
"value.converter":
"org.apache.kafka.connect.converters.ByteArrayConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false
}
}
Cloud Pub/Sub configs

28. Reference
• GCP Pubsub Connector
• Kafka Connect
• Creating Apache Kafka Connectors with the Kafka Connect API
• Source and Sink for Kafka and PubSub