Choosing the Right Streaming Protocol With Sami Ahmed & Amanda Gilbert | Current 2022 You’ve decided to build a real-time application – great! Now which data streaming communication protocol is right for you – polling, websockets, or server sent events (SSE)? Furthermore, how do these protocols integrate with Kafka? How can you gracefully handle if/when your client disconnects from Kafka? In this session, we will set the stage by talking about the strengths and weaknesses of each protocol, and then dive into how Kafka can be leveraged with these different protocols. We will demo different approaches you might take, for example using a Kafka connector with SSE vs. just having a client handle these real-time workloads directly – and also perform some chaos testing to test resiliency. By the end of this session, you should have an understanding of these different streaming protocols at your disposal, and how to select the appropriate one for your application.

1. TO POLL OR NOT TO POLL
AMANDA GILBERT, SENIOR SOLUTIONS ENGINEER
SAMI AHMED, SENIOR SOLUTIONS ENGINEER (FSI-NY)

2. why are a couple of backend devs
like you talking about front end
connection protocols?

3. TO POLL OR NOT TO POLL
Client (xhr) polls frequently if “realtime” is requirement
HTTP requests have overhead
Network latency implications
OLD WAY:
App request/response
NEW WAY:
Server push
New data automatically pushed by server to client
Less HTTP overhead
Efficient use of network
Includes long polling, SSE, and websockets

4. ARCHITECTURE

5. LOOK
MOM I'M
A WEB
DEV

6. LONG POLLING
Client opens long poll
request
Server sends response,
request is closed
New long poll opened
1.
2.
3.
Device support
More efficient compression due to
batching
Standard behavior for Kafka
consumers based on Kafka Protocol
How do we manage client
connections?
How do we scale without losing data
order?
Headers sent with response
Higher average latency
PROS
CONS[IDERATIONS]

8. LONG POLLING: SERVER SIDE
Requires us to
cache our
historical
messages
Batch responses:
how do we do
this
intelligently?

9. LONG POLLING: CLIENT SIDE

10. WEBSOCKETS
HTTP handshake begins
HTTP handshake
completes
Bidirectional Websocket
protocol ensues
1.
2.
3.
Opens a persistent Bi-Directional TCP
connection
Headers not sent with response
Lower latency
Terminated connections don’t
automatically recover
Less efficient compression (per
message)
How do we deal with historical data?
How do we cache our connections?
PROS
CONS[IDERATIONS]

12. WEBSOCKETS: SERVER SIDE

13. WEBSOCKETS: CLIENT SIDE

14. SSE
Client request
Server response with
custom text/event
stream
Server streams (ongoing)
event data
1.
2.
3.
Ends when server side sends "closed"
event
EventSource Browser client (auto)-
reestablishes dropped connection
Low latency
Low overhead
Can be run over TLS connection
Unidirectional TCP/IP connection from
server to client
Handles only UTF-8 data
Battery life
Pros
Cons

16. SSE CLIENT SIDE

17. SSE SERVER SIDE

18. CHAOS
TEST
SETUP

19. TEST 1 FAILURE & RECOVERY
pumba -l info netem --duration 1000s delay --time 100000 websocket-app sse-app

20. TEST 2 THROUGHPUT
pumba -l info netem --duration 2m delay --time 1000 -j 500 websocket-app sse-app

21. TEST 3 PACKET CORRUPTION
pumba netem --duration 5m corrupt --percent 8 websocket-app sse-app

22. QUESTIONS?
AMANDA GILBERT | AGILBERT@CONFLUENT.IO
SAMI AHMED | SAHMED@CONFLUENT.IO
HTTPS://GITHUB.COM/SAMI2AHMED/TO-POLL-OR-NOT-CURRENT22