https://devoxx.be/talk/?id=52363 CQRS, Event streaming, Event sourced, log management, Kafka, RabbitMQ... The all ecosystem is now working on event management, event sourcing, and CQRS. Kafka trend is growing, in a king of modern style ESB. This all trned now allow the emergence of new software. One of the new event log, stream and storage, is Apache Pulsar, a great Apache project, using Zookeeper and Bookeeper, coming from Yahoo! team. This talk will help to understand the architecture, the good points, the differences, and compare it with SQS, Kafka, RabbitMQ Iron or Redis listen. There will be example using java code. The two speakers are coming from two different companies, using Pulsar on production.

1. #Devoxx #Pulsar101 @waxzce @GwinizDu
Apache Pulsar 101

2. #Devoxx #Pulsar101 @waxzce @GwinizDu
Who are we?

3. #Devoxx #Pulsar101 @waxzce @GwinizDu
@waxzce on twitter
CEO @ Clever Cloud

4. #Devoxx #Pulsar101 @waxzce @GwinizDu
@GwinizDu on twitter
Engineer @ OVHcloud

5. #Devoxx #Pulsar101 @waxzce @GwinizDu

6. #Devoxx #Pulsar101 @waxzce @GwinizDu

7. #Devoxx #Pulsar101 @waxzce @GwinizDu
● Built on top of Bookkeeper
● Using Zookeeper
● In production since 2015
● Donated as OSS by Yahoo in 2016
● Apache Top Level Project in 2018

8. #Devoxx #Pulsar101 @waxzce @GwinizDu
Good packaging
http://pulsar.apache.org/docs/en/standalone/

9. #Devoxx #Pulsar101 @waxzce @GwinizDu
Layered

10. #Devoxx #Pulsar101 @waxzce @GwinizDu
Layered
Bookie Bookie Bookie Bookie Bookie
Broker Broker BrokerMessaging
Storage

11. #Devoxx #Pulsar101 @waxzce @GwinizDu
Layered
Bookie Bookie Bookie Bookie Bookie
Broker Broker Broker
Messaging
(pulsar)
Storage
(Bookkeeper)
Stateless
Scale-out
Not tight to storage
constraints

12. Multi Model Messaging platform

13. Distributed Pub-Sub

14. #Devoxx #Pulsar101 @waxzce @GwinizDu
Distributed Log
1 2 3 4 5 6 9998979695...Log Stream
Oldest Newest
Ordered Sequence of Immutable Records

15. #Devoxx #Pulsar101 @waxzce @GwinizDu
Distributed Log
1 2 3 4 5 6 9998979695...Log Stream
1 2 3 4 5 6 9998979695...Log Stream
Oldest Newest
1 2 3 4 5 6 9998979695...Log Stream

16. #Devoxx #Pulsar101 @waxzce @GwinizDu
Distributed Log
1 2 3 4 5 6 9998979695...Log StreamTopic Partition#1
1 2 3 4 5 6 9998979695...Log Stream
Oldest Newest
Partition#2
1 2 3 4 5 6 9998979695...Log StreamPartition#0

17. #Devoxx #Pulsar101 @waxzce @GwinizDu
A partitioned topic
oldest data newest dataSegment 1 Segment 2 Segment 3 Segment .. Segment n
Bookie Bookie Bookie
Segment 1 Segment 1 Segment 1
Bookie
Segment 2 Segment 2 Segment 2
Segment 3 Segment 3Segment 3

18. #Devoxx #Pulsar101 @waxzce @GwinizDu
Distributed Log
WAL
Non Blocking Ingress

19. #Devoxx #Pulsar101 @waxzce @GwinizDu
Architecture
Bookie
Broker Broker Broker
Bookie Bookie Bookie...
ZK
ZK
Object Storage
Presto cluster
ProxyProxy

20. #Devoxx #Pulsar101 @waxzce @GwinizDu
Protocols
Protocol buffer (broker)
+
HTTP (admin)
WebSocket + More to come ;-)
https://github.com/apache/pulsar/wiki/PIP-46:-Next-gen-Proxy

21. Distributed Pub-Sub

22. #Devoxx #Pulsar101 @waxzce @GwinizDu
Unified API
Cumulative vs Selective Acking
PubSub

23. #Devoxx #Pulsar101 @waxzce @GwinizDu
Choose your Delivery Guarantee
Server side Deduplication
Server side Dead lettering
Messaging semantic

24. #Devoxx #Pulsar101 @waxzce @GwinizDu

25. #Devoxx #Pulsar101 @waxzce @GwinizDu

26. #Devoxx #Pulsar101 @waxzce @GwinizDu

27. #Devoxx #Pulsar101 @waxzce @GwinizDu
Comes with virtual Topics
(*no limit to topics number)

28. #Devoxx #Pulsar101 @waxzce @GwinizDu
Comes with virtual multiples Topics

29. #Devoxx #Pulsar101 @waxzce @GwinizDu
Schema registry

30. #Devoxx #Pulsar101 @waxzce @GwinizDu
Schema registry

31. #Devoxx #Pulsar101 @waxzce @GwinizDu
Schema registry

32. #Devoxx #Pulsar101 @waxzce @GwinizDu
Write paths (DbLedgerStorage)
Journal file
(transaction logs)
Writes

33. #Devoxx #Pulsar101 @waxzce @GwinizDu
Write paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
Writes

34. #Devoxx #Pulsar101 @waxzce @GwinizDu
Write paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
Writes

35. #Devoxx #Pulsar101 @waxzce @GwinizDu
Write paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
periodically sort
and flush
as Entry Log files
Writes

36. #Devoxx #Pulsar101 @waxzce @GwinizDu
Write paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
periodically sort
and flush
as Entry Log files
location of entries
indexed in RocksDB
(ledgerId, entryId) -> (entryLogId, offset)
Writes
read cache

37. #Devoxx #Pulsar101 @waxzce @GwinizDu
Read paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
periodically sort
and flush
as Entry Log files
location of entries
indexed in RocksDB
(ledgerId, entryId) -> (entryLogId, offset)
Writes
read cache
Reads ?

38. #Devoxx #Pulsar101 @waxzce @GwinizDu
Read paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
periodically sort
and flush
as Entry Log files
location of entries
indexed in RocksDB
(ledgerId, entryId) -> (entryLogId, offset)
Writes
read cache
Reads

39. #Devoxx #Pulsar101 @waxzce @GwinizDu
Read paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
periodically sort
and flush
as Entry Log files
location of entries
indexed in RocksDB
(ledgerId, entryId) -> (entryLogId, offset)
Writes
read cache
Reads

40. #Devoxx #Pulsar101 @waxzce @GwinizDu
Read paths (DbLedgerStorage)
Journal file
(transaction logs)
Write cache
periodically sort
and flush
as Entry Log files
location of entries
indexed in RocksDB
(ledgerId, entryId) -> (entryLogId, offset)
Writes
read cache
Reads

41. #Devoxx #Pulsar101 @waxzce @GwinizDu
Durability strategies
● Non Persistent ● Persistent ● Compacted
● (Deduplicated)

42. #Devoxx #Pulsar101 @waxzce @GwinizDu
...
Messages retention & expiry
... ...
Deleted Retention Policy Withing TTL TTL expired
Acknowledged messages Unacknowledged messages

43. #Devoxx #Pulsar101 @waxzce @GwinizDu
Tiered Storage
https://jclouds.apache.org/reference/providers/

44. #Devoxx #Pulsar101 @waxzce @GwinizDu
Security

45. #Devoxx #Pulsar101 @waxzce @GwinizDu
Proxy

46. #Devoxx #Pulsar101 @waxzce @GwinizDu
Multi tenancy
> Tennant
> Namespace
> topics

47. #Devoxx #Pulsar101 @waxzce @GwinizDu
● Multi tenancy
● Proxy
● Pluggable Auth
○ Biscuit
● ACL
● Attenuation
● Decentralized

48. #Devoxx #Pulsar101 @waxzce @GwinizDu
Multi Cluster Management
Bookie
Broker Broker Broker
Bookie Bookie Bookie...
ZK
ZK
Object Storage
Presto cluster
ProxyProxy

49. #Devoxx #Pulsar101 @waxzce @GwinizDu
Multi Cluster Management
BrokerBrokerBroker
Bookie
Bookie
Bookie
ZK
ZK

50. #Devoxx #Pulsar101 @waxzce @GwinizDu
Multi Cluster Management
Broke
r
Broke
r
Broker
Boo
kie
Boo
kie
ZK
ZK
BK
Broke
r
Broke
r
Broker
Boo
kie
Boo
kie
ZK
BK
Broke
r
Broke
r
Broker
Boo
kie
Boo
kie
ZK
BK
EU-FR-1
EU-DE-1
US-EAST-1
Discovery

51. #Devoxx #Pulsar101 @waxzce @GwinizDu
Geo Replication
Broke
r
Broke
r
Producers
EU-FR-1
EU-DE-1
US-EAST-1
Broke
r
Broke
r
Consumers
Broke
r
Broke
r
Consumers
Broke
r
Broke
r
Producers
Topic Topic
Topic

52. #Devoxx #Pulsar101 @waxzce @GwinizDu
Integrations

53. #Devoxx #Pulsar101 @waxzce @GwinizDu
SQL integration

54. #Devoxx #Pulsar101 @waxzce @GwinizDu
SQL integration

55. #Devoxx #Pulsar101 @waxzce @GwinizDu
SQL integration

56. #Devoxx #Pulsar101 @waxzce @GwinizDu
Functions

57. #Devoxx #Pulsar101 @waxzce @GwinizDu
Functions

58. #Devoxx #Pulsar101 @waxzce @GwinizDu
Functions
Composable: implement the function worker model
• Threaded
– inside the function worker java thread (java
code)
• Process
– Executed in a separate process by function,
can be java or python (golang is nearly
here)
• Kubernetes
– here be dragons 🐉
• more?

59. #Devoxx #Pulsar101 @waxzce @GwinizDu
Virtual Machine
Pulsar function: Another execution level
Clever Cloud new serverless to come
Web Assembly (WASM)
(from rust, typescript, kotlin, golang, c++...)
Unikernel
Scheduler
Bare Metal

60. #Devoxx #Pulsar101 @waxzce @GwinizDu
Operating
● REST API
● Prometheus metrics
● Load Balancer

61. #Devoxx #Pulsar101 @waxzce @GwinizDu
Clients

62. #Devoxx #Pulsar101 @waxzce @GwinizDu
Community
https://apache-pulsar.herokuapp.com/
https://github.com/apache/pulsar

63. #Devoxx #Pulsar101 @waxzce @GwinizDu
Thank you for listening
That's all, folks!
Ping us on twitter:
@GwinizDu & @waxzce

64. Annexes
Thank you for listening

73. Event Sourcing

74. Tables / Text

75. Divider slide with color
Subtitle goes here

76. Divider slide with color
Subtitle goes here

77. Divider slide with color
Subtitle goes here

78. Divider slide with color
Subtitle goes here