Please find a better version here: https://www.slideshare.net/FlorentRamiere/back-to-database-fundamentals-aka-the-origin-of-the-streaming-platform ... aka the origin of the streaming platform. The streaming platforms rely on a component that has been part of our most precious systems. Let's rediscover that, and see why the streaming platform is an evolution of our current systems.

1. 1
Back to database fundamentals
(… aka the origin of the streaming platform)
Florent Ramiere florent@confluent.io

2. 22
What happened in 1978 in
our industry ?

3. 3

4. 4

5. 5

6. 6

7. 7
Materialize events into an opinionated structure

8. 8
So you get the power of SQL

9. 99
What if you need to scale ?

10. 10
Master handles all the
writes
Replication
Replica can now
handle the reads!

11. 11
Master
Replication
Slaves
Replication

12. 12
Leader
Replication
Follower
Replication

13. 1313
Replication leveraged
the events too!

14. 1414
But what if you want to
leverage these events
yourself ?

15. 15
SQL on tables
CDC
SQL on events

16. 16
Load events and
impact another
system
CDC

17. 17
ETL
CDC

18. 1818
Could we simplify ?

19. 19
CDC

20. 20
CDC
CDC

21. 21
CDC
CDC
CDC

22. 2222
What about transformation ?

23. 23

24. 24

25. 25
?!

26. 26
?

27. 27

28. 28

29. 29

30. 3030
Why are we talking about that
now ?

31. 3131
Why can’t use use a
big shared global database
that does it all ?

32. 32
Like this one ...

33. 33
Shared database
Application refactoring
Direct schema coupling
Scaling challenge
Single point of failure

34. 3434
File Transfer
Latent Data
Direct database load
Consistency Challenges
E
X
P
O
R
T
I
M
P
O
R
T

35. 35
RPC/REST/SOA
Direct coupling
Application refactoring
Availability concerns
Scaling Concerns
Function call
answer

36. 3636
Pub/Sub messaging
Loosely coupled
High scalability
High availability
Real-time
E
V
E
N
T

37. 37
Central nervous system
Focus on the essence of our systems

38. 3838
Back to Database

39. 3939
What problem was solved in
1998 ?

40. 40

41. 41

42. 42

43. 43

44. 44
CREATE TABLE sales (
prod_id NUMBER(6),
cust_id NUMBER,
time_id DATE,
channel_id CHAR(1),
promo_id NUMBER(6),
quantity_sold NUMBER(3),
amount_sold NUMBER(10, 2)
) PARTITION BY RANGE (time_id) INTERVAL ( NUMTOYMINTERVAL(1,'MONTH'))
CREATE TABLE order_items (
order_id NUMBER(12) NOT NULL,
line_item_id NUMBER(3) NOT NULL,
product_id NUMBER(6) NOT NULL,
unit_price NUMBER(8, 2),
quantity NUMBER(8),
CONSTRAINT order_items_fk FOREIGN KEY(order_id) REFERENCES orders(order_id)
) PARTITION BY REFERENCE(order_items_fk);
CREATE TABLE dept (
deptno NUMBER,
deptname VARCHAR(32)
) PARTITION BY HASH(deptno) PARTITIONS 2;

45. 45
Logical
Physical

46. 46
Table
Partition

47. 47

48. 48

49. 4949
How do we query ?

50. 50
SELECT * FROM dept WHERE deptno=47;

51. 51
SELECT * FROM dept WHERE deptno=47;
HASH(deptno) PARTITIONS 2

52. 52
SELECT * FROM dept WHERE deptno=47;
47 % 2

53. 53
SELECT * FROM dept WHERE deptno=47;
47 % 2
SELECT * FROM dept WHERE deptno=47;

54. 54
SELECT * FROM dept WHERE deptno=47;
SELECT * FROM dept WHERE deptno=47;

55. 55
SELECT * FROM dept WHERE deptno=47;

56. 5656
We are querying the state
produced by the processor
executing the commit log

57. 5757
We just re-created
materialized views

58. 5858
To express an always up to
date business view

59. 59

60. 60

61. 6161
We are querying the state
produced by the commit log
processor

62. 6262
We are querying the state
produced by the commit log
processorBusiness
Events
Output

63. 6363
We are querying the state
produced by the commit log
processorProducer/Consumer
Kafka Stream
KSQL
...
Events
State store

64. 64

65. 6565
We get to choose
the right tool !

66. 6666
This is streaming

67. 6767
The foundational component
has already been serving you
for years!

68. 68

69. 6969
Confluent
is the streaming platform
with Kafka at its core
as the commit log

70. 70
14 novembre à Paris - Gratuit

71. 71
THANK YOU
@framiere
florent@confluent.io
cnfl.io/blog