Data is being collected more and more every year. Cloud applications, including IoT, web, and mobile send torrents of bits at our data centers that have to be processed and stored. In addition, users expect an always-on experience, with little room for error. Numerous companies are successfully doing this every day. In this webinar, you will learn about the convergence of complementary technologies: Spark, Mesos, Akka, Cassandra and Kafka (SMACK), how Apache Kafka can help you get your data under control and the critical role Kafka plays in your data pipeline. Webinar recording: https://youtu.be/uwYlwLyv-1s Webinar Q&A will be posted shortly.

1. @PatrickMcFadin
Patrick McFadin
Chief Evangelist for Apache Cassandra
How to Build Data Pipelines for Real-Time
Applications with SMACK & Apache Kafka
1

2. The problem
Your
Magical
App

3. Sad solutions

4. SMACK

5. Spark
Mesos
Akka
Cassandra
Kafka

6. CassandraAkka
SparkKafka
Organize Process Store
Mesos
KafkaKafkaKafka
SparkSparkSpark
AkkaAkkaAkka
CassandraCassandraCassandra

7. CassandraAkka
SparkKafka
Organize Process Store

8. Managing Weather Data
Windsor California
67.3 F
Rainfall total: 1.2cm
Today:
High: 73.4F
Low : 51.4F
Yesterday:
High: 75.2F
Low : 52.3F
Our
Magical
App
Reactive and immediate
Batch

9. KillrWeather
KillrWeather
Windsor California
67.3 F
Rainfall total: 1.2cm
Today:
High: 73.4F
Low : 51.4F
Yesterday:
High: 75.2F
Low : 52.3F

10. https://github.com/killrweather/killrweather

11. Spark
Mesos
Akka
Cassandra
Kafka

12. Kafka

13. Kafka decouples data pipelines

16. The problem
Kitchen
Hamburger
please
Meat disk
on bread
please

17. The problem
Kitchen

18. The problem
Kitchen
Order Queue
Hamburger
please
Order

19. The problem
Kitchen
Order Queue

20. The problem
Kitchen
Order Queue
Meat disk
on bread
please
You mean a
Hamburger?
Uh yeah.
That.
Order

21. Order from chaos
Producer
Consumer
Topic = FoodOrder

22. Order from chaos
Producer
Topic = Food
Order
1
Consumer

23. Order from chaos
Producer
Topic = Food
Order
1
Order
Consumer

24. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer

25. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order

26. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3

27. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3

28. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3

29. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order

30. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4

31. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order

32. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5

33. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5

34. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5

35. Order from chaos
Producer
Topic = Food
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5

36. Scale
Producer
Topic = Hamburgers
Order
1
Order
2
Consumer
Order
3
Order
4
Order
5
Topic = Pizza
Order
1
Order
2
Order
3
Order
4
Order
5
Topic = Food

37. Kafka
Producer
Topic = Temperature
Temp
1
Temp
2
Consumer
Temp
3
Temp
4
Temp
5
Collection API
Temperature
Processor
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Precipitation
Processor
Broker

38. Kafka
Producer
Topic = Temperature
Temp
1
Temp
2
Consumer
Temp
3
Temp
4
Temp
5
Collection API
Temperature
Processor
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Precipitation
Processor
Broker
Partition 0
Partition 0

39. Kafka
Producer Consumer
Collection API
Temperature
Processor
Precipitation
Processor
Topic = Temperature
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Partition 1
Temperature
Processor

40. Kafka
Producer Consumer
Collection API
Temperature
Processor
Precipitation
Processor
Topic = Temperature
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Partition 1
Temperature
Processor
Topic = Temperature
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Partition 1
Topic Temperature
Replication Factor = 2
Topic Precipitation
Replication Factor = 2

41. Kafka
Producer
Consumer
Collection API
Temperature
Processor
Precipitation
Processor
Topic = Temperature
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Partition 1 Temperature
Processor
Topic = Temperature
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Topic = Precipitation
Precip
1
Precip
2
Precip
3
Precip
4
Precip
5
Broker
Partition 0
Partition 0
Tem
p
1
Temp
2
Tem
p
3
Temp
4
Temp
5
Partition 1
Temperature
Processor
Temperature
Processor
Precipitation
Processor
Topic Temperature
Replication Factor = 2
Topic Precipitation
Replication Factor = 2

42. Guarantees
Order
•Messages are ordered as they are sent by the
producer
•Consumers see messages in the order they were
inserted by the producer
Durability
•Messages are delivered at least once
•With a Replication Factor N up to N-1 server failures
can be tolerated without losing committed messages

43. CassandraAkka
SparkKafka
Organize Process Store
Mesos

44. Coming soon!
• May 4: How to Achieve High Throughput for Real-Time Applications with SMACK,
Apache Kafka and Spark Streaming
• May 18: How to Build Data Pipelines with SMACK: Storage Strategy using
Cassandra and DSE
• June 1: How to Build Data Pipelines with SMACK: Analyzing Data with Spark
• For the latest schedule of webinars, check out our Webinars
page: http://www.datastax.com/resources/webinars

45. Go get your SMACK on
Thank you!
Follow me on twitter: @PatrickMcFadin