This webinar discusses building streaming and fast data applications with technologies like Spark, Mesos, Akka, Cassandra and Kafka. It covers how microservices and fast data architectures are converging due to similar design problems and data becoming the dominant problem. The webinar also introduces Lightbend's Fast Data Platform for building streaming data systems and microservices with best practices, sample applications and machine learning-based monitoring and management.

1. WEBINAR
Building Streaming and Fast Data Applications
with Spark, Mesos, Akka, Cassandra and Kafka
Sean Glover (@seg1o), Senior Consultant at Lightbend

2. A Bit of History:
IT Services vs. Big Data

3. Big DataServices
Some Overlap: Concerns, Architecture
The Recent Past…

4. Microservices
& Fast Data
Much More Overlap
The Future?

5. Drilling Down:
Monoliths to Microservices

6. lightbend.com/reactive-microservices-architecture

7. •Tangled responsibilities, lead to
infrequent, “big-bang” deployments
•App lifetimes months to forever!
Monoliths

9. •Each does one thing, so must be
message driven & asynchronous
•Updates easier, deployments frequent
•App lifetimes: minutes! to forever
Microservices

10. •But be careful:
•Message overhead is much better
than function calls!
Microservices

11. Reactive Systems
reactivemanifesto.org

12. The Reactive Platform

13. Drilling Down:
Big Data

14. Hadoop

15. Hadoop 2013:
Embrace Spark

16. Go beyond batch?

17. The Emergence of Fast
Data:
(Time Is Money)

18. Respond to change

19. IoT

20. 20

21. Fast Data and Microservices;
Are they Converging?

22. •Each [stream app or μservice]:
•does one responsibility
•ingests unending [data or
messages]
Synergies
22

23. •Each [stream app or μservice]
must:
•operate asynchronously
•offer never-ending service
23
Synergies

24. •These architectures are
converging:
1.Similar design problems
2.Data becomes dominant
problem 24
Thesis

25. Lightbend
Fast Data Platform

26. 1. An accelerated on ramp for building
streaming data systems, data
applications, and other microservices.
Value Three Ways

27. 2. Best practices guidance for solving
specific design problems:
- Sample apps
- Documentation
- Enablement services
Value Three Ways

28. 3. Machine learning-based monitoring
and management:
-Keep your systems resilient,
scalable, and responsive with
minimal user intervention.
Value Three Ways

38. •Low latency? How low?
•High volume? How high?
Streaming Tradeoffs (1/3)
38

39. •Which kinds of data processing &
analytics are required?
•How will this processing be done?
•Individual processing of events?
•Bulk processing of records?
Streaming Tradeoffs (2/3)
39

40. •Which tools and data sources/
sinks must interoperate with your
streaming tool?
Streaming Tradeoffs (3/3)
40

41. 41

42. 42
•Low latency
•Low volume
•Complex flows
•Complex
Event
Processing

43. 43
•Med. latency
•High volume
•Data flows,
SQL
•En masse
processing

44. 44
•Low latency
•High volume
•Data flows,
correctness
•En masse
processing

45. 45
•Low latency
•Med. volume
•ETL, “tables”
•Data flow or
per event

48. Kafka & Spark Metrics

49. Correlated Troubleshooting

51. •Only metrics are uploaded, not
sensitive domain data.
•One less service for you to manage.
•…
Why Only Hosted
Services?

52. •…
•We can rapidly evolve this service.
without impacting your environment.
•You benefit from aggregated
knowledge from all FDP clusters.
Why Only Hosted
Services?

54. Upgrade your grey matter!
Get the free O’Reilly book by Dr. Dean Wampler,
VP of Fast Data Engineering at Lightbend
bit.ly/lightbend-fast-data

55. Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS, Marathon
On Premise or Cloud
FDP
Kafka
Connect
Storage
HDFS,
S3, …
SQL/
NoSQL
ES
Kafka
Streams
Cloud Hosted
Graviton
Cluster
Analysis
Machine
LearningMicroservices
ProducCon Suite
Machine Learning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Management…Consoles
For More Information:
lightbend.com/fast-data-platform