This document discusses how containers can be used for big data workloads. It notes that containers provide lightweight virtualization similar to virtual machines. The document outlines how containers can help with distributed processing and storage of big data. It discusses using containers to ship computation to data and schedule processes based on data locality. Overall, the document argues that containers are well-suited for big data applications by allowing distributed, short-lived processes to be run efficiently near related data.

1. Big Data and Containers
Charles Smith
@charles_s_smith

2. Netflix / Lead the big data platform architecture team
Spend my time / Thinking how to make it easy/efficient to work with big data
University of Florida / PhD in Computer Science
Who am I?

3. “It is important that we know where we come from, because
if you do not know where you come from, then you don't
know where you are, and if you don't know where you are,
you don't know where you're going. And if you don't know
where you're going, you're probably going wrong.”
Terry Pratchett

4. Database Distributed Database Distributed Storage
Distributed Processing
???

5. Why do we care about containers?

6. Containers ~= Virtual Machines
Virtual Machines ~= Servers

7. Lightweight
fast to start
memory use
Secure
Process isolation
Data isolation
Portable
Composable
Reproducible
Everything old is new

8. Microservices and large architectures

9. Datastorage
(Cassandra, MySQL, MongoDB, etc..)

10. Operational
(Mesos, Kubernetes, etc...)

11. Discovery/Routing

12. What’s different about big data.

13. Data at rest
Data in motion

14. Customer Facing
Minimize latency
Maximize reliability

15. Data Analytics
Minimize I/O
Maximize processing

16. Ship computation to data

17. The questions you can answer aren’t predefined

18. Hive/Pig/MR
Presto
Metacat
Hive
Metastore

19. That doesn’t look very container-y
(or microservicy-y for that matter)

20. Datastorage - HDFS (Or in our case S3)

21. Operational - YARN

22. Containers - JVM

23. So what happens when you want to do something else?

25. But is that really the way we want to approach containers?

26. What’s different about big data.

27. Running many different short-lived processes

28. Running many different short-lived processes
Efficient container construction, allocation, and movement

29. Groups of processes having meaning

30. Groups of processes having meaning
How we observe processes needs to be holistic

31. Processes need to be scheduled by data locality
(And not just data locality for data at rest)

32. Processes need to be scheduled by data locality
(And not just data locality for data at rest)
A special case of affinity (although possibly over time)
but...

33. We do need a data discovery service.
(kind of… maybe… a namenode?)

34. SELECT
t.title_id,
t.title_desc,
SUM(v.view_secs)
FROM
view_history as v
join title_d as t on
v.title_id =
t.title_id
WHERE
v.view_dateint > 20150101
GROUP BY 1,2;
LOAD LOAD
JOIN
GROUP

35. Data
Discovery
Query Compiler
Query Planner
Metadata
DAG
Watcher

36. Bottom line
Containers provide process level security
The goal should be to minimize monoliths
This isn’t different from what we are doing already
Our languages are abstractions of composable-distributed processing
Different big data projects should share services
No matter what we do, joining is going to be a big problem

37. Questions?