Talk given in NYC on 7/20/2015

1. Demystifying Data
Engineering

2. Data engineering
• Software engineering with an emphasis on
dealing with large amounts of data
• A “specialty” of software engineering

3. Why now?
• Always value in scale, but it was previously
too difficult / expensive
• Economics and technology advances make
these scales accessible

4. Enable others to answer questions on
dataset within latency constraints

5. Data engineering
• Distributed systems – consensus,
consistency, availability, etc.
• Parallel processing
• Databases
• Queuing

6. Data engineering
• Human-fault tolerance
• Metrics and monitoring
• Multi-tenancy

7. BackType
• When I joined:
• Comment search by keyword
• Comment search by user
• Basic stats on commenters
• Link search on Twitter

8. BackType
Kyoto
Cabinet
Custom
workers
Custom
crawlers

9. BackType
• Inflexible
• Prone to corruption
• Heavy operational burden
• Not scalable
• Not fault-tolerant

10. BackType
• Enable asking any question (with high
latency)
• Allows exploration and experimentation
• Establishes human-fault tolerance

11. Collector
Collector
Collector
Collector

13. ElephantDB
• Export results of MapReduce pipelines for
querying
• Low latency querying but out of date by
many hours
• Incredibly simple

15. • Infrastructure
• Data pipelines
• Abstractions
Data engineering

16. Data pipeline example
Tweets
(S3)
Normalize
URLs
Compute
hour bucket
Sum by
hour/url
Emit
ElephantDB
indexes

17. Data pipeline example
Tweets
(Kafka)
Normalize
URLs
Compute
hour bucket
Update hour/
url bucket
Cassandra

18. Abstraction example
MapReduce Cascading Cascalog

20. Infrastructure
• HDFS
• MapReduce
• Kafka
• Storm
• Spark
• Cassandra
• HBase
• ElephantDB
• Zookeeper

21. Streaming compute
team at Twitter
• Started streaming compute team at Twitter
• One shared Storm cluster for entire
company

22. Multi-tenancy
• Independent applications on same cluster
• Topologies should not affect one another

23. Resource allocation
• Topologies should be given an appropriate
amount of resources

24. Initial approach
• Use Mesos to provide resource guarantees
• Users include resources needed as part of
topology submission

26. Solution
• Implement new scheduler which gives
production topologies dedicated hardware
• Only Storm team can configure production
topologies
• Left-over machines are used as failover or
for in-development topologies

28. Data Engineering vs Data Science
• Well-defined problems
• No special statistics skills required
• Larger scope
• Not just analytics

29. Open source
• Almost all major Big Data tools are open
source (e.g. Hadoop, Storm, Spark, Kafka,
Cassandra, HBase, etc.)
• Many have commercial support

30. Open source
• Very important for recruiting data
engineers
• Strong developers want to work at places
where they can be involved with open
source

31. Open source
• Develop a technology brand for company
(in conjunction with a tech blog)
• Creating a popular open source project can
give you access to lots of strong engineers

32. Open source
• Identify strong engineers in the community
you may want to recruit
• Learn best practices and get help from the
people who know the tools the best
• *Do not* expect to get “free work” on
your projects

33. Ideal data engineer
• Strong software engineering skills
• Abstraction
• Testing
• Version control
• Refactoring

34. Ideal data engineer
• Strong software engineering skills
• Strong algorithm skills

35. Ideal data engineer
• Strong software engineering skills
• Strong algorithm skills
• Good at digging into open source code

36. Ideal data engineer
• Strong software engineering skills
• Strong algorithm skills
• Good at digging into open source code
• Good at stress testing

38. Finding strong data engineers
• Standard “coding on the whiteboard”
interviews are near useless
• Use take home projects to gauge general
programming ability
• The best is to see projects that require
data engineering

39. Questions?