* Quick Intro to Bigtop * Trend Micro Big Data Platform * Mission-specific Platform * Big Data Landscape (3p) * Bigtop 1.1 Release (6p)

1. 葉祐欣 (Evans Ye)
Big Data Conference 2015
Trend Micro Big Data Platform
and Apache Bigtop

2. Who am I
• Apache Bigtop PMC member
• Apache Big Data Europe 2015 Speaker
• Software Engineer @ Trend Micro
• Develop big data apps & infra
• Has some experience in Hadoop, HBase, Pig,
Spark, Kafka, Fluentd, Akka, and Docker

3. Outline
• Quick Intro to Bigtop
• Trend Micro Big Data Platform
• Mission-specific Platform
• Big Data Landscape (3p)
• Bigtop 1.1 Release (6p)

4. Quick Intro to Bigtop

5. Linux Distributions

6. Hadoop Distributions

7. Hadoop Distributions
We’re fully open sourced !

8. How do I add patches?

10. From source code
to packages
Bigtop
Packaging

11. Bigtop feature set
Packaging Testing Deployment Virtualization
for you to easily build your own Big Data Stack

12. Supported components

13. • $ git clone https://github.com/apache/bigtop.git
• $ docker run
--rm
--volume `pwd`/bigtop:/bigtop
--workdir /bigtop
bigtop/slaves:trunk-centos-7
bash -l -c ‘./gradlew rpm’
One click to build packages

14. • $ ./gradlew tasks’

15. Easy to do CI
ci.bigtop.apache.org

16. RPM/DEB packages
www.apache.org/dist/bigtop

17. One click Hadoop provisioning
./docker-hadoop.sh -c 3

18. bigtop/deploy image
on Docker hub
./docker-hadoop.sh -c 3
One click Hadoop provisioning

19. bigtop/deploy image
on Docker hub
./docker-hadoop.sh -c 3
puppet apply
puppet apply
puppet apply
One click Hadoop provisioning
Just google bigtop provisioner

20. Should I use Bigtop?

21. If you want to build your
own customised
Big Data Stack

22. Curves ahead…

23. Pros & cons
• Bigtop
• You need a talented Hadoop team
• Self-service: troubleshoot, find solutions, develop patches
• Add any patch at any time you want (additional efforts)
• Choose any version of component you want (additional efforts)
• Vendors (Hortonworks, Cloudera, etc)
• Better support since they’re the guy who write the code !
• $

24. Trend Micro
Big Data Platform

25. • Use Bigtop as the basis for our internal custom
distribution of Hadoop
• Apply community, private patches to upstream
projects for business and operational need
• Newest TMH7 is based on Bigtop 1.0 SNAPSHOT
Trend Micro Hadoop (TMH)

26. Working with community
made our life easier
• Knowing community status made TMH7 release
based on Bigtop 1.0 SNAPSHOT possible

27. Working with community
made our life easier
• Contribute Bigtop Provisioner, packaging code,
puppet recipes, bugfixes, CI infra, anything!
• Knowing community status made TMH7 release
based on Bigtop 1.0 SNAPSHOT possible

28. Working with community
made our life easier
• Leverage Bigtop smoke tests and integration tests
with Bigtop Provisioner to evaluate TMH7

29. Working with community
made our life easier
• Contribute feedback, evaluation, use case
through Production level adoption
• Leverage Bigtop smoke tests and integration tests
with Bigtop Provisioner to evaluate TMH7

30. Hadoop YARN
Hadoop HDFS
Mapreduce
Ad-hoc Query UDFs
Pig
App A App C
Oozie
Resource
Management
Storage
Processing
Engine
APIs and
Interfases
In-house
Apps
Trend Micro Big Data Stack
Powered by Bigtop
Kerberos
App B App D
HBase
Wuji
Solr
Cloud
Hadooppet (prod) Hadoocker (dev)Deployment

31. Hadooppet
• Puppet recipes to deploy and manage TMH
Big Data Platform
• HDFS, YARN, HA auto-configured
• Kerberos, LDAP auto-configured
• Kerberos cross realm authentication auto-configured
(For distcp to run across secured clusters)

33. • A Devops toolkit for Hadoop app developer
to develop and test its code on
• Big Data Stack preload images
—> dev & test env w/o deployment
—> support end-to-end CI test
• A Hadoop env for apps to test against new
Hadoop distribution
• https://github.com/evans-ye/hadoocker
Hadoocker

34. internal Docker registry
./execute.sh
Hadoop server
Hadoop client
data
Docker based dev & test env
TMH7
Hadoop app
Restful
APIs
sample data
hadoop fs put

35. internal Docker registry
./execute.sh
Hadoop server
Hadoop client
data
TMH7
Hadoop app
Restful
APIs
sample data
hadoop fs putSolr
Oozie(Wuji)
Dependency service
Docker based dev & test env

36. Mission-specific Platform

37. Use case
• Real-time streaming data flows in
• Lookup external info when data flows in
• Detect threat/malicious activities on streaming data
• Correlate with other historical data (batch query) to gather
more info
• Can also run batch detections by specifying arbitrary start
time and end time
• Support Investigation down to raw log level

38. Lambda Architecture

39. receiver

40. receiver
buffer

41. transformation,
lookup ext info
receiver
buffer

42. batch
streaming
receiver
buffer
transformation,
lookup ext info

43. transformation,
lookup ext info
batch
streaming
receiver
buffer

44. • High-throughput, distributed publish-subscribe
messaging system
• Supports multiple consumers attached to a topic
• Configurable partition(shard), replication
factor
• Load-balance within same consumer group
• Only consume message once
a b c

45. • Distributed NoSQL key-value storage, no SPOF
• Super fast on write, suitable for data keeps coming in
• Decent read performance, if design it right
• Build data model around your queries
• Spark Cassandra Connector
• Configurable CA (CAP theorem)
• Choose A over C for availability and vise-versa
Dynamo: Amazon’s Highly Available Key-value Store

46. • Fast, distributed, in-memory processing engine
• One system for streaming and batch workloads
• Spark streaming

47. Akka
• High performance concurrency framework for Java and Scala
• Actor model for message-driven processing
• Asynchronous by design to achieve high throughput
• Each message is handled in a single threaded context
(no lock, synchronous needed)
• Let-it-crash model for fault tolerance and auto-healing system
• Clustering mechanism to scale out
The Road to Akka Cluster, and Beyond

48. Akka Streams
• Akka Streams is a DSL library for streaming computation on Akka
• Materializer to transform each step into Actor
• Back-pressure enabled by default
Source Flow Sink
The Reactive Manifesto

49. No back-pressure
Source Fast!!! SinkSlow…
(＞﹏＜)’v(￣︶￣)y

50. No back-pressure
Source Fast!!! SinkSlow…
(＞﹏＜)’’’’’v(￣︶￣)y

51. With back-pressure
Source Fast!!! SinkSlow…

52. With back-pressure
Source Fast!!! SinkSlow…
request 3request 3

53. Data pipeline with Akka Streams
• Scale up using balance and merge
source: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-cookbook.html#working-with-flows
worker
worker
worker
balance merge

54. • Scale out using docker
Data pipeline with Akka Streams
$ docker-compose scale pipeline=3

55. Reactive Kafka
• Akka Streams wrapper for Kafka
• Commit processed offset back into Kafka
• Provide at-least-once delivery guarantee
https://github.com/softwaremill/reactive-kafka

56. Message delivery guarantee
• Actor Model: at-most-once
• Akka Persistence: at-least-once
• Persist log to external storage (like WAL)
• Reactive Kafka: at-least-once + back-pressure
• Write offset back into Kafka
• At-least-once + Idempotent writes = exactly-once

57. • Spark: both streaming and batch analytics
• Docker: resource management (fine for one app)
• Akka: fine-grained, elastic data pipelines
• Cassandra: batch queries
• Kafka: durable buffer, fan-out to multiple consumers
Recap: SDACK Stack

58. Your mileage may vary

59. we’re still evolving

60. Remember this:

61. The SMACK Stack
Toolbox for wide variety of data processing scenarios

62. SMACK Stack
• Spark: fast and general engine for large-scale data
processing
• Mesos: cluster resource management system
• Akka: toolkit and runtime for building highly concurrent,
distributed, and resilient message-driven applications
• Cassandra: distributed, highly available database designed
to handle large amounts of data across datacenters
• Kafka: high-throughput, low-latency distributed pub-sub
messaging system for real-time data feeds
Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka

63. Reference
• Spark Summit Europe 2015
• Streaming Analytics with Spark, Kafka,
Cassandra, and Akka (Helena Edelson)
• Big Data AW Meetup
• SMACK Architectures (Anton Kirillov)

64. Big Data Landscape

65. • Memory is faster than SSD/disk, and is cheaper
• In Memory Computing & Fast Data
• Spark : In memory batch/streaming engine
• Flink : In memory streaming/batch engine
• Iginte : In memory data fabric
• Geode (incubating) : In memory database
Big Data moving trend

66. • Off-Heap storage is a JVM process memory
outside of the heap, which is allocated and
managed using native calls.
• size not limited by JVM (it is limited by physical
memory limits)
• is not subject to GC which essentially removes
long GC pauses
• Project Tungsten, Flink, Iginte, Geode, HBase
Off-Heap, Off-Heap, Off-Heap

67. Pig
Hadoop YARN
Hadoop HDFS
Resource
Management
Storage
Processing
Engine
(Some) Apache Big Data
Components
Slider
Flink Spark
Flink ML,
Gelly
Streaming,
MLlib, GraphX
Kafka
HBase
Mesos
Tez
Hive Phoenix
Ignite
APIs and
Interfases
Geode
Trafodion
Solr
}
messaging system in memory data grid search engine
Bigtop
Ambari
Hadoop
Distribution
Hadoop
Management
Cassandra
NoSQL

68. Bigtop 1.1 Release
Jan, 2016, I expect…

69. Bigtop 1.1 Release
• Hadoop 2.7.1
• Spark 1.5.1
• Hive 1.2.1
• Pig 0.15.0
• Oozie 4.2.0
• Flume 1.6.0
• Zeppelin 0.5.5
• Ignite Hadoop 1.5.0
• Phoenix 4.6.0
• Hue 3.8.1
• Crunch 0.12
• …, 24 components included!

71. Hadoop 2.6
• Heterogeneous Storages
• SSD + hard drive
• Placement policy (all_ssd, hot, warm, cold)
• Archival Storage (cost saving)
• HDFS-7285 (Hadoop 3.0)
• Erasure code to save storage from 3X to 1.5X
http://www.slideshare.net/Hadoop_Summit/reduce-storage-
costs-by-5x-using-the-new-hdfs-tiered-storage-feature

72. Hadoop 2.7
• Transparent encryption (encryption zone)
• Available in 2.6
• Known issue: Encryption is sometimes done
incorrectly (HADOOP-11343)
• Fixed in 2.7
http://events.linuxfoundation.org/sites/events/files/slides/
HDFS2015_Past_present_future.pdf

73. Rising star: Flink
• Streaming dataflow engine
• Treat batch computing as fixed length streaming
• Exactly-once by distributed snapshotting
• Event time handling by watermarks

74. • Integrate and package Apache Flink
• Re-implement Bigtop Provisioner using
docker-machine, compose, swarm
• Deploy containers on multiple hosts
• Support any kind of base image for deployment
Bigtop Roadmap

75. Wrap up

76. • Hadoop Distribution
• Choose Bigtop if you want more control
• The SMACK Stack
• Toolbox for variety data processing scenarios
• Big Data Landscape
• In-memory, off-heap solutions are hot
Wrap up

77. Questions ?
Thank you !