hadoop-summit-2016

1. Apache
• 来自eBay的分布式实时Hadoop数据安全引擎
蒋吉麟 | 赵晴雯
eBay

2. Agenda
•About Eagle
•Front End
– Evolution
– Modularization
– Features
•Back End
– Architecture
– Tech Highlights
– Integration
•Q & A
2

3. 3
Apache Eagle is a distributed real-time monitoring and
alerting engine for Hadoop from eBay
Open sourced as Apache Incubator Project on Oct 26th 2015
See http://eagle.incubator.apache.org or http://goeagle.io

4. Hadoop @eBay
4
1-10 nodes
2007
100+ nodes
1000 + core
1 PB
2010 2011
1000+ node
10,000+ core
10+ PB
4000+ node
40,000+ core
50+ PB
2013
2015
10,000+ nodes
150,000+ cores
150+ PB
2009
10+ nodes

5. •swf
•exe
5

6. Features
•common
•metadata
•classification
•metrics
6

7. common
•Policies
•Alerts
7

8. metadata
8

9. classification
•Tree View
•Table View
9

10. metrics
10

11. Architecture
11
STREAM PROCESSING
ENGINE
User Profile
based Anomaly
detection
Policy evaluation
based
Framework
Eagle Storage
(Metadata,
metrics,
alerts…
User Profile
training
Eagle Query
DataCollection(Kafka,Yarn
API…)
Had
oop
jmx
DataSink(email,Kakfa…)
Other
Remediation
Systems
…

12. Tech Highlights
•Data Collection
•Stream Processing DSL
•Distributed Policy Engine
•ML-based anomaly detection
•Query Framework
NOTE {NAME}-{NUMBER} like HDFS-6914 means open source project ticket id contributed by us
12

13. Apache Eagle – Data Collection
Decoupled with Apache Kafka
• high-throughput distributed messaging
• Easy to inject various kinds of data sources
• Python/Java/C++ Kafka clients
Current data sources support
• Hadoop data
 HDFS, HBase audit log
 GC logs
 JMX metrics
 History/Running MR job data
• …
• Generic format data
13

14. Apache Eagle – Stream Processing DSL
Easy use
– Easily assemble data transformation, filtering, join…
Flexibility
– Physical execution platform independent
14
STREAM PROCESSING ENGINE
STREAM PROCESSING ENGINE
.flatMap(AuditLogTransformer)
.groupBy(_.user)
.flatMap(UserProfileAggregator);
env.fromKafka (KafkaConfig)
.alert.persistAndEmail
val env = ExecutionEnvironment.getStorm()
env.execute()

15. Apache Eagle – Stream Processing DSL
15
.flatMap(AuditLogTransformer)
.groupBy(_.user)
.flatMap(UserProfileAggregator);
env.fromKafka (KafkaConfig)
.alert.persistAndEmail
val env = ExecutionEnvironment.getStorm()
env.execute();
Distributed Streaming Cluster Environment
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Alerts
Real-time
Event Stream
Stream_{1}
Stream_{*}
Stream
Processing
env.execute()

16. Apache Eagle - Distributed Real-time Policy
Engine
Features
• Extensibility
• Usability
• Real-time
• Scalability
• Metadata-driven
16
METADATA MANAGER
Distributed Streaming Cluster Environment
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Real Time
Alerts
Alerts
Policy
Management
Policy
Dynamical Policy Deployment
Real-time
Event Stream
Stream_{1}
Stream_{*}
Dynamical Stream Schema
Stream
Processing

17. Apache Eagle – Distributed Real-time Policy
Engine
17
Distributed Real-time Policy Engine
Siddhi CEP
Policy
Evaluator
Machine
Learning Policy
Evaluator
Extensibility
• Default is WSO2 Siddhi CEP
• Powerful SQL-Like event stream
processing
• Open to other customized policy engine
Extensible Policy
Evaluator
public interface PolicyEvaluatorServiceProvider {
public String getPolicyType(); // literal string to identify one type of policy
public Class<? extends PolicyEvaluator<T>> getPolicyEvaluator(); // get policy evaluator
implementation
public List getBindingModules(); // policy text with json format to object mapping
}
public interface PolicyEvaluator {
public void evaluate(ValuesArray input) throws Exception; // evaluate input event
public void onPolicyUpdate(AlertDefinitionAPIEntity newAlertDef);// policy update
public void onPolicyDelete(); // invoked when policy is deleted
}
METADATA MANAGER
Policy/Metadata

18. Apache Eagle – Distributed Real-time Policy
Engine
18
METADATA MANAGER
Distributed Streaming Cluster Environment
Real Time
Alerts
Alerts
Policy
Management
Policy
Dynamical Policy Deployment
Usability
• Powerful SQL-Like CEP CQL
for Policy Definition
• Dynamical Policy Lifecycle
Management
(Deployment/Update)
• Easy-to-use Policy
management and Alert
analytics UI
from metricStream[(name == 'ReplLag')
and (value > 1000)] select * insert into
outputStream;

19. Apache Eagle – Distributed Real-time Policy
Engine
19

20. Apache Eagle – Distributed Real-time Policy
Engine
20
Real-time
• Stream events are
processed and alerts are
evaluated during
streaming
Distributed Streaming
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Real Time
Alerts
Alerts
Stream_{1}
Stream_{*}
Stream
Processing
Real-time
Event Stream

21. Apache Eagle – Distributed Real-time Policy
Engine
21
Metadata-Driven
• Stream Schema:
AlertStreamSchemaEntity
• Policy Definition: AlertDefinitionAPIEntity
@Table("alertdef")
@ColumnFamily("f")
@Prefix("alertdef")
@Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME)
@JsonIgnoreProperties(ignoreUnknown = true)
@TimeSeries(false)
@Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"})
@Indexes({
@Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true),
})
public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{
@Column("a")
private String desc;
@Column("b")
private String policyDef;
@Column("c")
private String dedupeDef;
METADATA MANAGER
Distributed Real-time Policy Engine
Dynamic Metadata Loading

22. Apache Eagle – Distributed Real-time Policy
Engine
22
Distributed Streaming Cluster Environment
AlertExecutor_{1}
AlertExecutor_{2}
…
AlertExecutor_{N}
Stream_{1}
Stream_{*}
Stream
Processing
Scalability
• Policy scalability: policy partitioning
• Event scalability: grouping
• Example: N Users with 3 partitions, M policies with 2 partitions, then 3*2 physical tasks

23. Apache Eagle – Query Framework
23
Query Syntax
• Full-function SQL-Like REST
Query (aggregation, sorting…)
Eagle Storage
• NOSQL storage like HBase
• RDMS
• Other storage systems

24. Apache Eagle – ML-based Anomaly Detection
24
User Activity Anomaly
Detection
• User profile feature
selection
• Offline user profile
generation
• Online Anomaly
detection
Useful link
• Eagle: User profile-
based anomaly
detection for securing
Hadoop clusters

25. Apache Eagle – Integration I
25
• Eagle in Apache Ambari
– natively be part of hadoop ecosystem
– http://eagle.incubator.apache.org/docs/ambari-plugin-install.html
• Eagle in Docker
– natively fly on Cloud/Container
– https://github.com/apache/incubator-eagle

26. Apache Eagle – Integration II
26
•Apache Ranger
– remediation engine
– Eagle data source
•Splunk
– Eagle alert consumer
– EAGLE alert output is the 1st abstraction of analytics and Splunk is the 2nd abstraction
• Dataguise, Apache knox
– Eagle data source

27. Learn more about Apache Eagle
27
• EAGLE: USER PROFILE-BASED ANOMALY DETECTION IN HADOOP CLUSTER (IEEE)
• EAGLE: DISTRIBUTED REALTIME MONITORING FRAMEWORK FOR HADOOP
CLUSTER

28. Q&A
28
apache/incubator-eagle
@TheApacheEagle
@ApacheEagle
http://eagle.incubator.apache.org