S4: Distributed Stream Computing Platform
•
6 likes•2,631 views
A presentation about Yahoo! S4 and Apache S4. I gave this presentation for Cloud Computing course of Dr. Payberah @ AUT fall 2014. The lecturer's references are Yahoo! S4 paper and Apache S4 website.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to S4: Distributed Stream Computing Platform
Similar to S4: Distributed Stream Computing Platform (20)
Recently uploaded
Recently uploaded (20)
S4: Distributed Stream Computing Platform
- 1. S4: Distributed Stream Computing Platform YAHOO LABS 2010 FARZAD NOZARIAN, MAZAHER BAZARI Cloud Computing CEIT@AUT 12/22/2014
- 2. /* Who we are! */ Farzad Nozarian fnozarian@aut.ac.ir Big Data Processing And Mining Mazaher Bazari mbazari@aut.ac.ir Mobile Cloud Computing 1
- 3. What is S4 Simple Scalable Streaming System Inspired by the MapReduce model! S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform for processing continuous unbounded streams of data 2
- 4. Motivation Real-time search High frequency trading Social networks 3
- 6. “cost-per-click” billing model Render the most relevant ads in an optimal position on the page Include user preferences from context: Processing thousands of queries per sec. recent user activity Geographic location Prior queries Prior clicks 5
- 7. Reinvent the Wheel! Extending the open source Hadoop platform to support computation of unbound streams But, Hadoop isn’t suitable! The Hadoop platform was highly optimized for batch processing MapReduce systems typically operate on static data by scheduling batch jobs. 6
- 8. Real world systems! Partition #1 Partition #2 Partition #... Partition #N Partition #3 Data Stream Latency Latency is proportional to Length of the segment Overhead of segmentation and initiate the processing jobs Fixed-size segments 7
- 9. Design goals Simple API Scale using commodity hardware Minimize latency by using local memory in each processing node 8
- 10. Design goals Decentralized and symmetric architecture Pluggable architecture Science friendly 9
- 11. S4 Model Avoiding the use of shared memory across the cluster Distributed operation on commodity hardware Actors model G. Agha, Actors: A Model of Concurrent Computation in Distributed Systems. 10
- 12. S4 model (cont.) Computation is performed by Processing Elements (PEs) Messages are transmitted between them in the form of data events The state of each PE is inaccessible to other PEs Event emission and consumption is the only mode of interaction between PEs The framework provides the capability to route events to appropriate PEs and to create new instances of PEs 11
- 13. Design Assumptions Lossy failover is acceptable! Nodes will not be added to or removed from a running cluster! 12
- 14. Design: Example What is the task? The task is to continuously produce a sorted list of the top K most frequent words across all documents with minimal latency 13
- 15. EV Quote KEY Null VALUE Quote=“I …” A keyless event (EV) arrives at PE1 with quote: EV Quote “I meant what I said and I said what I meant.”, Dr. Seuss PE1 QuoteSplitterPE (PE1) counts unique words in Quote and emits events for each word. EV WordEvent KEY word="said" VALUE count=2 PE2 PE3 PE4… EV WordEvent KEY word=“i" VALUE count=4 PE2 PE3 PE4… PE3 EV UpdatedCountEv KEY sortID=2 VALUE word=said count=9 EV UpdatedCountEv KEY sortID=9 VALUE word="i" count=35 WordCountPE (PE2-4) keeps total counts for each word across all quotes. Emits an event any time a count is updated. EV PartialTopKEv KEY topk=1234 VALUE words={w:cnt} MergePE (PE8) combines partial TopK lists and outputs final TopK list. 14
- 16. Design: Processing Elements PE Functionality Types of events Keyed attribute of events Value of the keyed attribute EV WordEvent KEY word="said" VALUE count=2 15
- 17. Design: Processing Elements (cont.) EV Quote KEY Null VALUE Quote=“I …” Keyless PEs No keyed attribute or value Consume all events of the type with which they are associated Typically used at the input layer of an S4 cluster where events are assigned a key Standard PE Count aggregate join 16
- 18. Design: Processing Node Processing Nodes (PNs) are the logical hosts to PEs. They are responsible for: listening to events executing operations on the incoming events dispatching events with the assistance of the communication layer emitting output events 17
- 19. Communication Layer Zookeeper Design: Processing Node (cont.) Processing Element Container PE1 PE2 PEn … Event Listener Dispatcher Emiter Routing Load Balancing Failover Management Transport Protocols 18
- 20. Programming Model High-level programming paradigm Generic Reusable Configurable Java Programming Language 19
- 21. Programming Model (cont.) Define Many ProcessEvent (polymorphism) Create PE Inherited from AbstractePE Implement ProcessEvent() Implement Output()(Optional) 20
- 23. Programming Model (cont.) PE1 PE2 PE3 PE4 PE5 PE1 PE2 PE3 PE4 PE5 Configuration 22
- 24. Programming Model (cont.) Configuration 23
- 25. Streaming Click-Through Rate Computation CTR = (ratio of the number of clicks )/(number of impressions) Two types of events Serve Event Click Event Serve is a search result page is returned to the user 24
- 26. Streaming Click-Through Rate Computation(cont.) Serve event contain: serveID query user Ads ….. Click event contain: Click information serveID Use a set of heuristic rules to eliminate suspicious serves and click 25
- 27. EV RawServe KEY Null VALUE _Serve_Data Event Flow of CTR Computation PE1 EV serve KEY Serve=123 VALUE Serve Data EV Click KEY Serve=123 VALUE Click Data PE4 EV JoinedServe KEY usr=Peter VALUE JoinedData EV JoinedClick KEY usr=Peter VALUE JoinedData EV FilteredServe KEY g-ad=Ipod-78 VALUE JoinedData EV RawClick KEY Null VALUE _Click_Data PE2 PE3 EV FilteredClick KEY g-ad=Ipod-78 VALUE JoinedData 26
- 28. Apache S4 27
- 30. Zookeeper Node1 Node2 Cluster 1 Repo S4 App 12 3 4 Apache S4 29
- 32. Zookeeper Node1 Node2 Cluster 1 Node1 Cluster 2 Repo 1 2 3 Apache S4 31
- 33. Apache S4: Commands s4 <command> <options> Command Purpose newApp Create a new application zkServer Start a ZooKeeper server newCluster Define an S4 cluster s4r Package an application deploy Deploy/configure an application node Start an S4 node status Get information about S4 infrastructure 32
- 34. Apache S4: Failover while preserving low processing latency High Availability State Recovery 33
- 35. Apache S4: Failover Zookeeper Node 2 Node 3 Node 1 Node 4 Standb Standby Standby Node 3 High Availability State Recovery 34
- 36. Apache S4: Failover PE recover a previous state() periodically checkpoint (uncoordinated and asynchronous) lazily recover PE 1 PE 2 Keyed Message Checkpoint Framework Hooks Storage Backend Storage S4 node 35
- 37. Summary S4: Simple Scalable Streaming System Design Processing Elements Processing Nodes Communication Layer Programming Model Apache S4 Deployment Failover 36
- 38. Thanks Q&A
Editor's Notes
- thousands of queries per second, which may include several ads per page. To process user feedback, we developed S4, a low latency, scalable stream processing engine.
- The main requirement for research is to have a high degree of flexibility to deploy algorithms to the field very quickly. The main requirements for a production environment are scalability and high availability
- Small segments will reduce latency, add overhead, and make it more complex to manage intersegment dependencies On the other hand, large segments would increase latency. The optimal segment size will depend on the application.
- Minimize latency by using local memory in each processing node and avoiding disk I/O bottlenecks.
- Decentralized architecture greatly simplifies deployment and maintenance. Use a pluggable architecture to keep the design as generic and customizable as possible.
- Upon a server failure, processes are automatically moved to a standby server. The state of the processes, which is stored in local memory, is lost during the handoff. The state is regenerated using the input streams.
- QuoteSplitterPE is a keyless PE object that processes all Quote events. For each unique word in a document, the QuoteSplitterPE object will assign a count and emit a new event of type WordEvent, keyed on word. If the WordCountPE object exists, the PE object is called and the counter is incremented, otherwise a new WordCountPE object is instantiated.
- S4 routes each event to PNs based on a hash function of the values of all known keyed attributes in that event.
- Communication Layer provides: Cluster management Automatic failover to standby nodes Maps physical nodes to logical nodes It uses a pluggable architecture to select network protocol Events may be sent with or without a guarantee It uses ZooKeeper to help coordinate between nodes