Lossless Stream Processing
•
35 likes•3,029 views
This document discusses real-time data processing and streaming technologies. It provides links to resources about real-time computing, streaming data, and Apache streaming frameworks like Storm and Spark Streaming. Key topics covered include exactly-once processing, distributed fault tolerance, and building real-time systems that can respond within targeted response times using streaming data pipelines.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Lossless Stream Processing
Similar to Lossless Stream Processing (20)
Recently uploaded
Recently uploaded (20)
Lossless Stream Processing
- 1. 7FWF R F W HW (). (, 0
- 3. https://voltdb.com/sites/default/files/real-time-data-report.pdf 2015 Real-time Data Report by VoltDB b 효과 같은 a b 능력 다른 a
- 4. b A FP WN b f4 r w 4 마감 시간(Deadline) ) y4 (( y4 https://www.soasta.com/blog/23-stats-mobile-web-performance-monitoring/ https://en.wikipedia.org/wiki/Real-time_computing b ? F A FP WN 4 F I N BS W
- 5. 4 4 BW F 6FWHM Unbound Bound Data Data X X v https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
- 6. >NH S GFWHM Unbound Data h e 작은 배치 처리 끊임없이
- 7. 5B5 d 실시간으로 l l ” 목표 응답 시간 b s 고객을 잃기 전까지 실시간 1 목표 스트림 1 방식 http://bigdatapage.com/4-really-real-meanings-of-real-time/
- 8. https://tsicilian.wordpress.com/2015/02/16/streaming-big-data-storm-spark-and-samza/ BTF n m A8 = A FI 8YFP NRW =SST m E5A? e BHFPF Exactly-onceb 목표 10초 이내
- 9. 사용 4 좋아 4 https://twitter.com/jaykreps/status/568499167686995969 https://twitter.com/miguno/status/568503505113247744 Storm vs. Spark Streaming 좋아 4 사용 4 w x
- 10. https://databaseline.wordpress.com/2016/03/12/an-overview-of-apache-streaming-technologies/ s o (). (
- 11. 분산 g w 장애 ! m t u l
- 12. 분산 장애 부분 전송 성공 At-most-once 1 At-least-once 1 Exactly-once 1
- 13. 8 FHWP SRH ) 8 FHWP SRH https://twitter.com/mathiasverraes/status/632260618599403520 8 FHWP SRH M F SRP W S MF I T SGP NR IN W NG W I W 1 8 FHWP SRH I PNY ) : F FRW I S I S FL 8 FHWP SRH I PNY
- 14. 8 FHWP SRH m t u l Idempotent exactly-once 1 in d 2 2 5W P F W SRH 8 FHWP SRH Transactional exactly-once 1 “ SPP GFH “ 76 s
- 15. YARN HDFS Rake Server Query Cache AppServer SQL thin JDBC client Apache Kafka Rake Library Rake API Logs LogAgent HTTPS Phoenix JDBC client (select) JDBC client (upsert) Kafka Consumer Kafka Producer Collector 6=8
- 16. 6=8 ) - g , g
- 17. YARN HDFS Web Server HQL Apache Kafka Kafka Consumer Kafka Producer Kafka Producer Collector YARN Hive Tez )) ⇔
- 18. )) ⇔ D G ) - g >SGNP / g D G ..((g >SGNP .,((g )) D3>( >3D Web Mobile
- 19. 수신, 처리, 저장 k 부분 8 FHWP SRH 전체 8 FHWP SRH 데이터 원천 비즈니스 행동 i t 목표 응답 시간 내 반응 실시간 시스템 구축 s c 데이터 파이프라인 배치 처리 pb l 스트림 처리 p
- 20. Appendix
- 21. https://lobste.rs/s/ecjfcm/why_is_exactly-once_messaging_not_possible_in_a_distributed_queue/ 8 FHWP SRH delivery N impossible. 8 FHWP SRH processing of messages N possible N WM T SH NRL HFR G FI idempotent. 8 FHWP SRH 8 FHWP SRH NI TSW RW 8 FHWP SRH