Apache HBase at Airbnb
•
15 likes•5,895 views
Jingwei Lu and Jason Zhang (Airbnb) AirStream is a realtime stream computation framework built on top of Spark Streaming and HBase that allows our engineers and data scientists to easily leverage HBase to get real-time insights and build real-time feedback loops. In this talk, we will introduce AirStream, and then go over a few production use cases.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apache HBase at Airbnb
Similar to Apache HBase at Airbnb (20)
More from HBaseCon
More from HBaseCon (20)
Recently uploaded
Recently uploaded (20)
Apache HBase at Airbnb
- 1. Apache HBase at Airbnb JINGWEI LU, LIYIN TANG, AND JASON ZHANG 1
- 2. Data Infrastructure at Airbnb
- 3. Event Logs MySQL Dumps Gold Cluster HDFS Hive Kafk a Sqoo p Silver Cluster Spark Cluster Spark ReAi r Airflow Scheduling S3 Presto Cluster AirPal Caravel Tableau Batch Infrastructure Yarn HDFS Hive Yarn Jingwei Lu, Liyin Tang, Jason Zhang 3
- 4. Streaming at Airbnb Event Logging MySQL BINLOG Cluster HDFS Hive Spinal tap Presto Cluster Yarn Kafk a HBase Spark Streaming Datadog Druid Kafka Jingwei Lu, Liyin Tang, Jason Zhang 4
- 5. Growing Pain
- 6. Stateless Jingwei Lu, Liyin Tang, Jason Zhang Computation SinkSource DStream DF DF
- 7. Stateful Jingwei Lu, Liyin Tang, Jason Zhang Computatio n Source DStream DF DF Sink1 Sink2 Sink N State Storage RDD
- 8. Multiple Streams Jingwei Lu, Liyin Tang, Jason Zhang DataFrame Sink1 Process A Sink2 Sink3 SinkN … DataFrame Sink1 Process N Sink2 Sink3 SinkN … Source DStream Align by Time DataFram e DataFram e State Storage Source DStream …
- 9. Streaming + Batch Jingwei Lu, Liyin Tang, Jason Zhang DataFrame Sink1 Process A Sink2 Sink3 SinkN … DataFram e State Storage Sourc e DStream Sourc e … Align by Time … DataFrame Sink1 Process A Sink2 Sink3 SinkN …
- 11. AirStream Architecture Jingwei Lu, Liyin Tang, Jason Zhang Sources Stream #1 Stream #N Hive Tables HBase Tables Virtual Table Views for Computation Sinks … Customized ComputationSpark SQL Simple Config HBase Services Streaming Sources Druid
- 12. AirStream Architecture Jingwei Lu, Liyin Tang, Jason Zhang Sources Stream #1 Stream #N Hive Tables HBase Tables Virtual Table Views for Computation Sinks … Customized ComputationSpark SQL HBase Services Streaming Sources Druid Same Computation for Batch processing
- 13. Stateful
- 14. Jingwei Lu, Liyin Tang, Jason Zhang State Store • Merge changes • Provide fast lookup • Fast persistent storage across streaming and batch jobs 14
- 15. Why HBase Jingwei Lu, Liyin Tang, Jason Zhang Rich Functionalities Rich Integration with Hadoop EcoSystem Easy Management Strong Community Reliable and Scalable
- 16. HBase State Store Operators in Airstream Jingwei Lu, Liyin Tang, Jason Zhang 16 Full Table Scan Simple Aggregation Bulk Upload Key/Prefix Lookup Update
- 17. Jingwei Lu, Liyin Tang, Jason Zhang Computation DAG 17 Input Data Left Outer Join Result Key Lookup
- 18. Jingwei Lu, Liyin Tang, Jason Zhang Key Space Design • Hash partition key space for load balance • Composite key for K -> V • Support full key lookup • Prefix lookup supported for all keys used in hash function Hash key1 key2 key3 Hash based on key prefix Hash key1 key2 Lookup based on key prefix key1 = ‘value1’ and key2 = ‘value2’ 18
- 19. • Partition based on key before write • Use bulk upload for large volume update Write Performance Jingwei Lu, Liyin Tang, Jason Zhang 19
- 20. Case Study Jingwei Lu, Liyin Tang, Jason Zhang Experiment realtime feedback Update Experiment Assignment Event Lookup HBase with TTL Booking Event Druid Datado g 20 one airstream job
- 22. Realtime Ingestion on HBase Data Infrastructure MySQL Analytica l Events Kafka Spark Streamin g HBase HDFS Presto/Hive/Spar k Source Inges t Realtime Query Snapsh ot Batch Query Jingwei Lu, Liyin Tang, Jason Zhang 22
- 23. Access Data in HBase Jingwei Lu, Liyin Tang, Jason Zhang HBase Hive Presto Spark SQL Spark Streaming Batch Jobs Interactive Query Streaming HDFS Snapshot Table Mapping/Unifed View on realtime data 23
- 24. Snapshot & Reseed Jingwei Lu, Liyin Tang, Jason Zhang HBase HDFS Snapshot（HFile Links) Bulk Upload 24
- 25. Case Study 1: Events Ingestion Jingwei Lu, Liyin Tang, Jason Zhang Kafka topic … topic topic Spark Executor 1 … Executor 2 Executor K HBase DeDup HDFS Region1…Region2R egion M Daily Snapshot Realtime Query Hive Presto Events Partition 25
- 26. Case Study 2: Streaming DB Export KafkaRDS Table1 … Spinalta p. Table1 … Table2 TableN Spinaltap. Table2 Spinaltap. TableN Spark Executor 1 … Executor2 Executor K HBase Region1 … Region2 Region M HDFS Region1…Region2R egion M Daily Snapshot Realtime Query Jingwei Lu, Liyin Tang, Jason Zhang 26
- 27. Case Study: Streaming DB Export Rows CF: Colums Version Value <ShardKey><DB_TABLE_#1><PK_a=A> id Fri May 19 00:33:19 2016 101 <ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 19 00:33:19 2016 San Francisco <ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 10 00:34:19 2016 New York <ShardKey><DB_TABLE_#2><PK_a=A’> id Fri May 19 00:33:19 2016 1 Jingwei Lu, Liyin Tang, Jason Zhang 27
- 28. Case Study: Streaming DB Export TXN 1 Commit_TS: 101 … TXN 2 Commit_TS: 102 TXN 3 Commit_TS: 103 TXN N Commit_TS: N’ Binlog Order Jingwei Lu, Liyin Tang, Jason Zhang 28
- 29. Case Study: Streaming DB Export TXN 1 Commit_TS: 101 … TXN 2 Commit_TS: 103 TXN 3 Commit_TS: 102 TXN N Commit_TS: N’ NTP Binlog Order Jingwei Lu, Liyin Tang, Jason Zhang 29
- 30. Case Study: Streaming DB Export TXN 1 Commit_TS: 101 … Binlog Order TXN 2 Commit_TS: 103 TXN 3 Commit_TS: 102 TXN N Commit_TS: N’ Point-in-Time Restore on TS 102 Jingwei Lu, Liyin Tang, Jason Zhang 30
- 31. Case Study: Streaming DB Export Rows CF: Colums Version Value <ShardKey><DB_TABLE_#1><PK_a=A> id bin100 101 <ShardKey><DB_TABLE_#1><PK_a=A> city bin101 San Francisco <ShardKey><DB_TABLE_#1><PK_a=A> city bin102 New York <ShardKey><DB_TABLE_#2><PK_a=A’> id bin100 1 Jingwei Lu, Liyin Tang, Jason Zhang 31
- 32. Case Study: Streaming DB Export Rows Version (Logical Offset) Value <ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100 <ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101 <ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103 <ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102 Jingwei Lu, Liyin Tang, Jason Zhang 32
- 33. Case Study: Streaming DB Export Rows Version (Logical Offset) Value <ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100 <ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101 <ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103 <ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102 Jingwei Lu, Liyin Tang, Jason Zhang 33
- 34. Summary Jingwei Lu, Liyin Tang, Jason Zhang Scalable and Reliable Rich Stateful Computation Rich Integration with Hadoop EcoSystem Easy Operation
- 35. 35
Editor's Notes
- *Disaster recovery *High Slow SLA job isolation
- Slide why Stateful process vs stateless
- Use diagram to show operators
- Realtime ingestion provides fast feedback loop. Advanced monitoring infrastructure Tracking changes instead of full snapshot for RDS dump
- What is the goal of realtime ingestion: *fast feedback loop for experiment to reduce testing cycle *provide realtime view of production database for many offline workload(for example, machine learning)
- Table mapping provide a unified view to access realtime ingested data.
- For snapshot using scan it takes 10-30 minutes per table. This does not scale. Take 10 minutes to do the link and restore. All tables can be accessed afterward.
- Backup based db export restore takes 9 - 12 hours and it is subject to AWS network situation. Long latency and fragile. We just need to track changes and apply to snapshot. Provide near realtime snapshot of db. Unify across mysql and dynamodb