Flink is used to build real time machine learning pipelines at Alibaba. I will go over the architecture challenges in building such a system. Hologres is proposed as a solution to such challenges. I will explain how we build Hologres and its applications.

1. Data Warehouse, Data Lakes
What’s Next?
Xiaowei Jiang
Apr 23rd, 2020

2. About Me
Stratify Inc.
2000 - 2002
Member of Technical Staff
2002
Microsoft
2002 - 2010
Principal Software Engineer
Facebook
2010 - 2014
Software Engineer
Alibaba Group
2014 - Now
Senior Director
2010 2014 Now

3. Use Case: Realtime Machine Learning
Kafka
Search &
Recommendation
Preprocessing
Feature Updates
Model Updates
Online Training
Log Data
Flink
HBase

4. More and more Complex Architecture
Kafka
Flink HBase Cassandra
Realtime
Ingestion
Dim Table Join
Subscription

5. Dashboards
Point Lookups
More and more Complex Architecture
Kafka
Flink HBase Cassandra
Realtime
Ingestion
Dim Table Join
Subscription

6. Dashboards
Point Lookups
More and more Complex Architecture
Kafka
Flink HBase Cassandra
Realtime
Ingestion
Dim Table Join
Subscription
Realtime Analytics
Realtim
e
Ingestion
ClickHouse Druid
Reports

7. Dashboards
Point Lookups
More and more Complex Architecture
Kafka
Flink HBase Cassandra
Realtime
Ingestion
Dim Table Join
Subscription
Hive
Archiving
Batch Processing
Realtime Analytics
Realtim
e
Ingestion
ClickHouse Druid
Reports

8. Presto
Drill
Federated Analytics
Batch Acceleration
Dashboards
Point Lookups
More and more Complex Architecture
Kafka
Flink HBase Cassandra
Realtime
Ingestion
Dim Table Join
Subscription
Hive
Archiving
Batch Processing
Realtime Analytics
Realtim
e
Ingestion
ClickHouse Druid
Reports

9. Presto
Drill
Federated Analytics
Batch Acceleration
Dashboards
Point Lookups
More and more Complex Architecture
Kafka
Flink HBase Cassandra
Realtime
Ingestion
Dim Table Join
Subscription
Redis
Mysql
Result
Caching
API
Applications
Hive
Archiving
Batch Processing
Realtime Analytics
Realtim
e
Ingestion
ClickHouse Druid
Reports

10. Lambda Architecture
Data
Collection
Batch Processing Batch View
Dashboards
Reports
Applications
API
Batch Layer Serving Layer
Speed Layer
Query
Stream
Processing
Realtime Data

11. Greenplum
Cassandra
HBase
ClickHouse
Druid
Kudu
Kylin
Data Islands
Data Islands
High Hardware Cost
Data Duplication
High Maitainance
Data Consistency
Steep Learning Curve
Multiple Systems
Hive

12. Simplified Architecture
Unified
Storage
Data
Collection
HDFS
Flink
Kafka
Dashboards
Reports
Online
Applications
Batch
Realtime

13. Data Lakes
HDFS
Alibaba OSS
AWS S3
Storage
Data Lake
Data Lake
Incremental
ETL Tables
Tables
Raw Ingest
Tables
Flink
Spark
Query
Hive

14. HSAP
Unified
Data Storage
Unified
Data Service
Hybrid Serving/Analytical Processing
Data Access Patterns
Batch Analytical Serving Transaction
HSAP

15. Simplified Architecture with HSAP
HSAP
Data
Collection
HDFS
Flink
Kafka
Dashboards
Reports
Online
Applications
Batch
Realtime
Unified Data
Storage & Service

16. Extensions
PostgreSQL Ecosystem
How we build it: Ecosystem
DocumentationTools

17. Holographic PostgresHologres = +

18. Hologres Architecture
Pangu/HDFS/OSS/S3
Blackhole
Hologres FE Hologres FE Hologres FE
LBS
Blackhole Blackhole Blackhole Blackhole
Hologres

19. Storage/Compute
Disaggregation Interconnection Network
Node1
MEM
Node2
MEM
Node3
MEM
Hologres: Cloud Native
Hologres

20. Asynchronous No Locking
Memory
Management
Hologres Storage
Hologres: Storage

21. Vectorization Optimized for Storage
Hologres Compute
Custom Scheduler
High Concurrency
Hologres: Compute

22. Use Case: Realtime Machine Learning Revisited
Kafka
Preprocessing
Feature Updates
Model Updates
Online Training
Log Data
Flink
Offline Training
Dashboards
Reports
Applications
Point Lookups
Result Caching
Realtime Analytics
Batch Acceleration
Federated Analytics
Search &
Recommendation
Hologres

23. Flink + Hologres
Realtime without Compromises
Take Away
Fresh Data,
Performance &
Scalability
Boost Business &
Reduce TCO
Simplified Architecture
Twitter: @HologreSQL
Discussions: hologres@googlegroup.com
Slack: https://bit.ly/2zf9EKn

24. THANKS