InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxDB IOx

1. Paul Dix
InfluxData – CTO & co-founder
paul@influxdata.com
@pauldix
Replication, durability and
subscriptions in InfluxDB IOx

2. InfluxDB IOx components
• Write Buffer
• Write segment persistence
• Subscriptions
• In-memory database
• Parquet persistence
• Catalog
• Configuration, partition summaries

4. Routing Layer
• Parse LP
• Split into shards
• Split into partitions
• Write one Entry per
shard

5. Possible Destinations
• Single server or
multiple servers
• Write Buffer
• In-memory DB

6. The Write Buffer
• Subscriptions
• Persistence
• Replication

7. The Write Buffer
• Router to Buffer

8. The Write Buffer
• Buffer assigns
clock value &
writer id
(unique)
• Makes request
to peers
• Note the
request from
Router to Buffer
still open

9. The Write Buffer
• Buffer 2 takes
the write,
updates its
clock to max
• Buffer 2
responds with
its clock value
• Buffer 1
updates its
clock to the
max of the two

10. The Write Buffer
• Buffer 1 sends
success
response to
routing layer
• Routing layer
responds to the
client (after all
shards succeed)

11. The Write Buffer
• Continues to
retry

12. The Write Buffer
• Consistency
checks

13. The Write Buffer
• Ask peers for
anything
missing
• Logical clock +
entry id
• High water mark

14. The Write Buffer
• Persistence to
object storage
• Entries in
segments
• Time or size

15. Subscriptions
• Everything
• Table (or regex)
• Predicate
• Push or Pull
• Or poll object
storage for
segments

16. Subscriptions
• Scale query tier

17. Catalog & Parquet
• Persist in larger
blocks

19. Thank You