Recommended
Recommended
More Related Content
Similar to MapMapReduce: Managing Big Geospatial Data
Similar to MapMapReduce: Managing Big Geospatial Data (20)
Recently uploaded
Recently uploaded (20)
MapMapReduce: Managing Big Geospatial Data
- 2. Whyarewehere? Stable data model for the customer Physical format
- 3. Thebiggerpicture Write-optimized data store Manual map edits Automatic map edits Read-optimized data store Physical output Cascades & quality checks ~ 5.000 changes / second ~ 1.5 billion changes / month
- 8. StoringfeaturesintoHBase feature table row key feature id family qualifier value ent type geometry entity geometry type type attributes associations att assoc
- 10. StoringfeaturesintoHBase 0 0,00 1 1 0,1 1,0 1,1 2 3 2 3 2,3 2,1 3,0 3,1 0,2 0,3 1,2 1,3 2,2 2,3 3,2 3,3 dec
- 11. StoringfeaturesintoHBase 00 000000 01 01 0001 0010 0011 10 11 10 11 1000 1001 1010 1011 0100 0101 0110 0111 1100 1101 1110 1111 bin
- 12. StoringfeaturesintoHBase 0 00 1 1 1 2 3 2 3 2 3 8 9 10 11 4 5 6 7 12 13 14 15 dec
- 13. StoringfeaturesintoHBase 0 000 1 1 01 02 03 2 3 2 3 20 21 22 23 10 11 12 13 30 31 32 33 base4
- 15. StoringfeaturesintoHBase feature table row key morton (base4) + feature id + feature type family qualifier value ent type geometry entity geometry type type attributes associations att assoc
- 20. Creatingaproductfromgeo-spatialdata Read-optimized data store Physical output Stable data model for the customer Physical format like shapefile or Avro-files to load into an RDBMS Fast-moving object model
- 21. Creatingaproduct:technologystack Current focus: batch, but we also target an incremental approach. We know it, we use it, we trust it. Popular, well-supported by Beam and Amazon EMR.
- 22. Write-optimized data store Manual map edits Automatic map edits Read-optimized data store Physical Output Cascades & quality checks ~ 5.000 changes / second ~ 1.5 billion changes / month Thecompletepicture
- 23. THANKYOU
Editor's Notes
- So how do we handle that multitude of concurrent writes? PostgreSQL cluster for transactional integrity as well However, we also need to read lots of features to create our output. How do we do this?
- So we can store the data in a read-optimized way. How can we read from it in a performant way?
- So we can store the data in a read-optimized way. How can we read from it in a performant way?
- So we can store the data in a read-optimized way. How can we read from it in a performant way?
- We are now capable of reading efficiently from HBase. So how do we transform that data?