Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 27

Building the Next-gen Digital Meter Platform for Fluvius

1

Share

Download to read offline

Fluvius is the network operator for electricity and gas in Flanders, Belgium. Their goal is to modernize the way people look at energy consumption using a digital meter that captures consumption and injection data from any electrical installation in Flanders ranging from households to large companies. After full roll-out there will be roughly 7 million digital meters active in Flanders collecting up to terabytes of data per day. Combine this with regulation that Fluvius has to maintain a record of these reading for at least 3 years, we are talking petabyte scale. delaware BeLux was assigned by Fluvius to setup a modern data platform and did so on Azure using Databricks as the core component to collect, store, process and serve these volumes of data to every single consumer and beyond in Flanders. This enables the Belgian energy market to innovate and move forward. Maarten took up the role as project manager and solution architect.

More Related Content

You Might Also Like

Building the Next-gen Digital Meter Platform for Fluvius

  1. 1. Building the next-gen digital meter platform for Fluvius Maarten Herthoge Team Lead Data Science & Engineering, delaware BeLux
  2. 2. 64% 29% 17% got measurable results is successful reaches production
  3. 3. It’s easy to understand why so many projects fail Deep Learning ETL vs ELT PaaS vs IaaS Data Visualisation Data Quality Master Data Big Data Databricks SMP vs MPP Data Catalog Spark Storm Data Mart No-SQL Cloud vs On-prem Velocity, Variety and VolumeSemantic Layer Predictive Prescriptive IoT Streaming AI
  4. 4. Start with the problem, not with the technology
  5. 5. The core data platform needs to support various use cases Portal access to 3 years of history Interfaces to allow scalable sharing of data
  6. 6. Digital meters collect data… a lot of data 7+ million IoT devices after full roll-out7M+ LEGAL 12TB Sending up to 12 TB of data every day GDPR and strict government regulations
  7. 7. The “golden hammer” does not exist, use a portfolio approach
  8. 8. Think about the types of use cases, processing flexibility and enablement you require Data consumers LOB CRM ERP #1 Increasing volume
  9. 9. Think about the types of use cases, processing flexibility and enablement you require Data consumers LOB CRM ERP #2 Need to collect and combine any data
  10. 10. Think about the types of use cases, processing flexibility and enablement you require Data consumers LOB CRM ERP #3 From data consumer to data explorer
  11. 11. Think about the types of use cases, processing flexibility and enablement you require Data consumers LOB CRM ERP #4 Real-time becomes the standard
  12. 12. Think about the types of use cases, processing flexibility and enablement you require Data consumers LOB CRM ERP #1 Increasing volume #3 From data consumer to data explorer #2 Need to collect and combine any data #4 Real-time becomes the standard
  13. 13. Prioritize software-as-a-service to create a data portfolio Managing all data Social LOB Graph IoT Image CRM INGEST STORE PROCESS SERVE Analyzing all data Integrating all data From GB’s to TB’s Seamless scaling 1000+ API requests/sec 7M+ devices
  14. 14. Avoid the data lake pitfall
  15. 15. Data lakes often presented as the go-to solution in these cases Massive scale Optimized performance Integration flexibility Cost effectiveness
  16. 16. But data lake implementations fail in 60% of the cases Massive scale Optimized performance Integration flexibility Cost effectiveness Becomes data swamp Capabilities mismatch Schema-on-read Misused as “strategy”
  17. 17. Focus on collaboration
  18. 18. Azure Databricks Aim for one unified analytics processing engine Cloud storage Data warehouses Big data storage IoT / streaming data ML models BI tools Data exports Data warehouses Collaborative Workspace DATA ENGINEER DATA SCIENTIST BUSINESS ANALYST Deploy and manage production jobs & workflows ETL / ELT JOB SCHEDULER MONITORING DEVOPS Single Runtime Foundation DATA LAKE MANAGEMENT SERVERLESS / SaaS SCALE WITHOUT LIMITS PAY-AND-SCALE- AS-YOU-GO
  19. 19. Narrow the scope and decouple storage from serve INGEST STORE PROCESS SERVE Why is this successful? Unified, consistent data ingest path Cost efficient for full detail Why is this successful? Allows for workload-specific platforms tailored at specific cases independent from the raw datastore SQL DB Service Fabric Data Lake Gen2 & Delta
  20. 20. Databricks acts as glue between a data lake and various workload specific application platforms INGEST STORE PROCESS SERVE Databricks Why is this successful? Unified runtime and collaboration One-click setup Native integration with Azure services Enterprise security & SLA SQL DB Service Fabric Data Lake Gen2 & Delta
  21. 21. Fluvius’ data portfolio drives innovation on the Flemish energy market Bringing consumption insights and awareness to millions of households Lower barrier for data- and market-driven change Portfolio projects TCO 20x lower compared to “golden hammer”
  22. 22. Focus on collaboration Avoid the data lake pitfall The “golden hammer” does not exist, use a portfolio approach Start with the problem, not with the technology Interested in more? Visit delaware.ai
  23. 23. Focus on collaboration Avoid the data lake pitfall The “golden hammer” does not exist, use a portfolio approach Start with the problem, not with the technology Thank you! Maarten Herthoge TL DS&E, delaware BeLux
  24. 24. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.

×