Meter Data Management Presentation

The Hidden Challenge in Meter Data
Management: What You Don't Know Will Hurt
You!

An InformationWeek Webcast
Sponsored by

Today’s Presenters

Jill Feblowitz,
Vice President, Utilities and Oil and Gas,
IDC Energy Insights

Kevin Brown,
Chief Architect,
IBM Informix Database

Dealing with
Smart Grid Data
Jill Feblowitz, Vice President,
IDC Energy Insights

Agenda

Smart Meter Outlook
Management Challenges
IT Challenges
The Way Forward
Recommendations

© IDC Energy Insights Page 5

It’s All About More Interaction


Smart Meter Outlook
A Decade of Global Growth in Smart
Metering

2015 AP and LA will
drive the global smart
2009 North meter marketplace
America overtakes
Europe in Smart
2005 North Meter shipments
2002 First AMI American utilities
deployments begin in begin large-scale
North America AMI deployments

2001 Enel
begins AMI
deployment


Smart Meter Outlook
Data Deluge

 85 million installed meters by 2015
according to the IDC Energy Insights
Smart Meter Tracker
 Each meter has the potential of producing
a vast amount of interval data: 12
consumption events for a year vs. 8760
billing events with a 1 hour interval
 Not to mention other on-going
communications and transactions


Smart Grid Outlook
IT Spending on Smart Grid, 2011
Hardware Services

Software
It is not just about
smart meters at the
home – it‟s about
making the grid
smarter, too.
 Smart meters at the
transformer
 Intelligence
electronic devices.


Change in Business Process… Meter
to Cash Before Smart Meters
60 to 90 days

Data Collection Bill Calc Fulfillment Payment &
Collections
Customer
Relationship
Management (CRM)

Customer Bill Print EBPP
information System
(CIS)
Meter Lock Box
(residential)
Bill Mail

Credit and
Complex Billing Collections
Electronic Bill
Advanced
Meter
Payment and Kiosk, Local Office,
(C&I) Meter data Presentment Mail
management (EBPP)

Meter Reading Accounting Accounting Accounting

Customer Service


Management Challenges (and Opportunities)
Change in Business Process… Meter to Cash
Future Customer Sees
The Data
7 to 90 days

Data Collection Data Transport Bill Calc Fulfillment Payment &
and Assessment Collections

CRM
Smart Phone Smart Phone

Web Portal Web Portal

In-home EBPP
Smart Display
Meter Advanced
Metering CIS Lock Box
Infrastructure EBPP

Bill Print Credit and
Bill Mail
Collections
Advanced
Meter Kiosk, Local Office,
(C&I) Mail
MDM

Network Operating Accounting Accounting Accounting
Center - Data
. Analysis Customer
Service

The business case must be supported.
Type of Benefit Source of Benefit
Revenue Assurance • Remote connect/disconnect
• Theft and tamper detection
• Analysis of billing data to detect unbilled accounts
• Pre-payment
Operational • Reduced truck rolls for connect/disconnect, high bill
Efficiency complaints
• More efficient deployment of workforce in outage
based on data
Deferred Capital • Support for demand response in capacity constrained
Investment areas
• Prioritization of equipment replacement
Increased Reliability • Predictive analytics applied to condition-based
monitoring
• Automated switching routines to minimize outage
impact


There Much More Left to Do with the
Meter Data
n=26

Source: IDC Energy Insights, Utility CIO Survey, 2010
n=26


Supporting New Pricing, Services
 Expanded offerings: energy  Still to come…..fast charge
efficiency, demand response, rates?
green energy rate
 New Pricing: Time-based,
critical peak pricing
 New Relationships: “Prosumer”
and net metering for PV
 Tools to Build Relationship and
Awareness: carbon footprint,
energy efficiency and savings
 Pre-pay and budget notification


IT Challenges
Data explosion calls for data storage and
handling and much more.

Managing Meter Data Managing T&D Grid Data
Achieving acceptable levels of production for Determining the balance between centralized or
billing and customer presentment distributed (device) computing
Optimizing performance and utilization of storage Optimizing performance and utilization of storage
specific to the workload specific to the workload
Making the right data available for production Making the right data available for operations
(billing and customer presentment) and analytics and analytics
Retaining and archiving billing data to meet Securing the smart grid telecommunications
regulatory requirements and satisfy business network from incursion by hackers
continuity
Managing data, alerting about data irregularities, Managing data, alerting about data irregularities,
and resolving inconsistencies and resolving inconsistencies

Protecting privacy of customer data Integrating old and new communication
infrastructure to support secure data
communications
Establishing consistent data synchronization, data Establishing consistent data synchronization, data
models, and protocols models, and protocols

Securing the AMI network from incursion by hackers Minimizing network traffic with high device data production
given new devices that produce high data volumes


IT Challenges
Examples of Volumes in our Study
Number of Intervals Frequency of Data Processed Rention in Active
Meters Currently Data Collection on a Daily Basis Database
Deployed (per day) (gigabytes) (years)
(million)

Utility A 1.32 15 minute 3 times 4.752 1.5
Utility B
1.40 Hourly 2 times 12 3.0
Utility C
0.70 Hourly 1 time 4 2.0
Utility D
2.00 15 minute 6 times 70 1.1

“Utilities are not accustomed to managing or processing this much
data, I mean there‟s been no reason to…Probably the largest amount of
data they work with has been in the billing world and maybe some GIS,
but this volume is much larger, especially on a daily basis, so it really
pushes on how well you architect something and you‟re using the right
tools in your systems and your databases are properly architected."


IT Challenges
Details, details, details…examples
 Time to process and speed of processing
– Re-interrogating the meters to get missing reads and avoid duplication
– “ If a system goes down or the network is down and you don’t get it resolved quickly,
then when you start piling up two days’ worth of data, then it becomes a challenge
because it takes longer.”
– Service levels mandated by PUCs or service level agreements between IT and the
business
 8:00 AM presentment of previous day’s usage
 Need to respond within four hours if system goes down
 Target of first time read success and presentation at SLA of 99.5%
– “Where the bigger challenge is going to be is managing flows on the system if
everybody wants to do load control simultaneously, that’s going to be a little more
challenging to us and it’s going to be interesting to see how those challenges work
out.”
– “We’ve had everything on the MDM from a bad index to a bad spot on the disk space
to we needed more horsepower.”
 Ease of access
– “If you wanted to see a month’s worth of data on the meter, you’d have to go through
30 different files. It’s not very user friendly.”
– “We have to go through essentially 60 millions rows of data per day.”


Looking Forward
Approaches
 Operational data stores
– To bring in other data
– To off load data so there is no impact on “production” of the MDM
 Changed business process
 Developed retention and archiving protocols
 “Changed how EDI moves data from one platform to another”
 “Changed database schemas on mainframe”
 Re-partitioning – “low hanging fruit”
 Adding servers and storage. Cluster servers.
 Considering high end compression techniques
“[We have been working with meter data for about 3 years, and in that time, we
have] “changed out systems two or three times.”
“We‟ve been pretty aggressive in providing a quality system to this market, so we
have already taken the steps to get what we think is the right level of hardware and
the software products…We‟ve spent some money, unfortunately.”


Looking Forward
Managing the Information: Wish
List
 Shorten processing time to meet
regulatory and internal SLAs
 Build analytics into the database
for faster querying
 Reduce hardware and software
costs related to server, storage
and RDBMS.
 “Loss less” storage is useful in
areas where time series data must
keep its fidelity, such as predictive
maintenance


Recommendations
 Develop a data retention policy and investigate what needs to
be kept online and for how long. Do not forget to include
customer needs for presentment. Start early to evaluate how
others in your organization (load research, capital planning,
etc.) will access data.
 Ask vendors for a “proof of concept”. Most vendors are willing
to help by running test data sets using their technology.
 Start by understand what current and future requirements for
processing speed, storage and data access will be for your
company and how these demands will ramp up over time.
 Do your due diligence. Based on scenarios, investigate your
options for processing and storage and the total cost of
ownership associated with these. Do not assume that by
adding more servers and storage is the most cost effective
approach.

Powering Large Volumes of Meter
Data with Informix
Kevin Brown, IBM Informix Chief Architect
kbrown3@us.ibm.com

Information Management

© 2011 IBM Corporation


Changing Storage Requirements
Changing Workloads For 10 Million Smart Meters:
Today – Each meter read once per month
Very soon – Each meter read once every 15 minutes
Regulations – Need to keep data on line for 3 years (PUC) and, perhaps, save for 7 years

Smart Meter Interval Data 350.4B

# of Records
Per Year for
10M meters

3.65B
120M

Frequency
Monthly Meter Daily Meter 15 Minute Meter of reads
Reads Reads Reads
22 © 2011 IBM Corporation


Keeping up with Smart Meter Data

Large amounts of data causes problems in 2 areas:
1) Storage management
• Large storage = Expensive
• Cumbersome to maintain
– Requires sophisticated partitioning schemes
– Reorganization often required, which leads to downtime

2) Query performance
• Compliance Reports must be completed before the end of each day
• Customer portal queries must be handled in a timely manner
• Customer billing must be completed each day



Informix TimeSeries: The Solution for Managing Meter Data

 Time Series:
– A logically connected set of records ordered by time

 Performance
– Extremely fast data access
• Up to 60 times faster than competition
– Handles operations hard or impossible to do in standard SQL

 Space Savings
– Saves at least 50% over standard relational layout

 Toolkit approach
– Develop algorithms that run directly in the database

 Easier
– Extremely low maintenance



Typical Relational Schema for Smart Meters Data
Index Smart_Meters
Table
Meter_id Time KWH Voltage ColN
1 1-1-11 12:00 Value 1 Value 2 …….. Value N
Table Grows



… … … … …….. …




… … … … …….. …

• Each row contains exactly one record = billions of rows in the table
• Additional indexes are required for efficient lookups
• Data is appended to the end of the table as it arrives
• Meter ID’s stored in every record
25
• No concept of a missing row
© 2011 IBM Corporation


Same Table using an Informix TimeSeries Schema
(logical view)
Smart_Meters
Table
Meter_id Series
1 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]




… …

Table grows

• Each row contains a growing set of records = one row per meter
• Data append to end of a row rather than to the end of the table
• Meter IDs stored once rather than with every record
• Data is clustered by meter id and sorted by time on disk
• Missing values take no disk space, missing interval reads take 2 bytes


Virtual Table Interface makes Time Series data appear
Relational
TimeSeries Table TimeSeries Virtual Table
Smart_meter SM_vt
mtr_id Series
(int) timeseries(mtr_data) mtr_id date col_1 col_2
1 [(Mon, v1, ...)(Tue,v1…)] 1 Mon Value 1 Value 2 ...
[(Mon, v1, ...)(Tue,v1…)] 1 Tue Value 1 Value 2 ...
2
1 Wed Value 1 Value 2 ...
3 [(Mon, v1, ...)(Tue,v1…)]
... ... ... ... ...
4 [(Mon, v1, ...)(Tue,v1…)]
3 Mon Value 1 Value 2 ...
5 [(Mon, v1, ...)(Tue,v1…)]
3 Tue Value 1 Value 2 ...
6 [(Mon, v1, ...)(Tue,v1…)] 3 Wed Value 1 Value 2 ...
... ... ... ... ...
7 [(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]
Execute procedure tscreatevirtualtable
8 („SM_vt‟, „Smart_meter‟);


100 Million Smart Meter Benchmark: Goals

1. Measure processing times for data collected over a 31 day period for:

 100 million meters at 30-minute intervals in an 8 hour day.

2. Demonstrate consistent processing times over the 31 day period.

3. Demonstrate linear storage growth of data stored over the 31 day period.

4. Complete one day’s billing cycle while simultaneously processing and loading meter data
for 100M within an 8 hour time period.

5. Demonstrate all processing can be done using a low-cost combination of commercially
available hardware, storage, and software.



100 Million Smart Meter Benchmark: Operations

 Operations performed each day
– Load interval and register data for 100M meters at 30 minute intervals
(49 records/day/meter)
• 49 records/day/meter * 100M = 4.9 Billion records/day
– Perform VEE on the data (validation/estimation/editing)
– Run a daily billing cycle on 6% of the meters
– Gather results for 31 days

Gather Results
Processing

Run daily billing cycle

Perform Validation, Estimation & Editing

Load & Register Data (100 million meters)

Processed over 30 minute intervals over 31 Days



100 Million Smart Meter Benchmark: Components

AMT-Sybex Affinity
Meterflow
Software Stack
Informix v11.70.xC3 with TimeSeries version 5.0
Monitor & Admin AMT-Sybex Affinity Meterflow Meter
AIX v7.1
Informix 11.70

AIX v7.1
Hardware Stack
IBM System P Series & IBM Power P750 XIV Storage System
Storage 32 cores (3.5 GHz) – 16 active 15 X 2TB storage
500 GB RAM 6 X 6 FC connections @ 4GB
1 GB LAN Fiber dual port adapter - 1 active
2 X 8GB FC dual port adapter - 4 active ports of
storage
Upstream Management

Storage Area Network (SAN)
Systems

Knowledge Applications

MDM, DMS, Data 4 X 8GB
NMS Management 6 X 4GB



100 Million Smart Meter Benchmark: Load Results
 An “end to end” run of 100 Million Meters at 30 minute intervals was performed for 31 days of data
 The result: all data was prepared, loaded, validated as well as a billing cycle run in less than 8 hours
 The average time to do these operations remained consistent over the 31 days
– Performance remained constant even as storage increase
 The billing cycle completed in less than 5 hours and ran concurrently

Process Avg. Elapsed Time Avg. Throughput Rate
Preparation and Technical 2 hrs 10 min 628,205 records/sec
Verification
Data Load 3 hrs 14 min 420,962 records/sec
Validation, Estimation, and Editing 2 hrs 11 min 623,409 records/sec
(VEE)

Total Time: 7 hours and 35 minutes!


Load Performance: Storage Comparison over Time

Load Time over 31 Days Total Process Time over 31 Days
100 Million Meters @ 30 minute intervals 100 Million Meters @ 30 minute intervals
Total Time - Minutes

Total Time - Minutes
No. of Days

Storage in TB over 31 Days
100 Million Meters @ 30 minute intervals
Disk Space in TB

No. of Days No. of Days



Summary: Informix is the Answer for Smart Meter Data

1) The enormous volume of smart meter data can be
overwhelming

2) Most smart meter data is time series data

3) Not all database servers are equal, choose one that
handles relational and time series queries equally well

For more information on our 100M Smart
Meter Benchmark: http://bit.ly/pfu2RW



(Kevin Brown – kbrown3@us.ibm.com)
(Jill Feblowitz - jfeblowitz@idc.com)


Meter Data Management Presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Meter Data Management Presentation

Similar to Meter Data Management Presentation (20)

Meter Data Management Presentation