While research assessing behavior-based energy interventions shows great promise, results vary widely and much is still unknown about the specific variables that impact program effectiveness. As utilities and regulatory agencies focus more attention on behavior-based energy interventions, it becomes critical to ensure that evaluations of such programs are rigorous and accurate. While the metric used to measure whether these various programs work (kWh) is fairly standard and easy to compare between studies, the metrics used to measure how and for whom they work have been left to individual researchers and evaluators. Standardization of assessment methods is common in related fields such as education and psychology, but has yet to take hold in energy program evaluation. This paper argues for a more systematic and comprehensive approach to the evaluation of behavior-based energy interventions, and describes a preliminary toolkit that is currently being developed and validated in conjunction with the International Energy Agency Demand Side Management Programme (IEA-DSM) Task 24 on Behavior Change as well as two large investor-owned utilities. Our approach is informed by theories and empirical research on behavior change as well as a content analysis of 85 behavior-based energy interventions. It includes questions on: context (demographics), user experience (ease of use, engagement), material culture (what people have), energy practices (what people do), and beliefs around energy use (what people think). Sample items for each construct and suggestions for implementation are presented. Broad use of such an instrument can improve and aggregate our overall knowledge across the countless additional studies expected to be conducted in the coming years.

1. Exploring Deep Savings:
A Toolkit for Assessing Behavior-Based Energy
Interventions
Beth Karlin
Transformational Media Lab
University of California, Irvine
Papers listed here. Contact me for details: bkarlin@uci.edu.

2. Is Feedback Effective?
 100+ studies conducted since 1976
 Total n = 256,536 (mean 119/study)
 Mean r-effect size = .1174 (p < .001)
 Average energy savings: 9%
Significant variability in effects
(from negative effects to over 20% savings)
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

3. Is Feedback Effective?
It depends…
Moderators identified in meta-analysis
• Study population (WHO?)
• Study duration (HOW LONG?)
• Frequency of feedback (HOW OFTEN?)
• Feedback medium (WHAT TYPE?)
• Disaggregation (WHAT LEVEL?)
• Comparison (WHAT MESSAGE?)
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

4. Methodological Limitations
1. Not naturalistic
 Participants generally recruited to participate
 May be different from “active adopters”
2. Not comparative
 Most studies tests one type of feedback (vs. control)
 Very few studies isolating or combining variables
3. Not testing mediation
 DV is energy use, but studies rarely test possible
mediators to explain effectiveness
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

5. Methodological Limitations
1. Not naturalistic
 Participants generally recruited to participate
 May be different from “active adopters”
2. Not comparative
 Most studies tests one type of feedback (vs. control)
 Very few studies isolating or combining variables
3. Not testing mediation
 DV is energy use, but studies rarely test possible
mediators to explain effectiveness
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

6. Will they come?
If you build it,

7. Program x Outcome
y
Does it work?
Does program x lead to outcome y?

8. Program x Outcome
y
Questions remain…
What is the
program?
What is going on
here?
How do we measure
outcomes?
Does program x lead to outcome y?

9. Program x Outcome
y
A theoretical approach
Hypothesis /
Theory
Clearly defined and
operationalized
Metrics tested for
reliability & validity
Does program x lead to outcome y?
How and for whom does program x lead to outcome y?

10. How and For Whom?

11. How and for Whom?

12. Toolkits in other fields

13. Toolkits in other fields

14. Question Bias

15. Question Bias

16. Question Bias

17. Psychometrics
• Theory and technique of measurement:
knowledge, abilities, attitudes, traits
• Construction and validation of instruments:
questionnaires, tests, assessments

18. Psychometrics
Psychometric Properties
1.Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity

19. Our Project
Initial
concept
paper
IEA DSM Task 24
Methods
review
Report for IEA
Karlin et al., 2015
Toolkit
Pilot
testing
Member
country
testing
Consultation
Feedback
SCE
Psychometric
testing
2013 201620152014
PG&E
Field testing
(scheduled)

20. Methodological Review
Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We
Know? A Review of Behaviour-Based Energy Efficiency Data Collection.
IEA-DSM Task 24 Subtask 3 Report.

21. Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We
Know? A Review of Behaviour-Based Energy Efficiency Data Collection.
IEA-DSM Task 24 Subtask 3 Report.
Data Collection Methods Used

22. Instruments Provided?
Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We
Know? A Review of Behaviour-Based Energy Efficiency Data Collection.
IEA-DSM Task 24 Subtask 3 Report.

23. Toolkit Development

24. Energy Cultures Frameworks
Material
culture
Energy
practices
Norms
and
aspiration
s
Have
DoThink

25. Toolkit Development
Heating devices Energy sources
Insulation House structure
Have

26. Toolkit Development
Social
expectations and
Environmental
concern
Expected
warmth levels
Maintaining
traditions
Think

27. Toolkit Development
Turning on
heater
Putting on jersey
Maintaining heating
technologies
Drawing
curtainsDo

28. Toolkit Development
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Intervention
Natural changes
User experience
Technology interaction
ControlGroupTreatmentGroup

29. Toolkit Development
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Intervention
Natural changes
User experience
Technology interaction
ControlGroupTreatmentGroup

30. Next Steps
1. Psychometric testing (SCE)
2. Local field testing (PG&E / SCE)
3. Member country review
4. Global field testing
5. Wide scale adoption? 

31. Beth Karlin
Transformational Media Lab
University of California, Irvine
bkarlin@uci.edu
Thank you!
(comments and suggestions welcome)