Ocial statistics are published by government agencies and other international institutes to provide infor- mation on the economy, living conditions, social development etc. These metrics are evaluated using dierent sources, primarily surveys and censuses and, in addition, data obtained from government administrations or private sector information. Several qualitative criteria are considered the basis for trustworthy ocial statistics, such as impartiality, transparency, relevance and independency. However, as the published metrics are derived from statistical analysis of imperfect and potentially incomplete data, errors and uncertainties are inevitable and, in some cases, require revisions or corrections that can lead to reduced condence in the overall process. It is also important to recognize that the uncertainties originate both from statistical errors, such as the use of limited raw data, and from bias induced by incomplete information or modeling assumptions. Understanding how dierent sources of errors lead to bias and variance helps us improve the overall process resulting in more accurate predictions. Quantitative measures of the variance errors are commonplace and easy to convey; among those the standard-error-in the mean is perhaps the most popular and results in the  symbol. Measures of the spread in the actual data are also easy to estimate and disseminated using the variance, or more frequently the standard deviation. More complete representation of the statistical spread in the row data leads to percentiles and, eventually, to reporting the complete probability distributions. Measures of bias, on the other hand, are not well developed because in many cases are not directly computable. In the engineering community rather than presenting the variance and bias error, the focus is to identify and rank the sources of uncertainties that explain the imprecision in the estimates. In this work we will discuss applications of two global sensitivity metrics, the Sobol indices and the active subspace variables as tools to describe the variance errors. Furthermore, we will discuss the distance metric as a strategy to assess bias errors derived from classical measures of discrepancies between probability distribution functions.

1. REPORTING
UNCERTAINTY:
TOO MUCH INFORMATION?
GIANLUCA IACCARINO
ARI FRANKEL
DAVID SHARP
DARIO BUONO
61st World Statistics Congress

2. THERE ARE MORE FAKE FLAMINGOS
THAT REAL FLAMINGOS

3. O. J. SIMPSON: NOT GUILTY!

4. MORE THAN 20% OF AMERICANS
(45+) TAKE CHOLESTEROL
LOWERING DRUGS

5. • Represent and summarize statistical
information obtained from a data collection/
analysis process
• The message is clear and accessible in non-
technical terms
WHAT IS SIMILAR ABOUT THOSE
STATEMENTS?

6. WHAT IS DIFFERENT ABOUT THOSE
STATEMENTS?
• The context and the consequence!

7. WHAT IS NOT EXPLICIT ABOUT
THOSE STATEMENTS?
• The characteristics (quality & quantity) of the
data used, i.e. the uncertainty
• The weight of the evidence and the relevance
to the context

8. WHY IS THERE UNCERTAINTY?
• Statistical statements are not absolute truth
• Uncertainty originates from
• Limited data/sample size
• Unaccounted bias, correlation
• Lack of knowledge
• Data analysis errors
• ….

9. CAN WE EVALUATE & REPORT THE
UNCERTAINTY?
Yes, but…
• It might be cumbersome or even confusing
• Might erode confidence in the producers of
official statistics
• By itself uncertainty does not provide
sufficient information – how much uncertainty
can be tolerated depends on the context!

10. OUR GOAL
• Introduce a framework to summarize data,
while reporting both context and uncertainty

11. THE INSPIRATION
• Engineering Safety
• Reliability Index
• Legal Proceedings – Court of Law
• Burden of proof
• Drug Approvals – Medical Trails
• Transparency on outcomes
Decision-making under uncertainty

12. QUESTIONS CALL FOR DATA; DATA
LEADS TO QUESTIONS.
1. What are the questions that we hope to
answer using the data?
2. Are the answers to these questions
supposed to inform a decision?
3. What are the consequences of a correct as
opposed to an incorrect decision?

13. QUESTIONS CALL FOR DATA; DATA
LEADS TO QUESTIONS.
QUESTION
CONSEQUENCE
DATA
DECISION
UNCERTAINTY

14. Question: what is P?
Data: P obtained through a statistical collection process
Decision: If P > LP then…

15. Question: what is P?
Data: P obtained through a statistical collection process
Decision: If P > LP then…

16. Question: what is P?
Data: P obtained through a statistical collection process
Decision: If P > LP then…
M / U is a natural measure of confidence

17. THE KEY IDEA: QMU SCORE
QMU: Quantification of Margins & Uncertainty
Q ( M , U )
The overall measure of
confidence/trust in the
evidence to make a
decision (Q=M/U)
The amount of
uncertainty in the data
The “operating”
margin from the
decision point
Report Q(M,U) for each decision…

18. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
Context: Natural hydrogen production in chemical waste can lead
to explosions…
Question: Is the time-to-flammability sufficiently high for first
responders to ventilate the tank?

19. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
Context: Natural hydrogen production in chemical waste can lead
to explosions…
Question: Is the time-to-flammability sufficiently high for first
responders to ventilate the tank?
Answer: The time-to-flammability is 12.5 days!
CONFIDENCE?
DECISION?

20. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
Context: Natural hydrogen production in chemical waste can lead
to explosions…
Question: Is the time-to-flammability MORE THAN 14days?
Tank M U Q
241-AZ-102 -1.37 0.523 -2.63
241-AY-102 3.60 0.45 8.08
241-AZ-101 10.8 2.03 5.33
241-AN-102 18.2 5.02 3.63
241-AN-107 21.9 4.78 4.57
241-AN-106 83.3 2.52 33.0
241-SX-103 60.8 28.2 2.16
241-AY-101 324 18.6 17.4
241-SX-105 137 67.7 2.03

21. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
Context: Natural hydrogen production in chemical waste can lead
to explosions…
Question: Is the time-to-flammability MORE THAN 14days?
Tank M U Q
241-AZ-102 -1.37 0.523 -2.63
241-AY-102 3.60 0.45 8.08
241-AZ-101 10.8 2.03 5.33
241-AN-102 18.2 5.02 3.63
241-AN-107 21.9 4.78 4.57
241-AN-106 83.3 2.52 33.0
241-SX-103 60.8 28.2 2.16
241-AY-101 324 18.6 17.4
241-SX-105 137 67.7 2.03

22. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
Context: Natural hydrogen production in chemical waste can lead
to explosions…
Question: Is the time-to-flammability MORE THAN 14days?
Next step: Target a specific Q score = Level of Confidence
Q = M / U

23. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
Context: Natural hydrogen production in chemical waste can lead
to explosions…
Question: Is the time-to-flammability MORE THAN 14days?
Next step: Target a specific Q score = Level of Confidence
Q = M / U
Achieving a given Q score
requires “control” of the
uncertainty….how?

24. EXAMPLE: UNCERTAINTY RANKING
Many sources of uncertainties….
• Composition of the chemical waste
• External conditions of the tank
• Chemical models used to estimate hydrogen generation
• ….
Many strategies to rank their importance
• Variance-based sensitivity indices (Sobol’ indices)
• Morris’ elementary effects
• Standardized regression coefficients
• Shapley values
• …

25. EXAMPLE: UNCERTAINTY RANKING
Variance-based sensitivity indices (Sobol’ indices)
P = P(x1, x2, . . . , xd)
VK = VxK
(Ex?K
(P|x?K))
SK =
VK
V(P)
x?K
Consider the quantity of interest
Define the conditional variance K
P = P(x1, x2, . . . , xd)
VK = VxK
(Ex?K
(P|x?K))
SK =
VK
V(P)
x?K
where the expectation is extended to all the independent variables but the k-th
The k-th Sobol index (primary effect) is
P = P(x1, x2, . . . , xd)
VK = VxK
(Ex?K
(P|x?K))
SK =
VK
V(P)
x?K

26. EXAMPLE: UNCERTAINTY RANKING
Many sources of uncertainties….
• Composition of the chemical waste
• External conditions of the tank
• Chemical models used to estimate hydrogen generation
• …. T (liquid)
T (gas)
T (air)
NO3- liquid
NO3- gas
NO2- liquid
NO2- gas
AL3+ liquig
AL3+ gas
Organics liquid
Organics gas
Other
Controlling the temperature more
important than knowing the composition
Sobol Indices

27. EXAMPLE: SAFETY RISK OF
CHEMICAL WASTE
GOAL: Target a the time-to-flammability >14days with a Q score of
2 by controlling the uncertainty in the temperature!
The margin M is achieved by increasing/decreasing the total
amount of waste in each tank!
Tank M U Q
241-AZ-102 1.35 0.672 2.01
241-AY-102 0.74 0.368 2.00
241-AZ-101 2.19 1.10 2.00
241-AN-102 3.54 1.76 2.01
241-AN-107 3.36 1.69 1.99
241-AN-106 0.66 0.33 2.00
241-SX-103 9.24 4.64 1.99
241-AY-101 0.75 0.37 2.00
241-SX-105 6.32 3.16 2.00

28. OPEN QUESTION
• Can we use the Q(M,U) score for decisions
regarding official statistics indices?
• GNI, GDP, HCPI, unemployment rates, etc.
are used in various situations that might
require different precision

29. MACROECONOMIC IMBALANCE
PROCEDURE (MIP 2015) SCOREBOARD

30. TAKE AWAY…
• Data availability does not eliminate the need to
be explicit about uncertainties
• Context is the key ingredient in understanding
and using data
• The QMU score provides a rational way to report
data in the a decision making environment
• Reporting uncertainties is the first step, reducing
or managing uncertainty is the GOAL
G. Iaccarino, R. Pecnik, J. Glimm, and D. Sharp. A QMU approach for characterizing the
operability limits of air-breathing hypersonic vehicles. Reliability Engineering and System
Safety, Vol. 96, pp. 1150-1160, 2011.
A. Frankel, G. Iaccarino, D. Sharp. Application of QMU to the design of a Chemical Waste
Storage Tank, submitted 2017