Chapter 10 - Statistic

1. Table of Contents
Chapter 10 (Forecasting)
An Overview of Forecasting Techniques (Section 10.1) 10.2–10.8
A Case Study: The Computer Club Warehouse Problem (Section 10.2) 10.9–10.13
Applying Time-Series Forecasting to the Case Study (Section 10.3) 10.14–10.30
The Time-Series Forecasting Methods in Perspective (Section 10.4) 10.31–10.35
Causal Forecasting with Linear Regression (Section 10.5) 10.36–10.40
Judgmental Forecasting Methods (Section 10.6) 10.41
© 2014 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any manner. This document may not be copied,
scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.

2. Forecasting at Fastchips
• Fastchips is a leading producer of microprocessors.
• Six months ago, it launched the sales of its latest microprocessor.
• Month-by-month sales (in thousands) over the initial six months have been
17 25 24 26 30 28
Question: What is the forecast for next month’s sales?
10-2

3. The Last-Value Forecasting Method
The last-value forecasting method ignores all data points in a time series except
the last one.
Forecast = Last value
Fastchips: Month-by-month sales (in thousands) over the initial six months:
17 25 24 26 30 28
Forecast = 28
10-3

4. The Averaging Forecasting Method
The averaging forecasting method uses all the data points in the time series and
simply averages these points.
Forecast = Average of all data to date
Fastchips: Month-by-month sales (in thousands) over the initial six months:
17 25 24 26 30 28
Forecast = (17+25+24+26+30+28) / 6 = 25
10-4

5. The Moving-Average Forecasting Method
The moving-average forecasting method averages the data for only the most
recent time periods.
n = Number of recent periods to consider as relevant for forecasting
Forecast = Average of last n values
Fastchips: Month-by-month sales (in thousands) over the initial six months:
17 25 24 26 30 28
Forecast (n=3) = (26+30+28) / 3 = 28
10-5

6. The Exponential Smoothing Forecasting Method
• The exponential smoothing forecasting method provides a more
sophisticated version of the moving-average method.
• It gives the greatest weight to the last month and then progressively smaller
weights to the older months.
• Exponential smoothing with trend adjusts exponential smoothing by also
directly considering any current upward or downward trend in sales.
10-6

7. Linear Regression
• Linear regression uses a two-dimensional graph with sales measured along
the vertical axis and time measured along the horizontal axis.
• After plotting the sales data, this method finds a line passing through the midst
of the data as closely as possible.
• The extension of the line into future months provides the forecast of sales in
these future months.
10-7

8. Measuring the Forecast Error
• The mean absolute deviation (called MAD) measures the average forecasting
error.
MAD = (Sum of forecasting errors) / (Number of forecasts)
• The mean square error (often abbreviated MSE) measures the average of the
square of the forecasting error.
MSE = (Sum of square of forecasting errors) / (Number of forecasts).
• The MSE increases the weight of large errors relative to the weight of small
errors.
10-8

9. The Computer Club Warehouse (CCW)
• The Computer Club Warehouse (CCW) sells computer products at bargain
prices by taking telephone orders (as well as website and fax orders) directly
from customers.
• Products include computers, peripherals, supplies, software, and computer
furniture.
• The CCW call center is never closed. It is staffed by dozens of agents to take
and process customer orders.
• A large number of telephone trunks are provided for incoming calls. If an
agent is not free when a call arrives, it is placed on hold. If all the trunks are in
use (called saturation), the call receives a busy signal.
• An accurate forecast of the demand for agents is needed.
Question: How should the demand for agents be forecasted?
10-9

10. 25 Percent Rule (Current Forecasting Method)
Since sales are relatively stable through the year except for a substantial increase
during the Christmas season, assume that each quarter’s call volume will be the
same as the preceding quarter, except for adding 25 percent for Quarter 4.
Forecast for Quarter 2 = Call volume for Quarter 1
Forecast for Quarter 3 = Call volume for Quarter 2
Forecast for Quarter 4 = 1.25(Call volume for Quarter 3)
Forecast for next Quarter 1 = (Call volume for Quarter 4) / 1.25
10-10

11. Average Daily Call Volume (3 Years of Data)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
A B C D E
CCW's Average Daily Call Volume
Year Quarter Call Volume
1 1 6,809
1 2 6,465
1 3 6,569
1 4 8,266
2 1 7,257
2 2 7,064
2 3 7,784
2 4 8,724
3 1 6,992
3 2 6,822
3 3 7,949
3 4 9,650
10-11

12. Applying the 25-Percent Rule
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
A B C D E F G H I
Current Forecasting Method for CCW's Average Daily Call Volume
Forecasting
Year Quarter Data Forecast Error Mean Absolute Deviation
1 1 6,809 MAD = 424
1 2 6,465 6,809 344
1 3 6,569 6,465 104 Mean Square Error
1 4 8,266 8,211 55 MSE = 317,815
2 1 7,257 6,613 644
2 2 7,064 7,257 193
2 3 7,784 7,064 720
2 4 8,724 9,730 1,006
3 1 6,992 6,979 13
3 2 6,822 6,992 170
3 3 7,949 6,822 1,127
3 4 9,650 9,936 286
4 1 7,720
4 2
4 3
4 4
10-12

13. Measuring the Forecast Error
• The mean absolute deviation (called MAD) measures the average forecasting
error.
MAD = (Sum of forecasting errors) / (Number of forecasts)
• The mean square error (often abbreviated MSE) measures the average of the
square of the forecasting error.
MSE = (Sum of square of forecasting errors) / (Number of forecasts).
• The MSE increases the weight of large errors relative to the weight of small
errors.
10-13

14. Considering Seasonal Effects
• When there are seasonal patterns in the data, they can be addressed by
forecasting methods that use seasonal factors.
• The seasonal factor for any period of a year (a quarter, a month, etc.) measures
how that period compares to the overall average for an entire year.
Seasonal factor = (Average for the period) / (Overall average)
• It is easier to analyze data and detect new trends if the data are first adjusted to
remove the seasonal patterns.
Seasonally adjusted data = (Actual call volume) / (Seasonal factor)
10-14

15. Calculation of Seasonal Factors for CCW
Quarter
Three-Year
Average
Seasonal
Factor
1 7,019 7,019 / 7,529 = 0.93
2 6,784 6,784 / 7,529 = 0.90
3 7,434 7,434 / 7,529 = 0.99
4 8,880 8,880 / 7,529 = 1.18
Total = 30,117
Average = 30,117 / 4 = 7,529
10-15

16. Excel Template for Calculating Seasonal Factors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
A B C D E F G
Estimating Seasonal Factors for CCW
True
Year Quarter Value Type of Seasonality
1 1 6,809 Quarterly
1 2 6,465
1 3 6,569
1 4 8,266 Estimate for
2 1 7,257 Quarter Seasonal Factor
2 2 7,064 1 0.9323
2 3 7,784 2 0.9010
2 4 8,724 3 0.9873
3 1 6,992 4 1.1794
3 2 6,822
3 3 7,949
3 4 9,650
10-16

17. Seasonally Adjusted Time Series for CCW
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
A B C D E F
Seasonally Adjusted Time Series for CCW
Seasonal Actual Seasonally Adjusted
Year Quarter Factor Call Volume Call Volume
1 1 0.93 6,809 7,322
1 2 0.90 6,465 7,183
1 3 0.99 6,569 6,635
1 4 1.18 8,266 7,005
2 1 0.93 7,257 7,803
2 2 0.90 7,064 7,849
2 3 0.99 7,784 7,863
2 4 1.18 8,724 7,393
3 1 0.93 6,992 7,518
3 2 0.90 6,822 7,580
3 3 0.99 7,949 8,029
3 4 1.18 9,650 8,178
10-17

18. Outline for Forecasting Call Volume
1. Select a time-series forecasting method.
2. Apply this method to the seasonally adjusted time series to obtain a forecast of
the seasonally adjusted call volume for the next time period.
3. Multiply this forecast by the corresponding seasonal factor to obtain a forecast
of the actual call volume (without seasonal adjustment).
10-18

19. The Last-Value Forecasting Method
• The last-value forecasting method ignores all data points in a time series
except the last one.
Forecast = Last value
• The last-value forecasting method is sometimes called the naïve method,
because statisticians consider it naïve to use just a sample size of one when
other data are available.
• However, when conditions are changing rapidly, it may be that the last value is
the only relevant data point.
10-19

20. The Last-Value Method Applied to CCW’s Problem
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
A B C D E F G H I J K
Last-Value Forecasting Method with Seasonality for CCW
Seasonally Seasonally
True Adjusted Adjusted Actual Forecasting
Year Quarter Value Value Forecast Forecast Error Type of Seasonality
1 1 6,809 7,322 Quarterly
1 2 6,465 7,183 7,322 6,589 124
1 3 6,569 6,635 7,183 7,112 543 Quarter Seasonal Factor
1 4 8,266 7,005 6,635 7,830 436 1 0.93
2 1 7,257 7,803 7,005 6,515 742 2 0.90
2 2 7,064 7,849 7,803 7,023 41 3 0.99
2 3 7,784 7,863 7,849 7,770 14 4 1.18
2 4 8,724 7,393 7,863 9,278 554
3 1 6,992 7,518 7,393 6,876 116
3 2 6,822 7,580 7,518 6,766 56
3 3 7,949 8,029 7,580 7,504 445
3 4 9,650 8,178 8,029 9,475 175
4 1 8,178 7,606
4 2
4 3
4 4
5 1 Mean Absolute Deviation
5 2 MAD = 295
5 3
5 4 Mean Square Error
6 1 MSE = 145,909
10-20

21. The Averaging Forecasting Method
• The averaging forecasting method uses all the data points in the time series
and simply averages these points.
Forecast = Average of all data to date
• The averaging forecasting method is a good one to use when conditions are
very stable.
• However, the averaging method is very slow to respond to changing
conditions. It places the same weight on all the data, even though the older
values may be less representative of current conditions than the last value
observed.
10-21

22. The Averaging Method Applied to CCW’s Problem
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
A B C D E F G H I J K
Averaging Forecasting Method with Seasonality for CCW
Seasonally Seasonally
True Adjusted Adjusted Actual Forecasting
Year Quarter Value Value Forecast Forecast Error Type of Seasonality
1 1 6,809 7,322 Quarterly
1 2 6,465 7,183 7,322 6,589 124
1 3 6,569 6,635 7,252 7,180 611 Quarter Seasonal Factor
1 4 8,266 7,005 7,047 8,315 49 1 0.93
2 1 7,257 7,803 7,036 6,544 713 2 0.90
2 2 7,064 7,849 7,190 6,471 593 3 0.99
2 3 7,784 7,863 7,300 7,227 557 4 1.18
2 4 8,724 7,393 7,380 8,708 16
3 1 6,992 7,518 7,382 6,865 127
3 2 6,822 7,580 7,397 6,657 165
3 3 7,949 8,029 7,415 7,341 608
3 4 9,650 8,178 7,471 8,816 834
4 1 7,530 7,003
4 2
4 3
4 4
5 1 Mean Absolute Deviation
5 2 MAD = 400
5 3
5 4 Mean Square Error
6 1 MSE = 242,876
10-22

23. The Moving-Average Forecasting Method
• The moving-average forecasting method averages the data for only the most
recent time periods.
n = Number of recent periods to consider as relevant for forecasting
Forecast = Average of last n values
• The moving-average forecasting method is a good one to use when conditions
don’t change much over the number of time periods included in the average.
• However, the moving-average method is slow to respond to changing
conditions. It places the same weight on each of the last n values even though
the older values may be less representative of current conditions than the last
value observed.
10-23

24. The Moving-Average Method Applied to CCW
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
A B C D E F G H I J K
Moving Average Forecasting Method with Seasonality for CCW
Seasonally Seasonally
True Adjusted Adjusted Actual Forecasting Number of previous
Year Quarter Value Value Forecast Forecast Error periods to consider
1 1 6,809 7,322 n = 4
1 2 6,465 7,183
1 3 6,569 6,635 Type of Seasonality
1 4 8,266 7,005 Quarterly
2 1 7,257 7,803 7,036 6,544 713
2 2 7,064 7,849 7,157 6,441 623 Quarter Seasonal Factor
2 3 7,784 7,863 7,323 7,250 534 1 0.93
2 4 8,724 7,393 7,630 9,003 279 2 0.90
3 1 6,992 7,518 7,727 7,186 194 3 0.99
3 2 6,822 7,580 7,656 6,890 68 4 1.18
3 3 7,949 8,029 7,589 7,513 436
3 4 9,650 8,178 7,630 9,004 646
4 1 7,826 7,279
4 2
4 3
4 4
5 1
5 2
5 3
5 4 Mean Absolute Deviation
6 1 MAD = 437
6 2
6 3 Mean Square Error
6 4 MSE = 238,816
10-24

25. The Exponential Smoothing Forecasting Method
• The exponential smoothing forecasting method places the greatest weight on
the last value in the time series and then progressively smaller weights on the
older values.
Forecast = a (Last value) + (1 – a) (Last forecast)
a is the smoothing constant between 0 and 1.
• This method places a weight of a on the last value, a(1–a) on the next-to-last
value, a(1–a)2 on the next prior value, etc.
– For example, when a = 0.5, the method places a weight of 0.5 on the last value,
0.25 on the next-to-last, 0.125 on the next prior, etc.
– A larger value of a places more emphasis on the more recent values, a smaller
value places more emphasis on the older values.
• The choice of the value of the smoothing constant a has a substantial effect on
the forecast.
– A small value (say, a = 0.1) is appropriate if conditions are relatively stable.
– A larger value (say, a = 0.5) is appropriate if significant changes occur frequently.
10-25

26. The Exponential Smoothing Method Applied to CCW
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
A B C D E F G H I J K
Exponential-Smoothing Forecasting Method with Seasonality for CCW
Seasonally Seasonally
True Adjusted Adjusted Actual Forecasting Smoothing Constant
Year Quarter Value Value Forecast Forecast Error a 0.5
1 1 6,809 7,322 7,500 6,975 166
1 2 6,465 7,183 7,411 6,670 205 Initial Estimate
1 3 6,569 6,635 7,297 7,224 655 Average = 7,500
1 4 8,266 7,005 6,966 8,220 46
2 1 7,257 7,803 6,986 6,497 760 Type of Seasonality
2 2 7,064 7,849 7,394 6,655 409 Quarterly
2 3 7,784 7,863 7,622 7,545 239
2 4 8,724 7,393 7,742 9,136 412 Quarter Seasonal Factor
3 1 6,992 7,518 7,568 7,038 46 1 0.93
3 2 6,822 7,580 7,543 6,789 33 2 0.90
3 3 7,949 8,029 7,561 7,486 463 3 0.99
3 4 9,650 8,178 7,795 9,199 451 4 1.18
4 1 7,987 7,428
4 2
4 3
4 4
5 1
5 2
5 3
5 4
6 1
6 2 Mean Absolute Deviation
6 3 MAD = 324
6 4
7 1 Mean Square Error
MSE = 157,836
10-26

27. A Time Series with Trend
(Population of a State over Time)
1995 2000 2005 Year
Population
(Millions)
4.8
5.0
5.2
5.4
Trend
line
10-27

28. Exponential Smoothing with Trend Forecasting Method
• The exponential smoothing with trend forecasting method uses the recent
values in the time series to estimate any current upward or downward trend in
these values.
Trend = Average change from one time-series value to the next
• The formula for forecasting the next value in the time series adds the estimated
trend.
Forecast = a (Last value) + (1 – a) (Last forecast) + Estimated trend
a is the smoothing constant between 0 and 1.
• Exponential smoothing also is used to obtain and update the estimated trend.
Estimated trend = b (Latest trend) + (1 – b) (Last estimate of trend)
b is the trend smoothing constant.
• The formula for forecasting n periods from now is
Forecast = a (Last value) + (1 – a) (Last forecast) + n (Estimated trend)
10-28

29. Exponential Smoothing with Trend Applied to CCW
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
A B C D E F G H I J K L M
Exponential-Smoothing with Trend Forecasting Method with Seasonality for CCW
Seasonally Seasonally
True Adjusted Latest Estimated Adjusted Actual Forecasting Smoothing Constant
Year Quarter Value Value Trend Trend Forecast Forecast Error a 0.3
1 1 6,809 7,322 0 7,500 6,975 166 b 0.3
1 2 6,465 7,183 -54 -16 7,430 6,687 222
1 3 6,569 6,635 -90 -38 7,318 7,245 676 Initial Estimate
1 4 8,266 7,005 -243 -100 7,013 8,276 10 Average = 7,500
2 1 7,257 7,803 -102 -100 6,910 6,427 830 Trend = 0
2 2 7,064 7,849 167 -20 7,158 6,442 622
2 3 7,784 7,863 187 42 7,407 7,333 451 Type of Seasonality
2 4 8,724 7,393 179 83 7,627 9,000 276 Quarterly
3 1 6,992 7,518 13 62 7,619 7,085 93
3 2 6,822 7,580 32 53 7,642 6,877 55 Quarter Seasonal Factor
3 3 7,949 8,029 34 47 7,670 7,594 355 1 0.93
3 4 9,650 8,178 155 80 7,858 9,272 378 2 0.90
4 1 176 108 8,062 7,498 3 0.99
4 2 4 1.18
4 3
4 4
5 1
5 2
5 3
5 4
6 1
6 2
6 3
6 4 Mean Absolute Deviation
7 1 MAD = 345
Mean Square Error
MSE = 180,796
10-29

30. MAD and MSE for the Various Forecasting Method
Forecasting Method MAD MSE
CCW’s 25 percent rule 424 317,815
Last-value method 295 145,909
Averaging method 400 242,876
Moving-average method 437 238,816
Exponential smoothing 324 157,836
Exponential smoothing with trend 345 180,796
10-30

31. Typical Probability Distribution of Call Volume
(Assumes Mean = 7,500)
7,500 7,750
7,250
Mean
10-31

32. Typically Probability Distributions of Call Volume
in the Four Quarters (Assumes Annual Mean = 7,500)
6,500 7,000 7,500 8,000 8,500 9,000
Quarter 2 Quarter 1 Quarter 3 Quarter 4
10-32

33. Comparison of Typical Probability Distributions
of Seasonally-Adjusted Call Volumes in Years 1 and 2
6,500 7,000 7,500 8,000
Year 1 Year 2
10-33

34. Comparison of the Forecasting Methods
• Last value method: Suitable for a time series that is so unstable that even the
next-to-last value is not considered relevant for forecasting the next value.
• Averaging method: Suitable for a very stable time series where even its first
few values are considered relevant for forecasting the next value.
• Moving-average method: Suitable for a moderately stable time series where
the last few values are considered relevant for forecasting the next value.
• Exponential smoothing method: Suitable for a time series in the range from
somewhat unstable to rather stable, where the value of the smoothing constant
needs to be adjusted to fit the anticipated degree of stability.
• Exponential smoothing with trend: Suitable for a time series where the mean
of the distribution tends to follow a trend either up or down, provided that
changes in the trend occur only occasionally and gradually.
10-34

35. The Consultant’s Recommendations
1. Forecasting should be done monthly rather than quarterly.
2. Hiring and training of new agents also should be done monthly.
3. Recently retired agents should be offered the opportunity to work part time on an on-call
basis.
4. Since sales drive call volume, the forecasting process should begin by forecasting sales.
5. For forecasting purposes, total sales should be broken down into the major components:
a) The relatively stable market base of numerous small-niche products.
b) Each of the few (perhaps three or four) major new products.
6. Exponential smoothing with a relatively small smoothing constant is suggested for
forecasting sales of the marketing base of numerous small-niche products.
7. Exponential smoothing with trend, with relatively large smoothing constants, is
suggested for forecasting sales of each major new product.
8. Seasonally adjusted time series should be used for each application of the methods.
9. The forecasts in recommendation 5 should be summed to obtain a forecast of total sales.
10. Causal forecasting with linear regression should be used to obtain a forecast of call
volume from this forecast of total sales.
10-35

36. Causal Forecasting
Causal forecasting obtains a forecast of the quantity of interest (the dependent
variable) by relating it directly to one or more other quantities (the independent
variables) that drive the quantity of interest.
Type of Forecasting
Possible Dependent
Variable
Possible Independent
Variable
Sales Sales of a product Amount of advertising
Spare parts Demand for spare parts Usage of equipment
Economic trends Gross domestic product Various economic factors
CCW call volume Call volume Total sales
Any quantity This same quantity Time
10-36

37. Sales and Call Volume Data for CCW
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
A B C D E
CCW's Average Daily Sales and Call Volume
Sales Call
Year Quarter ($thousands) Volume
1 1 4,894 6,809
1 2 4,703 6,465
1 3 4,748 6,569
1 4 5,844 8,266
2 1 5,192 7,257
2 2 5,086 7,064
2 3 5,511 7,784
2 4 6,107 8,724
3 1 5,052 6,992
3 2 4,985 6,822
3 3 5,576 7,949
3 4 6,647 9,650
10-37

38. Adding a Trendline to the Graph
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
A B C D E
CCW's Average Daily Sales and Call Volume
Sales Call
Year Quarter ($thousands) Volume
1 1 4,894 6,809
1 2 4,703 6,465
1 3 4,748 6,569
1 4 5,844 8,266
2 1 5,192 7,257
2 2 5,086 7,064
2 3 5,511 7,784
2 4 6,107 8,724
3 1 5,052 6,992
3 2 4,985 6,822
3 3 5,576 7,949
3 4 6,647 9,650
10-38

39. Linear Regression
• When doing causal forecasting with a single independent variable, linear
regression involves approximating the relationship between the dependent
variable (call volume for CCW) and the independent variable (sales for CCW)
by a straight line.
• This linear regression line is drawn on a graph with the independent variable
on the horizontal axis and the dependent variable on the vertical axis. The line
is constructed after plotting a number of points showing each observed value
of the independent variable and the corresponding value for the dependent
variable.
• The linear regression line has the form
y = a + bx
where
y = Estimated value of the dependent variable
a = Intercept of the linear regression line with the y-axis
b = Slope of the linear regression line
x = Value of the independent variable
10-39

40. Excel Template for Linear Regression
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
A B C D E F G H I J
Linear Regression of Call Volume vs. Sales Volume for CCW
Time Independent Dependent Estimation Square Linear Regression Line
Period Variable Variable Estimate Error of Error y = a + bx
1 4,894 6,809 6,765 43.85 1,923 a = -1,223.86
2 4,703 6,465 6,453 11.64 136 b = 1.63
3 4,748 6,569 6,527 42.18 1,780
4 5,844 8,266 8,316 49.93 2,493
5 5,192 7,257 7,252 5.40 29 Estimator
6 5,086 7,064 7,079 14.57 212 If x = 5,000
7 5,511 7,784 7,772 11.66 136
8 6,107 8,724 8,745 21.26 452 then y= 6,938.18
9 5,052 6,992 7,023 31.07 965
10 4,985 6,822 6,914 91.70 8,408
11 5,576 7,949 7,878 70.55 4,977
12 6,647 9,650 9,627 23.24 540
10-40

41. Judgmental Forecasting Methods
• Manager’s Opinion: A single manager uses his or her best judgment.
• Jury of Executive Opinion: A small group of high-level managers pool their
best judgment to collectively make the forecast.
• Salesforce Composite: A bottom-up approach where each salesperson
provides an estimate of what sales will be in his or her region. These estimates
are then aggregated into a corporate sales forecast.
• Consumer Market Survey: A grass-roots approach that surveys customers
and potential customers regarding their future purchasing plans and how they
would respond to various new features in products.
• Delphi Method: A panel of experts in different locations who independently
fill out a series of questionnaires. The results from each questionnaire are
provided with the next one, so each expert can evaluate the group information
in adjusting his or her responses next time.
10-41