NERC's Cloudy Crystal Ball
How much confidence do NERC demand forecasts warrant?
March 2004
By Tom Replogle
Independent consultants must properly estimate peak demand growth if they are to provide clients with reasonable market analysis. Some consultants defer to the North American Electric Reliability Council (NERC) on this matter because NERC bases its projections on utility-specific estimates developed with more information than consultants typically can access. NERC recently rolled out new demand growth forecasts, so the time seems right to explore whether this confidence is justified or misplaced.
Each year NERC provides actual demand for the prior year, along with a 10-year forecast for demand growth. Table 1 presents NERC forecasts for U.S. summer peak demand growth since 1983, and compares them with actual demand growth over the same periods. Table 1 shows that for periods where historical data are available, NERC always has underestimated the U.S. growth rate.
Table 2 compares NERC 5-year projections against actual demand growth. The shorter term allows the comparison of more forecast periods as well as the opportunity to evaluate NERC near-term forecast accuracy. Here, too, NERC underestimated peak demand in every forecast.
A test for bias1 seems appropriate given the apparent tendency of NERC to underestimate peak-demand growth. Figure 1 presents 5-year forecasted and actual growth rates as a scatter plot.
If NERC forecasts were completely accurate, every prediction of growth would match actual growth, and all the observations above would fall along the "Expected Best Fit" line.2 Even if forecasts are inaccurate, observations should still fall around the "Expected Best Fit" line, provided the observations are unbiased. This is because the expected value of the forecast error would be zero.
In practice, all the observations fall above the "Expected Best Fit" line, meaning actual growth has been above forecast. A regression analysis provides the "Observed Best Fit" line.3
The slope of the "Observed Best Fit" line means actual growth generally has been 37 percent above what NERC has forecasted. The test for bias evaluates whether that difference reasonably can be attributed to chance, or whether it is evidence of bias in NERC projections. The test will examine the null hypothesis (H0) below alongside the alternate hypothesis (H1):
H0: The actual slope of the NERC forecast population is equal to the slope of the "Expected Best Fit" line.
H1: The actual slope of the NERC forecast population is not equal to the slope of the "Expected Best Fit" line.
The t-statistic for the test is [(1.366-1)/0.0465] or 7.87. With 13 degrees of freedom this two-tailed test implies the null hypothesis can be rejected in favor of the alternate hypothesis at a 99.999 percent confidence level. Stated another way, one can be 99.999 percent confident that NERC systematically under-forecasts peak demand. Using 10-year NERC forecasts the confidence level is 99.99 percent.
Independent consultants making use of NERC forecasts should carefully consider the very strong statistical evidence of bias. Analysts underestimating peak demand growth will predict market recovery at a later date, forecast lower near-term revenues for existing generation assets, and advocate less new construction than is needed to meet planning reserves.
Endnotes
- The word "bias" is used in a statistical sense, describing a consistent tendency for actual growth to be greater than forecast. This does not imply or suggest any intent of NERC, the utilities, or their staff to misrepresent the conditions that are deemed most likely to occur in the future.
- The "Expected Best Fit" line has a slope of 1.
- The "Observed Best Fit" line has a slope of 1.366 and a standard error of 0.0465. The line passes through the origin (meaning that bias is reflected in the slope rather than as a constant, or is divided between slope and constant).
Tom Replogle is an associate at ICF Consulting. Contact him at treplogle@icfconsulting.com.
Articles found on this page are available to subscribers only. For more information about obtaining a username and password, please call our Customer Service Department at 1-800-368-5001.
Public
Utilities Fortnightly & Spark
