Bending a Spear into a Pruning Hook

The biggest problem with the current method of weather normalization is that it uses a linear modeling approach. In the last post, I talked about the Degree Day method of weather summary, and how it requires a linear model. However, when you use hourly data, linear models are not the only option.

See the graph below, which shows a linear model applied to a graph of temperature and load.

Above: the problem.

Above: the problem.

Demand does not respond to weather in a linear fashion, so this approach will never work correctly. A square peg does not fit into a round hole, and a straight line can’t follow a curve.  The best you can hope for with a linear model is that the errors will roughly cancel out. In the long run errors will likely balance, but it takes hundreds of observations for it to be achieved.

Running a linear model is essentially like closing your eyes and hoping for the best—not a very analytical approach. A good model will accurately describe weather response across the entire temperature spectrum—even at the extreme ends. See the model below, which was created by Bellweather using the same data as the chart at the top.

The Bellweather model is optimized to work across all temperatures. For the given data set, it would not be possible to have a model with a better fit. However, more can be done to improve accuracy.

In a subsequent post I’ll talk about subsetting data to maximize overall model quality.

A Modern Approach to Weather Data

In the last post I talked about the outdated Degree Day weather model.  The problem is that Degree Days are a summary of temperature information, and summaries create information loss. This isn’t a secret; any analyst knows that Degree Days are an obstacle to improving accuracy. For the industry to progress, utilities need to give up using degree Degree Days and begin working with real hourly weather data.

The chart below shows daily Cooling Degree Days and daily load for a mid-sized utility in the Northeast during June 2013.

Load and CDD.png

The best candidate to appropriately fit the dataset above is a linear model. There are not enough data points to identify a more complex relationship. Compare the chart above to the following chart, which displays hourly data for the same utility over the same period of time.

The best fit for the data above would be a 5th degree polynomial. Because the shape displayed in the two charts is not the same, it is clear that Degree Days don't preserve the true weather/load relationship, which indicates an essential loss of information.

For a long time, Degree Days may have been the best option available, due to limited availability of hourly data. Fortunately, this is no longer the case. There are over 1,000 stations in the publicly accessible NOAA system that record hourly data and most have records going back 20 years.

With the present availability of hourly data we shouldn’t be using a method of information shorthand that is a holdover of a bygone era. Hourly temperatures themselves are not the solution to creating better weather analysis, but once hourly data becomes the basis for that analysis, there is the possibility option to use an advanced approach.

Read my next post about how hourly data allows us to use non-linear models.


Whats Wrong With Degree Days

Strachey.  Pictured with his greatest invention: "neck-burns."

Strachey.  Pictured with his greatest invention: "neck-burns."

Heating Degree Days and Cooling Degree Days are measures of temperature used by virtually every electric utility. However, the concept of Degree Days actually predates the existence of utilities. Degree Days were invented by Sir Richard Strachey in 1878.  Strachey was a member of the Meteorological Council to the Royal Society placed in charge of investigating a famine in India. Strachey’s inquiry led him to invent the concept of Degree Days as a way of estimating the amount of heat necessary for growing crops. Degree Days were shortly adopted by utilities for estimating energy demand based on weather and they are still used for this purpose today.

To put it in perspective: the basic process we use for analyzing weather was invented the same year that Thomas Edison invented the phonograph. Audio technology has successfully progressed over the last 130 years, but somehow weather analysis has not. I'm honestly not sure if it would be possible to find another household/commercial/industrial practice that has persisted as long; everything we did in the 1800's has been replaced by new and better technologies... but utilities are still using Degree Days. And I don't believe this is a case of "if it ain't broke don't fix it." Degree Days are genuinely detrimental for utilities and I'm prepared to explain why.

Read for my next post to learn about a modern approach to weather data.