Incentives help ensure that National Grid manage the high-voltage electricity transmission network in England and Wales efficiently, but how do you make sure that the incentives reward the right behaviour?
The idea behind incentives is that, with the right behaviour, you get paid a reward, but with the wrong behaviour, you have to pay a penalty. Ofgem, who regulate companies running the gas and electricity networks, recently ran a consultation on a proposed incentive scheme1. The Smith Institute reviewed these proposed incentives for National Grid2, to help them to respond to Ofgem’s consultation.
The Smith Institute’s work has provided valuable insights which have helped us provide a neutral view to Ofgem which enabled us to mitigate the random component of the incentive. This work was delivered to incredibly tight timescales in a highly professional way.Jeremy CaplinEnergy Forecasting Manager, Commercial Operations, National Grid
In Ofgem’s proposed incentive scheme, the incentive described in point 4.36 of1 aims to reward electricity demand forecasting models with symmetric errors, favouring neither over- nor under-forecasting. Their proposed incentive consists of a triangular payoff function: with 50% over-forecasts and 50% under-forecasts in a month, National Grid receives the maximum reward of £42,000, but with 70% or more over-forecasts, or 70% or more under-forecasts, National Grid pays the maximum penalty of £42,000. There is a linear variation between these extremes, as shown in the figure below. National Grid produces forecasts for different “cardinal points”, which are different points during the day, such as the evening peak, forecasting for 10 cardinal points in winter and 12 in summer. If any one of these cardinal points is consistently (70% or more forecasts in a month) over- or under-forecast, then, in the proposed incentive scheme, National Grid pays the maximum penalty of £42,000. This incentive aims to reward models that favour neither over- nor under-forecasting, but instead have symmetric errors.
We spotted something strange about this incentive – it penalises the type of models that it seeks to reward! If we look carefully at the probabilities, we find that the incentive is likely to penalise models with perfectly symmetric error terms (models where, in each forecast, the error has probability one half of being positive – an over-forecast – and one half of being negative – an under-forecast). If, for a single cardinal point, 30 forecasts are made in a month, then with probability of more than 4% there will be either at least 70% over-forecasts or at least 70% under-forecasts. In the winter months, with 10 cardinal points, the probability that National Grid will have to pay the maximum penalty (because one or more of the 10 cardinal points has been consistently over- or under-forecast) is over 35%. This rises in the summer months, when there are 12 cardinal points, to a probability of more than 40%. National Grid would therefore expect to pay the maximum penalty in, on average, 4 months of each year, even when using models with perfectly symmetric error terms (which is what the incentive is trying to reward)!
National Grid highlighted this problem in their response3 to Ofgem’s consultation, and, to address the issue, Ofgem changed the incentive4. The new incentive needs two individual cardinal point forecasts, rather than just one, to be consistently over- or under-forecast before the maximum penalty is hit. This reduces the probability of having to pay the maximum penalty when using a model with a perfectly symmetric error term. In the winter months, the probability changes from greater than 35% to less than 7%, and in the summer months it changes from greater than 40% to less than 10%. While this is much lower than it was, it is questionable whether it is the “very low probability” of National Grid paying the maximum fee when using a perfectly symmetric model that Ofgem claim. In fact, combining the probabilities for summer and winter, there is a 62% probability that National Grid will pay the maximum penalty some time during 12 months.
The incentive for forecasting models with symmetric errors also apply to the 48 daily half-hourly wind generation forecasts. In the original proposal, there would have been an 87% chance for perfectly symmetric wind forecasting models to hit the maximum penalty in a month. In the final scheme, the probability drops to 61%. The difference in probabilities between the demand forecast incentive and the wind generation forecast incentive is that only 10-12 demand forecasts are made every day whereas there are 48 daily wind generation forecasts. To get similar probabilities for wind generation forecasts, the incentive should require 5 or more forecasts in a month to have at least 70% over or under-forecasts.