This is the third and final post in this three-part series on opportunity cost and how it affects offers and bids in electricity markets, as evidenced in the NEM. In my first post, I showed that prices mostly stayed within the variable costs of genreation, most offers were not and this was consistent with the economic concept of opportunity cost. In my second post, I provided a range of reasons why offer prices diverge from variable cost and according to opportunity cost. In this last post of the series, I explore policy implications now and anticipate how things will change in the continuing transition to a low-emissions generation mix.

Competition and the rebidding process is crucial for keeping prices in check

First, let’s explore the implications that arise out of what we’ve learned from the first two articles.

The data presented in the first post indicates that, if prevailing prices are not in zone 3 (that is, not above the variable cost of the most expensive peaker), there is likely to be a sizeable portion of generation priced higher in case market conditions worsen. If market conditions do become tight, but not so tight that generation is short of meeting demand, then prices are more likely to settle somewhere between variable costs and the market price cap via the rebidding process.

In the second post, we see generation offers are complicated by recovery of the fixed costs of peakers and inflexibility constraints (i.e. time to run up to minimum operating level, minimum runtime and start-up costs) and the iterative nature of the rebidding process. Together, these complications help explain some of the offers on the vulture’s perch, and their relative positions on it.

Note that competition between all the peaking resources on the vulture’s perch (and potential for demand-side response) limits the price peaking resources might achieve. The outcome of that competition is a function of how tight market conditions are expected to be (it dictates how much spare generation remains after demand is met) and how long they are likely to persist (another angle on the size of the prize).

Offers in zone 3 (i.e. above the highest variable cost) should provide at least a rough indication of the supply of generation in rough order of the ability to generate if market conditions unexpectedly tighten and cause them to be dispatched for a single dispatch interval. It would be operationally problematic and disrupt dispatch if generators submitted offers at the low end of the vulture’s perch to capture unexpected price spikes if they are too slow to respond to them (although this issue is mitigated if they submit accurate ramp rate limits).

If the pre-dispatch schedule indicates the sudden price spike is likely to persist, then the size of the prize increases and the generators higher in the offer stack, missing out on generating at such a high price, would be expected to rebid downward to compete. The more persistent the spike, the greater the opportunity for less flexible generation to compete and the greater the competition, the more downward pressure there is on prices.

We should review the rebidding process

The rebidding process consists of traders shifting generating quantities between the ten offer prices they are required by the rules to submit at the beginning of the trading day (4am) and stick to for the rest of that trading day. They increase the share of their available generating capacity in prices below or above the marginal prices signalled in pre-dispatch schedules to alter their output and reflect changes in their willingness to generate. This practice causes the prevailing market price to increase or decrease, but only around the offer prices submitted by market participants at the start of the trading day.

The second chart in my first post shows, perhaps surprisingly for market sceptics, this rebidding process is competitive enough that prices in the NEM very much tend to lie between the upper and lower bounds of variable costs. The challenge I offer you is to interrogate the offer data and present it in a way that reveals the role rebidding behaviour plays in achieving that outcome – how often does the rebidding process return errant sheep back into the fold? In other words, how often does iterative rebidding bring about this outcome?

How the transition will affect existing bidding arrangements

So how is the transition likely to affect this behaviour and set of arrangements? Here are my thoughts:

• Opportunity cost will have a greater impact on generation offers and prices
• Weather is an inherently less predictable source of fuel so market conditions and prices become more uncertain
• Monitoring market power becomes trickier
• How the existing arrangements adapt as thermal generators retire
• There are some rules that should be reviewed:- The cumulative price threshold regime- The requirement to specify ten offer prices at the beginning of the day- Collection and reporting of fuel constraints of all generation types- Incorporation of fuel constraints in pre-dispatch.

Opportunity cost will affect more offers (and bids)

As the transition deepens, we can expect generation offers in the NEM to diverge from variable cost more often as the generation mix changes away from thermal and toward renewable generation. In the lead-up to their retirement and as their numbers dwindle, greater variation in wholesale prices is forcing owners to make more opportunity cost-based decisions – is it profitable to operate or not, given low wholesale prices are expected in the next week or month or season (for example, see this news article).

Having no fuel storage, solar PV and wind generation generate whenever they can, which should lead them to conform with variable cost offer pricing to a greater extent than baseload thermal power stations. However, in some cases, they are also dragging prices down below variable cost as some of them are operating under contracts that include feed-in tariffs, which becomes a lost opportunity cost if they are not generating electricity.

These renewable, intermittent resources are increasingly being mixed with more fuel-storage limited resources (eg hydro, pumped hydro and batteries). The bids and offers from these fuel-limited resources will tend to help reduce more extreme abundance and scarcity associated with very high and low wholesale prices (see the Andrés-Cerezo and Fabra paper in the sources below for how it works in theory). However, this means prices will more often diverge from variable costs and according to expectations about scarcity now and in the future.

As prices diverge from variable costs more often, the magnitude and duration of wholesale price variations expected in the future are not a sign the market is in jeopardy or broken. They indicate the resource(s) that will provide the most profitable new investment. Market participants anticipate and react to the opportunities and threats caused by changing patterns of wholesale prices, which indicate the relative value and profitability of different assets and manifest in the investment in more of some resources and the potential retirement of others. For example:

• greater variation in wholesale prices encourages more investment in pumped hydro and batteries as this variation increases arbitrage opportunities (the market for which is regularly examined in section 1.3.6 of AEMO’s quarterly energy dynamics reports)
• higher prices on average encourages more investment in wind generation (assuming there is little correlation between wind and time of day)
• lower prices during the day due to the duck curve invites higher consumption of electricity.

Weather and weather forecasts become increasing important

Given an expectation that these investment market dynamics continue and the share of renewable resources increases, generation capability becomes increasingly dependent on the weather and on weather forecasts. Being increasingly reliant on the weather, generation forecasts become both more important and more uncertain, compared with the status quo, as fuel forecasts for traditional thermal fuels are less uncertain than they are for wind, solar PV, hydro and, by extension, batteries.

Unlike bids and offers based on variable costs, offers and bids based on opportunity cost include consideration of both present and future trading intervals – participants must take account of future conditions and prices. When offers and bids depend on expectations of future market conditions and prices, they inherently rely on the intentions of every other participant in the market. Consequently, an increase in the uncertainty of that future and every other participant’s intentions makes their own account of future conditions and prices more prone to error.

It follows that, because our power systems are increasingly reliant on the weather, which is inherently difficult to forecast, fuel-limited generators can and will get it ‘wrong’. Without the benefit of hindsight, they can underestimate or be surprised by the magnitude of scarcity risks. They have to make a call about future conditions and they will sometimes price their fuel too low and deplete their fuel stocks faster when they might have reduced total wholesale market costs (and increased their profits) by raising their offer prices earlier. At other times, generators will overprice their fuel if they expect conditions to be worse than they turn out, which can unfortunately provide ‘evidence’ of inappropriate price gouging (unfair or not).

The fundamental problem is, given the unpredictability of the weather, no one knows how bad an event might be until it is over and the stakes can be high. While this all might seem scary, it is something some markets, those dominated by hydro generation, for instance, have managed for decades. As an example of how important managing adverse weather events is in New Zealand, there was an unusually dry period at the end of 2020 and into 2021 that was the subject of a review by the Electricity Authority. Prices rose to high levels during the period as lake levels were well below what is considered prudent to get through winter months, when much of the rainfall sticks to the surrounding mountains in the lake catchments as snow. In the end, while lake storage got uncomfortably low, rainfall arrived that filled up the lakes and things went back to normal. The review found that the system worked pretty much as it should and made a few recommendations to improve the communication of the changing situation.

In the future, dunkelflaute events (periods of low solar and wind output) could be of similar importance and concern in Australia, given its increasing reliance on these renewable generation technologies and storage.

Monitoring market power

Policy makers and regulators want confidence that offer behaviour will result in goldilocks prices that adequately (accurate seems too high a hurdle) reflect relative scarcity. Allowing generators to submit offers and set prices up to the market price cap in the NEM demonstrates a degree of faith in competition that many other policy makers have not. The data in my first post suggests this faith is justified. Nevertheless, policy makers will continue to be concerned about the degree to which the natural market failure of imperfect information might hide the systemic misuse of pricing power.

One undesirable potential scenario is for fuel-limited generators to submit offer prices that suggest a higher likelihood of running out of fuel than is credible, which is inconsistent with efficient and competitive outcomes. This happens in any market where suppliers can ‘fake’ scarcity to keep prices high (for examples, see this article on the art of creating artificial scarcity).

This behaviour was found to have occurred in New Zealand during what the Electricity Authority determined to be an undesirable trading situation in November 2019. Meridian Energy was generating electricity priced at over $100/MWh despite data and video evidence showing torrents of water being spilled over the dam at one of their power stations. Note how crucial it is to monitor storage and spill so the regulator can take appropriate action to correct past and influence future actions to assure people they can trust that prices adequately reflect relative scarcity.

Another relevant example, referred to in the last section, is a more reasonable one where the reviewer found that prices were acceptably high, given the circumstances.

The other, even more undesirable situation, is one where fuel-limited generators are overly optimistic of the future and run dry before a weather event is over. There can even be a temptation for policy makers and regulators to be so concerned about the first scenario that generators can credibly claim this has contributed to their overly optimistic pricing.

Policymakers might mitigate against these worse-case scenarios by tasking the regulator with developing a model(s) for establishing a credible value or range of values for the remaining storage, given existing and forecast data. The Electricity Authority reviewed competition in New Zealand’s wholesale market and calculated water values from 2016-2021, publishing a review paper on the topic in October 2021.

How the arrangements have to adapt to 100% renewables

Fuel-limited generators and storage assets currently offer their quantities below and above thermal generator offers set at variable costs to manage their fuel stocks, which helps explain why wholesale prices in the NEM tend to be in within zone 2 (the plausible bounds of variable costs) most of the time. It makes sense that fuel-limited generators shift their output around generators that are not so fuel-constrained.

If market conditions are such that supply is high, relative to demand, then fuel-limited generators output will vary around low variable cost generation like brown coal. As conditions tighten, they shadow black coal and CCGTs and when conditions are very tight, they shadow price around OCGT offer prices. The Snowy Hydro submission in the sources below provides greater insight into how it considers opportunity cost in its bidding decisions.

While there is still a long way to go to achieve a renewables-dominated power system in Australia, people are already considering how things work or need to change once all thermal generators are gone. When contemplated this problem, there are two pieces of good news that help me sleep at night:

1. The thermal generating stations or units are not retiring all at once, but in an orderly fashion over the next decades.
2. Other markets (across the Tasman in New Zealand, for instance) that are much closer to the goal of 100% renewable generation can provide valuable insight.

In theory, the NEM can function as it does now while new investment in renewable generation and storage covers the loss of thermal generation retiring. Prices should rise above variable cost more often as the prospect of being short of generation increases – how quickly and by how much is the question. While thermal generation offers have long been the price setters in electricity markets, it is the cost of non-supply that has always been the ultimate benchmark. The competition between generation resources keeps supply and prices below the level at which generation runs short and demand management is required. Again, if scarcity events increase then this creates an opportunity for more competition (or demand flex), which lowers the risk of scarcity and the associated prices.

As we approach a 100% renewable power system, it would appear increasingly beneficial to become more sophisticated with respect to the consideration and treatment of the demand for electricity and how we match it with generation. It is no longer a simple matter of investing in and operating generation to meet instantaneous demand. How much demand for electricity is more flexible in time – that is, what is the opportunity cost of delaying electricity consumption? Old examples are domestic hot water, commercial chilling. New examples are all the various devices that operate on rechargeable batteries. The new trick is to find mutually agreeable schedules for market generation and demand. An abundance of sunshine and solar PV should reduce prices and encourage more consumption during daylight hours, including large and very time-flexible loads like hydrogen, ammonia, or methanol production. Large time-flexible loads are invaluable to flex when storage is decreasing and the risk of scarcity rises.

New Zealand is closer to 100% renewable generation mix and it operates a wholesale market very similar to the NEM and continues to do so. However, the Electricity Authority asked the stakeholder Market Development Advisory Group (MDAG) to tackle the problem of market design in a 100% renewable generation future and it published its final reportand recommendations in December 2023. As part of its work, the MDAG commissioned a report by E Grant Read that I also recommend reading (it is in the list of sources below) as he explains in detail the challenges and options for the New Zealand wholesale electricity market in a 100% renewables power system.

Review of the cumulative price threshold regime

The cumulative price threshold (CPT) is a financial safety valve in the NEM, designed over 20 years ago to reduce the cost of summer heatwaves, that is also expected to be of benefit to consumers. Once the sum of wholesale prices over a rolling week reaches a specified threshold, AEMO reduces the market price cap from its usual, eye-wateringly high level ($16,600/MWh) to $600/MWh (administered price).

The Reliability Panel recommends a market price cap high enough to drive new investment in generating resources to satisfy consumers’ reliability expectations, with respect to generation adequacy. However, the CPT regime was designed to limit the size of the windfall gains generators make when conditions are tight over the morning/evening demand peaks for a few days (specifically for summer heat waves).

The CPT is essentially a ‘stop-loss’ mechanism that reduces the liability (and associated costs) of retailers and consumers facing wholesale spot prices. This necessarily reduces the revenues for generation to be available in situations where the CPT kicks in and the incentive for retailers and consumers facing wholesale prices to buy contract cover for these events.

The logic for the CPT is that, by the time it kicks in, the generators have already made large windfall gains, which means the reduction in the market price cap will have little or no effect on investment and the resources deployed so there is little to no reduction in the reliability enjoyed by consumers.

However, this logic only works when the resources deployed during short scarcity events are thermal peakers. While the CPT and administered price have been raised to cope with high fuels prices since the NEM suspension in June 2022, the regime remains unsuitable for dunkelflaute events (several days or more of low wind and sunshine), which become a significant risk once storage and intermittent generation become a much larger share of generating resources.

Review of the rebidding rules

The problem is that fixing ten price bands at the beginning of the trading day (4am) is no longer suited for a trading environment with more varied diurnal market conditions and more opportunity cost-driven participants. Also, because the market conditions in which prices end up being set above and below variable cost are relatively rare, the circumstances are different each time. They also generally develop after the start of the trading day. Consequently, it is unlikely generators have submitted a set of prices in price below and above the range of variable costs that turn out to be efficient and this means consumers are more likely to come out worse off from these restrictions.

The chart (below) displays offer prices submitted by generators in the NEM for 30 September 2022.

Chart 3: Distribution of offer prices in the NEM on 30 September 2022

The chart shows a decent majority of offer prices submitted for that day (2,059 out of 3,529 or 58%) lay somewhere between -$100/MWh and $700/MWh. The chart also reveals price ranges that are sparsely populated with offers. In particular, there are relatively few (258 out of 3,529) offer prices in the large range between $1,500/MWh to $12,000/MWh, which makes it far less likely the rebidding process will end up with a price in this range. Instead, tight conditions are more likely to result in much higher prices.

It is natural that generators submit more offer prices in the zone where prices are likely to settle than outside this range, where they might settle only occasionally. However, on the other hand, the vulture’s perch could be a product of the restrictions on rebidding offer prices. If so, the restrictions might have had a perverse effect, leading to higher wholesale prices rather than lower ones, as participants might be arbitrarily picking prices near the market price cap to reflect their preference not to generate. This, less than ideal, offer distribution is creates a much larger degree of tolerance around the efficiency of prices when market conditions stray out of the range where variable costs are marginal.

Allowing generators to change both their price bands and their quantities has the potential for both beneficial and harmful effects on efficient price discovery. It could help traders with opportunity cost considerations (fuel-limited resources, most importantly) to offer prices that better reflect the value of limited fuel and reserve scarcity, which would be more efficient. Generators could re-bid both their prices and quantities as conditions changed, until a marginal price was reached where generators were ambivalent about altering their output any further.

On the other hand (I am an economist, after all) the price discovery process might become less stable as the extra degree of freedom could induce or even necessitate more rebidding iterations for the market to settle at a more efficient price. Such an effect has the potential to reduce the potential efficiency gains from the relaxation of the requirement. However, in mitigation of this negative potential, wholesale trading is a repeat game so we should expect traders to adapt and optimise according to the marginal costs and benefits.

Finally, given the increasing number of parties bidding and the huge increase in the amount of rebidding there has been since the NEM became a five-minute market, there is an opportunity to greatly reduce the amount of data traffic involved. The rebidding process is crucial to the outcomes in the NEM and so we should make it easy for people to capture and analyse the data to monitor how well it is working.

The existing offer/bid form is inefficient, given that bids are deemed to be good until changed. Presently, the format of bids is a two-dimensional file containing repeated information where values don’t change. This could be changed so that only data the trader wants to change is sent, but I imagine this option comes at considerable cost.

There is an alternative, much lower hanging prize.  AEMO could publish a view of the bidding data that eliminates all the repeat, redundant data. This is not a difficult task for a data administrator and would greatly reduce the size of the datafiles, which I understand are presently are over 400 million rows a day. I would be surprised if removing the redundant data didn’t at least halve the number of rows and size of the file.

Review of what we collect and publish on fuel-constraints and available storage

Given the increasing share and importance of storage in the control of many participants, wholesale prices are increasingly dependent on the collective view of the effect of future market conditions to value the remaining stores of energy. Therefore, it would be beneficial to make changes to the scheduling process so everyone is aware of the collective state of charge/fuel supply limits in the pre-dispatch timeframe. This extra piece of information should improve price discovery and efficiency. It will also provide an important piece of information that help builds confidence that wholesale prices adequately reflect scarcity.

At present, the rules require generators to rebid to ensure they can fulfil schedules produced under the pre-dispatch and other PASA (projected assessment of system adequacy) processes. However, we simply do not know the degree to which outcomes in each schedule are feasible with respect to the limitations in the capacity of each battery and pumped hydro station to generate/discharge and charge. As the amount of battery and other storage resources increase, the integrity of pre-dispatch schedules decreases as infeasible outcomes are more likely to occur (I.e., NEMDE charging or discharging batteries that are already full or empty) and more rebidding effort is required to adjust them.

Adding a requirement on traders to submit and keep state of charge and fuel supply (in MWh) accurate and include in their bids their upper and lower limits could help NEMDE maintain the integrity of the schedules derived from the pre-dispatch and short-term PASA process. Publishing total MWh storage in each NEM region over the scheduling period could also help all participants better understand important limitations that affect the supply situation. This should help them rebid according to the risk of scarcity and overabundance (I.e., running out of energy or having so much it must be spilled).

Finally, it is important, from a social acceptance perspective, for the market operator to publish all the information pertinent to the bidding process and for market monitoring agents (public and private) to assess and present how well or poorly storage is being managed to keep prices low and reliability that meets societal expectations.

Sources

Short run marginal cost – Technical paper, Adam McHugh, ERAWA, 11 January 2008

Opportunity cost – the road not taken, EconClips on Youtube, 14 June 2018

Bidding in energy-only wholesale electricity markets – Professor George Yarrow, assisted by Dr Chris Decker, written for AEMC, Nov 2014

Water is valuable: the allocation of water and other resources in the New Zealand electricity market, Diana Tam with Prof. Lew Evans, June 2013

Structure matters – storage in electricity markets, David Andrés-Cerezo and Natalia Fabra, December 2020

Submission on opportunity cost methodology from Snowy Hydro for AEMC consultation on Snowy Hydro Limited compensation claim, 13 October 2023

Opportunity Costing in the NZEM – Implications of decarbonisation, E Grant Read, prepared for the Market Development Working Group of the Electricity Authority, 19 January 2022