Historically, weather has always had a strong influence on Australia’s National Electricity Market (NEM). Peak demand events are primarily driven by heating and cooling needs, and extreme weather can damage transmission and distribution networks. Today, as more renewable generation connects to the system, weather also has a strong supply side impact. This has made forecasting and managing weather driven uncertainty an essential part of system operation. At the same time, the Australian Energy Market Operator (AEMO) projects that Queensland’s utility-scale storage capacity should increase from 1.5GW/7GWh to 5GW/60GWh by 2035 (see below).

Projected capacity growth in Queensland according to the Integrated System Plan (ISP). Data taken from the Optimal Development Pathway (CDP 14).

Understanding how these storages will manage weather-driven uncertainty will be key to enabling a cost-effective energy transition. This research, undertaken at Queensland University of Technology in partnership with Powerlink and the Bureau of Meteorology, explores how we can address this challenge.

Why Perfect Foresight Matters for Storage Operation

Unlike traditional generators, storage operators bid based on opportunity cost as their ability to discharge (and charge) is constrained by their available stored energy. Operators must balance immediate returns with preserving energy for future high price periods. This requires anticipating periods of energy scarcity and surplus, while also managing the risk of forecast errors and other uncertainties.

This complex decision-making is difficult to capture in energy market models. Since the best available information is always changing (see the sample forecasts below), models which simulate limited foresight must re-optimise for each time step. Accounting for uncertainty can also add further computational complexity. These factors can be extremely limiting for long-term studies, such as the ISP.

Evolution of QLD1 PASA wind availability forecast for April 07, 2024, compared with what a perfect foresight model would see (actuals). Only every 12^th run is shown for legibility.  

Because of these constraints, market models typically assume perfect foresight, where dispatch and/or capacity expansion decisions are made with full knowledge and full certainty of the future. While this approach has been reasonable given storage’s limited role to date, ignoring the impact of more realistic imperfect foresight risks underestimating the firming required to deliver a cost-effective and reliable future grid.

Current approaches to address these issues, such as derating storages and limiting model lookahead, still model storage dispatch with perfect certainty. Also, while they apply flat constraints on storage behaviour, actual uncertainty varies significantly depending on weather and climate conditions.

The Limits of Real-World Weather Forecasting

While a perfect foresight model can ‘predict’ the weather well in advance, the plot below, from the European Centre for Medium-Range Weather Forecasts, shows that despite improvements in forecast accuracy over time, average uncertainty remains high beyond seven to ten days. Because of this, forecasters often rely on seasonal and climate projections for longer outlooks, which focus on broad-scale drivers and take a probabilistic approach.

Similarly, within the NEM, AEMO’s short term forecasts only provide a seven-day day lookahead (current plus the next six trading days). This poses a significant challenge for storage operation, as managing medium to long-term fluctuations in renewable generation may require operating beyond the accurate limits of forecast models.

Average Anomaly Correlation Coefficient (ACC) in the southern and northern hemispheres over time. ACC is a measure of how well weather model predictions fit actual data.

Simulating a Future Grid with Historic Forecasts

This study explores the impact of these forecasting limitations by modelling the least-cost dispatch of a potential Queensland system using historic market data.

• Generation and storage capacity modelled are taken from 2035 ISP results.
• Demand, wind and solar availability are based on historic data from the last financial year (2023/24) and scaled to match increased capacity.
• Dispatch is optimised based on archived demand and renewable generation forecasts.

This approach has three key advantages:

• Modelling the broader network provides insight into the system-level cost and reliability benefits of storage and generation assets.
• Using historic forecasts preserves the weather-driven relationships between renewable generation, demand, and uncertainty.
• Using historic forecasts also highlights the value of further improvements in renewable generation and demand forecasting.

As the aim is not to design a future grid, but instead to explore the impact of imperfect foresight on large-scale storages in Queensland, some simplifications were made:

• The Queensland network is modelled without interconnection to the NEM.
• Historic demand data is used, meaning the study does not account for how demand will evolve in future due to electrification and Consumer Energy Resources.
• Since current Queensland wind generation is concentrated in the Far North and Darling Downs, the historic wind data used may contain more extreme variability than would be expected from a more diverse mix.

A snapshot of the dispatch model is shown in the figure below. Note how early in the sample, wind and solar generation can meet all available load, while during a later low wind period, a mix of storage and gas generation ensures reliability.

QLD1 technology level dispatch using historic weather data from April 4^th – April 8^th, 2024. Negative values indicate charging load. Synchronous unit requirements not modelled.

Three Different Approaches to Uncertainty

Three models with varied levels of foresight and different approaches to uncertainty were tested.

1. A perfect foresight model. This model dispatches with full knowledge of outcomes for the entire simulated year, resulting in optimal system operation.
2. A deterministic model, which does not have full knowledge of future outcomes and instead optimises dispatch based on the best available information (central forecast) at each point in time.
   1. Each interval the model plans operation for the next seven days, locks in dispatch for the current period, and then steps forward and re-optimises for the next interval based on an updated forecast.
   2. The key weakness of this model is that it assumes the forecast values are correct and operates accordingly, without considering uncertainty.
3. A stochastic model, which uses probabilistic forecasts to capture a range of potential wind, solar, and demand outcomes.
   1. By minimising the expected dispatch cost across this range of scenarios, the stochastic model takes a risk-aware approach to decision-making.

A summary of all three approaches is shown below:

The three models discussed in this article, and how they are optimised.

Each model simulated dispatch for the full 2024 financial year. The perfect foresight model serves as a benchmark value, representing ideal system operation, while the deterministic and stochastic models illustrate different approaches to uncertainty.

Case Study: Storage Dispatch During a Wind Drought

Since the cost of forecasting errors is not symmetrical (underestimating scarcity is far more costly than overestimating it) imperfect foresight models must strike a balance between holding back a reserve and using stored energy efficiently. This trade-off becomes especially critical during renewable energy droughts, which can deplete storage levels, reducing the margin for forecast errors. These dynamics are best illustrated through how each model managed storage levels during a wind drought from June 2024.

The perfect foresight model enters the week with a high state of charge (65%), as it knows a difficult period is coming. During the evenings of the 3^rd and 4^th of June it discharges constantly to meet demand, reaching 0% stored energy on the morning of the 6^th of June. It then must navigate the remainder of the week with limited stored energy. As it has perfect foresight, it does so successfully, avoiding any unserved energy (USE).

Perfect foresight storage dispatch (June 01 – June 08). For clarity, storage levels for all Battery and Pumped Hydro units are aggregated.

In contrast, the deterministic model enters the week with a lower state of charge (30%) as the central forecast initially doesn’t predict the coming wind drought. Once it recognises the need for stored energy, on around midnight on the 2^nd of June, it begins charging.  Due to the depth of storage and lack of excess generation, this takes several days, and the deterministic model is only able to reach a storage level of around 85%. This lower storage level, combined with imperfect operation during the low reserve period at the end of the week leads to three periods of substantial USE, shown in red.

Similarly to the perfect foresight model, the stochastic model begins the sample period with a high state of charge (80%). This is not because it knows a difficult event will eventuate, but rather because a difficult event may eventuate, and it must be prepared. This risk-aware behaviour reduces the likelihood of unserved energy, as the model only fully depletes storage when necessary. However, it comes at a cost, as the model holds back energy even when it could be used immediately.

Despite this conservative approach, the stochastic model is also not immune to costly errors – on June 7th, a brief period of USE occurred as wind generation fell below the forecast range and not enough stored energy was available to respond. This was the only instance of USE within the stochastic model for the entire simulated year.

Summary of Results

Across the cases modelled, the cost of imperfect foresight (perfect foresight vs stochastic) ranged from 7-17%. This cost difference was largely driven by the minimal USE shown above, higher utilisation of peaking generators and more conservative storage operation. In contrast, the perfect foresight model maximised the value of storages and optimally dispatched cheaper but less flexible CCGT generation.

Adjusting the objective function to incentivise proactive charging, rather than simply minimising cost, eliminated USE in both imperfect foresight models. However, even with these adjustments, the deterministic model had 6-9% higher costs compared to the stochastic model. Without proactive charging behaviour, the deterministic saw much higher cost impacts. This shows clear value in accounting for uncertainty, as well as in the continued use of probabilistic forecasts within the NEM.

Further sensitivity analysis showed that while additional firming capacity, such as peaking plant or long-duration storage, reduced dispatch costs, this effect was much stronger in the stochastic model. This is because additional firming reduces the chance of USE, allowing the conservative stochastic model to minimise the cost of generation rather than holding back a reserve to manage reliability risks. This suggests perfect foresight and deterministic models, which do not account for uncertainty, may underestimate the value of firming assets.

Conclusion

As levels of renewable energy and energy storage within the network continue to grow, energy market modelling will need to evolve with it. Modelling a highly renewable Queensland system, we found that imperfect foresight leads to higher dispatch costs. Accounting for the impact of uncertainty also highlighted the benefits of flexible firming assets such as gas and longer-duration storage. Additionally, stochastic methods, and other formulations (such as robust optimisation) which account for uncertainty and/or risk, have clear potential to reduce costs and improve reliability.

Sample demand forecasts developed using set scaling factors (top) and ensemble models (bottom). Ensemble model plot sourced from this presentation from AEMO.

While this work has shown a clear effect, further work must be done to fully quantify the impact of imperfect foresight. Modelling transmission constraints and losses, interconnection with the NEM, and assessing the impact of future demand uncertainty would all provide further insights. Stochastic and other risk-aware models could also be improved by incorporating ensemble forecasts (like those above), which may better capture how uncertainty varies over time and space.

---

About our Guest Author

---