Do you know what’s causing some increased latency on P5MIN data files on MarketNet in recent weeks (Potential Tripwire #4)?

It’s Sunday 15th August 2021, and Five Minute Settlement is now less than 7 weeks away.

Today’s article is not the usual type of ‘observations of market outcomes’ type of article that we focus on here on WattClarity®, but it’s still quite relevant to our readers – and will become increasingly important as we move to shorter trading periods, and it relates to the data files of the form ‘PUBLIC_P5MIN_*.csv’ published by AEMO on MarketNet.

For those not so familiar – these files are related to the P5 Predispatch process which updates every 5 minutes, and provides a useful forecast of price (and dispatch target, for participants) for the next 11 dispatch intervals following the current dispatch interval.

With respect to these files, in recent weeks we’ve seen the File Creation time blow out quite frequently, as summarised in the following trended daily stats:

We see that the minimum file creation time each day is typically 30-40 seconds into the start of the dispatch interval (remember there are 288 of these files created each day – one every 5 minutes), and that most times the maximum latency might be 60 seconds or less.  However in recent weeks it has been significantly more, on a significant number of days.

Note that these stats are just for the creation of the file, and don’t take account of an additional lag in the file being made available on each participant file share on MarketNet.

Once the file is made available on MarketNet, there are additional steps required to make any useful decision in relation to these new pieces of data.  Here’s one example of what might follow with indicative timing (but keep in mind that there are other possible decisions that might be made … such as for spot exposed energy users, or solar generators, to switch off if the price forecast is not to their liking):

With dispatch intervals (and the new trading periods) being only 5 minutes long, and with participants needing to make material decisions within each of these dispatch intervals, we can see that some of these lags in recent weeks might have made it next-to-impossible-to-respond within the timeframe required to be considered for the next dispatch interval.

We wonder if any of our readers (especially at AEMO) can help us understand what’s happening here, and whether this is just something transient or if it’s going to persist into October?

—————-

For comparison purposes, here’s the File Creation Latency for the ‘PUBLIC_DISPATCHIS_*.csv’ files we’ve scanned from one of** the MarketNet feeds we maintain on behalf of various clients (i.e. the ones that contain the actual dispatch prices – as distinct from the P5 forecasts).

Note (**) that we assist a number of clients with respect to these data files – and in different ways – but that the File Creation latency is always the same, as this is a central function at the AEMO.

There’s clearly a more intensive (and slower) process involved in producing the P5 Predispatch forecasts at the AEMO, which helps to illustrate why there was hesitation early in the Five Minute Settlement transition process to extend the P5 forecast horizon past 11 dispatch intervals – despite the fact that this is ultimately what’s required to address the challenges in Potential Tripwire #1 and Potential Tripwire #2.

—————-

PS1 on Monday 16th August 2021

We thought it was worth highlighting one other factor relating to P5 Predispatch that might be related to the increased latency seen above – and we’d be keen to hear back from those who do know if they can confirm this for us, please?

1)  As part of Five Minute Settlement, the AEMO is introducing some sensitivities around the ‘base case’ demand forecast underlying P5 predispatch

2)  From Wednesday 7th July 2021 these data sets began to be produced in AEMO’s Production Systems.

3)  However this process was stopped on Friday 23rd July 2021 due to a bug uncovered in the process – as described in Market Notice 88762 at the time:

________________________________________________________________________________________________
Notice ID            88762
Notice Type ID        Status of Market Systems
Notice Type Description    MARKET
Issue Date            Friday, 23 July 2021
External Reference        CHG0067492 – Fault Remediation
________________________________________________________________________________________________

 Change Number:  CHG0067492

Notification issued to:  Market Notice

Notification Type:  Initial

Change Type: Normal

Service/ Component: EMMS – Dispatch

Change Title:   Fault Remediation

Change Start Date & Time:  23/07/2021 15:00:00 AEST

Change End Date & Time:  23/07/2021 16:00:00 AEST

Change Description:

Fault Remediation

Environment:  Production

Participant Impact and Requirements:

Participants are advised that AEMO has discovered an error in the formulation for 5-minute pre-dispatch (5MPD) scenarios. The scenario offsets that were intended to be applied at the start of each 5MPD scenario have instead been applied cumulatively to each interval over the forecast horizon. AEMO will cease publishing 5MPD scenarios until this error is remedied.

AEMO is currently remediating this as a priority, but the necessary re-certification of the NEMDE formulation and software, followed by a period in pre-production, means that 5MPD scenarios are unlikely to be restored to production for several weeks. AEMO apologises for this inconvenience.

Publication of the 5MPD base case, including fast start inflexibility profiles, is unaffected by this change.

Support Information:
Report any impacts to the AEMO Information and Support Hub or on External 1300 236 600.

4)  I would like to stress, here, that the bug found is related to an error in the way in which the calculation was being performed, and the suspension of publication was not related to any increased latency issues (which might be entirely unrelated).

5)  To highlight that these two events (both related to P5 Predispatch) don’t really seem well correlated, see here:

6)  The sensitivities are still offline and being corrected:

6a)  Envisaged to be back in Pre-Production 25th August 2021

6b)  Envisaged to be back in Production late in September (i.e. just before commencement of Five Minute Settlement)

—————-

PS2 on Monday 16th August 2021

We understand AEMO is investigating this – and will look forward to hearing further (and sharing here).