NEMDE under strain? Escalation in File Creation times for P5MIN

Over the past two weekends, we’ve become aware of a blow-out in file creation times in the ‘PUBLIC_P5MIN_*’ files by NEMDE at AEMO.

1)  Some of our clients have, also, and have been asking us questions.

2)  One client’s questions are worth noting:

(a)  This client is one of the 15 different Bidding Technology providers we identified in Appendix 22 of the recently released GenInsights21 report … some highlighted here)

(b)  They have let us know about problems with their auto-bidder process, which has been impacted by the (sometimes very late) arrival of P5MIN files from the AEMO.

3)  Worth noting, for completeness, that there were also some delays in publication of DISPATCH files at AEMO recently as well:

(a)  Significant in percentage terms compared to ‘normal’ expectations;

(b)  But not as great, in absolute terms, as what’s shown below.

So we’ll leave DISPATCH for its own article at some point, and have taken a bit more of a look at P5MIN …

(A)  Brief background

We’ve been aware of some issues here for a while, and have occasionally posted articles on WattClarity about this:

On 15th August 2021 we asked ‘Do you know what’s causing some increased latency on P5MIN data files on MarketNet in recent weeks (Potential Tripwire #4)?’  – in this article:

1)  looking back at the span of time from 15th May 2021 to Friday 13th August 2021; and

2)  noting some days in August showed maximum File Creation time of in excess of 180 seconds from the start of the dispatch interval.

2)  a delay which we noted would make it tight for participants to complete an effective rebid process in time before ‘Gate Closure’ on the rebid prior to the start of the next dispatch interval.

… seeing the scale of these delays it became obvious why it was not possible to extend the time horizon for P5MIN beyond the 12 x 5-min limit still used with the introduction of Five Minute Settlement … hence leaving the market to grapple with the challenges in Tripwire #1 and Tripwire #2 in the 4.5 months now since the commencement of 5MS.

On 16th November 2021 we returned with ‘Ongoing file creation delays in P5 predispatch data at AEMO’ to highlight what had been the experienced in the period 8th November 2021 to 15th November 2021:

… in that week, the deviations from ‘normal’ were not quite reaching as high as 180 seconds.

Then on 4th December 2021 we had some good news to share, with ‘Two recent improvements (in late November 2021) by AEMO in the dispatch process’, with the improvements being:

1)  Improvements in the file creation time from ~16:00 on Thursday 23rd November 2021  – particularly:

(a)  A 5-6 second step change improvement (i.e. reduction) in the ‘minimum’ times; and

(b)  it’s also worth nothing that in the time window shown through until 3rd December, the ‘worst’ excess was under 120 seconds.

2)  That article also mentioned a reduction in the Gate Closure on rebids.

(B)  What’s been happening since

Prompted by our observations that the latency had seemed to be getting worse in recent weeks, and spurred further by some client questions, we extracted some updated data and present the following charts:

The raw data shown above spans part-way through Thursday 16th December 2021 to part-way through Monday 14th February 2022.  There’s clearly been some pronounced spikes in latency  … in particular, well in excess of 180 seconds over Sunday 13th February 2022 – with the most recent weekend being as bad as the examples shown from August 2021).

I also wondered whether my eyesight was ageing quicker than I had hoped, as it seemed to me that the minimum latency points in each cycle seemed to be trending upwards as well…

(B1)  Trended Daily Statistics

As with the trend in August 2021 we’ve taken the time to compile the stats on a trended daily basis here (lopping off 16th December and 14th February as incomplete days), and for particular client context have labelled the two most recent weekend excursions with the Case Numbers we tagged in conversation with our clients:

In this chart, we’ve also included the data labels for the daily minimum … which do confirm that the minimum daily point has increased by as much as 8 seconds:

1)  from lows of 34 seconds in mid-December;

2)  To now reaching 42 seconds in mid-February.

So it seems that the gains noted in December 2021 might have disappeared!

(B2)  Time of Day Statistics

Flipping the data to look by time of day, the chart is a bit of a spaghetti mess – but it does clearly show that (apart from some outliers, which I have identified for future reference) almost all of the extreme periods have been occurring in daylight hours.

Distilling this ‘time of day’ data into similar statistics, this chart makes it clearer what the situation has been recently:

It’s clearer to see here that the extreme periods are skewed more towards morning daylight hours than afternoon daylight hours.

(B3)  Speculation about the possible cause

We understand that AEMO is conducting its own investigations into these types of issues … but that does not stop us scratching our heads to wonder what’s going on – and (perhaps more importantly) what it means for the future.

1)  Clearly there are times when NEMDE is struggling to produce a feasible solution for the P5MIN forecasts.

2)  There might have been some gains made in December 2021 – but these seem to have evaporated.

———

Readers should be very clear that what I am doing now is purely speculation … and may well be wildly incorrect!

———

3)  We wonder …

(a)  it seems one of the most complex things NEMDE needs to do is co-optimise across the hundreds of constraint equations used in dispatching the market (with slightly different forms of these constraints used in P5MIN and P30 predispatch).

(b)  so we speculate the source of the latency might be to do with constraint equations?!

4)  If this is the case (i.e. if constraints are the culprit) then we wonder why the problems occur most commonly during daylight periods (particularly mornings) and occasionally overnight?

(a)  Clearly sunshine – and hence solar output – is coincident here.

i.  But before those at either extreme of the Emotion-o-Meter arc up at thinking ‘solar’s to blame!’ or ‘WattClarity is blaming solar!’, keep in mind that…

ii.  Correlation is not necessarily causation!

(b)  We wonder, if solar is somehow actually involved, how would that happen?  …

(c)  … and then we think about a new range of constraint equations recently introduced into the market to address the challenges of Large Solar (and also Wind) connecting into ‘weak’ parts of the network…

(d)  … with the ‘X5 constraint’ (which Allan described in some detail on 23rd November 2020) being one of those.

(e)  Perhaps on days with high solar output, the solve time is affected?

(f)  If this is the case, then perhaps it is occasions of low demand (e.g. on weekends – like seen on 12/13 February, and 5/6 February beforehand) that further exacerbate solve times?

… or, alternatively, we may be well off the mark!

We will certainly look forward to understanding more about what’s actually involved here, and how it’s going to be resolved (and what market participants might do about it in the meantime)!