Piecing together what happened on Sunday 24th January 2021 with the ‘Q>>NIL_CLWU_RGLC’ constraint equation (Part 3)

Summer 2020-21 was far less eventful than prior summers – though not totally quiet. For instance, Sunday 24th January 2021 was an interesting day because of a couple discrete incidents – including two that were of particular interest to me:

Incident #1) There was a price spike in the the 16:35 (to $2,173.06/MWh) and 16:40 (to $15,000/MWh) dispatch price for the QLD region.

1a) It wasn’t the only price spike in QLD in recent times … but it was one of particular interest

1b) As noted here in the first article for the day, we determined that the ‘Q>>NIL_CLWU_RGLC’ constraint equation rapidly reduced the output of many generators in QLD of varied fuel types. The resultant drop in supply meant that prices could only go one way – up, and quickly!

1c) What made this even more interesting was that this happened through the period where the AEMO experienced a total SCADA Failure, which will have impacted on their operations in various ways.

Incident #2) Later the same day, there were also stresses in the SA region, as discussed in this 2nd article for the day…. however that’s not the subject of this particular article.

On 15th February 2021 the AEMO published a ‘Preliminary Report – Total loss of SCADA systems on 24th January 2021’, in which they communicated a number of things – including that the SCADA outage lasted from 15:46 through until 16:56 (both NEM Time). With use of that AEMO Preliminary Report, I’ve attempted to advance our analysis to the point where I can post this article as a sort of ‘Part 3’ in a sporadic investigation into what happened

WARNING: There may well be errors in the analysis behind this article, given the complexities involved (including the difficulty for me in ascertaining whether InitialMW numbers are actually metered or just assumed at various points through the sequence of events). If you do find errors, it would be great if you could let us all know in comments below the article!

(A) Why do we invest in completing investigations like these?

There are a number of reasons why we engage in forensic analysis like this particular one – including these two big reasons. This event, in particular, has already revealed a couple of things we can do in our ez2view software to continue to enhance its capability – which we’ll look forward to delivering as part of the move towards version 9.1 and then into the future…

Of course, if you could help us in these ways (or know of people who can help us) please let us know?

(B) High-level sequence of events (a current understanding)

In the following table I have presented our current understanding of a high-level sequence of results for a couple of hours spanning the price spike and disruption to SCADA feed:

Click on the image (or here) for a larger – and more readable – view of this table.

As we continue through the article, I will be seeking to explain some of what is noted in this table. With respect to the two columns based on the 4-second SCADA data, refer to this prior (i.e. ‘Part 2’) article to see graphically how the SCADA:

(1) Did not disappear completely;

(2) And indeed seems to have had problems for a longer time range than the AEMO indicated in their Preliminary Report .

… this is just one of a number of (confusing) factors that has made it difficult to try to understand what actually happened on this afternoon.

In this article we outline our current understanding in the hope that the more learned readers can help us understand things we might be missing (or errors we have made).

(C) About the ‘Q>>NIL_CLWU_RGLC’ constraint equation

In the table above we see that it was the 16:40 dispatch interval that saw the dispatch price spike (to $15,000/MWh, the current Market Price Cap).

It also saw the ‘Q>>NIL_CLWU_RGLC’ constraint equation violate, with a negative headroom (-170MW) because the LHS ended up greater than the RHS. Much of this article pertains to what happened to this constraint equation, as it was the one that ‘caused’ the price spike – hence let’s start by viewing some aspects of this constraint equation in the 16:40 dispatch interval using the very useful ‘Time-Travel’ function and a combination of Widgets in our ez2view software:

(C1) A quick reminder about the form of constraint equations

As noted on the Glossary Page, on most occasions (with FCAS constraints being a notable exception, but not the only one) the form of the constraint is as follows:

LHS ≤ RHS

This is the case for most constraint equations (with a notable exception being FCAS constraints, along with several others).

—

This particular constraint equation is for the purpose of ‘avoiding overload of the Raglan to Larcom Creek (8875) on trip of Calvale to Wurdong (871) line’ (these lines are all in Central Queensland). It’s a member of a ‘System Normal’ constraint set, so always invoked.

(C2) The LHS of the ‘Q>>NIL_CLWU_RGLC’ constraint equation

Because NEMDE is a linear program (LP), the Left-Hand-Side (i.e. LHS) must be a linear expression. This means that it must be of the following form:

LHS = Factor1 x Var1 + Factor2 x Var2 + … FactorN x VarN

… and the Var1, Var2 … VarN are DUIDs and interconnector elements that NEMDE gives Dispatch Targets to.

The LHS is determined in the NEMDE run, and applies to the end of the dispatch interval. In other words, the Dispatch Targets are given to the DUIDs (i.e. Var1, Var2 … VarN) and interconnector elements to indicate where they are expected to be at the end of the dispatch interval.

—

For the 16:40 dispatch interval we have highlighted some of the relevant information in the 3-widget display here:

We can see the start of the LHS terms and factors in the window above. In the interests of completeness, here is the full list:

0.979 x Barcaldine GT (ENERGY)
+ 0.9998 x Townsville GT (Yabulu) unit 1 (ENERGY)
+ 0.9998 x Sun Metals solar farm (ENERGY)
+ 1 x Mt Emerald Wind Farm (ENERGY)
+ 0.1135 x Millmerran unit 1 (ENERGY)
+ 0.1135 x Millmerran unit 2 (ENERGY)
+ 0.9797 x Lilyvale Solar Farm (ENERGY)
+ 0.9999 x Barron Gorge hydro unit 1 (ENERGY)
+ 0.9999 x Barron Gorge hydro unit 2 (ENERGY)
+ 0.1385 x Braemar 1 GT unit 1 (ENERGY)
+ 0.1385 x Braemar 1 GT unit 2 (ENERGY)
+ 0.1385 x Braemar 1 GT unit 3 (ENERGY)
+ 0.1404 x Kogan Creek (ENERGY)
+ 0.1385 x Braemar 2 GT unit 5 (ENERGY)
+ 0.1385 x Braemar 2 GT unit 6 (ENERGY)
+ 0.1385 x Braemar 2 GT unit 7 (ENERGY)
+ 0.1377 x Darling Downs GT (ENERGY)
+ 0.1377 x Darling Downs solar farm (ENERGY)
+ 0.8309 x Callide B unit 1 (ENERGY)
+ 0.8309 x Callide B unit 2 (ENERGY)
+ 0.8309 x Callide C unit 3 (ENERGY)
+ 0.8309 x Callide C unit 4 (ENERGY)
+ 1 x Daydream Solar Farm (ENERGY)
+ 1 x Hayman Solar Farm (ENERGY)
+ 0.9995 x Clare Solar farm (ENERGY)
+ 0.1396 x Condamine CCGT (ENERGY)
+ 0.999 x Collinsville Solar Farm (ENERGY)
+ 0.1499 x Coopers Gap Wind Farm (ENERGY)
+ 0.9998 x Townsville GT (Yabulu) unit 2 (ENERGY)
+ 1 x Haughton Solar Farm (ENERGY)
+ 0.9999 x Kareeya hydro unit 1 (ENERGY)
+ 0.9999 x Kareeya hydro unit 2 (ENERGY)
+ 0.9999 x Kareeya hydro unit 3 (ENERGY)
+ 0.9999 x Kareeya hydro unit 4 (ENERGY)
+ 0.9702 x Emerald Solar Farm (ENERGY)
+ 0.9781 x Middlemount Solar farm (ENERGY)
+ 0.979 x Clermont Solar Farm (ENERGY)
+ 0.9996 x Mackay GT (ENERGY)
+ 0.995 x Rugby Run Solar Farm (ENERGY)
+ 0.09239 x Yarranlea Solar Farm (ENERGY)
+ 0.09239 x Maryrorough Solar Farm (ENERGY)
+ 0.09235 x Warwick Solar Farm 1 (ENERGY)
+ 0.09235 x Warwick Solar Farm 2 (ENERGY)
+ 0.9998 x Mt Stuart GT unit 1 (ENERGY)
+ 0.9998 x Mt Stuart GT unit 2 (ENERGY)
+ 0.9998 x Mt Stuart GT unit 3 (ENERGY)
+ 0.09239 x Oakey GT unit 1 (ENERGY)
+ 0.09239 x Oakey GT unit 2 (ENERGY)
+ 0.1396 x Roma GT unit 7 (ENERGY)
+ 0.1396 x Roma GT unit 8 (ENERGY)
+ 0.9998 x Kidston solar farm (ENERGY)
+ 0.9998 x Ross River Solar Farm (ENERGY)
+ 0.9993 x Hamilton solar farm (ENERGY)
+ 0.9993 x Whitsunday solar farm (ENERGY)
+ 0.9942 x Stanwell unit 1 (ENERGY)
+ 0.9942 x Stanwell unit 2 (ENERGY)
+ 0.9942 x Stanwell unit 3 (ENERGY)
+ 0.9942 x Stanwell unit 4 (ENERGY)
+ 0.09239 x Oakey 1 Solar Farm (ENERGY)
+ 0.09239 x Oakey 2 Solar Farm (ENERGY)
+ 0.1334 x Tarong North (ENERGY)
+ 0.1334 x Tarong unit 1 (ENERGY)
+ 0.1334 x Tarong unit 2 (ENERGY)
+ 0.1334 x Tarong unit 3 (ENERGY)
+ 0.1334 x Tarong unit 4 (ENERGY)
+ 0.1401 x Gangarri Solar Farm (ENERGY)
+ 0.1205 x MW flow north on the QNI AC Interconnector
-0.4937 x Childers Solar Farm (ENERGY)
-0.4722 x Susan River Solar Farm (ENERGY)
-0.8142 x Gladstone unit 1 (ENERGY)
-0.8142 x Gladstone unit 2 (ENERGY)
-0.8048 x Gladstone unit 3 (ENERGY)
-0.8048 x Gladstone unit 4 (ENERGY)
-0.8142 x Gladstone unit 5 (ENERGY)
-0.8142 x Gladstone unit 6 (ENERGY)

In particular note that:

1) most DUIDs (65 of 73) are associated by positive factors

(a) these are the ones that will be ‘constrained down’ when the constraint binds

(b) also important to note that some factors (e.g. for MEWF1, at 1.0) are ~10 times as large as for other DUIDs (e.g. 0.09239 for OAKEY1SF) – this impacts on which units are ‘constrained down’ first (assuming their bids are equal).

2) the dispatch target of QNI is also included … with a positive factor (i.e. the constraint will try to reduce flow northwards, or increase flow south).

3) there are 8 DUIDs with negative factors – which are the ones that this constraint will try to ‘constrain up’.

(a) These 8 x DUIDs are all south of the Raglan to Larcom Creek (8875) Line on the coastal route into Brisbane.

(b) It’s easy to see this collection in the ‘QLD Schematic’ within ez2view, as shown in the example further below.

(C3) The RHS of the ‘Q>>NIL_CLWU_RGLC’ constraint equation

The Right-Hand Side (RHS) reflects the things that generally NEMDE can’t control, but essentially treats as “Constant” for the given dispatch interval. The RHS is calculated at the start of the dispatch interval using meter readings of certain parameters taken at the start of the dispatch interval.

Flipping to the ‘Plain English Translation’ tab in the ez2view widget above we see the form of the RHS used in the dispatch time horizon (it’s a little different for P5 predispatch and P30 predispatch):

As annotated on the image, we can look at this RHS formulation as two groups of terms:

1) Group 1 are the ‘Feedback Terms’ which are the exact same (74 in total) terms and factors shown above for the LHS … with the difference being that (in the RHS) the metered InitialMW values are used for start of interval values.

2) Group 2 are the ‘Other Terms’ which are four terms in total, three of which (i.e. 2 x flows and 1 x dynamic line rating) are dynamically updated by Powerlink and supplied to AEMO using the interconnected SCADA system … with the 4th term being the constant (30).

For completeness, the ‘Other Terms’ are constituted as:

3.935 x (Qld: 8875 Raglan-Larcom Creek 275kV Emergency Rating

– MVA flow on 8875 275kV line at Raglan

– 0.4235 x [MVA on 871 275kV feeder from Calvale, Line end switched MW]

– 30 {Operating_Margin})

Annotating the AEMO Map, here’s some quick locations in central Queensland referenced in the RHS:

(D) Dispatch Intervals in sequence

With the above as background (and also keeping in mind we will look to drill in further in a Part 4 to the Case Study – time permitting), let’s start at the 15:45 dispatch interval and walk through the key dispatch intervals in some detail (considerable detail for the first few), to explore what (we currently believe) happened:

(D1) The 15:45 dispatch interval

This dispatch interval starts at 15:40 (or shortly beforehand) as the AEMO conducts a sweep of available SCADA data in order to ascertain the starting position of the DUIDs and network elements that it monitors. The starting position is recorded as ‘InitialMW’ for each DUID for the 15:45 dispatch interval:

These ‘start of interval’ values are then processed through the RHS, then (via NEMDE) to the LHS in order to establish Dispatch Targets for all 73 x DUIDs (plus QNI) for where they are expected to be at the 15:45, as shown above. The DUIDs strive to deliver their targets as indicated below:

… however as illustrated below, the AEMO’s 4-second SCADA data (used for determining cost allocation for Regulation FCAS using a ‘Causer Pays’ approach) contains a number of holes during this 15:45 dispatch interval:

Refer to the table at the top of this article to see that there were 80% of the expected data updates were missing in this 5-minute period (i.e. only 15 arrived). It appears* that this had some effect on the outcome for the next dispatch interval, as illustrated next…

* Very happy for those who are more knowledgeable than us to point out any error in this understanding!

(D2) The 15:50 dispatch interval

As we tick into the 15:50 dispatch interval, the AEMO tries to complete the same kind of ‘SCADA Sweep’ process to set InitialMW … but this does not appear to have worked.

Instead, out understanding is that the AEMO (following procedure) assumes that the InitialMW was equal to the prior Dispatch Target for each of the units:
(a) This was certainly the case for the 73 x DUIDs in this equation;
(b) In this instance it appears that QNI was also affected (i.e. the ‘Metered Flow’ is identical to the prior Target).
(c) I did not check all the other DUIDs around the NEM.

However what’s a bit baffling at this point is that it appears that the AEMO was able to update the ‘Other Terms’ in the constraint equation … specifically:

Network Data Point #1) The emergency rating for the 8875 Raglan-Larcom Creek 275kV line (which we understand is one of the ones dynamically supplied by Powerlink)

Network Data Point #2) The MVA on 8875 275kV line at Raglan

Network Data Point #3) The MVA on 871 275kV feeder from Calvale (Line end switched MW)

I say that ‘it appears that’ because the aggregate number produced for ‘Other Terms’ here (i.e. 611MW):
(a) was similar to the number for 15:45 … and
(b) distinctly different from what happened at 15:55.

In the following image we see what happened within the interval, in terms of dispatch … with a couple questions flagged, about use of AGC (e.g. Regulation FCAS) and also functioning of AWEFS and ASEFS forecasts (and Targets delivered via SCADA to Wind Farms and Solar Farms).

Through this Dispatch interval, there were 79% of expected 4 second readings missing (i.e. just 1 fewer than the prior dispatch interval):

(D3) The 15:55 dispatch interval

In the 15:55 dispatch interval, there are more things that we don’t really understand, as highlighted:

As noted here, we see something different than for the prior dispatch interval:

1) We see that the DUID Initial MW values (and QNI flow) are different from the prior Dispatch Target … suggesting that the ‘SCADA Sweep’ was able to work and hence Aggregate ‘Feedback Terms’ appears real;

2) However something quite odd appears to have happened with the Aggregate ‘Other Terms’ as this now calculates to be –239MW (which is a net change of –840MW on what it had been for the prior dispatch interval).

The net effect of this is that it drives the RHS down by 790MW which is a large amount – which binds the ‘Q>>NIL_CLWU_RGLC’ constraint equation.

In terms of price outcomes, the dispatch price spikes to $277.89/MWh. However this snapshot from ez2view at the 15:55 dispatch interval shows that the elevated price was seen in other regions, and that the price outcome in QLD was due to a Snowy Hydro unit setting the price (mostly).

We’ll look at some of these further (time permitting) in the up-coming Part 4 of this Case Study.

(D4) The 16:00 dispatch interval

Dispatch price drops to $60.80/MWh as the constraint unbinds.

It appears that this occurs coincident with (perhaps partly because of?) whatever was going awry with SCADA and jamming the ‘Other Terms’ collectively at –239MW has been alleviated, raising that figure to a more ‘normal’ level of 566MW.

(D5) The 16:05 dispatch interval

The dispatch price remains at ‘normal’ level – at $55.68/MWh.

The Aggregate ‘Other Terms’ calculates out to be +565.51MW … so more like what it had been before 15:55 (but this is the last time it happens for a while).

(D6) The 16:10 dispatch interval

Dispatch price climbs to $100.00/MWh.

We see the Aggregate ‘Other Terms’ is again jammed down at –238.9MW (i.e. exactly the same as 15:55 – and also the same as subsequent dispatch intervals). Because of this subtraction from the RHS calculation, the RHS value drops and the constraint is bound (Marginal Value is only –65.01, a relatively minor amount).