It seems like ages ago now, but Sunday 24th January 2021 was an interesting day because of a couple discrete incidents – including two that were of particular interest to me
Incident #1) There was a spike in the the 16:35 (to $2,173.06/MWh) and 16:40 (to $15,000/MWh) dispatch price for the QLD region.
1a) As noted here in the first article for the day, we determined that the ‘Q>>NIL_CLWU_RGLC’ constraint equation rapidly reduced the output of many generators in QLD of varied fuel types. The resultant drop in supply meant that prices could only go one way – up, and quickly!
1b) What made this even more interesting was that this happened through the period where the AEMO experienced a total SCADA Failure, which will have impacted on their operations in various ways.
Incident #2) Later the same day, there were also stresses in the SA region, as discussed in this 2nd article for the day…. however that’s not the subject of this particular article.
Since that time I have been trying to steal some time (here and there) to piece together some analysis that would help me understand how that constraint equation actually worked in the absence of any SCADA data feed, which has been a challenge (especially because of only small amounts of time here-and-there, and plenty of other distractions). However I have been getting closer to a clearer picture!
Thankfully, the AEMO made my job easier when (on 15th February 2021) they published this ‘Preliminary Report – Total loss of SCADA systems on 24th January 2021’:
It does not directly answer my question – but it does help to clarify some of the key timings, which I have copied into the table below for ease of reference.
(A) Sequence of Events
With the benefit of this report, and a few other things published by AEMO, my understanding of the sequence of events is as follows:
| Time of Event/Action on Sun 24th Jan 2021 |
What happened? |
| 15:46 |
The report says ‘From approximately 1546 hrs on 24 January, AEMO’s internal Supervisory Control and Data Acquisition (SCADA) service failed ’ |
| Time Unknown?
(perhaps before the Market Notices?)
|
The report says: ‘In accordance with AEMO’s internal procedures: I presume this would have taken some time to call around all affected parties individually? See comments at (C) below of a few things I would like to know more about… |
| 16:30:20 (approx)
(i.e. 44 minutes after SCADA was lost) |
NEMDE publishes NEMDE outcomes for the 16:35 dispatch interval – which (of interest to me) included the price in QLD spiking to $2,173.06/MWh. |
| 16:33
(i.e. 47 minutes after SCADA was lost) |
Market Notice 82330 was issued, to advise the market of the SCADA issues: _________________________________________________ Notice ID : 82330 AEMO is currently experiencing failure of SCADA in all regions. AEMO is investigating the failure. Manager NEM Real Time Operations The 47 minutes in which the SCADA was lost but the Market Notice had not been issued is of interest to me, in terms of seeing what type of responses participants might have been making if they had been unaware of the issues. |
| 16:35:20 (approx)
(i.e. 49 minutes after SCADA was lost) |
NEMDE publishes NEMDE outcomes for the 16:40 dispatch interval – which (of interest to me) included the price in QLD spiking to $15,000/MWh |
| 16:56
(i.e. 70 minutes after SCADA was lost |
The report says ‘Normal operation of SCADA was restored by 1656 hrs on 24 January 2021, and the relevant generating units were returned to normal AGC operation’ |
| 17:08
(i.e. 82 minutes after SCADA was lost |
Market Notice 82340 was issued, to advise the market the SCADA issue had been resolved: __________________________________________________ Notice ID : 82340 Refer AEMO Electricity Market Notice 82330 AEMO SCADA service has been returned to normal in all regions at 1646 hrs. Manager NEM Real Time Operations |
I’ve been using the above to try to piece together what happened, in particular, with the ‘Q>>NIL_CLWU_RGLC’ constraint equation on the day.
(B) Review of the 4-second data
In order to understand what’s happened, we have extracted some of the AEMO’s 4-second data that we have been accumulating for some time now…. however even that is delivering some further questions…
In the following chart, we plotted the number of seconds between consecutive timestamps through the period of the afternoon that’s of interest:
So long as I have interpreted this correctly (and that’s a big caveat!) it would appear that:
1) The problems began to occur some time before the AEMO notes as the start time (i.e. 15:46)
2) The problems appear to have been resolved by 16:41 (i.e. 15 minutes prior to the 16:56 noted by the AEMO), at least in terms of this data I’m looking at; and
3) Between start and end the problem appears to be intermittent:
(a) There is some sporadic continuation in the interim period
(b) The worst delay were 3 instances of approximately 7 minutes (i.e. 420 seconds) between timestamps.
I’m taking the above into consideration in my exploration of the data, so it would be really great if someone who knows more about this data than me could let me know if I have misinterpreted?
(C) Possible implications?
My particular interest is the ‘Q>>NIL_CLWU_RGLC’ constraint equation through the afternoon – however it’s occurred to me that there are other questions I am curious about as well:
| Question? |
Why am I interested? |
| About AGC, and provision of Regulation FCAS |
Q1) What happened with the provision of regulation FCAS? I assume that this meant that no signals were provided for raise/lower … in which case frequency would have wandered (except to the extent that PFR helped)? Q2) What happened to AGC signals? I presume that signals stopped for those generators – so did the DUIDs just sit still, or did they revert to following targets in their EMMS? The AEMO reports notes that generators were instructed to disconnect AGC. How was this done? I did not see anything in a market notice about this … but I might have missed it? Was it by phone call? |
| How were AWEFS/ASEFS affected? |
Over recent years, as the number of Semi-Scheduled generators we serve (primarily with our ez2view software) has continued to grow, we’ve been consciously trying to learn all that we can about the detailed nuances of the way this particular category worked, in order that we can help our clients better. It certainly helped us to have Marcelle join almost exactly a year ago, and then Linton in August 2020. As noted on 30th December 2020, we also recently invested some seed funding to help start a new start-up business, Overwatch Energy Pty Ltd, which has a particular focus on serving these types of generators in a way that complements what’s possible with our software. As such, I’m really interested to ensure I understand the specifics of how this SCADA outage will have affected these Semi-Scheduled plant: Q1) Was their any disruption to ASEFS and AWEFS forecasts due to the SCADA issues; Q2) My understanding is that a number of Wind Farms and Solar Farms receive their dispatch targets (and SDC) via SCADA link to plant, over and above the MarketNet feed into their EMMS. 2a) Hence it seems plausible that there would have been some of these also not following targets, if relying on automated operations from the SCADA link. 2b) Am I correct in this hypothesis? |
The AEMO’s preliminary report says
‘AEMO intends to further investigate and report on:
• The cause of the SCADA failure and the immediate actions taken to resolve the issue.
• Any further changes that may be required to prevent software bugs impacting AEMO’s SCADA systems.
• The impact on the power system and whether the power system remained in a secure operating state.
• The impact on the market, including a review of pricing outcomes, the impact on the demand forecast process, the impact on constraint equations, and the performance of the 5-minute forecasts for renewable generation.
• The impact of primary frequency response on power system frequency performance.
• The provision of information to the market.
• The application of procedures associated with market suspension.
AEMO will include any additional findings in its final report on this event.’
… so, in a way, I guess I am now in a race with the AEMO to see how much we can explore, on our own first.
(C1) Why do we conduct these investigations?
More seriously, though, it might be worthwhile reiterating that we are happy to invest a non-trivial amount of time into investigations like these because (apart from just satisfying our burning natural curiosity) they help to identify how we can serve out clients better in future.
We’ve found that (time permitting) it’s definitely better to be buried in the data ourselves, rather than just reading someone else’s report. More specific details are here.
(C2) Do you know of people who can help us?
Let us know if you can help us identify others who can work with us (internally, or externally) to help us do this better in the future?
Leave a comment