Adding a Slice of Cheese


The South Australian Blackout of 28th September 2016 is not as simple as it looks.

All engineering failures typically have multiple causes.  As the cheddar cheese metaphor implies, it is often the case that a series of events all line up (like the holes in a block of cheese) to cause a major disaster like a state wide black out.  At the time or immediately after the event it is often unclear as to what happened and when until a detailed analysis is done.  At the time of writing, AEMO have produced four reports on the event (now linked individually here for ease of reference).

This article highlights just one issue relating to the Blackout event.

Arguably it may not be as important compared to some of the other issues that occurred on the day, but in terms of its overall importance to the power system in general, I believe it deserves a lot more attention than it has been given to date.

The issue is “poor system frequency control” caused by inappropriate generator governor settings.

It is arguable that better frequency control could have prevented the SA blackout of 2016.

The final report produced by AEMO (press release here) contains the following statements pertaining to the most important synchronous generators that were on line that day – three units from Torrens Island Power station.

  • The three TIPS B generating units were each enabled to provide 15 MW of 6-second contingency FCAS prior to the event. However, the frequency in SA prior to separation was only just below the level where FCAS delivery is required to begin, until after the loss of the Heywood Interconnector at 16:18:15.8
  • The rapid decline in system frequency following loss of the Heywood Interconnector did not allow time for more substantial governor response from these units, as it can take up to six seconds for these generating units to increase their active power output when they participate in the contingency FCAS market.

It is the position of this article that these points should be examined and expanded on much more closely than they have been to date; not necessarily because of the events that occurred on the 28th September 2016, but because frequency control across the whole NEM is very important and has been very poorly served by the existing FCAS arrangements.

Hypothetically if good governor control had been enabled in SA that day, it is possible that the load increase across the Heywood Interconnector would have been mitigated, and this may have been enough to prevent the trip of the interconnector and subsequent collapse and blackout.   It is also possible that (assuming the Heywood interconnector does trip) fast acting governor control may have slowed the rate of frequency decline and allowed time for load shedding systems to operate, thus avoiding total system black.

Frequency Control

As both scenarios are entirely hypothetical we will never know if either intervention would have been enough, but that is not the issue.  The key issue is that poor frequency control across the NEM has been allowed to develop. If good frequency control had been available it might have helped or even saved the day which would have been the case if generator governors had been active and properly tuned.  This also applies for any possible future events.

The paper presented by Ms K. Summers at the Wind energy Forum of 2017 recently organised by the Clean Energy Council has demonstrated that the following has occurred:

  • Synchronous generation has progressively detuned its governor control systems, increased governor dead bands or even switched off speed governing altogether in order to avoid the possibility of dispatch errors and incur a possible financial penalty.
  • Reliance has increased on Area grid control to regulate system frequency which is (by design) a much slower control loop compared to local speed droop governing. This combined with Energy management system  (EMS) communication and processing delays have led to control system hunting behaviour often  resulting in  continuously slowly oscillating system frequency which can be easily confirmed by viewing the FACS causer pays four second data.
  • The FCAS market has sometimes led to market regions being left with no frequency control enabled, most notably in South Australia immediately prior to the blackout event of 2016, but also (more worryingly for the whole system  security) for the major load and generation centre of NSW.

The control of frequency is very important to the NEM as a whole. Anything (FACS markets included) which loosens this control puts the whole system at risk.

We would support a comprehensive review of the FCAS market to investigate whether an overhaul would address the undesirable outcomes identified above. However, given that the financial value of FCAS markets is small compared to that of the energy markets, we would question whether even a perfectly-designed FCAS market would address and reduce the cost impact of infrequent, but high cost system disturbances such as (in the worst case) a system wide black out.

These risks to the power system are obvious to power system engineers but we appreciate that this may not be the case for the general public, or even for engineering professionals whose expertise lies in other fields.  As a result we have to fall back on analogies to explain the issues.

Imagine if you had a situation which provided incentives for car manufacturers to disconnect the accelerator pedal and the brakes.  For this situation to be practical everything would be fine as long as the car is always able to roll along at constant speed, but what happens if you need to change speed by hitting the brakes or accelerating? Without brakes or accelerator your risk of crashing the vehicle is very much higher.

In effect this is the situation the FCAS and energy markets are jointly incentivising.  Most generators on the system avoid contributing to the control of system frequency so that when an event occurs the whole system is put at greater risk.  Region wide black outs are more likely to be the end result of system events which would normally only result in localised loss of supply.

It can be argued that all we need to do is modify the FCAS markets to ensure the incentives are there for generators to provide appropriate governing support, but in practice this interferes with the main energy market, and nearly two decades of market design has failed to resolve these issues.

A Simpler Approach

There are currently rule changes under consideration which aim to introduce new FCAS markets (inertia and fast frequency response) which are aimed at addressing these issues. These appear to add complexity to an already overly complex market design and recent history shows that complex markets are unlikely to lead to desirable outcomes.

Accordingly it may be preferable to consider a much simpler approach.  Why don’t we ensure (via appropriate rules and technical standards) that all new generation technologies contribute to frequency control and adjust the market to remove penalties that currently exist for generators when they act to provide  good frequency control?  This would be fair to all generation participants and result in much better system control which would ultimately be of benefit to all users of the system.

This general approach is already adopted for reactive power and voltage control, so a precedent already exists.

The prevailing dogma argues that an FCAS market will deliver an efficient outcome for frequency control. As detailed in the paper presented at the wind industry forum of 2017, it is obvious that nearly two decades of FCAS markets has so far achieved the opposite outcome. An annual cost of several million dollars allocated to FCAS services (and related relaxation of standards) has actually resulted in worse frequency regulation than existed prior to the introduction of the NEM.

To revert back to analogies, you may get a slightly cheaper car if you buy one without brakes, but it would be the pedestrian you mow down who pays the ultimate price of this dogmatic acquiescence to current economic fashions which seem to dictate you need a market for everything.  Security – be it for power systems or for general police forces don’t need to be directly marketable. They simply provide a well maintained background environment in which a market can operate.

A well maintained and secure system is needed to allow energy to be traded across the market. If the ancillary services are not there this cannot occur, but it is not necessary and we believe recent history shows may not even be desirable that the ancillary services themselves be procured via market mechanisms – even if these mechanisms were perfect which for the NEM they demonstrably are not.

In the context of the NEM, it is not individual pedestrians at risk, but whole regions, or even the whole east coast of Australia exposed to the increased risk of blackout –consequential damage to equipment (significantly  this would include generators in particular) and to the population at large.

About our Guest Author


Bruce Miller is an Electrical Engineer from a High Voltage Power systems background. He has experience in design, commissioning and project management and the development of mathematical and software tools.

He currently manages a team of highly skilled engineering and software specialists who carry out planning, technical and commercial studies for clients to assist them in obtaining low cost, clean energy which is safe and reliable.

You can find Bruce on LinkedIn here.

6 Comments on "Adding a Slice of Cheese"

  1. “Why don’t we ensure (via appropriate rules and technical standards) that all new generation technologies contribute to frequency control and adjust the market to remove penalties that currently exist for generators when they act to provide good frequency control?”

    You mean like also ruling that no generator can tender anymore power to the grid than they can reasonably (ie short of mechanical breakdown) guarantee 24/7, 365 days a year for a level playing field market? Now you’re talking but who’s going to point out to our current overlords that it’s a fundamental tenet of engineering that you can’t build a reliable system from unreliable componentry? Unfortunately we just have to wait for the train wreck for them to finally comprehend a national power grid doesn’t run on emotion.

    • I am hopeful that a well designed system will emerge from the current unsatisfactory situation, and that a reliable system will be built from reliable componentry – but only if people pay more attention to the physics and a bit less attention to the fashionable economic theories of the day.

      • Well in defence of market economics I’d say it can provide some useful tools for the greatest good for the greatest number but that will depend on the underlying assumptions.

        For example it may be the science of Catastrophic Anthropogenic Global Warming alerts the economic practitioner to some social cost that is not being reflected in the much lower private cost of fossil fuel use and there’s a simple remedy. Just call the guru who knows the additional price that needs to be added in order to bring the level of CO2 in the atmosphere back to the level said guru knows it should be. Having imposed that on all players in say the business of producing reliable, 240V, 50Hz electricity as required by consumers is then another set of issues which we would probably agree needs more physics rather than political interference. The economist would say set those basic level playing field parameters and the various players will then come up with the economic solutions for us.

        But back to the CO2 assumptions and that guru’s estimate of social cost. With only around 150 yrs of reliable thermometer records our guru has had to rely on such temperature proxies as tree rings, ice cores and varves along with a host of computer models to form an estimate of CAGW and there is some debate about the scientific veracity of all that. However there is one measure of global temperature that clearly rules them all and that’s sea level rise since sea level rises with temperature for only two reasons. Thermal expansion of oceans or runoff from ice melt on land and the polar caps.

        Well you wouldn’t want to rely on a tide gauge in Venice or the side of a volcano in Hawaii but here in our own stable ancient weathered continent we have a couple of very reliable sea level gauges. Take the one at Port Arthur(one of the longest man made in the world but certainly in the SH) and it shows us 13.5cm sea level rise from 1841 to 2000 or an average sea level rise of 0.85mm/year. But there is another at Hallett Cove Conservation Park in SA (so proclaimed by none other than our Premier Jay Weatherll as Environment Minister in 2010 because of its world geological significance) and that shows there was an ice age 20000 years ago but around 15000 years ago the ice began to melt and the sea level off SA rose 130 metres until 6-7000 years ago, which could be an average increase of 16.25mm/yr for 8000 years.

        So what conclusion did I logically draw from that? There are either some very very clever scientists and their computer models that can extract the anthropogenic CO2 signal from the current Port Arthur temperature proxy, or else aboriginal cooking fires and traditional burnoffs to flush out game have an awful lot to answer for with that Hallett Cove temperature proxy. Now where were we? Oh yes, economics and the underlying assumptions.

        • I agree with you that market economics can provide useful tools which can be used to benefit society as a whole. I will leave it to others to determine for themselves whether they believe it has actually been applied to this end.

          My main point is that some things can easily be traded – e.g. commodities like food, minerals, fuel and (specific to the point of this article) power. For these things designing markets is easy.

          However, there are more esoteric, less easily defined benefits that derive from concepts that cannot be easily turned into commodities. In the power system world this consists of all types of ancillary services. These things are not easily expressed in a market paradigm.

          Markets that are set up to deliver these types of services easily lead to unforseen consequences which we have seen playing out in the frequency ancillary services, so that now we have worse frequency regulation and a less secure system than we did in 1990. This contributed (albeit only peripherally) to the first state wide black out we have had in approximately 50 years.

          With respect to your comments on climate change, I will just treat that as commentary. I am happy to reply to any questions related to Power systems which is my field of expertise and what the article is about. Climate change is not my field of expertise and therefore I suggest you refer such comments to a climate scientist.

          • Climate change is not my particular field either, but facts are when I go for a stroll through a Conservation Park and my interest is aroused as to just what are the broader facts of the matter. As a result I naturally continue to believe in climate change, just not CAGW anymore after previously reserving judgement on some worrying critiques of computer modelling of lesser proxy variables-
            Science and the scientific method don’t fit in neat little compartments or become the preserve of anyone. The facts speak for themselves until they’re contradicted or otherwise interpreted.

            Nevertheless particular fields of endeavour can require certain expert technical expertise to fully comprehend the bigger picture for the policymaker so with that in mind can I ask your thoughts on this interpretation of where we’re headed on the current path with the grid-

            There is a legitimate role for good Govt overseeing areas of market failure with externalities (the tragedy of the common and the like) but there is a sense with our national grid that no-one is in charge anymore with a history of incrementalism and the science of muddling through. Whatever level playing field social cost we put on CO2 and bearing equally on all inputs it should (concrete or steel for thermal or wind generators), that still leaves facilitating reliable voltage and frequency at the flick of a switch for consumers and policymakers need to understand and address the obvious shortcomings now. In that respect you’re talking of frequency, I’m talking of reliability of supply while others are beginning to talk about the voltage-
            There must be strong incentive to produce all three at the same time with no beggar thy neighbour outcomes on any score.

Leave a comment

Your email address will not be published.