Plant startup is only the first step.[1] It's load pickup.
Here is a PJM training module on load pickup during system restoration.[2] It gives a sense of how touchy the process is. Power network control has good control over generation and transmission, but limited control over load.
When load is turned on, there are transient loads with different time constants. There's a huge load for the first second as inductors, capacitors, and incandescent filaments start up. That tails off in under a second. There's a second load as motors wind up to speed. Ten seconds or so. Then there's "cold load", where everything in HVAC starts trying to get temperatures back to normal. Maybe half an hour.
There's no mention of computer control. Listening to this, you visualize people running around reading meters and throwing big switches. It's probably people looking at display boards and sending commands to remote big switches, but the concept is the same.
Botching this means voltage or frequency goes out of tolerance, protective devices shut things down, and the system operators have to start over.
More PJM training modules on related subjects.[3]
Unclear what caused this yet. Something caused enough system instability to trip protective devices, but there's no good info yet. Once everybody has a chance to compare all the logging data from different points, it will make more sense.
The groundwork for the blackout will inevitably come back to a buildout and reliance of renewable sources that do not have spinning mass that can do frequency synchronization.
There are costly means to compensate for the lack of spinning baseload but actually building these devices have been neglected, to no ones surprise.
Sounds like these systems are extremely vulnerable to sabotage. But adding more complexity such as distributed digitally controlled loads will not necessarily make the situation much better.
Dropping from 32 to 8 GW usually means that an interconnected grid has fragmented into islands. If an island has a blackout, you need to black start it and resynchronise it with other islands.
Oh man, this reminds me of factorio. When you miscalculate your power needs and your generators eat up the fuel that isn't being produced enough. I've had to make a bootstrapping mechanisms and power switches for some of my powerplants to ensure I can get it back up in case it goes down and I can then re-enable my factory bit by bit.
Unfortunately, you can't simply hand-feed 100 coal into all the Spanish thermal plants and then plop down two more to be on the safe side until you've fixed supply...
It's even worse in satisfactory imo, because the game is build around your fabs consuming wildly varying amounts of power throughout their production cycle... And after a power loss, they'll not only need the initial burst (can be handled via batteries) but will also now all be synchronized to have their power spikes at exactly the same moment.
Dyson sphere program is the by far most forgiving in that regard. You can overdraw to something like 200% before it becomes really problematic, and even that quickly resolves itself the moment you get more power online
Can't answer for aluminium, but I can answer for window glass: One of our plants was threatened by massive flooding 2021, threatening the float glass oven (which doesn't float itself despite the name). It was closed off with insulation, turned off, and everyone hoped for the best and no steam explosion. In the end, only the cellars with the control equipment flooded and it could be gradually (as the control cabinets were checked and restarted) brought up again without loosing too much heat a day later.
In short: Heat Inertia of large molten bodies is massive with good insulation. If the time is too long tho, only dynamite will dismantle a solid chunk of material again.
In some cases they don't have any practical ability to.
The Texas winter storm resulted in a lot of scrap at semiconductor lines due to power loss. There are industries that are completely dependent on grid power. You can't generate enough on site to back up 100% of your operation. Think about how much power one EUV source consumes.
The only reason Samsung is building additional factories in Texas is because the local utilities are effectively treating them as a critical load. The new plants are right by the ERCOT operation center and likely have access to the same cranking paths that the grid uses for black starts.
I can only imagine the difficulties facing technicians and engineers black starting a nation-scale grid.
I operate a microgrid facility in Hispaniola and have wonderfully cooperative users backed by a separately powered communications and control system. Even for us, a facility serving a small neighborhood and farm, a black start must be performed as a careful choreography of systems and loads…and we can just pick up a radio and tell people to turn the main breaker of their house on or off, and to leave their AC units off until we finish bringing everything up. In 12 years we’ve only had to do it twice, but even for us it’s a tedious process.
That was a fun 10 hours or so that we were without power in most of Barcelona. Some barrios had ~some~ power but most of us were without.
The one thing that surprised me is how quickly rumors started about power being out in Portugal (true), France (true to an extent), Belgium (false), and the UK (false). Walking back home from work you heard the names of Trump being mentioned as well.
It's a bit scary to personally experience how quickly people can start panicking. On the other hand the bars that had some power had a great afternoon with tons of beer being served before it got too warm :-)
I am in Barcelona too. The combination of no power and patchy internet was the perfect recipe for rumors to spread, which was fun to observe. I went for a walk since I couldn't work and saw tons of people socializing, drinking beers (more than usual), and young people playing without their phones. It'll be a special day to remember, I think.
A security guard at my workplace actually told me they had "received a call" about power being out in Romania as well. I'm not sure how that one rumour spread.
How warm is "too warm"? I find cold beer, and cold soft drinks as well, quite repulsive. I usually try to order without ice entirely, and if I receive any bottles or cans, I permit them to come to room temp before consumption.
Refrigeration is sort of a scourge to those who enjoy food. We are so used to simply freezing our esophagus and not tasting anything! Food is kept cold because it lasts longer, not because it tastes better.
Sure, refrigeration makes restaurants and grocery stores possible. But it also presents challenges to cooking and it's very resource-intensive -- think about it -- basically everyone everywhere is always running their refrigeration -- that's a huge load on any system.
I am no expert, however with solar power and more grid batteries (I don’t think Spain has many grid batteries) wouldn’t it be possible to charge the batteries and use that to drive a ‘black start’.
Sounds simple, and I appreciate from experience a huge amount of prep and validation needs to be done in the background.
At an extremely abstract level, it is conceptually simple, but most inverters are designed with safeguards to prevent exactly this. They only produce power when the grid is up, and in phase with it.
Because if not sequenced and interlocked, islanding behavior can result in back-powering parts of the grid that are supposed to be disconnected and down for maintenance, or creation of desynchronized islands that then cause more damage when reconnected.
This is absolutely a problem that operators need to solve. Radio synchronization is possible, neighborhood islanding could be beneficial, etc. There are entire village-wide micro-grids in remote places where everyone's inverters do perform these functions, it just hasn't been embraced in monopoly environments yet.
For instance, relays that close isolated parts back onto the main network would need to be phase-aware, to only reconnect if it's safe to do so. You could mandate that all island-mode inverters center themselves around 50.1 or 60.1Hz in the absence of outside influence so there's certain to be phasing opportunities. Distribution networks would need more switches to provide positive isolation during maintenance. And all of these things would need huge amounts of interoperability testing. It's possible, but it has a cost.
Initial estimates went from "No idea" to "Could take days".
In the end it took about 12 hours to restore most of the power in two countries. There are some knock-on effects from trains being in the wrong places, but a solid result overall.
What caused it? Nowadays when anything like this happens in the EU, I immediately wonder if it was Russian saboteurs.
They have certainly been destroying infrastructure here, eg. undersea internet and power cables.
Spain is mostly back up.
Plant startup is only the first step.[1] It's load pickup.
Here is a PJM training module on load pickup during system restoration.[2] It gives a sense of how touchy the process is. Power network control has good control over generation and transmission, but limited control over load.
When load is turned on, there are transient loads with different time constants. There's a huge load for the first second as inductors, capacitors, and incandescent filaments start up. That tails off in under a second. There's a second load as motors wind up to speed. Ten seconds or so. Then there's "cold load", where everything in HVAC starts trying to get temperatures back to normal. Maybe half an hour.
There's no mention of computer control. Listening to this, you visualize people running around reading meters and throwing big switches. It's probably people looking at display boards and sending commands to remote big switches, but the concept is the same.
Botching this means voltage or frequency goes out of tolerance, protective devices shut things down, and the system operators have to start over.
More PJM training modules on related subjects.[3]
Unclear what caused this yet. Something caused enough system instability to trip protective devices, but there's no good info yet. Once everybody has a chance to compare all the logging data from different points, it will make more sense.
[1] https://pjm.adobeconnect.com/_a16103949/p622tuwooba/
[2] https://pjm.adobeconnect.com/p6e5csm81ter/
[23 https://www.pjm.com/training/training-resources
The groundwork for the blackout will inevitably come back to a buildout and reliance of renewable sources that do not have spinning mass that can do frequency synchronization.
There are costly means to compensate for the lack of spinning baseload but actually building these devices have been neglected, to no ones surprise.
Sounds like these systems are extremely vulnerable to sabotage. But adding more complexity such as distributed digitally controlled loads will not necessarily make the situation much better.
But according to production data it has never dropped to zero? Why it would be a black start?
It dropped from about 32GW to about 8GW
source: https://transparency.entsoe.eu/generation/r2/actualGeneratio...
though admittedly https://en.wikipedia.org/wiki/Black_start mentions that partial shutdown may be also requiring a black start?
Dropping from 32 to 8 GW usually means that an interconnected grid has fragmented into islands. If an island has a blackout, you need to black start it and resynchronise it with other islands.
Oh man, this reminds me of factorio. When you miscalculate your power needs and your generators eat up the fuel that isn't being produced enough. I've had to make a bootstrapping mechanisms and power switches for some of my powerplants to ensure I can get it back up in case it goes down and I can then re-enable my factory bit by bit.
Unfortunately, you can't simply hand-feed 100 coal into all the Spanish thermal plants and then plop down two more to be on the safe side until you've fixed supply...
It's even worse in satisfactory imo, because the game is build around your fabs consuming wildly varying amounts of power throughout their production cycle... And after a power loss, they'll not only need the initial burst (can be handled via batteries) but will also now all be synchronized to have their power spikes at exactly the same moment.
Dyson sphere program is the by far most forgiving in that regard. You can overdraw to something like 200% before it becomes really problematic, and even that quickly resolves itself the moment you get more power online
If you're interested in the process of a black start
https://www.youtube.com/watch?v=uOSnQM1Zu4w
from Practical Engineering
We had something similar happen in Tunisia in 2002. The reliance on diesel turbines meant a very long rebooting process.
As a result, the government invested heavily in installing gas turbines all over the country.
When we had another blackout event in 2018(?), it only took an hour or so to get things back again.
Why is that? Are turbines easier to speed regulate than diesel engines?
I imagine a lot of sunshine in Tunisia (not too familiar with your country) , is there a big transition to solar?
How do energy intensive processes like aluminium smelters or window glass producers preven their machines from self destruction ?
Can't answer for aluminium, but I can answer for window glass: One of our plants was threatened by massive flooding 2021, threatening the float glass oven (which doesn't float itself despite the name). It was closed off with insulation, turned off, and everyone hoped for the best and no steam explosion. In the end, only the cellars with the control equipment flooded and it could be gradually (as the control cabinets were checked and restarted) brought up again without loosing too much heat a day later.
In short: Heat Inertia of large molten bodies is massive with good insulation. If the time is too long tho, only dynamite will dismantle a solid chunk of material again.
In some cases they don't have any practical ability to.
The Texas winter storm resulted in a lot of scrap at semiconductor lines due to power loss. There are industries that are completely dependent on grid power. You can't generate enough on site to back up 100% of your operation. Think about how much power one EUV source consumes.
The only reason Samsung is building additional factories in Texas is because the local utilities are effectively treating them as a critical load. The new plants are right by the ERCOT operation center and likely have access to the same cranking paths that the grid uses for black starts.
Power guarantees and insurance.
They are also frequently situated close to reliable power sources such as nuclear or hydro, usually fed by more than one generating station.
A friend of mine lived close to a brick plant his power NEVER went out.
I can only imagine the difficulties facing technicians and engineers black starting a nation-scale grid.
I operate a microgrid facility in Hispaniola and have wonderfully cooperative users backed by a separately powered communications and control system. Even for us, a facility serving a small neighborhood and farm, a black start must be performed as a careful choreography of systems and loads…and we can just pick up a radio and tell people to turn the main breaker of their house on or off, and to leave their AC units off until we finish bringing everything up. In 12 years we’ve only had to do it twice, but even for us it’s a tedious process.
That was a fun 10 hours or so that we were without power in most of Barcelona. Some barrios had ~some~ power but most of us were without.
The one thing that surprised me is how quickly rumors started about power being out in Portugal (true), France (true to an extent), Belgium (false), and the UK (false). Walking back home from work you heard the names of Trump being mentioned as well.
It's a bit scary to personally experience how quickly people can start panicking. On the other hand the bars that had some power had a great afternoon with tons of beer being served before it got too warm :-)
I am in Barcelona too. The combination of no power and patchy internet was the perfect recipe for rumors to spread, which was fun to observe. I went for a walk since I couldn't work and saw tons of people socializing, drinking beers (more than usual), and young people playing without their phones. It'll be a special day to remember, I think.
Spain and Portugal had to deny a fake social viral that said power was out across most of Europe and the Russians were cutting undersea cables.
On the upside, more people at all levels are going to be more prepared if it happens again.
I'm close to a refinery and seeing black smoke coming out 5 minutes later of the blackout was a bit of a scare tbh.
A security guard at my workplace actually told me they had "received a call" about power being out in Romania as well. I'm not sure how that one rumour spread.
Most information was spread by car radios. Rest were people asking and talking to each other.
As long as the beer is cool, you only need CO2! Great off grid mechanism. Or N2 in case of Guinness.
I never have any cash on me. That would’ve sucked.
How warm is "too warm"? I find cold beer, and cold soft drinks as well, quite repulsive. I usually try to order without ice entirely, and if I receive any bottles or cans, I permit them to come to room temp before consumption.
Refrigeration is sort of a scourge to those who enjoy food. We are so used to simply freezing our esophagus and not tasting anything! Food is kept cold because it lasts longer, not because it tastes better.
Sure, refrigeration makes restaurants and grocery stores possible. But it also presents challenges to cooking and it's very resource-intensive -- think about it -- basically everyone everywhere is always running their refrigeration -- that's a huge load on any system.
I am no expert, however with solar power and more grid batteries (I don’t think Spain has many grid batteries) wouldn’t it be possible to charge the batteries and use that to drive a ‘black start’.
Sounds simple, and I appreciate from experience a huge amount of prep and validation needs to be done in the background.
At an extremely abstract level, it is conceptually simple, but most inverters are designed with safeguards to prevent exactly this. They only produce power when the grid is up, and in phase with it.
Because if not sequenced and interlocked, islanding behavior can result in back-powering parts of the grid that are supposed to be disconnected and down for maintenance, or creation of desynchronized islands that then cause more damage when reconnected.
This is absolutely a problem that operators need to solve. Radio synchronization is possible, neighborhood islanding could be beneficial, etc. There are entire village-wide micro-grids in remote places where everyone's inverters do perform these functions, it just hasn't been embraced in monopoly environments yet.
For instance, relays that close isolated parts back onto the main network would need to be phase-aware, to only reconnect if it's safe to do so. You could mandate that all island-mode inverters center themselves around 50.1 or 60.1Hz in the absence of outside influence so there's certain to be phasing opportunities. Distribution networks would need more switches to provide positive isolation during maintenance. And all of these things would need huge amounts of interoperability testing. It's possible, but it has a cost.
This article was published when over half of power was already restored.
99% up now. It's pretty impressive how fast they can get the system up and running again considering how seldom they need to do these types of things.
Initial estimates went from "No idea" to "Could take days".
In the end it took about 12 hours to restore most of the power in two countries. There are some knock-on effects from trains being in the wrong places, but a solid result overall.
What caused it? Nowadays when anything like this happens in the EU, I immediately wonder if it was Russian saboteurs. They have certainly been destroying infrastructure here, eg. undersea internet and power cables.
[dead]
[dead]