Texas grid flags risks as data centers, crypto sites fail voltage tests
reuters.comFor those who dont know why this is important:
The loads are slowing down the generators that are burning a well metered amount of fuel to stay at 60Hz. This is a delicate balance since the phase angle must also be spot on.
If a generator and the local line disagree on f, phase or V, you have a short circuit.
If you lose a large amount of load, your generator will spin up with the excess fuel until the control system re-establishes the right amount of fuel.
But now your generators are out of sync! No worry, for small disturbances the dissipative losses sync everything up like syncros on a manual transmission.
But the disturance cant be too big!
Rotating machines are big and heavy, so the first line of defense is their inertia. But this is a finite (and precious) resource.
Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse. They generate large amounts of power while providing no inertia.
You'd think it isn't a big deal since the DC-AC converter can just synthesize whatever is needed. Heck just keep it rigid at 60 Hz with no phase change.
Well the later doesn't work - the rest of the grid is no longer at that phase and frequency so you got yourself a short.
Furthermore, the DC-AC converter, despite their manufacturers' promise, has no good way to establish what f and phase it should be at during a disturbance (and these magic codes are closed source, believe it or not)
Anywho, a large enough loss of load causes the grid to enters into unstable oscillations, causing protective relays to trip causing a zipper effect where the grid goes down.
Now restart will take a few days depending on the energy mix (fastest for hydro heavy)
Long story short - this is not a trivial problem, and the data-centers can't be allowed to just dump load willy nilly.
EDIT: made it clear that the grid killing disturbance is not caused by renewables; not exclusively anyway. Everyone has to play nice or the grid goes down.
That’s a simplified and somewhat outdated version, there’s a huge range of technology to mitigate each of these issues but infrastructure generally isn’t free. A turbine does provides inertia for free where you need to pay for a flywheel or battery system.
In the end a rentable heavy grid can be extremely resilient due to all the batteries, but smart battery systems need to be incentivize or mandated because ‘dumb’ batteries are cheaper.
10+ MW voltage-source converters that can't do up to around 80% of their nominal capacity as mostly-reactive apparent power with stabilizing synthetic inertia scaled as desired/specified are a mostly software issue, stemming from lack of regulatory pressure incentivizing the engineering complexity of that.
Though if you want to do a smoothing action on real power flux you'll have to colocate battery capacity with the converter. Which to be clear is fairly cheap to do as long as you get compensated for the substantial frequency stabilization capacity this represents. I'm talking like 15~120 minutes at converter nominal AC power of battery capacity.
The first 10~20% of reactive power are almost free from the converter electronics, btw....
Would demanding that large spikey users of energy like data centers implement some sort of demand ramping/isolation from the grid in the form of a massive capacitor bank or flywheel generator between them and the grid help reduce the risk here?
Some of us in Texas are all too familiar with the problem of balancing load with generation, the risk of a cascade failure causing a slow restart.
During winter storm Uri, they did a duty cycle where we only had power available for ~6-12 hours at a time on the days it was available. This was apparently to avoid that very problem.
So far as I know, the obvious mitigations like winterizing NG generation and/or peering with neighbor grids have not been performed.
Grid scale battery systems are also used to maintain proper synchronization.
How about an across the board $1/W hook up fee for new customers? Thats about the price of installed capacity per watt.
New house with 200 A panel an assumed 30% utilization rate? $3600.
New data center with 80% utilization rates at 100MW? $80 million dollars.
New 10 GW data center? That'll be $8 billion.
It's outrageous that I'm paying an extra fee to export energy to a neighboring state to power a datacenter.
All projects have an interconnection fee.
Large projects will have a large load interconnection tariff that’s supposed to shoulder the costs of upgrading the infrastructure to support these new projects.
Data center discussions are weird right now because people assume these things don’t exist and propose them as solutions. They already exist.
The problem in the article is something different: The sites they’re talking about are designed to disconnect from the grid and use backup power when the voltage drops, which can be a problem because now there’s too much energy being supplied to the grid and not enough load to absorb it.
> It's outrageous that I'm paying an extra fee to export energy to a neighboring state to power a datacenter.
I don’t find anything outrageous whatsoever about building out transmission to strengthen regional interconnections. Lord knows we are decades behind the curve on this. If AI datacenters are what finally lets us maybe start to get a tiny bit ahead of the game I’m all for it. When the bubble pops there will finally be a bit of pressure off, and perhaps we can get back to a tiny bit of overbuilt capacity to cover for exceptional events again. And more reasonable projects maybe won’t be sitting in a decade long interconnection queue.
At some point we get to all collectively pay for our parents lack of investment in energy infrastructure. We have been living off our (great?) grandparents investments into the future and wide scale deindustrialization of the economy since I’ve been alive. At some point you run out of inertia.
Could always move to Texas if you hate the idea of your state ever possibly spending a dollar that benefits someone across the border. They islanded their grid pretty much precisely due to this attitude.
Watching the PJM and MISO interconnections over the past decade or so has been illuminating. It’s been a very slow moving disaster in the making that nearly no one is paying attention to.
As for your proposed solution I doubt anyone - including the datacenter operators - would argue since that is largely what tends to happen as it is. Just change utilization factor to capacity factor and sounds good to me!
Texas has an independent energy grid, so are not exporting energy to them.
My rant was generally about what ought to be done to data centers.
In fact, I don't live near Texas. My state is having me finance a power line that will increase demand for our electricity.
Your math on the house seems off by a factor of 4? But 30% utilization seems high, as that would be a $2k/mo electric bill at 20 cents/kWh.
Reframing what you're saying, that would be a connection fee that worked out to just under 12 cents per kWh used for the first year, which seems both somewhat reasonable but also probably not going to move the needle much on the large deployments?
I guessed what the utilization factor for a house was, purposely going for a very high number to show that it wouldn't add too much to the house.
Note, In proposing a one time fee, on the capacity to generate power, not the energy.
Yes, but using the 30% utilization factor I get 200A * 240V * 30% = 14.4kW , meaning a $14400 connection fee rather than $3600 ? $3600 would be a 7.5% utilization factor.
And yes, I got the one time fee aspect. My point is that if you're basing this off of expected utilization (rather than say the size of the service regardless of how much energy is used), and that expected utilization is roughly correct, then that one time fee can be thought of as amortized cost per unit energy over a specified amount of time. And that cost isn't even that high if amortized over a year.
This all seems "routine".
Also the little sound bite about peak demand, Texas has enormous capacity during the summer as well; far, far exceeding demand. It makes total sense to be testing and connecting multi-GW consumers during these months.
Do we really need to keep slamming the grid and killing the planet for pseudonymous casino chips?
Can someone please merge crypto with llm training/inference somehow?
I’m curious how this works for other large consumers. Do they have some kind of artificial load that lets them gradually reduce consumption instead of doing it all at once?
If you're large enough your connection to the grid is a negotiation with an engineering team.
The utility will force you to put equipment to correct for power factor (massive capacitor bank), resistive load, etc.
The utility also charges commercial users for apparent power (includes reactive power, or that sloshes around setting up a steady state), as opposed to just real power charged to residential users.
EDIT: in case your wondering, yes resistor loads is just glorified bunch of short circuits and a fan.
That’s interesting. For the rest of us I guess it’s just the law if large numbers that our air conditioners don’t cycle off at the same time
Pretty much. Except the law of large numbers breaks down in correlated events.
Like a black out.
What really hurts is are all the rotating machines, especially the one-phase ones (fridge, AC, blower). Those have transients that last several cycles and are electrical shorts until the back emf is setup.
Utilities will try to roll the power back on in sections to avoid instability, starting with hospitals and those near hydro plants.
But you can help put by turning off your stuff during the blackout and when the light come back on walking around the house to turn them back on.
No. I imagine they are less sensitive and generally stay online and are only de-energized if the utility has an outage
But they can have their own internal fault and have to shutdown.
I look forward to hearing from the usual suspects how the inevitable failure is the fault of wind and solar power.
This almost seems like a straw man to me. Isn't the much larger problem the actual increased energy usage and making sure that all of this massive extra cost doesn't just get dumped on consumers?
I am a huge proponent of AI actually, but very suspicious that "financiers" are suddenly creating what amounts to an energy tax by finding legal ways to sneak extra fees or rates into our electricity bills to cover build out and commercial usage costs.
But as far as smoothing out demand, my (admittedly a layperson) theory is that we need to force them to adapt more solar and wind and at the same time more facilities for handling the variable production from that. Such as more large batteries and a shift to large scale long term storage of renewable fuel like hydrogen or other fuels produced directly from renewable sources.
If you have a large production and storage of renewable fuel, then maybe you can build that in such a way that it can handle significant input variations that could include excess grid power.
Data centers are pretty useful for adding more renewables to the grid. They generally have both a UPS (for short-term outages) and backup generators (for long-term outages). UPS means they're already paying for inverters and some amount of batteries, and then in combination with a more renewable grid, it makes sense to put the grid storage batteries there, and they have the incentive to do it to take advantage of time of day metering by actually running on them during peak demand hours. Likewise, if you have a huge capacity backup generator and the grid is stressed because renewable generation is low (and therefore prices get high), you can turn it on. Which allows the grid to get more power from renewables the other 95% of the time.
Solar and wind would make it worse.
What matters is peak capacity. Using the grid as a free battery for when your intermittent sources of energy go away against your fairly constant loads only makes the grid weaker overall. The best part of these huge predictable load centers is utilization factor.
Storage of some sort would probably help some, but overall the best type of load for a grid is predictable constant usage. Bonus points if it can reasonably be part of a load shedding/demand response program.
Texas is the standing proof that "making sure that all of this massive extra cost doesn't just get dumped on consumers" is not a problem. Texas has the most load, the most marginal load added every year, and the cheapest retail power. Load grew 15% in the 5 years to 2024, and retail prices fell by 1¢.
I don’t think that is true.
https://www.electricchoice.com/historical-electricity-pricin...
Do t they also have like the most solar?
Maybe in absolute terms? I haven't checked in a while. In terms of fraction of energy from solar Texas is still far behind California.
This sounds like a good reason to have a lot of batteries.
Texas was the leading state in new grid battery installs for the last few years.
A lot would probably be an understatement. Aren't most batteries in data centers designed to just hold load for seconds or a few minutes at a time while generators start up?
If the problem is an instant disconnect of a large load, switching that large load into charging batteries at that instant, for a minute or so, feels like a solution.
Yeah, different batteries, different purposes.
Keeping the data center up is completely different from keeping the grid up.
Not only are the batteries too small; they're also on the wrong side of the disconnect.
I'd rather see legislation banning crypto mining and AI data centers from the public grid entirely. No sense in forcing the broader public to subsidize them.
I'd rather see legislation banning everyone from the grid. Why have a common resource if its no longer common. Just get rid of it. Ban everyone from using the roads, who knows, they might be transporting evil computer hardware to an evil data centre using a road. Farms obviously have to go, they supply food to people who write code that gets run in datacentres. In fact, just ban all trade and commerce to be on the safe side*
*The US should implement this policy for real for my personal amusement.
The problem with that is one of the best things we have to control pollution at power plants is the rules that go into place when connecting to the US grid (I know TX is different).
I really don’t want to incentivize private power plants that aren’t on grid. Or just running tons of industrial sized generators instead.
If we’re going to allow enough of this stuff to be built that it can destabilize things why not require they behave and don’t stop off like that? Some sort of organized draw down?
And if they don’t? Mandatory cutoff for X amount of time. Weeks/months.
Ban private fossil generators above a certain size without a license. You can just do things. They can build as much solar and batteries as they want.
https://www.datacenterdynamics.com/en/news/google-confirms-1...
https://www.datacenterdynamics.com/en/news/google-announces-...
SpaceX is running a bunch of “portable”, high pollution gas generators in Memphis, TN specifically to get around the regulation you’re describing.
Elon definitely got the “you can just do things” memo.
The proposed regulation they're describing doesn't have that loophole.
It’s what I was thinking of when I said “lots of industrial generators” too, but I couldn’t remember where it was.
Right, that’s what I was thinking of when I wrote my comment. Regulate back. If there is no will to do so, well, that’s a choice. Write the law, pass the law, aggressively enforce civil and criminal penalties for violations. They haul gas generators in without a license? Confiscate and tear them down for scrap (which will be painful, as these turbines are in short supply and their manufacturers are backlogged years into the future), in accordance with law you pass. Hold the utility liable if they provide a fossil gas pipeline connection. Humans like Elon may not care, but utilities have something to lose. Find “one throat to choke” as the saying goes.
It is not politically easy, but it is logistically straightforward.
If they weren't on the public grid, they would just slap in a bunch of gas turbines and run one of the noisier, more polluting sources of electricity. I think it would be better if we required them to replace the power they used, but do so on the grid so that it benefits everyone.
Didn’t Texas pretty much do the exact opposite thing recently?
More scarce resources used to feed the surveillance state's fancy chat bots
I think it's time to put data centers on a power budget. If they want to make more money, they need to become more efficient and eliminate AI fraud, waste, and abuse.