One-minute frequency disturbances from synchronization in power systems
nature.comThe corresponding concept in gears is to avoid meshing two gears unless the number of teeth are co-prime, because this will cause uneven wear patterns. If the tooth counts are coprime, then each tooth will contact every tooth on the other gear the same number of times, effectively "wear leveling" the teeth.
In this instance, one fix is to fire the timer every 59.XXX seconds, so it will naturally drift out of phase with the minute hand of the clock.
If work needs to be done exactly every 60 seconds, this is fixed by having a WAIT(RAND(60e6)) microseconds during the first timer initialization, so if 10,000 units start simultaneously they don't try all at once. This is good practice in most (all?) systems. Hint: test power and infra outage/reconnect, which can unintentionally sync timers.
This is why many software will do automated updates every 17 hours: 17 is coprime with 24.
Classic "thundering herd" mitigation tactics.
Edit: Tempting to blame IoT, but (since this is news) I don't think anyone knew there was something to fix here. We have the tools however.
Absolutely fascinating comment regarding gears and coprimes. This is the first time I have heard of either.
My mind goes somewhere completely unrelated to the comment though: cicadas!
I believe it is a mystery why some species of cicada have have a 17-year life-cycle. Theories have included avoiding predation etc.
However, I wonder if the 17-year cycle (prime) avoids "even wear patterns" somehow in their DNA??
coincidentally, earlier on HN: https://news.ycombinator.com/item?id=50333323
I get a disturbing amount of equipment that does not take into account the inrush current of starting many loads at the same time. While each individual device is protected by a breaker that can handle the inrush of it's load the main breaker for the equipment is not sized to handle every load starting at the same time. The fix is to include logic that starts things up one at a time but convincing engineers, project managers, and machine owners that they didn't think of everything is an uphill battle.
We used to have a turbopump that would trip the breakers every time you turned it on from cold. The pump seemingly attempted to spin up to max RPM before settling down to a more sustainable load. The only way to get the thing to stay on was to turn it on once, let it warm itself up on the initial attempt, trip the breakers, and try it another cycle now that it had some momentum and would pull slightly fewer amps. Sometimes took a few attempts to get that rolling start working.
That's a great way to lose the breaker though. If you're lucky the latch will fail. If you're unlucky it will weld shut.
Not ideal, but it was the only thing that worked. This was attached to a million dollar piece of equipment, so any remedial electrical replacement work would have been slightly inconsequential. Thankfully, it was one of those machines that you turned off as infrequently as possible, so we did not have to do this procedure too many times.
The vendor did get the hint, and the second iteration of the device did slowly warm up the pumps so this process would no longer be required.
Yes, soft start is the way to go. Also: proper industrial breakers allow for temporary overload. I have a couple of pieces of gear that have inrush currents well over the limit of a normal breaker so I put them on a slow one and now it is fine.
https://onccy.com/how-to-choose-between-type-a-b-c-and-d-min...
> Also: proper industrial breakers allow for temporary overload
Notoriously a crescent wrench makes an excellent 1000A slow blow fuse
A 5/32" flat-blade screwdriver makes an excellent fast-blow fuse when dropped across exposed bus bars. And you can re-use the remains of the handle to stir your coffee once you've retrieved it from across the room.
Soft-start is in theory the way to go, but it's very dependent on the motor type, there's a whole pile of different methods required for induction motors depending on whether they're capacitor-start, permanent split capacitor, shaded-pole, etc, so you can do it in some cases by lowering the voltage, or with a VFD, or a VSD, or the old-fashioned way with a centrifugal switch but that requires very careful handling. If as the OP said it's running million-dollar equipment you probably don't want to monkey with it at that level.
Reminds me of computer fans, some bioses seem to do that ramp up for no reason thing as well.
Fans have a minimum speed needed to spin up when cold starting from 0 RPM. So it's safer to start at near max RPM and spin down to the desired target RPM if the controller doesn't know the exact operating characteristics of each fan.
You can observe this with a speed control knob/ESP32 where a running fan will happily run forever with the knob turned down to e.g. a 30% duty cycle. But after turning to 0% and waiting for the blades to fully stop, if set back to the same 30% they won't move at all or you'll maybe hear a sketchy clicking/whine sound with the blades intermittently twitching.
I've always assumed it was to calibrate the PWM sensor for fan speed, so it starts with no sensing active, runs up to max, and then the sensor kicks in and settles it down. You don't need to run a fan at max speed to get it to start, most will start with very little power applied.
not exactly relevant to computer fans, but in case of your usual standing fan you need a way larger kick to get it going. if it fails to start but stays powered it will stay stuck, heat up and fail. this is why fans usually have the highest option as the first one, so you have to start it at full speed at first, and only then reduce it.
You also want to be sure that, in the absense of fan-control, the CPU won't overheat. Full-on is safer than nothing until a driver is loaded.
I assume you are talking beyond what (in the DIN rail world) type B/C/D breakers are meant to handle?
It's sort of a solved problem - the ones I'm aware of are things like LynTec sequenced panels built exactly for this, and Siemens' SENTRON semiconductor CBs where you can control then curve to your spec.
Convincing stakeholders though, is almost a job of its own.
These were all across-the-line, 3-phase 600v starters so there was no option to slowly start things up. Soft-starters would have been ideal but they are 5x the cost.
what? Inrush current is well known in electronics engineering.
In other words, convincing ignorants is an uphill battle. Those who know what "inrush current" means don't require convincing about the value of rolling starts.
59 seconds will still have second alignment. Maybe it needs to be prime milliseconds or microseconds.
But seconds are 60X higher frequency than minutes. This is almost 6 octaves higher. The grid will low-pass-filter that out much better than the minute period. (You can see that trend in the diagrams given in the article: the amplitudes all get lower with decreasing period to the left, where the 60 second blip is a an outlier.)
Secondly, the amplitude drops. Events that were concentrated on the xx:xx:00 times are now distributed into xx:xx:xx times. If the overall number of events is the same, each :xx time slot now receives 1/60th of the events that were concentrated on the :00 time. If power decibels are relevant here (20 log 10 x), that's a 35 dB reduction in amplitude of the driving signal.
35 dB less amplitude, shifted to a 6 octave higher frequency that the grid doesn't react to very well can be expected to basically make the problem vanish.
This is much like the situation of soldiers avoiding rythmic marching when crossing a bridge to randomize their footstrikes.
It depends on a lot of factors, but even second granularity gives you a 60x reduction in the thundering herd. Whether more is needed or even possible often depends on the specific problem domain. For example, if your task takes 20ms, us granularity doesn’t help. Indeed it makes it worse if you’re using randomization because there’s a lot more microseconds than seconds so you can end up with a lot of work that ends up in practice still getting scheduled together due to other delays.
You want your jitter to be significantly >> than the time your task takes as otherwise you get non intuitive degradation that can reintroduce the problem you’re trying to solve. It’s a balancing act picking the jitter that drops off the thundering herd enough and one that is still greater than the time your work takes.
I think you should re-read the post you're responding to. It said 59.XXX seconds, and given the reasoning immediately after that, it's clear that XXX is intended to be a randomly chosen number of microseconds.
That was edited in later
Small nit: Wait(50+rand(10)) is probably closer to what you want to get about a minute between executions. As stated your proposed execution has an average about 30s execution interval with a variance of about 17s. If you’re talking about a random sleep on a 1 minute interval, this actually results in a possibility of 120s between tasks, also not what you want.
But yeah, as soon as you start using Rand most people make all sorts of mistakes. And if you’re using C Rand, don’t. It has other statistical problems.
He’s still allowing for exactly 60s between executions. He’s saying stagger the startup by a random amount up to 60s. Then we get what we want every 60s, but they’re all just staggered a random amount.
My bad. I misread it. Delaying the start like that does divide the thundering herd peak by 60. The point about c Rand stands.
The other thing to watch out for is you’ve still got 1/60 of the thundering herd at the second level. If the tasks are <1s and you have enough of them (as you probably do at grid scale), then doing it finer grained is warranted like offsetting the timer by a millisecond sleep of 10 * uniform_distribution<int> between 0 and 6000. That then gives you 10ms granularity, distributing the workload by two more orders of magnitude.
I almost wrote 59.999 seconds too, but I figured someone would nitpick that the timing often won't be that accurate or will get coalesced. ;)
I should have said, you almost always want to use the randomly staggered timers on initialization, not just when the exact timer interval matters! It's also important to test electricity or backend outages, which can unintentionally synchronize all the timers.
You have good advice and I missed the “on startup” for the wait.
A common aggregation is about 10ms although most OSes can give you finer granularity accuracy if you ask for it and have permissions. It really depends on how long the task takes and whether there’s any non-uniformity in its processing to make finer granularity helpful or harmful.
Unfortunately some people do read advice like it’s mana from the gods instead of thinking through the implication on their specific problem.
Coprime meaning "no common factors".
I'm skeptical that this one minute jitter period is solely attributable to one minute timers in power inverter equipment. It's likely the net scale effect of all cron-like scheduling of actions in everything with a microprocessor. All the long-term scheduling that is done with minute precision (i.e. seconds field is implicitly :00) regardless of its individual period. Like if one device is scheduling at x:17 and x:47 every hour, another one is doing x:06, x:09, x:12, x:15, ... and numerous other combinations, at scale, the common divisor will be one minute. If wall clock time is used, all the devices that are synchronized to a network time will have cron-like scheduling events firing at xx:xx:00 for all values of x.
This type of scheduling should be randomized at the seconds level, perhaps. Unfortunately, it would require a massively coordinated effort across numerous operating system and application stacks.
Perhaps easier, now it's identified it can be anticipated.
Like TV pickup, where advert breaks in television cause surges in power demand.
The classic example is everyone making a cup of tea at the same time during the World Cup half time break.
Something as common as predictable as every minute should be simple to model in.
> The classic example is everyone making a cup of tea at the same time during the World Cup half time break.
Are we talking soccer? I can't even imagine the british drinking tea while watching :)
What's most fascinating about that wiki page is that claim that mass coordinated shut offs of lights etc can cause more co2 to be released.
> I'm skeptical that this one minute jitter period is solely attributable to one minute timers in power inverter equipment. It's likely the net scale effect of all cron-like scheduling of actions in everything with a microprocessor
The article says as much:
> The identified synchronization mechanism is not specific to UPSs or battery systems, but reflects a general property of digitally controlled devices operating with external timing signals or periodic software routines
Maybe too many things are started from crontab. Public NTP servers observe large spikes in traffic on minute marks due to admins putting ntpdate in crontab instead of running a proper NTP service.
I know my older APC UPSes do weekly self tests, but I guess newer equipment does more frequent self testing? And of course they're all internet-connected meaning clocks are within a few hundred ms of each other.
It also made me think about how many GitHub Actions I have that kick off on Sunday morning at a specific time. I try to vary the time/day for those jobs but I'm guessing there are enough things that kick off at specific time intervals to make an impact at the top of an hour, or at :15, :30, :45...
This is why systemd gives you the RandomizedDelaySec option. (Other cron-like facilities may have similar features, I'm not discussing them here). Set that to N and each time the timer reaches its trigger (say, once per day), its actual start will be up to N seconds after the trigger. E.g. if a once-per-day trigger is at midnight but you have RandomizedDelaySec=300 set, then it will run sometime between 0:00:00 and 0:05:00, with a different actual start time each day.
If enough people used that, with a high enough value of N (say 1800 for jobs that really don't have to start at any particular time), it would greatly spread out the spikes in accidentally-coordinated usage.
Does this also apply to my small UPS here in my office?
I just checked the power meter dashboard of the building and can see small peaks in the minute range.
In this case it's not counting in too much on the grid level as there's a periodically charged batterie in between.
I'm not sure I agree with the conclusion that this is merely caused by UPS system control logic. That is an extreme amount of coordination across a lot of makes, models and configurations.
The FFT indicates this is a periodic signal, not a once a day or week thing. It's happening continuously all day every day. I don't know of any UPS systems that perform tests with this kind of regularity.
In any case, I refuse to believe an external sync signal is causing this. The more likely explanation is that the active elements are pulling each other to a common mode due to being physically coupled on the same grid. This is like how clocks on the same beam will synchronize.
The coordination is organic. Simply start with this assumption: "each device will check [something] every 60 seconds". Within a few years, all systems will be synced.
When the power goes out on the grid, everything is turned off at the same time. When the power comes back on, everything turns back on at the same time. Every machine that had a 60 second timer is now operating from the same starting time.
The external sync here is NTP. Synchronized clocks + timer-scheduled actions → globally synchronized behavior
From the article:
> The origin of these current peaks was independently confirmed by the manufacturer to be periodic self-tests executed by the battery EMS, confirming our interpretation. Similar minute-scale diagnostic routines are also documented for other commercial UPS systems (cf. Supplementary Note 8).
https://www.snaptec.com.au/akkutec-4801-din-user-manual.pdf
> battery circuit testing 1 x per minute for 1 sec
https://adelsystem.com/hubfs/PDF%20DC-UPS/Instruction%20Manu...
> The battery testing is carried out automatically every 60 sec.
So, the two devices you refer to have no time-of-day clock, no network connection, no TCP/IP stack, and certainly no NTP clients.
Their testing on the enterprise-class equipment shows the difference that NTP makes, and with enterprise-class equipment comes wider deployment and more powerful models.
But most of the consumer units would not have this stuff. Surely, the manuals would describe how to set it up.
Counting mains power cycles is an adequate clock with a 0 of whenever it was last closed in. That 0 could conceivably be correlated to an hourly schedule if that process is automated. Not saying it’s common, but signals don’t have to come from the network.
Potentially related phenomenon: Across multiple locations >200km apart in France, I've personally noticed that mains powered lights reliably and visibly pulse a few times at the beginning of every hour, and half-hour throughout the day. From memory, it's roughly one or two 1-2s long sequences of pulsed dimming, at around 5-10 pulses/second. The pulse frequency doesn't seem to change, and it starts and stops abruptly. Does anyone know what this could be?
Generally this is a control signal injected into the power to control/inform loads about the current rate (e.g.: peak, off-peak), and other tasks. Pulsadis is a common system in France for this.
Perhaps, however Pulsadis uses a 175 Hz carrier, and neither that nor the bit periods are anywhere close to the pulse frequency I've observed. The amplitude of the dimming does however seem to be consistent with that of the Pulsadis signal.
Cool, I thought grid stability attacks would rely on large numbers of IoT devices but it looks like computing infrastructure is the better target.
Whether it's by buying and using a trillion tokens strategically or by inserting synchronized malicious jobs it should be possible to find some resonant frequencies in the line frequency control system. By loading and unloading multi-megawatts of draw you could get the local supply throttling up and down enough desynchronize from the larger grid. If you could get real time measurements of the line frequency you could use that to tune your malicious load and do the whole 'pushing a kid at the peak of their swing' thing.
To mitigate you would have to put dummy loads or some kind of minute-scale power storage onsite or nearby to compute loads.
Multiple MW is noise level, even to a single moderately sized power production facility, let alone the whole grid region. This is not a viable attack. I can have an entire aluminum smelter trip off or start up and it won't phase my supply, a data center is going to be trivial unless you can spend enough to control all the data centers in which case youve just made the least efficient grid attack in history.
Further, there is no single resonant frequency, only the particular resonance of that exact moment in time. The grid is composed of many generating units, many transmission lines, many transformers. The resonant frequency of the grid is changing moment to moment as those things come online, get loaded partially, go offline, heat up, etc.
This is not a child on a swing set problem.
Ones and twos of MW are indeed tiny loads, there is a 1MW motor where I work that gets flicked on and off like a light switch, but hyperscale canters are in the >100MW range, that's >1% of even the largest generating stations in the USA.
And you're not aiming for some physical resonant frequency of the entire grid, just the controller on your local generator, getting it to ramp up and down chasing just behind your changing load.
I live in Ontario and in the past week the difference between our highest and lowest demand was ~4.5GW, but those changes take place over hours. If you can ramp up and down a 0.1GW load over minutes there is going to be a point where some control loop can't keep up. Generators have physical inertia but compute loads do not.
I agree that it's not trivial, not cheap, and not efficient but with the state of the world it would not be a surprise if some nation thought it would be useful to burn resources in order to disrupt an important event, maybe elections?
The whole point of the grid is that it is effectively one giant electrical machine, physically. It isn't possible to target your local generations n any more than it's possible to target a particular generator three towns over.
In any case, a 4.5gw load imbalance from moment to moment is not a difficult problem to solve. This is why all utilities are required to have 'spinning reserve' available to serve new load at very short notice. In the meantime the grid would just drop frequency very slightly (like 0.1 hz would be an enormous instantaneous deviation)
Van Jacobsen TCP synchronisation at queue points in the Internet. Led to protocol changes to de-cohere with a small random backoff.
That is exactly what this reminded me of.
A very interesting article, but it is disappointingly vague about what these UPS self tests are, and exactly how they cause disruption to the power grid. Figure 5b shows that their building's UPS has 20-amp spikes exactly every minute, which is a pretty big spike. My guess is that a UPS performs a self test by switching from line power to battery power. If most of the UPS units in a country do that exactly every minute, synchronized by NNTP [edit: NTP], yeah I can see that it would mess up the grid having all that load dumped at once. (Although the observed frequency disturbance of 1.5 mHz is extremely small, 3/1000 of a percent.) Now that the problem is identified, it should be trivial for UPS manufacturers to eliminate it.
(By the way, the article is in Nature Energy, not Nature which the URL suggests.)
> If most of the UPS units in a country do that exactly every minute, synchronized by NNTP,
Thankfully, usenet usage is way down, so we won't need to worry about this. ;p
It's both up and down since 500TB of binary data is posted every day but nobody uses it for anything else