EPYC 7002 CPUs may hang after 1042 days of uptime

It feels a little bit different.

One creates uncertainty in all floating point results, given you don’t know when it happens. The other requires you to reboot maybe every ~3 years and you know exactly when it happens.

I’m not saying we should tolerate a defect, but it doesn’t feel nearly as problematic.

There are always bugs in silicon, just like there are bugs in software. They mostly show up under "a highly specific and detailed set of internal timing conditions". There are 40 documented erratas on EPYC 7002s alone; there are 35 in the 13gen Intel CPUs, including, curiously, RPL038, "Processor Exiting Package C6 or C8 May Hang". Mobile ARM chip manufacturers are notoriously bad at documenting their bugs, so who knows how many they have.

This one is interesting because its preconditions are so trivial, and it will affect many more people than usual.

> Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

This bug only applies to servers that haven’t been rebooted for 3 years and have the CC6 sleep state enabled. It can be worked around by disabling CC6 sleep state or rebooting once every 3 years.

If you think operators of these servers can’t be bothered to update and reboot their machines once in 3 years or change a single BIOS setting, what makes you think they’d be interested in tearing down their servers, physically replacing the CPU, and reassembling all of them with the associated downtime and inevitable accidental damage to some units? Nothing about that makes sense from a business perspective.

A key difference between then and now is how much easier it is to distribute software/firmware workarounds or fixes. From an end users perspective replacing the CPU might be seen as far easier than updating their software. A software fix would affect performance, so of course it isn't as simple as that, but this difference is part of the dynamic.

Also, as a direct user of the CPU, if the fdiv bug would impact you it would affect you often rather than once every three years which is the impact frequency of this fault.

Another matter that affected the fdiv bug is that the Pentium line was the first time a CPU had been aggressively marketed directly at the general public in quite the way it was. Prior to that only manufacturers and techies would have known about it and they were used to errata for hardware components. The public more generally had an impression that hardware (at least undamaged hardware) was reliable and only software had bugs, and the fdiv bug invalidated that view of reality causing a bit of a panic.

These types of bugs have been in hardware forever. Nobody is going to replace hundreds of EPYC servers even if they could get a free replacement from AMD.

There are definitely cases where hardware should be exchanged with fixed chips, particularly the small business/consumer/hobbyist range where exchanging CPUs is worth the time and effort. The RDRAND problem with Ryzen chips was much worse because it actually happened all the time and there is still no microcode fix available for some motherboards (though AMD already makes the fix available so it's more of an issue about a lack of motherboard support than broken hardware).

> today we seem too willing to put up with being sold broken stuff.

i remember reading that when hard disks just came into the mass market they were so expensive that having some bad sectors was not such a big deal... and so hard disk would usually come with a sheet of paper listing the known broken sectors (detected at QA stage, i guess).

maybe someone older than me (i guess somebody in their 50ies or 60ies) could confirm that.

I'm way too young to remember it clearly but from what I was told it was nothing of the sort. Intel announced that they had identified a bug and would review on a case by case basis to see who was affected and would determine if you were worthy of getting a CPU that was fixed.

Again, this is secondhand but from people who worked directly in the industry at the time.

Don't divide, Intel inside!

I had a very similar issue with some AMD-based servers (bulldozer, I think) about ten years ago. There was a bug where Xen-based virtual machines could set a C-state on cores it was assigned, but for whatever reason it wasn't able to wake them up. It was fun trying to figure out what the heck was going on.

EPYCs have many cores, and most applications (including those with long uptime requirements) use only a subset of the cores continuously. So it is totally normal for some of the cores to go to deep sleep C6 during phases of lower load. It will cause server operators headaches, when those cores don't come back eventually. Reboots help, disabling C6 in the (already running) OS also helps.

Please note, that we are not talking about a core sleeping for three years. We are talking about a core going to deep sleep, when the system has been up for three years or longer.

You've had a Ryzen 7000 series CPU running for nearly 3 years already?

I dont understand opinons like this

Just because it would be dangerous for your nodejs web_app.exe running on ubuntu behind apache fully exposed on the internet

then there are billion other ways to use computers, like even air gapped systems.

So, dont try to justify obvious flaw

3 years is irresponsible? To quote Logan Roy, you, software developers, "are not serious people" [1]. Just out of curiosity looked for a list of longest running electrical devices [2]:

    1840 - The Oxford Electric Bell
    1871 – Souter Lighthouse in South Shields, UK
    1896 – The Isle of Man’s Manx Electric Railway
    1902 – The Centennial Bulb

[1] https://www.youtube.com/watch?v=LZTaXjt2Ggk

[2] https://www.drax.com/electrification/4-of-the-longest-runnin...

You don't need full system resets to get security updates. Kexec, live patching, userspace reboot.

> A machine staying up for almost 3 years is irresponsible in this day and age. [...] but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

You don't need to reboot a machine to update ssh.

You only need to reboot the machine to update the kernel; for everything else, you just have to restart the corresponding user-space processes (and even PID1 can re-exec itself). Most kernel vulnerabilities are not remotely exploitable, so as long as you can trust your user-space processes (and keep them updated), it should be safe enough.

Air gapped machines and kernel live patching both exist.

Are you perhaps a Windows user? In the Linux world updates don't necessarily require reboots.

1042 days ought to be enough for anybody

Kernel bugs are rare. Most (almost every single) vulnerability can be patched without rebooting.