Martin's Blog

My Server Froze After 1044 Days of Uptime

linuxhetzneramdsysadmin
Contents

One day SSH to my dedicated server simply stopped responding. No error, no timeout message, just nothing. Ping still worked perfectly. I asked Hetzner for a remote KVM console and found a screen full of kernel soft lockup messages.

The cause turned out not to be a failing disk, bad RAM or a kernel bug in the usual sense. It was a documented AMD CPU erratum that is triggered by nothing more than uptime.

The machine

  • Hetzner dedicated server, AMD Ryzen 7 3700X (Zen 2, “Matisse”)
  • ASRock B450 Pro4, 64 GB ECC RAM, two NVMe drives in md RAID1
  • Debian 12, kernel 6.1.0-13-amd64

That kernel version is a hint in itself: the box had been running since late 2023 without a reboot. Yes, I know.

What the console showed

watchdog: BUG: soft lockup - CPU#8 stuck for 29009s! [kworker/8:2:2852286]
RIP: 0010:smp_call_function_many_cond+0xeb/0x2f0
...
rcu: INFO: rcu_preempt self-detected stall on CPU

Two CPUs kept showing up, and both were stuck in smp_call_function_many_cond. That function sends an inter-processor interrupt (IPI) to other cores and waits until all of them confirm they have handled it.

  • CPU 8 was a kernel worker toggling a static key, which means patching live kernel code. That needs every CPU to acknowledge.
  • CPU 11 was a thread of a userspace application calling madvise() to free memory. That triggers a TLB shootdown, which also needs every CPU to acknowledge.

Both were victims. Some other core never answered, so they spun forever. CPU 8 had been stuck for 29009 seconds, about 8 hours. The core actually causing the problem did not appear anywhere in the output.

Why ping worked but SSH didn’t: ICMP echo replies are generated inside the kernel’s network handling on cores that are still alive, so no userspace is involved. sshd has to accept, fork and map memory, and those memory operations need the same cross-CPU acknowledgements, so it blocked on the same stuck core.

There was nothing to do on the running system, so I did a hardware reset from Hetzner Robot.

Reading the logs of the dead boot

After the reset, journalctl -k -b -1 showed the first sign of trouble well before the soft lockups:

INFO: task khugepaged:121 blocked for more than 120 seconds.
...
 __flush_work.isra.0+0x173/0x280
 __lru_add_drain_all+0x14f/0x1f0
 khugepaged+0x63/0x970

__lru_add_drain_all queues a small job on every CPU and waits for all of them to finish. One CPU was not running its queued work anymore. Working back from the two-minute hung task check interval, khugepaged got stuck at around 11:29. The userspace logs agree: the last cron entry is from 11:29:01, and a Docker healthcheck that ran every 30 seconds stops after 11:28:33. journald itself ended up stuck in uninterruptible sleep, and systemd’s SIGKILL had no effect on it.

What was missing from the logs turned out to be the most useful clue:

  • No hard lockup report, even though the NMI watchdog was enabled. A core spinning with interrupts disabled would have been caught.
  • The RCU stall report only named the waiting CPU. A core busy in the kernel would have been listed too.
  • No machine check or EDAC errors, with ECC confirmed active.

This all fits a core that the kernel believes is idle but that never wakes up again. An idle core counts as quiescent for RCU, and a halted core fires neither its own soft lockup timer nor NMI watchdog events. It is invisible to every detector.

The uptime

The kernel timestamps on the console were at about 90,242,176 seconds when I took the screenshots, and the hang had started roughly 9 hours earlier. That puts the uptime at the moment of the hang at about 90.21 million seconds, or 1044.1 days.

1044 days is a very specific number.

AMD erratum 1474

In 2023 AMD published erratum 1474 in the revision guide for Family 17h Models 30h-3Fh (EPYC 7002 “Rome”). In short: roughly 1044 days after the last system reset, a CPU core can fail to come back out of the CC6 power state and hang. The exact timing depends on spread spectrum and REFCLK frequency. AMD’s workaround is to either disable CC6 or reboot before that point, and no fix is planned. Community analysis at the time traced it to a counter overflow slightly before 1043 days (ServeTheHome has a good write-up).

The erratum was first listed for EPYC Rome, but my 3700X is the same Zen 2 core. The Xen developers later extended their workaround to all Family 17h models (Zen, Zen+ and Zen 2), and VMware ESXi disables CC6 automatically after 1000 days of uptime in recent versions.

A core stuck in CC6 ignores interrupts, so it never acknowledges IPIs or runs queued work. Every other core eventually waits on it. That matches everything in my logs.

The good news: a reset fixes it

The counter starts over at every system reset, so after the hardware reset the machine is fine again. Without another reboot, the next danger window would be around July 2029. I would rather not rely on remembering that, so I disabled CC6.

Disabling CC6 on Linux

Linux doesn’t know CC6 by that name. It only sees the ACPI idle states the firmware offers:

$ grep . /sys/devices/system/cpu/cpu0/cpuidle/state*/desc
/sys/devices/system/cpu/cpu0/cpuidle/state0/desc:CPUIDLE CORE POLL IDLE
/sys/devices/system/cpu/cpu0/cpuidle/state1/desc:ACPI FFH MWAIT 0x0
/sys/devices/system/cpu/cpu0/cpuidle/state2/desc:ACPI IOPORT 0x414

On Zen boards, the I/O-port based C2 state (ACPI IOPORT 0x414) is the one where the core drops into CC6. C1 (MWAIT) only halts the core. So disabling C2 keeps the cores out of CC6. On my box, C2 was where the cores spent most of their idle time.

You could also disable C-states in the BIOS, but on a rented server that means another KVM session, and it’s easier to do from the OS.

Right away, without a reboot

apt install linux-cpupower
cpupower idle-set -d 2
cpupower idle-info

C2 is now listed as C2 (DISABLED). To confirm it’s really unused, run cpupower idle-info again after a minute: the C2 Usage counter should stay the same while the C1 counter keeps growing.

This only lasts until the next reboot.

Permanently, via the kernel command line

The kernel parameter processor.max_cstate=1 limits the ACPI idle driver to C1. Instead of editing /etc/default/grub directly, I used a drop-in file, which is easy to find and easy to remove later:

mkdir -p /etc/default/grub.d
cat > /etc/default/grub.d/99-disable-cc6.cfg <<'EOF'
# AMD erratum 1474: keep Zen 2 cores out of CC6 (ACPI C2)
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT processor.max_cstate=1"
EOF
update-grub
grep -c 'processor.max_cstate=1' /boot/grub/grub.cfg   # should be > 0

Future kernel updates run update-grub automatically, so new kernels pick up the parameter too.

After a reboot, only POLL and C1 are left:

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.1.0-44-amd64 root=UUID=... ro consoleblank=0 processor.max_cstate=1

# ls /sys/devices/system/cpu/cpu0/cpuidle/
state0  state1

# cpupower idle-info
CPUidle driver: acpi_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 2
Available idle states: POLL C1
POLL:
Flags/Description: CPUIDLE CORE POLL IDLE
Latency: 0
Usage: 335813
Duration: 8301606
C1:
Flags/Description: ACPI FFH MWAIT 0x0
Latency: 1
Usage: 468640
Duration: 39865747

ACPI IOPORT 0x414 is gone, so there is no way for a core to enter CC6 anymore.

To undo it, delete the file, run update-grub and reboot.

The cost is slightly higher idle power and temperature. For a server in Hetzner’s datacenter that doesn’t matter to me.

Making the next hang less painful

The server sat dead for about 9 hours before I noticed, and that is the part that actually cost me. The hung task detector had fired about 25 minutes before the first soft lockup, so the machine was telling anyone who looked that something was wrong at 11:33. Nobody was looking.

The tempting fix is to have the kernel reboot itself. Linux can turn its own hang detectors into a panic:

kernel.softlockup_panic = 1
kernel.panic_on_rcu_stall = 1
kernel.hung_task_panic = 1

hung_task_panic would have caught this one at 11:33 and turned 9 hours of downtime into about a minute. I decided against it anyway.

The reason is that the detectors are heuristics, not verdicts. hung_task_panic reboots the machine when any task sits in uninterruptible sleep for 120 seconds, and on a Docker host that is not as rare as it sounds: a heavy fsync, an overlayfs stall, a backup saturating the NVMe. Soft lockups and RCU stalls can likewise fire on transient overload that would have cleared by itself. I would be trading a hang that can no longer happen for unplanned reboots in the middle of real work, with in-flight writes lost and no warning. That is a bad trade once CC6 is disabled, because the specific failure these knobs would catch is already gone.

What I did set is the one that has no false positives:

echo 'kernel.panic = 30' > /etc/sysctl.d/90-panic.conf
sysctl --system

Debian leaves kernel.panic at 0, which means a panicking kernel sits there forever. On a machine in someone else’s datacenter that is the wrong default: by the time the kernel has panicked the box is dead either way, so rebooting 30 seconds later costs nothing and can only help. Note what it does not do, though. It acts only after an actual panic, and this hang never panicked. Those are two different problems, and it is easy to mistake one for the other.

Worth knowing: an automatic reboot throws away the evidence. A panic that reboots in 30 seconds never reaches the disk, so journalctl -k -b -1 will not have it. /sys/fs/pstore/ is worth checking after a crash, and netconsole to another host is the usual answer if you care about post-mortems.

The real fix for “dead for 9 hours” was never a kernel setting. It was that nothing external was watching. An uptime check from another machine would have told me in two minutes, and then rebooting is my decision rather than a heuristic’s.

Lessons

  • Reboot your servers. My kernel was almost three years old; after the reset the box came up on 6.1.0-44. Regular kernel-update reboots alone would have prevented this completely.
  • “It responds to ping” doesn’t mean the machine is fine. Kernel-level networking can keep working on healthy cores while everything else is deadlocked.
  • If an older AMD Zen box hangs, check the uptime first. If it’s around 1042-1044 days, you probably already know the cause.