SIGN IN SIGN UP

fix(aws-pcs): stop needrestart from restarting slurmd and killing running jobs (#1165)

* fix(aws-pcs): stop needrestart from restarting slurmd and killing running jobs

apt-daily-upgrade updates base libraries (e.g. glibc); needrestart then
auto-restarts every service linked against them. slurmd links libc, so it is
restarted mid-upgrade, which tears down slurmstepd and kills the jobs running
on the node (the job then requeues from scratch). This matched a customer
report of jobs being killed at the same time apt-daily-upgrade.timer fired.

Add a needrestart drop-in via CNG UserData that excludes slurmd/slurmctld/
slurmdbd from automatic restart. Security upgrades still install; only the
Slurm-daemon auto-restart is suppressed, so a running job is never taken down.

Verified end-to-end on real hardware: with the guard, a long-running job
survives an apt upgrade + needrestart that updates glibc; without it, slurmd
restarts and the job is killed/requeued.

* fix(aws-pcs): use ASCII dash in needrestart guard comment

Avoid a non-ASCII em-dash in the heredoc that gets written to
/etc/needrestart/conf.d/90-pcs-slurm.conf (rendered as an escaped code point
on disk). Comment-only change; the override_rc line is unaffected.

* fix(aws-pcs): apply needrestart slurmd guard to GPU CNG templates too

The needrestart/slurmd job-kill guard was only added to add-cng.yaml. Apply the
same drop-in to the P5/P6-B200/P6-B300 GPU templates, which run the long
distributed-training jobs most affected by an unattended slurmd restart.

* docs(aws-pcs): document needrestart/slurmd job-kill issue and fix (OPERATIONS 6.2)

Explain how apt-daily-upgrade -> needrestart -> slurmd restart kills running jobs,
the shipped needrestart drop-in that excludes the Slurm daemons, the heavier
timer-disable alternative, and why the drop-in is forward-compatible.

* docs(aws-pcs): reframe OPERATIONS 6.2 as user-facing guidance

Rewrite the needrestart/slurmd section from an incident write-up into a note for
users: state plainly that needrestart restarting slurmd stops running jobs, that
the templates already include the mitigation so no action is needed, and that the
drop-in stays harmless (no conflict/error) if the platform fixes this upstream.

* fix(aws-pcs): narrow needrestart guard to slurmd only

slurmctld/slurmdbd are not systemd services on PCS login/compute nodes (PCS runs
the controller managed-side), so excluding them was a no-op. Guard only slurmd,
which qr(^slurmd) also matches for the versioned units (slurmd-25.11 etc.).
Update OPERATIONS 6.2 accordingly.

* docs(aws-pcs): tighten OPERATIONS 6.2

Condense the needrestart/slurmd note to three short paragraphs and drop the
'disable automatic upgrades entirely' alternative — unnecessary since the
templates already guard slurmd.
D
Daisuke Miyamoto committed
ac667129f74fdda385f0babe05bf020389ccddeb
Parent: 3e4edac
Committed by GitHub <noreply@github.com> on 7/7/2026, 9:00:18 AM