fix(aws-pcs): stop needrestart from restarting slurmd and killing running jobs (#1165)
* fix(aws-pcs): stop needrestart from restarting slurmd and killing running jobs apt-daily-upgrade updates base libraries (e.g. glibc); needrestart then auto-restarts every service linked against them. slurmd links libc, so it is restarted mid-upgrade, which tears down slurmstepd and kills the jobs running on the node (the job then requeues from scratch). This matched a customer report of jobs being killed at the same time apt-daily-upgrade.timer fired. Add a needrestart drop-in via CNG UserData that excludes slurmd/slurmctld/ slurmdbd from automatic restart. Security upgrades still install; only the Slurm-daemon auto-restart is suppressed, so a running job is never taken down. Verified end-to-end on real hardware: with the guard, a long-running job survives an apt upgrade + needrestart that updates glibc; without it, slurmd restarts and the job is killed/requeued. * fix(aws-pcs): use ASCII dash in needrestart guard comment Avoid a non-ASCII em-dash in the heredoc that gets written to /etc/needrestart/conf.d/90-pcs-slurm.conf (rendered as an escaped code point on disk). Comment-only change; the override_rc line is unaffected. * fix(aws-pcs): apply needrestart slurmd guard to GPU CNG templates too The needrestart/slurmd job-kill guard was only added to add-cng.yaml. Apply the same drop-in to the P5/P6-B200/P6-B300 GPU templates, which run the long distributed-training jobs most affected by an unattended slurmd restart. * docs(aws-pcs): document needrestart/slurmd job-kill issue and fix (OPERATIONS 6.2) Explain how apt-daily-upgrade -> needrestart -> slurmd restart kills running jobs, the shipped needrestart drop-in that excludes the Slurm daemons, the heavier timer-disable alternative, and why the drop-in is forward-compatible. * docs(aws-pcs): reframe OPERATIONS 6.2 as user-facing guidance Rewrite the needrestart/slurmd section from an incident write-up into a note for users: state plainly that needrestart restarting slurmd stops running jobs, that the templates already include the mitigation so no action is needed, and that the drop-in stays harmless (no conflict/error) if the platform fixes this upstream. * fix(aws-pcs): narrow needrestart guard to slurmd only slurmctld/slurmdbd are not systemd services on PCS login/compute nodes (PCS runs the controller managed-side), so excluding them was a no-op. Guard only slurmd, which qr(^slurmd) also matches for the versioned units (slurmd-25.11 etc.). Update OPERATIONS 6.2 accordingly. * docs(aws-pcs): tighten OPERATIONS 6.2 Condense the needrestart/slurmd note to three short paragraphs and drop the 'disable automatic upgrades entirely' alternative — unnecessary since the templates already guard slurmd.
D
Daisuke Miyamoto committed
ac667129f74fdda385f0babe05bf020389ccddeb
Parent: 3e4edac
Committed by GitHub <noreply@github.com>
on 7/7/2026, 9:00:18 AM