feat(server): log in-flight jobs when the worker event loop stalls (#24767)
## Context The Query Read Timeout investigation (Aug 25) proved worker pods freeze their event loop for 10-30s (Sentry breadcrumb silence ending at the exact error millisecond, BullMQ lock-renewal failures in the same windows), timing out every in-flight query on the pod. What we cannot yet name is which job holds the loop. ## What this does Adds `EventLoopStallMonitorService`: a 500ms unref'd interval measures its own scheduling drift; when a tick fires more than 5s late, it logs the stall duration plus every job in flight at that moment (queue, job name, workspaceId, elapsed). Jobs register/unregister in `MessageQueueExplorer` around processing. A stalled loop cannot observe itself from inside — a late timer reveals both the stall and its suspects. Sample output (one WARN per stall, so tens of lines per day fleet-wide): ``` WARN [EventLoopStallMonitorService] Event loop stalled for 27431ms with 4 job(s) in flight: messaging-queue/MessagingMessagesImportJob workspace=cbee49b7 elapsedMs=31204; entity-events-to-db-queue/UpsertTimelineActivityFromInternalEvent workspace=d46df29e elapsedMs=27811; workflow-queue/RunWorkflowJob workspace=6f6b2cc0 elapsedMs=27515; calendar-queue/CalendarEventsImportJob workspace=21fc27f4 elapsedMs=2103 ``` Jobs whose elapsed matches or exceeds the stall duration were running through the freeze (suspects); small elapsed means they started just before the late tick (bystanders). An earlier commit added env-gated Sentry continuous profiling as a second instrument; it was reverted in-branch — we first try to attribute the freezes with the existing transaction profiles and this stall log.
C
Charles Bochet committed
2ae1dfbce8100f6c4128f6cd64805dfcb4c017dc
Parent: 4883eaa
Committed by GitHub <noreply@github.com>
on 8/25/2026, 5:57:50 PM