Max Executor Threads (#566)
# Overview For one of my services, we use `async DBOS.scheduled workflows` with a high frequency of every 30s for about 10 different jobs, and gradually the Python process keeps accumulating memory (even if there are no spikes in concurrent workflows being run). # Details After forking the repository, I noticed that the `max_workers` was set to `sys.maxsize`, which on 64-bit arch platforms is `9223372036854775807`. I'm not sure if this is intentional, but `ThreadPoolExecutors` default `min(32, (os.process_cpu_count() or 1) + 4)` is much safer to prevent large memory consumption over time (since `ThreadPoolExecutor` doesn't free idle threads until the executor is destroyed). # Solution I removed the fixed `sys.maxsize` for the number of workers and added a configurable parameter called `max_executor_threads` which, if specified, is used; else the default value for `ThreadPoolExecutor` is used. ## Note I haven't had much time to dig into the repo, but I believe the root cause of the gradual increase in threads is that some references are lingering, preventing idle threads from being utilized by the `ThreadPoolExecutor`, particularly when using async scheduled workflows. This leads the executor to spawn new threads periodically.
M
Mahit Mehta committed
c68119a18eedb2a252a4d0044fec130ed26845c5
Parent: c1ba8e3
Committed by GitHub <noreply@github.com>
on 1/26/2026, 5:52:40 PM