Merge branch 'slab/for-7.3/kfree_rcu_nolock' into slab/for-next
Merge series "mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage" from Harry Yoo. From the cover letter [1]: This series improves kmalloc_nolock() and kfree_nolock() coverage in slub_kunit and introduces kfree_rcu_nolock() for unknown context as suggested by Alexei Starovoitov. Unknown context means the caller does not know whether spinning on a lock is safe (e.g., a BPF program attached to an arbitrary kernel function or in NMI context). The slab allocator already supports unknown context via kmalloc_nolock() and kfree_nolock(), but te slab allocator does not support freeing objects by RCU in unknown context. It is not ideal to have completely separate batching for unknown context because the worst scenario where spinning on a lock would lead to deadlock is very rare, and in most cases, it is safe to use the existing mechanism (kfree_rcu_sheaf()). Since most part of the slab allocator already supports unknown context and sheaves support batching kvfree_rcu() calls for slab objects, implement kfree_rcu_nolock() with minimal changes by teaching kfree_rcu_sheaf() how to support unknown context and making it a little bit harder to allocate an empty sheaf, instead of making intrusive changes to the existing kvfree_rcu batching logic. kfree_rcu_nolock() tries to free the object to the rcu sheaf if trylock succeeds. Once the rcu sheaf becomes full, it is submitted to RCU via call_rcu() if spinning is allowed or IRQs are enabled (to avoid calling call_rcu() in the middle of call_rcu()). Otherwise, call_rcu() is deferred via irq work. When there is no sheaf available, kfree_rcu_sheaf() falls back to defer_kfree_rcu(). It submits the object to kvfree_rcu batching via irq work. To do this, patch 6 converts kvfree_rcu to use kvfree_rcu_head without visible changes to the API for now. Unlike kfree_rcu(), only the 2-argument variant is supported. This is because the last resort of the 1-arg variant is synchronize_rcu(), which cannot be used in an unknown context. As suggested by Alexei Starovoitov, kfree_rcu_nolock() can be used with struct kvfree_rcu_head (8 bytes), which is smaller than struct rcu_head (16 bytes). Link: https://lore.kernel.org/all/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org/ [1]
V
Vlastimil Babka (SUSE) committed
564ed40708ebc60a78f280944799cbe8e3401816