Making large AI models cheaper, faster and more accessible
[pipeline]: fix p2p comm, add metadata cache and support llama interleaved pp (#5134)
* test: add more p2p tests * fix: remove send_forward_recv_forward as p2p op list need to use the same group * fix: make send and receive atomic * feat: update P2PComm fn * feat: add metadata cache in 1f1b * feat: add metadata cache in interleaved pp * feat: modify is_xx_stage fn * revert: add _broadcast_object_list * feat: add interleaved pp in llama policy * feat: set NCCL_BUFFSIZE in HybridParallelPlugin
W
Wenhao Chen committed
4fa689fca1ecf50b8e905cf1c74d4a2c08219daf
Parent: af95267
Committed by GitHub <noreply@github.com>
on 12/22/2023, 2:44:00 AM