Qualcomm AI Engine Direct - Model sharding for LLM (#4923)
For LLM, model size is too large to fit in device memory for inference. Therefore, we need to divide the model into a few parts in order to avoid inference time out-of-memory errors. Summary: - Use custom fallback op to split graph - Add splill fill feature - Add model sharding argument for qnn
S
shewu-quic committed
4116cb24dbf811d820ad7c6024dea738162f6c4a
Parent: 35e2302
Committed by GitHub <noreply@github.com>
on 8/28/2024, 9:16:10 AM