Qualcomm AI Engine Direct - improve llama3.2 3B TPS (#20903)
### **User description** ### Summary - Update num_sharding from 4 to 3, TPS change from 26.27 to 28.95. - Remove dummy Quantize/Dequantize due to tag_ios failure + remove dummy mul 1, TPS change from 28.95 to 29.34. - Minor fix before change: <img width="991" height="242" alt="image" src="https://github.com/user-attachments/assets/0d04eeb9-0934-405e-a21c-e2bf31a77211" /> after change: <img width="1045" height="242" alt="image" src="https://github.com/user-attachments/assets/b7992e63-1677-4e8a-a974-3702e4d12d4c" /> #### Minor fix - Update cli tool option - Update unit test for spill_fill for newer version of sdk (in newer version of sdk, the spill_fill become 0 and fail the unit test) - Update unit test root (examples... -> executorch.examples...) ### Test plan ``` python ./examples/qualcomm/oss_scripts/llama/llama.py --artifact llama3_2_3b_instruct --build_folder build-android --soc_model SM8850 --checkpoint ~/.llama/checkpoints/Llama3.2-3B-Instruct/consolidated.00.pth --params ~/.llama/checkpoints/Llama3.2-3B-Instruct/params.json --tokenizer_model ~/.llama/checkpoints/Llama3.2-3B-Instruct/tokenizer.model --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --decoder_model llama3_2-3b_instruct --model_mode kv --max_seq_len 4096 --device ef5e4029 --host localhost --system_prompt "You are a helpful assistant." --max_context_len 4096 --calib_tasks wikitext ```
Q
qti-chenweng committed
fcc1020905fcc6fbfe2fd2a11b15e01b6b985080
Parent: 5828778
Committed by GitHub <noreply@github.com>
on 7/21/2026, 11:31:46 PM