Fix mask dtype mismatching query dtype properly (#1984)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/1984 Some refactoring on the mask handling in llama. In our quantized group embedding implementation we directly call llama_quantized.embedding_type but that operator always return fp32. We need to convert it back to fp16 based on the dtype we got in the QuantizedGroupEmbedding module. Reviewed By: kimishpatel Differential Revision: D53796343 fbshipit-source-id: 09537724bc648a1e2bf00bb9be2c56b19894acd1
M
Mengwei Liu committed
4b9e8a4ae1511ef14d080ab8fb4e782177259f36
Parent: dd01c6d
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 2/16/2024, 6:21:51 AM