Do not print eos (#4654)
* allow models to use customized token ids during export (#4649) Summary: LLama3.1's [bos and eos](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct/blob/main/tokenizer_config.json) are different from what is hardcoded in the code. This PR updates the export flow to allow read customized token ids instead of hardcoded ones. It also deletes a few metadata entries that are not used by the runner. Pull Request resolved: https://github.com/pytorch/executorch/pull/4649 Differential Revision: D61044259 Pulled By: helunwencser * Do not print eos Summary: We don't want to print eos in the response because some eos tokens could be `<|end_of_text|>`. Differential Revision: D61048254 --------- Co-authored-by: Lunwen He <lunwenh@meta.com>
L
Lunwen He committed
3e0eb0ff0dfd74eb373a79ddf5787441f3745a44
Parent: 728a29d
Committed by GitHub <noreply@github.com>
on 8/12/2024, 9:55:02 PM