SIGN IN SIGN UP

Do not print eos (#4654)

* allow models to use customized token ids during export (#4649)

Summary:
LLama3.1's [bos and eos](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct/blob/main/tokenizer_config.json) are different from what is hardcoded in the code. This PR updates the export flow to allow read customized token ids instead of hardcoded ones.

It also deletes a few metadata entries that are not used by the runner.

Pull Request resolved: https://github.com/pytorch/executorch/pull/4649

Differential Revision: D61044259

Pulled By: helunwencser

* Do not print eos

Summary: We don't want to print eos in the response because some eos tokens could be `<|end_of_text|>`.

Differential Revision: D61048254

---------

Co-authored-by: Lunwen He <lunwenh@meta.com>
L
Lunwen He committed
3e0eb0ff0dfd74eb373a79ddf5787441f3745a44
Parent: 728a29d
Committed by GitHub <noreply@github.com> on 8/12/2024, 9:55:02 PM