Fix eval (#5955)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/5955 This PR fixes a bunch of issues in the eval pipeline: - Use the right token for `eot_token_id` - Do not add `bos` and `eos` during `tok_encode` based on this [discussion](https://fburl.com/code/uifmt746). - Update `executorch/examples/models/llama2/tokenizer/tiktoken.py` to be synced with llama 3.1's official [code](https://github.com/meta-llama/llama-models/blob/main/models/llama3/api/tokenizer.py). Majorly updated set of special tokens. Reviewed By: mergennachin Differential Revision: D62198560 fbshipit-source-id: 856a625ef0d607ab0e910eefe455a6dff1240584
L
Lunwen He committed
2027a14369b253eea5cd38cce62b53e0f5d09f76
Parent: 80afaf2
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 10/9/2024, 7:28:09 PM