SIGN IN SIGN UP

Fix eval (#5955)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/5955

This PR fixes a bunch of issues in the eval pipeline:
- Use the right token for `eot_token_id`
- Do not add `bos` and `eos` during `tok_encode` based on this [discussion](https://fburl.com/code/uifmt746).
- Update `executorch/examples/models/llama2/tokenizer/tiktoken.py` to be synced with llama 3.1's official [code](https://github.com/meta-llama/llama-models/blob/main/models/llama3/api/tokenizer.py). Majorly updated set of special tokens.

Reviewed By: mergennachin

Differential Revision: D62198560

fbshipit-source-id: 856a625ef0d607ab0e910eefe455a6dff1240584
L
Lunwen He committed
2027a14369b253eea5cd38cce62b53e0f5d09f76
Parent: 80afaf2
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 10/9/2024, 7:28:09 PM