Fix discrepancy on eos and bos tokens for different pte files (#3923)
Summary: Currently the logic is that if pte file doesn't contain metadata such as vocab size, bos id and eos id, use a default value. This doesn't work well because llama2 and llama3 are using different tokenizer artifacts, therefore they have different default values. Instead assuming a default value at the runner level, we should deduce the values for vocab size bos id and eos id from the tokenizer aritifact, and let the metadata in the pte file override it. This PR fixes this issue. Pull Request resolved: https://github.com/pytorch/executorch/pull/3923 Test Plan: demo app works for pte file without vocab size and eos id bos id:  Reviewed By: shoumikhin Differential Revision: D58393326 Pulled By: larryliu0820 fbshipit-source-id: e306493c569d2350fe6df6617200c337c627024c
M
Mengwei Liu committed
159375669872da608921dfa7251d94d2ca0aa223
Parent: 10de8b2
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 6/12/2024, 12:06:12 AM