SIGN IN SIGN UP

Change the way tokenizer and model are being serialized (#1657)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/1657

In practice we seem to value the BOS and EOS from the model instead of tokenizer. For example `sentencepiece` tells me the tokenizer is having `BOS` being 1 and `EOS` being 2, but the model tells me the `BOS` should be 3 and `EOS` should be two 3s.

To avoid the confusion, only serialize `BOS` and `EOS` into pte file instead of the tokenizer.

Reviewed By: mikekgfb

Differential Revision: D52926490

fbshipit-source-id: 29db53ac12f1ecf97ec245b8edd6150cfbfe057e
M
Mengwei Liu committed
e2369c99df957ffaeefd13fcf14d1e1249cd08a7
Parent: bc317e1
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 1/22/2024, 3:38:25 AM