Change the way tokenizer and model are being serialized (#1657)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/1657 In practice we seem to value the BOS and EOS from the model instead of tokenizer. For example `sentencepiece` tells me the tokenizer is having `BOS` being 1 and `EOS` being 2, but the model tells me the `BOS` should be 3 and `EOS` should be two 3s. To avoid the confusion, only serialize `BOS` and `EOS` into pte file instead of the tokenizer. Reviewed By: mikekgfb Differential Revision: D52926490 fbshipit-source-id: 29db53ac12f1ecf97ec245b8edd6150cfbfe057e
M
Mengwei Liu committed
e2369c99df957ffaeefd13fcf14d1e1249cd08a7
Parent: bc317e1
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 1/22/2024, 3:38:25 AM