0 0 1 Python

Bug fix in bpe tokenizer (#4149)

Summary:
- Record bos/eos in the binary format
- Updated tests

Pull Request resolved: https://github.com/pytorch/executorch/pull/4149

Reviewed By: larryliu0820

Differential Revision: D59349794

Pulled By: guangy10

fbshipit-source-id: ecdd5bd22dfcdc60429d179f07a61d46c832ef87

Guang Yang committed 2y ago

d948c1acd7d8a2b7e41bcdc28155f2ceca2af0e1

Parent: 5f2ab0e

Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 7/8/2024, 10:45:22 PM