SIGN IN SIGN UP

Add a tokenizer python script (#1611)

Summary:
Add a tokenizer python script that adds some post processing to the vanila `sentencepiece` tokenizer model. This comes in handy when we want to consume it in C++.

Pull Request resolved: https://github.com/pytorch/executorch/pull/1611

Reviewed By: mikekgfb

Differential Revision: D52821402

Pulled By: larryliu0820

fbshipit-source-id: 91fef7074c22d2a8c51a42922c493423eb6fb401
M
Mengwei Liu committed
78ccd2ec1437ec010c566ebe63ce807258d7f31c
Parent: fa50ded
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 1/19/2024, 8:04:57 AM