Add a tokenizer python script (#1611)
Summary: Add a tokenizer python script that adds some post processing to the vanila `sentencepiece` tokenizer model. This comes in handy when we want to consume it in C++. Pull Request resolved: https://github.com/pytorch/executorch/pull/1611 Reviewed By: mikekgfb Differential Revision: D52821402 Pulled By: larryliu0820 fbshipit-source-id: 91fef7074c22d2a8c51a42922c493423eb6fb401
M
Mengwei Liu committed
78ccd2ec1437ec010c566ebe63ce807258d7f31c
Parent: fa50ded
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 1/19/2024, 8:04:57 AM