0 0 1 Python

Add a tokenizer (#1641)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/1641

Add a tokenizer in examples. This needs to consume the artifact generated by `tokenizer.py`.

Reviewed By: mikekgfb

Differential Revision: D52894344

fbshipit-source-id: e3f03fa5cd2d2607ef2bfd371125de1f72adc968

Mengwei Liu committed 2y ago

6aab49e49c85035924b70ee2292699996dcc9037

Parent: 78ccd2e

Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 1/19/2024, 8:04:57 AM