Move tokenizer into extension/llm/tokenizer (#4278)
Summary: This PR moves the titoken and bpe tokenizers into `extension/llm/tokenizer` such that they can be reused by other models. Note: Currently the tiktoken has two sets of unit tests based on llama2's tokenizer: - default - multimodal This PR only moves the default unit test into extension and keeps the multimodal's unit tests inside llama2/tokenizer. Pull Request resolved: https://github.com/pytorch/executorch/pull/4278 Test Plan: - test/run_oss_cpp_tests.sh examples/models/llama2/tokenizer/test - test/run_oss_cpp_tests.sh extension/llm/tokenizer/test Reviewed By: larryliu0820 Differential Revision: D59822702 Pulled By: helunwencser fbshipit-source-id: 5d51ba3e44c9b2d9dc77b9f4349b58947ed68502
H
helunwencser committed
0cde6b8967e1f227c34ef51c1fd536f42e590ed6
Parent: 740a0a5
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 7/17/2024, 6:12:07 AM