SIGN IN SIGN UP

Move tokenizer into extension/llm/tokenizer (#4278)

Summary:
This PR moves the titoken and bpe tokenizers into `extension/llm/tokenizer` such that they can be reused by other models.

Note: Currently the tiktoken has two sets of unit tests based on llama2's tokenizer:
- default
- multimodal

This PR only moves the default unit test into extension and keeps the multimodal's unit tests inside llama2/tokenizer.

Pull Request resolved: https://github.com/pytorch/executorch/pull/4278

Test Plan:
- test/run_oss_cpp_tests.sh examples/models/llama2/tokenizer/test
- test/run_oss_cpp_tests.sh extension/llm/tokenizer/test

Reviewed By: larryliu0820

Differential Revision: D59822702

Pulled By: helunwencser

fbshipit-source-id: 5d51ba3e44c9b2d9dc77b9f4349b58947ed68502
H
helunwencser committed
0cde6b8967e1f227c34ef51c1fd536f42e590ed6
Parent: 740a0a5
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 7/17/2024, 6:12:07 AM