SIGN IN SIGN UP

Simplify tokenizer header (#4283)

Summary:
The dependency and includes for tokenizer are unnecessarily complicated. This PR deletes some unused and updates some to use/depend on what it really uses.

Pull Request resolved: https://github.com/pytorch/executorch/pull/4283

Test Plan:
- test/run_oss_cpp_tests.sh examples/models/llama2/tokenizer/test
- test/run_oss_cpp_tests.sh extension/llm/tokenizer/test

Reviewed By: larryliu0820

Differential Revision: D59837081

Pulled By: helunwencser

fbshipit-source-id: 4c69f01a868a52207725ae39f65fa62ead7b6749
H
helunwencser committed
844a69f268f8b04bbc52555aa12f56578c3d2f2f
Parent: 6dbb4dc
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 7/22/2024, 7:52:29 PM