Simplify tokenizer header (#4283)
Summary: The dependency and includes for tokenizer are unnecessarily complicated. This PR deletes some unused and updates some to use/depend on what it really uses. Pull Request resolved: https://github.com/pytorch/executorch/pull/4283 Test Plan: - test/run_oss_cpp_tests.sh examples/models/llama2/tokenizer/test - test/run_oss_cpp_tests.sh extension/llm/tokenizer/test Reviewed By: larryliu0820 Differential Revision: D59837081 Pulled By: helunwencser fbshipit-source-id: 4c69f01a868a52207725ae39f65fa62ead7b6749
H
helunwencser committed
844a69f268f8b04bbc52555aa12f56578c3d2f2f
Parent: 6dbb4dc
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 7/22/2024, 7:52:29 PM