0 0 1 Python

Add tiktoken OFF/ON options for android prebuilt aar (#4203)

Summary:
BPE (tiktoken OFF) is for llama2, and tiktoken ON is for llama3.

Pull Request resolved: https://github.com/pytorch/executorch/pull/4203

Reviewed By: cccclai

Differential Revision: D59608775

Pulled By: kirklandsign

fbshipit-source-id: b1bb9411982b078b752eb67a48e84d308db5ac07

Hansong Zhang committed 2y ago

e9aa542ef4002e6811b26a8b808335e953c92f85

Parent: d8c4afa

Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 7/12/2024, 8:45:10 PM