0 0 1 Python

Update setup.sh for tokenizer selection (#3207)

Summary:
For LLAMA3, users need to use tiktoken. Add a option to load from env var.

Pull Request resolved: https://github.com/pytorch/executorch/pull/3207

Reviewed By: cccclai

Differential Revision: D56430637

Pulled By: kirklandsign

fbshipit-source-id: cc1cc50100d6142510a455ca29d56a810942f90b

Hansong Zhang committed 2y ago

1a93deebdadd983fdda1313122b0750c8c6b8e61

Parent: 67123b6

Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 4/22/2024, 8:08:18 PM