SIGN IN SIGN UP

move get tokenizer to export_llama_lib (#4451)

Summary:
Move the get tokenizer logic to export_llama_lib so we can use it during export as well. It's needed in the second diff when to run pt2e for calibration

Pull Request resolved: https://github.com/pytorch/executorch/pull/4451

ghstack-source-id: 235865854
exported-using-ghexport

Reviewed By: larryliu0820

Differential Revision: D60419366

fbshipit-source-id: c9d6d7b6c2615bfa957791e13b2f640f0dbf3158
C
Chen Lai committed
d207eb0bec46866542ce378a8907d8e02660a083
Parent: c1d53ba
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 8/1/2024, 7:12:33 PM