Add time to first token for llama runner (#2141)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/2141 Add time to first generated token & other features - Since we're measuring the first token time, the token rate is measured both at the * Model Load Time - just a timer around ET_CHECK_OK_OR_RETURN_ERROR(load()); * Total inference time - Immediately after model load until the end of the inference loop * >>First token time - From immediately after the model load until the first generated (not prompt) token is printed. * >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts * >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop * >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens. * Sample time - amount of time spent sampling per token (present in llama.cpp) bypass-github-executorch-ci-checks bypass-github-pytorch-ci-checks Reviewed By: digantdesai, Jack-Khuu Differential Revision: D54223564 fbshipit-source-id: 3846903b56d20e2d4159fae63de3d471e5677c51
V
Varun Puri committed
caee336d6842713937daf717882aa146edb29ace
Parent: 9e83fde
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 3/14/2024, 8:31:29 PM