SIGN IN SIGN UP

Add time to first token for llama runner (#2141)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/2141

Add time to first generated token & other features

- Since we're measuring the first token time, the token rate is measured both at the

* Model Load Time - just a timer around   ET_CHECK_OK_OR_RETURN_ERROR(load());
* Total inference time - Immediately after model load until the end of the inference loop
* >>First token time - From immediately after the model load until the first generated (not prompt) token is printed.
* >>>>Prompt eval - (comparable to llama.cpp prompt_eval_time) prompt array allocation and tokenization. Ends right before the inference loop starts
* >>Remaining tokens - immediately after the first token is outputted until the end of the inference loop
* >>Net eval time - (comparable to llama.cpp eval_time) Total time spent generating tokens.
* Sample time - amount of time spent sampling per token (present in llama.cpp)

bypass-github-executorch-ci-checks
bypass-github-pytorch-ci-checks

Reviewed By: digantdesai, Jack-Khuu

Differential Revision: D54223564

fbshipit-source-id: 3846903b56d20e2d4159fae63de3d471e5677c51
V
Varun Puri committed
caee336d6842713937daf717882aa146edb29ace
Parent: 9e83fde
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 3/14/2024, 8:31:29 PM