mudler / LocalAI UNCLAIMED

LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

0 0 66 Go

feat: stream tokens usage (#4415)

* Use pb.Reply instead of []byte with Reply.GetMessage() in llama grpc to get the proper usage data in reply streaming mode at the last [DONE] frame

* Fix 'hang' on empty message from the start

Seems like that empty message marker trick was unnecessary

---------

Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

mintyleaf committed 1y ago

2bc4b56a79e375839e78e3f60e41aff9aa3ef542

Parent: fc920cc

Committed by GitHub <noreply@github.com> on 12/18/2024, 8:48:50 AM