feat: Add endpoints for tokenize, detokenize and count tokens (#1136)
* Add endpoint to count tokens * Add tokenize and detokenize endpoints * Change response key to tokens for tokenize endpoint * Fix dependency bug * Cleanup * Remove example added by mistake * Move tokenize, detokenize, and count to Extras namespace. Tag existing endpoints --------- Co-authored-by: Andrei Betlen <abetlen@gmail.com>
F
Felipe Lorenz committed
c139f8b5d50f6f416a24c0ba65983a3fb84bf2f3
Parent: 1f3156d
Committed by GitHub <noreply@github.com>
on 3/9/2024, 2:09:00 AM