Morph MORPH
SIGN IN SIGN UP
ggml-org / llama.cpp UNCLAIMED

LLM inference in C/C++

0 0 0 C++

Tokenizer SPM fixes for phi-3 and llama-spm (#7375)

* Update brute force test: special tokens
* Fix added tokens
  - Try to read 'added_tokens.json'.
  - Try to read 'tokenizer_config.json'.
  - Try to read 'tokenizer.json'.
* Fix special tokens rtrim

Co-authored-by: Georgi Gerganov <[email protected]>
* server : fix test regexes
J
jaime-m-p committed
917dc8cfa67a72fb7c8bf7392270da3bf4833af4
Parent: fabf30b
Committed by GitHub <[email protected]> on 5/20/2024, 6:15:57 PM