Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) (#7425)
* Update brute force test: add_special * Update brute force test: default values for add_bos_token and add_eos_token * Enable rtrim when pre-inserting BOS Co-authored-by: Georgi Gerganov <[email protected]> * Revert "server : fix test regexes"
J
jaime-m-p committed
d7e852c1bc8e85bf62a6f1aede08cd2de723404a
Parent: 917dc8c
Committed by GitHub <[email protected]>
on 5/21/2024, 12:39:48 PM