tests : multi-thread the tokenizer tests (#5474)
* tests : multi-thread the tokenizer tests ggml-ci * unicode : fix data race for unidentified codepoints ggml-ci * unicode : minor style fixes ggml-ci
G
Georgi Gerganov committed
cf45252a7cfcb998bade46a886e20477cecc538a
Parent: 03bf161
Committed by GitHub <noreply@github.com>
on 2/13/2024, 1:14:22 PM