COMMITS
/ examples/tokenize/tokenize.cpp October 10, 2024
D
common : use common_ prefix for common library functions (#9805)
Diego Devesa committed
September 15, 2024
G
common : reimplement logging (#9418)
Georgi Gerganov committed
August 15, 2024
Z
common : remove duplicate function llama_should_add_bos_token (#8778)
Zhenwei Jin committed
July 27, 2024
S
ggml : reduce hash table reset cost (#8698)
slaren committed
July 11, 2024
C
tokenize : add --no-parse-special option (#8423)
compilade committed
July 4, 2024
D
tokenize : add --show-count (token) option (#8299)
Daniel Bevenius committed
May 25, 2024
M
Make tokenize CLI tool have nicer command line arguments. (#6188)
Mikko Juola committed
April 9, 2024
J
BERT tokenizer fixes (#6498)
Jared Van Bortel committed
February 16, 2024
B
ggml : add numa options (#5377)
bmwl committed
November 18, 2023
K
tokenize example: Respect normal add BOS token behavior (#4126)
Kerfuffle committed
November 17, 2023
G
tokenize : fix trailing whitespace
Georgi Gerganov committed
Z
examples : add tokenize (#4039)
zakkor committed