dhruvhead/tiktoken-rs - Morph

SIGN IN SIGN UP

dhruvhead / tiktoken-rs UNCLAIMED

0 0 0 Rust

3 branches 33 tags

bpe openai rust tokenizer

Roger Zurawicki Return Option from get_context_size, improve error messages, bump to 0.11.0 (#160)

b115197 · 8d ago · 199 Commits

	.github
	scripts
	tiktoken-rs
	vendor
	.editorconfig		282 B
	.gitignore		336 B
	.gitmodules		123 B
	Cargo.toml		60 B
	CODE_OF_CONDUCT.md		4.8 KB
	CONTRIBUTING.md		10.3 KB
	Justfile		1.3 KB
	LICENSE		1.0 KB
	README.md		23 B
	renovate.json		173 B
	rustfmt.toml		22 B
	SECURITY.md		599 B