dhruvhead/tokenizer - Morph

SIGN IN SIGN UP

dhruvhead / tokenizer UNCLAIMED

0 0 0 Go

2 branches 22 tags

deep-learning golang-tokenizer nlp tokenizer

sugarme Merge pull request #83 from olewinther/fix/transform-range-bounds-check

6e0a309 · 1mo ago · 289 Commits

	decoder
	example
	model
	normalizer
	pretokenizer
	pretrained
	processor
	spm
	util
	.gitignore		80 B
	.travis.yml		399 B
	added-vocabulary_test.go		5.5 KB
	added-vocabulary.go		15.9 KB
	bpe_test.go		6.4 KB
	CHANGELOG.md		2.2 KB
	config_test.go		557 B
	config.go		2.4 KB
	coverage.out		4.2 KB
	encoding_test.go		4.8 KB
	encoding.go		18.2 KB
	example_test.go		1.1 KB
	file-util.go		7.1 KB
	go.mod		502 B
	go.sum		2.4 KB
	init.go		584 B
	LICENSE		10.5 KB
	pretokenizer_test.go		519 B
	pretokenizer.go		9.6 KB
	README.md		2.9 KB
	tokenizer.go		29.6 KB
	util.go		4.2 KB