🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
remove the redundant non maintained jieba and use rjieba instead (#40383)
* porting not maintained jieba to rjieba * Fix format * replaced the line with rjieba instead of removing it * cut_all is not included as a parameter. cut_all is a seperate function rjieba * rev * jieba remove installation * Trigger tests * Update tokenization_cpm.py * Update tokenization_cpm_fast.py --------- Co-authored-by: Yih-Dar <[email protected]>
D
dsinghvi committed
1363fceeec955db125141c3d7768f8ef345ed288
Parent: 36fddeb
Committed by GitHub <[email protected]>
on 8/30/2025, 11:28:52 AM