fix: tokenization of special characters: (#850)
It should behave like llama.cpp, where most out of the box usages treat special characters accordingly
A
Antoine Lizee committed
4d4e0f11e292984320e59f1dc1ecf9a0cc437a75
Parent: 952e4cc
Committed by Andrei Betlen <abetlen@gmail.com>
on 11/2/2023, 6:28:14 PM