llama : refactor unicode stuff (#5992)
* llama : refactor unicode stuff ggml-ci * unicode : names * make : fix c++ compiler * unicode : names * unicode : straighten tables * zig : fix build * unicode : put nfd normalization behind API ggml-ci * swift : fix build * unicode : add BOM * unicode : add <cstdint> ggml-ci * unicode : pass as cpts as const ref
G
Georgi Gerganov committed
83796e62bc9f6caae6228168e359890f51e60fee
Parent: 828defe
Committed by GitHub <noreply@github.com>
on 3/11/2024, 3:47:47 PM