Make Tokenizer methods const (#4014)
Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/4014 The Tokenizer class supports encode() and decode() functions. They (and all sub functions) have to be 'const' in order to guarantee the Tokenizer object itself isn't mutated during those calls, which means it can be used in a multi threaded environment and is stateless, e.g. 2 decoding runs (parallel or sequential) have no influence on each other. Reviewed By: larryliu0820 Differential Revision: D58780400 fbshipit-source-id: 2265a7c5639af600a3ae37e1cee4c7c77ba781ee
T
Thilo Koehler committed
c6d4e8baad5312230b4109296254e6670b9ca4e9
Parent: 51f6455
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
on 6/19/2024, 11:07:09 PM