SIGN IN SIGN UP

Make Tokenizer methods const (#4014)

Summary:
Pull Request resolved: https://github.com/pytorch/executorch/pull/4014

The Tokenizer class supports encode() and decode() functions. They (and all sub functions) have to be 'const' in order to guarantee the Tokenizer object itself isn't mutated during those calls, which means it can be used in a multi threaded environment and is stateless, e.g. 2 decoding runs (parallel or sequential) have no influence on each other.

Reviewed By: larryliu0820

Differential Revision: D58780400

fbshipit-source-id: 2265a7c5639af600a3ae37e1cee4c7c77ba781ee
T
Thilo Koehler committed
c6d4e8baad5312230b4109296254e6670b9ca4e9
Parent: 51f6455
Committed by Facebook GitHub Bot <facebook-github-bot@users.noreply.github.com> on 6/19/2024, 11:07:09 PM