🚨 Refactor a bit add_tokens logic: fix bytelevel decode of added tokens + less memory deserialization 🚨 (#1995)

* we don't need  the model....

* fmt

* nit

* better version?

* test + the only viable fix

* up

* fmt

* nit

* fix

* fmt + clippy

* update the added vocab bench to reflect real world usecases

* big update, squeeze even more perfs

* profile exampl.e

* fmt

* bench: reduce normalizers to 2 (none, nfkc) for faster runs

* bench: fix sample_size to 10 (criterion minimum)

* bench: reduce to 100k tokens only for practical CI runtime

* perf: remove daachorse, keep algorithmic improvements using aho-corasick

The daachorse library swap is now isolated in PR #1999. This restores
aho-corasick while keeping all the algorithmic improvements:
- refresh_added_tokens uses added_tokens_map_r directly (no token_to_id loop)
- MatchingSet is Option<(AhoCorasick, Vec<u32>)> for empty-trie fast-path
- AddedVocabulary no longer maintains redundant added_tokens/special_tokens Vecs

* mut not required

* just fmt

* nits

* fix: single-pass partition in refresh_added_tokens, fix add_tokens signature

Map to (content, id, normalized) in one pass then partition, dropping
the boolean by destructuring. Also fix truncation_benchmark add_tokens
call to pass owned values as required by the iterator bound.

* proper fix

* nits

* update

* fix: targeted property check for duplicate token skip, simplify trie errors

Skip token only when content AND (normalized, lstrip, rstrip, special) match
an existing entry — avoids full struct equality. Replace .expect() with
.unwrap() on trie build since duplicates/empties are structurally impossible.

* current changes

* fix serialzie

* fmt + fix test

* refactor: proper error propagation, remove double iteration, use PartialEq

- Replace .unwrap()/.expect() with ? in add_tokens, refresh_normalized_tokens,
  refresh_added_tokens, and with_normalizer (return Result instead of panicking)
- Remove seed_normalized_cache and double iteration during deserialization;
  let add_tokens compute normalization directly
- Use PartialEq for duplicate token comparison instead of manual field checks
  (also fixes missing single_word comparison)
- Update all callers in tests, benchmarks, examples, and Python/Node bindings

* refactor: use ToPyResult for cleaner error conversion in Python bindings

* up?

* patch

* order

* ci: pin macOS Python to 3.13 (3.14 breaks abi3 linking)

* ci: add -undefined dynamic_lookup for macOS abi3 cross-compilation linking

* fix: rustfmt

* fix: remove missing profile_added_vocab_deserialize example from Cargo.toml

* fix: rustfmt node bindings

* fix: ci_benchmark add_tokens signature compat

* fix: clippy unused Result warning

* fix: support TOKENIZERS_DATA_DIR env var in Python benchmarks

* fix: don't serialize normalized_cache — always recompute from normalizer

The normalized cache is a derived value from content + normalizer.
Serializing it is wrong because the normalizer may have changed since
the file was saved. The cache is always rebuilt during add_tokens()
using the active normalizer.

* Update tokenizers/Cargo.toml

* refactor: add_special_tokens takes impl IntoIterator<Item = AddedToken>

Consistent with add_tokens — consumes tokens instead of borrowing,
avoiding clones in the internal forwarding call.

* Apply suggestion from @ArthurZucker