# Migrating from HuggingFace `tokenizers`

`tokenizers-moonbit` mirrors the HuggingFace `tokenizers` API closely. This guide
maps common Python usage to MoonBit. The same `tokenizer.json` works in both.

Chinese version: [`docs/zh/migration-from-hf.md`](./zh/migration-from-hf.md)

## Loading

| HuggingFace (Python) | MoonBit |
|---|---|
| `Tokenizer.from_file("tokenizer.json")` | `@tokenizer.from_file("tokenizer.json")` |
| `Tokenizer.from_str(s)` | `@tokenizer.Tokenizer::from_str(s)` |
| `Tokenizer.from_pretrained(id, local_files_only=True)` | `@tokenizer.from_pretrained(id)` or `@tokenizer.from_pretrained_cached(id, cache_dir=...)` |
| `Tokenizer.from_pretrained(id)` | `@hub.from_pretrained(id)` on native/js, or host fetch + `@tokenizer.from_pretrained_downloaded(id, json)` |
| `tok.save("dir/tokenizer.json")` / directory workflows | `tok.save(path)` (pretty by default) or `tok.save_pretrained(dir)` |
| `BPE.read_file(vocab, merges)` / `BPE.from_file(...)` / model artifacts | `@model.Model::bpe_read_file(vocab, merges)` / `bpe_from_file(...)`, or the explicit `from_bpe_files` loader; WordPiece/WordLevel provide matching `*_read_file` / `*_from_file` aliases |

```python
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
```

```moonbit
let tok = @tokenizer.from_file("tokenizer.json")
```

`from_str` takes JSON text and does no file IO, so it works on every backend.
`from_file` uses `moonbitlang/x/fs`. Core `@tokenizer.from_pretrained` is the
all-backend offline loader: it can load a local directory/file or resolve an
existing HF Hub cache snapshot via `$HUGGINGFACE_HUB_CACHE`, `$HF_HUB_CACHE`,
`$HF_HOME/hub`, or `$HOME/.cache/huggingface/hub`. Online download is provided by the optional
`@hub` package on native/js: it fetches `tokenizer.json`, writes the same cache
layout, and then reuses the core loader. It uses HuggingFace/tokenizers-like
request headers on native, discovers auth tokens from `HF_TOKEN`, `HF_TOKEN_PATH`
or `$HF_HOME/token`, and supports mirror endpoints via
`HF_ENDPOINT` or `HubDownloadOptions::new(endpoint="https://hf-mirror.com")`;
truthy `HF_HUB_OFFLINE` defaults hub options to local-only mode. Wasm/wasm-gc callers
can fetch JSON in the host environment and call
`@tokenizer.from_pretrained_downloaded`.
`save_pretrained(dir)` writes `dir/tokenizer.json`, so saved artifacts can be
loaded back with `from_pretrained(dir)`.

## Encoding

| HuggingFace (Python) | MoonBit |
|---|---|
| `tok.encode(text)` | `tok.encode(text)` |
| `tok.encode(text, add_special_tokens=False)` | `tok.encode(text, add_special_tokens=false)` |
| `tok.encode(a, b)` | `tok.encode_pair(a, b)` |
| `tok.encode_batch([(a, b), ...])` | `tok.encode_pair_batch([(a, b), ...])` |
| `tok.encode(words, is_pretokenized=True)` | `tok.encode_pretokenized(words)` |
| `tok.encode_batch([words, ...], is_pretokenized=True)` | `tok.encode_pretokenized_batch([words, ...])` |
| `tok.encode_batch([(words_a, words_b), ...], is_pretokenized=True)` | `tok.encode_pretokenized_pair_batch([(words_a, words_b), ...])` |

`add_special_tokens=false` / `False` only suppresses special tokens injected by
the configured post-processor. Non-special post-processing effects still run,
including Template/BERT `type_ids`, pair `sequence_ids`, and ByteLevel/RoBERTa
offset trimming.

Pre-tokenized migration keeps HF's added-token extraction semantics: a special
or added token embedded inside an input word is still split out before the
ordinary span is passed to the model, while the configured pre-tokenizer itself
is skipped.

```python
enc = tok.encode("Hello world")
enc.ids            # [15496, 995]
enc.tokens         # ['Hello', 'Ġworld']
enc.attention_mask # [1, 1]
```

```moonbit
let enc = tok.encode("Hello world")
enc.ids            // [15496, 995]
enc.tokens         // ["Hello", "Ġworld"]
enc.attention_mask // [1, 1]
```

Pre-tokenized APIs skip the tokenizer's pre-tokenizer but still apply
normalization, model tokenization, post-processing, truncation and padding.
Offsets are measured against normalized words joined by one ASCII space.

### `Encoding` fields

| HuggingFace | MoonBit | Notes |
|---|---|---|
| `enc.ids` | `enc.ids` | `Array[Int]` |
| `enc.tokens` | `enc.tokens` | `Array[String]` |
| `enc.type_ids` | `enc.type_ids` | `Array[Int]` |
| `enc.attention_mask` | `enc.attention_mask` | `Array[Int]` |
| `enc.special_tokens_mask` | `enc.special_tokens_mask` | `Array[Int]` |
| `enc.offsets` | `enc.offsets` | `Array[(Int, Int)]`; currently char offsets |

## Decoding

| HuggingFace (Python) | MoonBit |
|---|---|
| `tok.decode(ids)` | `tok.decode(ids)` |
| `tok.decode(ids, skip_special_tokens=True)` | `tok.decode(ids, skip_special_tokens=true)` |
| `tok.decode_batch(batch)` | `tok.decode_batch(batch)` |
| `s = tok.decode_stream(False); s.step(id)` | `let (s2, chunk) = s.step(id)` after `tok.decode_stream(skip_special_tokens=false)` |

MoonBit `DecodeStream::step` returns the updated stream explicitly instead of
mutating in place, so rebind the stream before feeding the next id.

## Vocabulary lookups

| HuggingFace (Python) | MoonBit |
|---|---|
| `tok.token_to_id("[CLS]")` | `tok.token_to_id("[CLS]")` → `Int?` |
| `tok.id_to_token(101)` | `tok.id_to_token(101)` → `String?` |
| `tok.get_vocab_size()` | `tok.get_vocab_size()` |

## Programmatic construction and added tokens

| HuggingFace (Python) | MoonBit |
|---|---|
| `Tokenizer(model)` | `@tokenizer.Tokenizer::new(model)` |
| `tok.normalizer = normalizer` | `tok.with_normalizer(Some(normalizer))` |
| `tok.pre_tokenizer = pre_tokenizer` | `tok.with_pre_tokenizer(Some(pre_tokenizer))` |
| `tok.model = model` | `tok.with_model(model)` |
| `tok.post_processor = processor` | `tok.with_post_processor(Some(processor))` |
| `tok.decoder = decoder` | `tok.with_decoder(Some(decoder))` |
| `tok.normalizer` / `tok.model` / ... | `tok.get_normalizer()` / `tok.get_model()` / ... |
| `AddedToken("<x>", single_word=True)` | `AddedToken::new("<x>", single_word=true)` |
| `tok.add_tokens([...])` | `tok.add_tokens_with_count([...])` or `tok.add_tokens([...])` |
| `tok.add_special_tokens([...])` | `tok.add_special_tokens_with_count([...])` or `tok.add_special_tokens([...])` |
| `tok.encode_special_tokens = True` | `tok.set_encode_special_tokens(true)` |
| `tok.encode_special_tokens` | `tok.get_encode_special_tokens()` |
| `tok.get_added_tokens_decoder()` | `tok.get_added_tokens_decoder()` |
| `tok.num_special_tokens_to_add(is_pair)` | `tok.num_special_tokens_to_add(is_pair=...)` |
| `tok.post_process(enc, pair, add_special_tokens=True)` | `tok.post_process(enc, pair=Some(pair), add_special_tokens=true)` |

MoonBit builders return updated tokenizer values, so rebind or chain them:

```moonbit
let (tok, count) = @tokenizer.Tokenizer::new(model)
  .with_pre_tokenizer(Some(@pretokenizer.PreTokenizer::whitespace_split()))
  .add_tokens_with_count([
    @tokenizer.AddedToken::new("<tag>").with_single_word(true),
  ])
```

The `*_with_count` variants return HF-style duplicate-aware counts. Ordinary
added tokens keep `special_tokens_mask=0`; `add_special_tokens` registers tokens
as non-normalized special entries and sets mask `1` when they are emitted. If
`encode_special_tokens` is enabled, input occurrences of those special token
strings stay on the ordinary model path and therefore receive mask `0`, matching
HF's switch for templating/chat use cases.

## Differences to be aware of

- **Booleans:** MoonBit uses `true`/`false` and named arguments use `name=value`.
- **Optionals:** lookups return `Int?` / `String?` (`Some`/`None`).
- **Offsets:** HuggingFace returns byte offsets by default; this port currently
  returns char offsets.
- **Configuration style:** HuggingFace mutates via `enable_truncation` /
  `enable_padding`; MoonBit uses chainable `with_truncation` / `with_padding`
  plus `TruncationParams::with_*` and `PaddingParams::with_*` builders for
  strategy, direction, stride, `pad_type_id`, and `pad_to_multiple_of`.
- **Training:** deterministic WordLevel training is supported, including custom
  pre-tokenizers, pre-tokenized token streams, `min_frequency`, `special_tokens`,
  `vocab_size`, and HF-style frequency/lexical vocab ordering. WordPiece, BPE,
  and Unigram trainer MVPs are available for the same input modes with common
  controls such as continuation-prefix, end-of-word suffix,
  `max_input_chars_per_word`, and `byte_fallback`; WordPiece/BPE also support
  `initial_alphabet`, `limit_alphabet`, and `max_token_length`, plus a
  `byte_level_alphabet()` helper matching HF
  `ByteLevel.alphabet()` workflows. Trainer defaults for `vocab_size`,
  `min_frequency`, BPE `unk_token`, and Unigram `unk_token` now follow HF
  defaults; pass explicit `unk_token=Some(...)` / `unk_piece=Some(...)` or
  historical thresholds when you need the older MoonBit behavior.
- **Regex components:** common HF `Split`/`Replace` regex families are handled by
  shared deterministic fast paths across wasm/js/native: `\s`, `\d`, `\w`,
  ASCII classes, Unicode letter/number/punctuation/symbol classes, anchored
  runs, exact `{2..4}` / minimum `{2,}`/`{3,}`/`{4,}` quantifiers across the
  same positive and inverse class families, `{1,n}` bounded runs, and `{2,3}` /
  `{2,4}` / `{3,4}` ranged runs. Arbitrary complex
  regexes are still out of scope; Split rejects unknown patterns explicitly,
  while Replace falls back to literal replacement.

## Verified models

With optional fixtures present, output is checked token-for-token against Python
`tokenizers` for 39 real models, covering BPE, byte-level BPE, byte_fallback BPE,
WordPiece, Unigram, WordLevel, Qwen/Llama-3/o200k Split patterns, coder,
multimodal, ModernBERT-style encoder and embedding tokenizers. See
[`PROGRESS.md`](../PROGRESS.md).