All pull requests

Pull requests

• Fix Kimi pretokenization for overlapping Han categories

  #2567·Aurantium-47 opened Oct 22, 2026·

  ·
  1

• feat(composition): implement stateless components

  #2566·SBrandeis opened Oct 21, 2026Collaborator·5/5·

  ·
  1

• feat(composition): Component boilerplate

  #2565·SBrandeis opened Oct 21, 2026Collaborator·4/5·

  ·
  1

• feat(composition): serialization utils

  #2563·SBrandeis opened Oct 21, 2026Collaborator·3/5·

  ·
  1

• feat(composition): TokenizerSpec: JSON config as source of truth

  #2562·SBrandeis opened Oct 21, 2026Collaborator·2/5·

  ·
  1

• feat(tk-convert): export fill_model_type

  #2560·SBrandeis opened Oct 21, 2026Collaborator·1/5·

  ·
  1

• fix(bitcannon): backwards compatibility: use onig to generate atom_tables

  #2558·SBrandeis opened Oct 21, 2026Collaborator·

  ·
  1

  1

• Restore Python tokenizer JSON serialization for v1

  #2556·xgbtxy opened Oct 20, 2026·

  ·
  1

  1

• fix(vocab): load an empty vocab

  #2554·SBrandeis opened Oct 19, 2026Collaborator·

  ·

  · v1.0.0
  3

• feat(python): Add support for DecodeStream in Python

  #2551·SBrandeis opened Oct 19, 2026Collaborator·

  ·
  1

  1

• fix(byte-level): drop unknown bytes + warn

  #2550·SBrandeis opened Oct 19, 2026Collaborator·

  ·
  1

  1

• tk-encode, tk-serialize: harden BPE merge loading, BucketVocabStore sizing, WordPiece scratch capacity, and JSON duplicate-key order

  #2549·professor-moody opened Oct 18, 2026·

  ·

• fix(parallelism): honor RAYON_NUM_THREADS, expose set_num_threads in Python

  #2546·nov3o opened Oct 17, 2026·

  ·

• fix(decoders): Strip panics on short tokens made only of content

  #2545·spicysharkco opened Oct 16, 2026·

  ·
  4

• feat(python): update PyO3 and rust-numpy to 0.30 and test Python 3.15(t)

  #2544·ngoldbaum opened Oct 16, 2026Contributor·

  ·

• feat(python): Support Python 3.15 via abi3t wheels

  #2543·ngoldbaum opened Oct 16, 2026Contributor·

  ·
  1

• perf(encode): encode one long input on every thread, and stream a file from Python with encode_file

  #2542·nov3o opened Oct 16, 2026·

  ·

• perf(bpe trainer): a small parallel trainer (a compact take on #2501)

  #2541·nov3o opened Oct 16, 2026·

  ·

• Restore special token masks in v1 encodings

  #2540·starx710 opened Oct 16, 2026·

  ·
  1

• Preserve token text in v1 encodings

  #2539·starx710 opened Oct 16, 2026·

  ·
  1

• Restore vocabulary lookups for v1 tokenizers

  #2535·tuzengji opened Oct 15, 2026·

  ·
  1

• Expose token-string decoding in Rust and Python

  #2531·starx710 opened Oct 15, 2026·

  ·
  1

• Fix Metaspace prefix before normalized added tokens

  #2529·starx710 opened Oct 14, 2026·

  ·
  1

• chore(tk-encode): drop the legacy Encoding

  #2509·SBrandeis opened Oct 14, 2026Collaborator·

  ·
  1

• perf(normalizers): Improve unicode and BERT normalizers

  #2508·SBrandeis opened Oct 13, 2026Collaborator·

  ·

  · v1.0.0
  1