[auryn-macmillan]

Summary

Expose the new exllamav3 prompt-cache and hybrid-state options in TabbyAPI, and prefill warm prefixes in the background after a model load.

Requires the corresponding exllamav3 PRs (turboderp-org/exllamav3#594, turboderp-org/exllamav3#595): the new aging_* constructor kwargs and the hybrid-state options are passed to AsyncGenerator at load, so an engine without them raises TypeError rather than silently ignoring the settings. The effective-settings log and the warm-prefix task guard the newer Generator attributes with getattr, so a partially-updated engine still loads.

What changed

• common/config_models.py + endpoints/core/types/model.py
  • memory.aging_lfu, aging_half_life_hours, aging_hit_cap_pages, aging_weight_cap,
    aging_persist_dir, aging_capture_prefixes, aging_prefix_min_tokens,
    aging_prefix_max_tokens, aging_prefix_max_entries, passed through to AsyncGenerator.
  • model.recurrent_frontier_checkpoints, recurrent_eviction ("lru"/"gds"),
    prefill_cost_coefficients [D, E], in both the model config and ModelLoadRequest.
• Warm prefixes — memory.warm_prefixes (prompt strings, tokenizer-encoded) and
  memory.warm_prefix_token_ids (exact IDs, no tokenizer step) are prefilled once in a background task after load, so their pages (and aging-LFU weights) are resident before the first request. Prefixes captured by the engine from earlier hits (pagetable.stored_prefixes()) are replayed too, in most-recently-written order, within memory.warm_prefix_max_tokens; configured prefixes are pinned and not budget-limited. A running warmup is cancelled on unload so no job outlives its generator, and the warmed token total is logged.
• Log-call fix (folded) — the effective-settings log line passed three format arguments to xlogger.info, which accepts at most two and crashed generator creation at model load. Format the line before logging.
• Warnings — frontier checkpoints enabled with sysmem_kv_cache: 0 now log that partial pages cannot be stored (only page-aligned checkpoints will be kept).

Example config

memory:
  sysmem_kv_cache: 8192          # partial pages of frontier checkpoints land here
  sysmem_recurrent_cache: 8192
  aging_lfu: true
  aging_persist_dir: /var/cache/tabby/aging
  warm_prefixes:
    - "<static system prompt + tool definitions>"

model:
  recurrent_frontier_checkpoints: true
  recurrent_eviction: gds
  prefill_cost_coefficients: [2.230e-5, 4.34e-6]   # ratio only

Testing

• python -m compileall -q backends common endpoints on the rebased branch.
• pytest tests/test_generator_recovery.py -q
• Loaded a 1.68 bpw hybrid model on 4 GPUs: settings logged as frontier=on, eviction=gds; 31,040-token follow-up resumed with 31,031 cached tokens and a 0.08 s prefill; unload cancels the warmup task cleanly.
• Verified against the engine with and without memory.aging_lfu / aging_persist_dir.