Skip to content

fix: let OpenAI-compatible models use their real context window - #441

Merged
ProfSynapse merged 2 commits into
mainfrom
fix/openai-compatible-context-window
Nov 12, 2026
Merged

ProfSynapse merged 2 commits into
mainfrom
fix/openai-compatible-context-window

Conversation

@ProfSynapse

Copy link
Copy Markdown
Owner

Fixes #439. OpenAI-compatible models were stuck at a 4,096-token context window, and the chat slider could only lower it.

What changes

  • Each model takes its window from, in order: the value the user typed, the value the server reported on Connect, then 4,096.
  • Connect reads the server's size from max_model_len, context_length, context_window, max_context_length or top_provider.context_length in /v1/models. A single-model llama.cpp server is asked for /props → default_generation_settings.n_ctx; its trained maximum (meta.n_ctx_train) is ignored.
  • Each model row has a size field and a Check button. Check sends one prompt of that size with max_tokens: 1 and reports one of three results:
    • it fits;
    • it's too large, with a Use N button when the server names its limit (if N is what the server already reports, the typed value is cleared instead);
    • the server shortened it, with no Use button, because the count is not the server's size: Ollama keeps about half its num_ctx.
  • Raising a model's window reaches existing chats. Each chat now records the model's full window next to its own budget. A chat that was using all of the old window follows the new one, whether it's open or reopened later. A smaller budget the user chose stays.
    • Chats saved before this change count as "whole window" only at the OpenAI-compatible 4,096 fallback.
    • Plugin-wide chat and agent overrides equal to the old maximum are also cleared.
  • Narrow widths: model rows keep their controls whole, and the check note stays below the window line.

Verification

  • Build and tests: npm run build passes (lint, mobile reachability, type check). Full Jest suite: 7,183 passed, 119 skipped, 0 failed.
  • The new tests catch the bugs: they fail when the old window, suggestion and "Use" behaviour is put back.
  • Live lane:
    • llama.cpp -c 8192: 3 of 3 pass. The oversized check returns too-large with serverLimit: 8192.
    • Ollama 0.40 (num_ctx 32,768): the Check cases pass. The oversized check returns truncated with promptTokens: 16386. Its discovery case originally failed because Ollama reports no size; it passes with the new COMPATIBLE_CONTEXT_REPORTED=none.
  • In Obsidian 1.15 (macOS), with the build installed in a test vault:
    • Connect showed "8,192 tokens, reported by your server".
    • Check reported 8,180 tokens read (about 9 s). The 32,768 check came back too large with Use 8,192.
    • Ollama models showed the 4,096 "did not report" default, and the chat slider reached 32,768 once that size was typed in.
    • An open chat moved from 4,096 to 32,768 when the window was raised. A chosen 32K budget survived a further raise and a reopen.
    • A real chat ran with a 7,581-token prompt against the 8,192 window and was not compacted.
    • Rows stayed clear of overlap at 600 px, 900 px and under phone emulation.
    • dev:errors stayed empty.

Notes

  • Check can be slow on local hardware. A check at 32,768 took 299 s against Ollama; the button's tooltip warns about this.
  • Ollama's MLX runner doesn't enforce num_ctx. It read 49,140 tokens with num_ctx 32,768, and Check reports what the server actually did.

🤖 Generated with Claude Code

ProfSynapse and others added 2 commits November 12, 2026 07:18
OpenAI-compatible models were fixed at a 4,096-token context window, and the
chat slider could only lower it. Each saved model now takes its window from
the user's value, then the server's report, then 4,096.

- Connect reads max_model_len, context_length, context_window,
  max_context_length and top_provider.context_length, and asks a single-model
  llama.cpp server's /props for its allocated n_ctx.
- Endpoint settings show each model's window and where it came from, with a
  field to set it.
- Check sends one prompt of that size with max_tokens 1 and reports fits,
  too large (offering the limit the server names), or silently shortened.
- Raising a window removes chat and agent overrides pinned at the old maximum,
  which chat settings saved even when the slider was never moved.

Verified against a real llama.cpp server started with -c 2048
(tests/debug/openai-compatible-context-live.test.ts).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Found verifying #439 in a running Obsidian against llama.cpp and Ollama.

- A chat committed its budget when the model was picked, so a chat that
  sat at the 4,096 fallback stayed there after the window was raised.
  Chats now record the model's whole window beside their budget; a budget
  at the whole window follows it when it grows, on reopen and in an open
  chat. A smaller budget the user chose is kept. Older chats count as
  "whole window" only at the OpenAI-compatible 4,096 fallback.
- A check that the server shortened no longer offers "Use N": Ollama
  keeps about half its num_ctx (16,386 of 32,768), so N understated it.
- "Use N" equal to the server's own report clears the user value, so the
  model keeps following the server instead of pinning a copy.
- The live lane takes COMPATIBLE_CONTEXT_REPORTED=none for servers whose
  model list reports no context (Ollama's /v1).
- Model rows keep their controls whole at narrow widths, and the window
  line and check note stay stacked when another plugin restyles setting
  descriptions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ProfSynapse
ProfSynapse merged commit 0c8d318 into main Nov 12, 2026
1 check passed
@ProfSynapse
ProfSynapse deleted the fix/openai-compatible-context-window branch November 12, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Not able to set context window for local models

1 participant