Skip to content

fix(evaluations): Model resolution for remote AI config fetch - #147

Merged
aknight-ld merged 1 commit into
mainfrom
fix/eval-config-fetch
Oct 7, 2026
Merged

fix(evaluations): Model resolution for remote AI config fetch#147
aknight-ld merged 1 commit into
mainfrom
fix/eval-config-fetch

Conversation

@aknight-ld

@aknight-ld aknight-ld commented Oct 7, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Contributor

Bugfix for a drift in functionality I noticed testing pre-release evaluations from code. Model resolution started experiencing drift and subsequent AI config fetch test scripts failed; this change restores them.

Note

Overview
Fixes model resolution when building generation config from a remotely fetched AI Config variation that links a model config.

When a variation has a linked model config, AIConfigVariation.from_api now always sets generation["model"] from the model config’s id (e.g. gpt-4o) instead of the variation’s modelName, which the management API exposes as the model-config key (e.g. OpenAI.gpt-4o) rather than a provider model ID. Unlinked variations still use the variation’s own modelName.

Tests were updated to mirror API shape (modelName as config key) and to assert linked vs unlinked behavior explicitly.

Reviewed by Cursor Bugbot for commit fe094a3. Bugbot is set up for automated code reviews on this repo. Configure here.

@aknight-ld
aknight-ld requested a review from donei003 October 7, 2026 17:07
@aknight-ld
aknight-ld merged commit 0b4df05 into main Oct 7, 2026
8 checks passed
@aknight-ld
aknight-ld deleted the fix/eval-config-fetch branch October 7, 2026 17:14
@github-actions github-actions Bot mentioned this pull request Oct 7, 2026
Merged
andrewklatzke pushed a commit that referenced this pull request Oct 9, 2026
@github-actions
chore: release main (#142)
920cdfe 
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-ai-server: 0.3.0</summary>

##
[0.3.0](launchdarkly-ai-server-0.2.4...launchdarkly-ai-server-0.3.0)
(2026-10-09)


### ⚠ BREAKING CHANGES

* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
* **evaluations:** take tools as a list of Tool

### Experimental

* **evaluations:** let tools= override a variation's tools
([90b293c](90b293c))
* **evaluations:** Model resolution for remote AI config fetch
([#147](#147))
([0b4df05](0b4df05))
* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
([4b8a819](4b8a819))
* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
([cad1547](cad1547))
* **evaluations:** take tools as a list of Tool
([0a92b13](0a92b13))
* **skills:** add experimental Agent Skills under
`launchdarkly_ai_server.experimental.skills`
([dfb519c](dfb519c))


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
* **client:** warn about unrecognized `init_client` options instead of
ignoring them silently
([dfb519c](dfb519c))


### Bug Fixes

* **client:** `init_client` adopts a client only after telemetry setup
succeeds, and closes a client it built when setup fails
([dfb519c](dfb519c))
* **client:** `shutdown()` releases the global OpenTelemetry tracer
provider the SDK registered, so a later `init_client` exports spans
again
([dfb519c](dfb519c))


### Documentation

* **evaluations:** call the credential an API key, not a token
([7a238e5](7a238e5))
</details>

<details><summary>launchdarkly-ai-python: 0.2.0</summary>

##
[0.2.0](launchdarkly-ai-python-0.1.8...launchdarkly-ai-python-0.2.0)
(2026-10-09)


### ⚠ BREAKING CHANGES

* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
* **evaluations:** take tools as a list of Tool

### Experimental

* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
([4b8a819](4b8a819))
* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
([cad1547](cad1547))
* **evaluations:** take tools as a list of Tool
([0a92b13](0a92b13))
* **skills:** add experimental Agent Skills under
`launchdarkly_ai_server.experimental.skills`
([dfb519c](dfb519c))


### Features

* **client:** warn about unrecognized `init_client` options instead of
ignoring them silently
([dfb519c](dfb519c))


### Bug Fixes

* **client:** `init_client` adopts a client only after telemetry setup
succeeds, and closes a client it built when setup fails
([dfb519c](dfb519c))
* **client:** `shutdown()` releases the global OpenTelemetry tracer
provider the SDK registered, so a later `init_client` exports spans
again
([dfb519c](dfb519c))
</details>

<details><summary>launchdarkly-ai-claude-agents: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-claude-agents-0.2.4...launchdarkly-ai-claude-agents-0.2.5)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
</details>

<details><summary>launchdarkly-ai-claude-messages: 0.2.4</summary>

##
[0.2.4](launchdarkly-ai-claude-messages-0.2.3...launchdarkly-ai-claude-messages-0.2.4)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
</details>

<details><summary>launchdarkly-ai-openai-agents: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-openai-agents-0.2.4...launchdarkly-ai-openai-agents-0.2.5)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))


### Bug Fixes

* **agents:** accept sync tool handlers on invoke and native graphs
([#134](#134))
([4ab5020](4ab5020))
</details>

<details><summary>launchdarkly-ai-openai-messages: 0.2.4</summary>

##
[0.2.4](launchdarkly-ai-openai-messages-0.2.3...launchdarkly-ai-openai-messages-0.2.4)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
</details>

<details><summary>launchdarkly-ai-langchain-agents: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-langchain-agents-0.2.4...launchdarkly-ai-langchain-agents-0.2.5)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))


### Bug Fixes

* **agents:** accept sync tool handlers on invoke and native graphs
([#134](#134))
([4ab5020](4ab5020))
* **AIC-3506:** strip tools from LangChain model constructor kwargs
([#133](#133))
([77c4979](77c4979))
</details>

<details><summary>launchdarkly-ai-langchain-messages: 0.2.4</summary>

##
[0.2.4](launchdarkly-ai-langchain-messages-0.2.3...launchdarkly-ai-langchain-messages-0.2.4)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))


### Bug Fixes

* **AIC-3506:** strip tools from LangChain model constructor kwargs
([#133](#133))
([77c4979](77c4979))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Release-only PR** (Release Please): bumps package versions,
`__version__` strings, `.release-please-manifest.json`, and
`CHANGELOG.md` entries—no application source changes in the diff.
> 
> **`launchdarkly-ai-server` 0.3.0** and **`launchdarkly-ai-python`
0.2.0** document **breaking evaluation tool APIs**: `Tool` → frozen
**`EvalTool`**, tools passed as a **list** (with inline definitions),
and **`tools.get` is async**. Also ships experimental **evaluations**
work (remote AI config model resolution, `tools=` override) and
**`launchdarkly_ai_server.experimental.skills`**, plus
**`init_client`/`shutdown`** telemetry lifecycle fixes and warnings for
unrecognized `init_client` options.
> 
> **Provider packages** (0.2.4–0.2.5) mainly add **SDK usage telemetry**
for which public helpers apps call (**AIC-3495**), with **LangChain**
fixes (strip `tools` from model ctor kwargs; accept **sync** tool
handlers on invoke/graphs) and the same sync-handler fix on **OpenAI
agents**.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
c2a6a78. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@donei003 donei003 donei003 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aknight-ld @donei003