[AIC-3363] Add support for evaluation data to be defined in code

[aknight-ld]

Adjusted approach for inline dataset definition. For users who wish to define their evaluation data directly from code, we now offer an alternate path where they provide the data directly by dropping the dataset key and instead providing rows.

Sample shape:

result = await evals.run(
    project_key="my-project",
    key="support-qa-2026-08-20",
    rows=[
        {"input": "How do I reset my password?", "expectedOutput": "Use the reset link."},
        {"input": "Where is order {{order_id}}?", "variables": {"order_id": "A-17"}},
    ],
    handler=create_openai_messages_handler(),
    generation={"provider": "OpenAI", "model": "gpt-4o"},
)

---

Note

Overview
Adds inline evaluation datasets so evals.run() accepts either a hosted dataset key or a list of row mappings on the same dataset parameter (wire fields: input, expectedOutput, variables, metadata, optional rowIdx).

For inline runs, rows are validated up front (non-empty input, JSON-safe variables/metadata, index must match list position), uploaded in batches of 500 to the run’s dataset-rows endpoint before generation/events, and template rendering for handlers is shared with hosted datasets via render_row. Runs are created without datasetId; generation/criterion events omit datasetId/datasetKey. Failed uploads cancel the run and surface the error without invoking the handler.

The management API client gains an idempotent POST option so dataset-row uploads and run cancel can be retried on 5xx or transport failures. README documents the inline flow; tests cover batching, validation, retries, cancel-on-failure, and unchanged hosted-dataset event identity.

^{Reviewed by Cursor Bugbot for commit d956bc2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jeffdupont]

Reviewed with the 1.0 freeze in mind. Inline rows are a useful addition, and since evals are additive and Python-only for 1.0 (the GA plan defers JS evals to 1.1), the lack of a JS counterpart isn't a blocker. JS has no eval code on main to add this to. make test passes at 196d0b1 (1418 passed, 11 skipped, exit 0) and make typecheck is clean.

Three things I'd like settled before merge:

1. No spec, and the PR goes against the one on main. TESTING.md §8.5 says the run create body is "exactly {source: \"api\", datasetId: <dataset id>}", and §8.6 lists datasetId as a run identity field on every event that an event type "must never drop". This PR drops it from the run body, the identity set, the eventId hash and the payloads, and adds a new request (POST …/runs/{id}/dataset-rows). The spec for the earlier approach, launchdarkly/ai-sdks-monorepo#15, was closed with #95, and I didn't find a replacement. Event payloads are a data contract with ingest, and JS will build against the spec in 1.1. So I think this needs a spec PR covering the upload request, batch size, retry rule, event identity without a dataset, and row validation before it merges.
2. A bad row can still fail after the records exist. The README says "a malformed row fails the run before any records are created", but values inside variables / metadata are never checked for JSON. I reproduced both cases at 196d0b1. A datetime in variables raises a bare TypeError from json.dumps, after POST evaluations and POST runs have both gone out. A math.nan in metadata goes out as "metadata": {"score": NaN}, which isn't valid JSON. §8.4 already requires this check, done up front, for inline tool schemas, for the same reason. #95 had it.
3. A failed upload leaves a run that never finishes. The upload happens after the run is created, one batch at a time. With 600 rows and a 400 on the second batch (reproduced), rows 0-499 are stored, the call raises, and the run stays open with a placeholder row count, so nothing will ever complete it. Should the SDK mark the run failed, or should the server time it out? Whichever it is should go in the spec. Retrying the batch POST on 5xx relies on the server applying each rowIdx at most once. I haven't checked that in the server, and the spec should say it too, because §8.3 currently says POSTs are never retried on 5xx.

Smaller notes:

• InlineDatasetRow is a new root export. It's only a type alias (DatasetRow | Mapping[str, Any]), so callers don't need it to use rows=. If evals move to experimental.evaluations as the lifecycle draft proposes, it should go with them. Either way I'd keep it out of the root __all__.
• Passing a DatasetRow means writing row_index= by hand, and it has to equal the row's position or the call raises. DatasetRow is also documented as "a rendered dataset row" for the handler. Reusing it as the input type freezes both roles together. A mapping-only input, or row_index made optional for inline rows, would be easier to change later.
• #95 also capped the row count (10k) and the rendered byte size. Does the server enforce limits on dataset-rows? If it does, checking them up front avoids the partial-upload case above. I haven't checked the server.
• The DatasetRef docstring says "the server drops any that name one as a mismatch". I haven't verified that. If it's true, it belongs in the spec, because it's the reason events can't carry a dataset id.

[jeffdupont]

Approving at c1e6b20. My three blockers are settled. The spec landed as launchdarkly/ai-sdks-monorepo#44 and covers the inline run body, the dataset-rows upload and batch size, idempotent POSTs, cancel-on-failure, and why inline events drop datasetId. Row JSON is now checked before any request, and a failed upload cancels the run before re-raising. InlineDatasetRow is out of the root __all__. pytest (1430 passed, 11 skipped), mypy and ruff are clean at this commit.

Two small things for a follow-up. Neither blocks the merge:

1. Non-string keys are still coerced. module.py:126 relies on json.dumps(value, allow_nan=False) to reject non-string keys, but it only raises for unusual keys like tuples. int, float, bool and None keys are silently turned into strings: {1: "a"} goes out as {"1": "a"}, {True: x} as "true", and {1: "a", "1": "b"} produces JSON with a duplicate key. I confirmed _normalize_inline_rows([{"input": "x", "variables": {1: "a"}}]) is accepted. §8.4 says "Do not coerce", so this needs a recursive check that every key is a str. The test at test_evaluations_run.py:3206 passes only because it uses a tuple key. An int key case would catch this. The A.11 Python cell in #44 has the same wrong claim ("raises TypeError for … non-string keys") and should be corrected too.
2. No generator test. §8.4 asks for a test that a generator of otherwise valid rows throws the "dataset key or a sequence of inline rows" error before any request. The behaviour is already right (_validate_dataset_source rejects it), so this is one more case in test_invalid_dataset_source_fails_before_any_request.

The DatasetRow dual-role note from my first review still stands but isn't blocking.