[VibeSWE]

JSON events with multiple arrays can multiply into an unbounded cartesian product before Arrow encoding. Checked, borrowed preflight accounting now rejects an event that would exceed a limit with HTTP 400 before its expanded rows or repeated field names are allocated. Earlier events may have been materialized within the shared budget, but no events or inferred schema from a rejected HTTP request are written. An ingest endpoint may still create an empty dataset before validation.

Changes

• Add configurable defaults of 1,000 rows per event, 10,000 total rows per request, and 64 MiB of expanded compact JSON per request. CLI flags and environment variables are documented in docs/json-flattening-limits.md.
• Share budgets across JSON/custom sources, partition groups, Kinesis records, and OTEL scopes/resources. Include generic intermediate rows, structural parent-name expansion, Kinesis metadata, and OTEL body prefixes. Protect schema detection too.
• Stream internal field-statistics rows directly from typed data, measuring each borrowed row before materializing it. An aggregate file can exceed one budget; an individually oversized row is warned and skipped while other rows continue. Preserve legacy fields, order, null handling, and partition validation.
• Preserve output order, collision behavior, empty-array/null handling, V0/depth fallbacks, and Kafka/OTEL metrics/traces behavior. The byte budget measures expanded compact JSON, not process RSS.
• Add differential tests, allocation assertions with a hard allocation guard, an actual-source comparison harness, and a bounded local HTTP smoke script.

Compatibility: these intentionally introduce rejection boundaries. The request row cap counts flat objects too: batches above 10,000 rows must be split or the configured limit raised. A single event above 1,000 rows is rejected by default, including wrappers containing large arrays. Input payload bytes and expanded-output bytes are independent limits; deployments with larger input settings may need to raise the output budget too. Rejections truncate nothing; clients that drop HTTP 400 batches should adjust batching before upgrading. Migration guidance is in the docs.

Validation

Revised sources committed as 7fe8a17e73e752493da46f970b63764aa413f016:

• 118 focused server unit tests passed, including JSON, OTEL, Kinesis, shared ingest utilities, configuration, and the five streaming-statistics regressions.
• 15 standalone allocation/differential tests passed; one intentionally ignored child test is executed by the subprocess guard test. Borrowed preflight rejection allocates zero bytes. Includes 10,000/10,001 flat-event controls.
• 76 live HTTP checks passed: HTTP 400, unchanged event counts, absent schema sentinels, escaped/long parent names, flat batch boundaries across JSON/partitions/Kinesis/OTEL, schema detection, and CLI precedence.
• Backend build, formatting, and clippy all-targets passed. Clippy retains 22 existing unrelated test warnings.

Fresh independent adversarial reviews by Opus 5.5 (max effort) and Sol 6.1 (high reasoning) both reported CLEAN, with no actionable issues, for exact head 7fe8a17e73e752493da46f970b63764aa413f016. Both reviewed the full diff against 04b73b57 and the changes since the first review. Earlier Opus findings led to explicit compatibility documentation, streamed statistics, and corrected per-event allocation guarantees. Merge remains conditional on CI passing for this reviewed head.

Performance

Three release microbenchmark runs against fixed comparison baseline 547c5118 are recorded in tools/flatten-bench/RESULTS.md. Flat batches are 41–45% slower at the conversion stage (about 0.13 ms extra per 1,000 rows). The moderate-array case is 3–5% faster and allocates 14% fewer bytes. These figures exclude parsing, Arrow conversion, and I/O; they are not end-to-end throughput measurements. The reviewed corrections leave the measured JSON algorithm files unchanged.

---

This PR has:

• Tested ingestion, schema detection, and rejection without event/schema writes.
• Added comments explaining the counting and allocation bounds.
• Added configuration, migration documentation, and reproducible verification tools.