feat(spec)!: an analytics dataset dimension's and measure's field is a column reference (#21220) - #21240
Conversation
…ference The dataset layer's field takes the accept set its cube members hold, from one shared declaration (data/analytics-column-reference.ts); a dimension refuses the row wildcard. D3 entry dataset-member-field-expression-refused. Co-authored-by: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01UtnxvdiN376GF3sgXwAw4d
…rrowed contract; regenerate the dataset reference The route parses every dataset first, so an expression field is now answered 400 VALIDATION_FAILED at its path; the service door's 403 stays pinned on unparsed (stored) fixtures. Co-authored-by: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01UtnxvdiN376GF3sgXwAw4d
… dataset field narrowing Co-authored-by: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01UtnxvdiN376GF3sgXwAw4d
…cription's prose Co-authored-by: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01UtnxvdiN376GF3sgXwAw4d
…taset-field-column-reference
📓 Docs Drift CheckThis PR changes 1 package(s): 4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 1 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 137 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin d1d069ec7bdf7b952ec7f0f77412a4734f8445a4 && git checkout d1d069ec7bdf7b952ec7f0f77412a4734f8445a4
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 3dc33b2d13a919db611bc077d42337df51ec626d fc4e91c09b028326af60f9a2cff08c156ffefaad && git checkout -B drift-repro 3dc33b2d13a919db611bc077d42337df51ec626d && git merge --no-ff fc4e91c09b028326af60f9a2cff08c156ffefaad
node scripts/docs-audit/affected-docs.mjs --json 3dc33b2d13a919db611bc077d42337df51ec626d
|
Contract reviewServed-tier: Isolated reviewer for card #21220 / PR #21240. Inputs: the card body and all four of its comments (triage ① Derived judgments
② Semver level
Check-runs on ③ Boundary flags
Implemented-by: VERDICT: FAIL |
…taset-field-column-reference
…conversion dataset-count-measure-empty-field-removed (retiredFromLoadPath, retiredAfter 17.5.0) drops field: '' from a count measure, which then compiles to COUNT(*); the D3 entry links it and states that the dataset door never judged an empty field. Changeset, ledger notes and rationale fragment corrected to match. Co-authored-by: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01UtnxvdiN376GF3sgXwAw4d
…taset-field-column-reference
…fter the base merge STEP18_RATIONALE fragment 57 -> 58 (57 is allocated to the in-flight ui-record-line-items-props-closed); MAJOR_18_CONVERSIONS entry 53 -> 54 (main landed form-field-public-picker at 53). Neither list requires unique orders. Co-authored-by: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01UtnxvdiN376GF3sgXwAw4d
Contract reviewServed-tier: Isolated reviewer for card #21220 / PR #21240, second review (the first, ① Derived judgments
② Semver level
Check-runs on ③ Boundary flags
Implemented-by: VERDICT: PASS |
Fixes #21220
Clause-②: yes
Dispatched by the PM claim
5938507454(PM loop round 1,domain:specseat 1), on the triage direction5938409101. An ADR-0021 dataset'sdimensions[].fieldandmeasures[].fieldnow take the column-reference accept set the cube members they compile to already hold since #20943 (PR #20998), from ONE shared declaration. A non-column value is refused at parse, atdimensions.N.field/measures.N.field, with a prescription naming the ADR-0021 form. The runtime door from PR #21190 is untouched and stays as defence in depth. The changeset carries the(narrowing)arm, the BREAKING banner atminor, and the ADR-0087 markerregistered dataset-member-field-expression-refused, dataset-count-measure-empty-field-removed.Patch round 2 — contract review
5940617829, item ①.6The review found one lossless sub-shape that the first round sent to D3 only: a dataset measure
{ aggregate: 'count', field: '' }. It parses on the base, and the #21190 door skips it (its!== ''guard). On SQLite's native path it compiled toCOUNT()and answered 200. Its producer is Studio's dataset inspector. This round:New D2 conversion
dataset-count-measure-empty-field-removedinMAJOR_18_CONVERSIONS(order: 54, inserted at its identifier's sort position, defined directly aboveelementFilterRemoved).mainlandedform-field-public-picker-removedat 53, so this entry takes the next free number.fieldfrom acountmeasure whosefieldis exactly''. Without the key,compileDatasetemitssql: m.field ?? '*', which isCOUNT(*): the row count.time-default-utc-suffix-dropped, not recalled:toMajor: 18,retiredFromLoadPath: true(ADR-0087's ratified pre-GA policy for a lossless repair), andretiredAfter: '17.5.0'(the spec's current version, the same value the precedent carries).stripKeyshelper and emits one notice per removed key.'', asumover'', a count over'*', a count with nofield, and a count over a column.Scope: only
count+''. A non-count measure with'', a dimension with''and every expression have no working row or no mechanical rewrite, so they stay D3-only. A padded value has no known producer and is out of scope.The D3 entry links the conversion with
conversionIds: ['dataset-count-measure-empty-field-removed'], the way18.time-default-zone-refused.tslinks its conversion. Itsreasonnow says what is true: the door refuses an expression, and it never judged''. It also says that D2 carries thecount+''repair and D3 the rest.acceptanceCriteria, theSTEP18_RATIONALEfragment, the changeset, the two ledger notes and thedataset.zod.tscomment are corrected to match.registry.tswas regenerated bygen:migration-registry.Pins. The new
src/conversions/dataset-count-measure-empty-field-removed.test.tscovers four things on a STORED row, throughapplyConversionsToStoredItem('dataset', row):The table-wide fixture replay in
conversions.test.tscovers before → after.What a NEW save does. The write path parses with the current schema and replays no conversion, so a new Studio save of
field: ''is still refused at save with the prescription to omit the key. The producer-side change stays objectui's. A row already stored that way is repaired on load at every stored-row seam.What changes
@objectstack/specpackages/spec/src/data/analytics-column-reference.tsdeclares the column path once (a bare identifier, then zero or more.identifierhops). It sits outside thedatabarrel, likeui/analytics-carrier-filter.ts, so it is not published API. It exports two anchored forms built from that one source string:ANALYTICS_COLUMN_REFERENCE: the path, or'*'.ANALYTICS_COLUMN_PATH: the same path without the'*'arm.data/analytics.zod.ts,CUBE_MEMBER_SQLis now that sameRegExpobject:const CUBE_MEMBER_SQL = ANALYTICS_COLUMN_REFERENCE. The declaration stays in this file on purpose. ADR-0021's 2026-10-01 note links toanalytics.zod.ts#CUBE_MEMBER_SQL, and ADRs are a governed surface this PR does not edit. The cube members' JSON-Schemapatternis byte-identical; the new pin asserts it.ui/dataset.zod.ts:DatasetMeasureSchema.fielduses.regex(ANALYTICS_COLUMN_REFERENCE). Admitted: a column, a relationship path, or'*'. A count may still omitfield.DatasetDimensionSchema.fielduses.regex(ANALYTICS_COLUMN_PATH), so a dimension also refuses'*'(see measurement 3)..regex(), not refinements, so the published JSON Schema carries each as apattern.dropped-refinements.baseline.jsonis untouched.invalid_format. Each prescription opens with the contract sentence and names ADR-0021.filterform andderived: { op, of: [...] }, with the 0–1 ratio scale.describe()texts now say "never a SQL expression".content/docs/references/ui/dataset.mdxis regenerated from them.migrations/entries/semantic/18.dataset-member-field-expression-refused.ts.registry.tswas regenerated bygen:migration-registryand never hand-edited inside the markers.STEP18_RATIONALEfragment, inserted at the id's sort position withorder: 58. Round 1 used 57. After the round-2 base merge it takes 58, because 57 is allocated to the in-flight PR feat(spec)!: record:line_items gets its ComponentPropsMap row, its columns are the inline grid column contract, and the showcase Tasks grid binds by name (#21142) #21244's fragment. Neither list requires unique orders:step18-rationale-merge.test.tsmodels two fragments sharing oneorderand only asserts a positive integer, andmainalready holds two 56s. So whichever of the two PRs lands first, neither re-orders.dataset-count-measure-empty-field-removed, for the one lossless sub-shape (acountmeasure'sfield: ''; see Patch round 2). The D3 entry carries the rest: an expression has no mechanical rewrite into a column.RETIRED_KEYS_BY_MAJORrow: no key left the shape.datasetledger rowsdimensions.fieldandmeasures.fieldstaylive. Each is re-verified on 2026-10-01, with the narrowing recorded in itsnote.content/docs/data-modeling/analytics.mdxgains one "Key rules" bullet saying thatfieldis a column reference.Ratchets, as expected for a value narrowing. The
api-surface,authorable-surface,json-schema.manifest,export-originsanddeclaration-mapartifacts are byte-identical.spec-changes.jsonand the upgrade guide stay at protocol 17, so major-18 entries do not project yet, and both checks are green.The PM's mechanism assumptions, measured
Confirmed.
dataset.zod.ts:125(dimension, required) and:189(measure, optional) were barez.string()at the based6d6e872.CUBE_MEMBER_SQLwas a module-privateconstatanalytics.zod.ts:240.Exporting
CUBE_MEMBER_SQLwould move the public surface.packages/spec/src/data/index.tsre-exports the whole module withexport * from './analytics.zod', so an exportedCUBE_MEMBER_SQLbecomes a new@objectstack/spec/dataexport and needsgen:api-surface. I took the non-public module instead.check:api-surface,check:export-originsandcheck:declaration-mapare green with zero changes to their artifacts.'*'on a dimension: measured, and refused. Readings come fromPOST /api/v1/analytics/dataset/querywith today's spec, through the real REST route, a realAnalyticsServiceand a real better-sqlite3SqlDriver, using a temporary probe test that was deleted afterwards (the tree is clean). The ObjectQL-strategy column bridgesexecuteAggregatestraight toSqlDriver.aggregate, not through the ObjectQL engine.field'*'DATABASE_ERROR(SELECT * AS ... GROUP BY *)DATABASE_ERROR(groupBy: ['*'])''''COUNT()it compiled to)'*'(control)'*'SUM(*))' amount'(sum)' industry'amount * 2PERMISSION_DENIED(PR fix(service-analytics)!: refuse a caller-supplied analytics member that is not a column reference at the door #21190's door)A
'*'dimension is never answered: it compiles to grouping by every column, which is not an axis. So the dataset dimension takes the same path pattern without the'*'arm. That is one pattern source with one stated restriction, not a second pattern; the pin proves the dimension's publishedpatternequals the cube's with only the\*|arm removed.⚠ Flagged, not silently chosen. The triage line reads "exactly the
CUBE_MEMBER_SQLaccept set" for both keys. This PR narrows the dimension one step further, as the dispatch's mechanism item 3 invited and the card's own pin wording ("*(on a measure)") suggests. The cubeDimensionSchema.sqlstill admits'*', per ruling D's execution parameters, and is untouched here.Census, repo-wide, with a lit control. A scan of every
field:value in tracked files that mention a dataset anddimensions/measures, includingpackages/**tests,content/docs/**andskills/**.examples116,content88,skills15,platform-objects6,service-analytics587,spec423,lint282,rest92.fields found: only the fixtures that PR fix(service-analytics)!: refuse a caller-supplied analytics member that is not a column reference at the door #21190 wrote on purpose to drive its door:restanalytics-16019-driver-declared-fault.test.ts(translate(...),lower(name)) andservice-analyticsinline-dataset-field-admission-door.test.ts(an expression constant and a template). Both were re-pinned (next section).platform-objects, the hand-written docs and the published skills. Every other non-column literal the scan caught is not a dataset field (driver-sql and protocol prose, filter paths,$fieldprose in a skill).spec,lint,service-analytics,metadata-protocolandrestare green on the narrowed contract.skills/**teaches an expressionfield, so no Tier H follow-up is owed.D2 for one sub-shape, D3 for the rest (corrected in round 2; the first round said "D3 only").
fieldalready answered 403 at the dataset door. On the REST route it is now refused one step earlier, at the route's ownDatasetSchemaparse, which the route runs on the inline and the saved branch alike. An expression has no mechanical rewrite.countmeasure withfield: ''answered 200 on SQLite's native path, and the door never judged it.dataset-count-measure-empty-field-removed, which every stored-row rehydration seam replays (applyConversionsToStoredItem, e.g.metadata-protocol'sconvertStoredItemDetailed). The runtime door is still reachable for a dataset handed toqueryDatasetunparsed: the build probe's dashboard-widget path (metadata-protocolbuild-probes.ts) passes the stored row as read.Fixture triage (two consumer tests the narrowing turns red; both re-pinned, not loosened)
service-analyticsinline-dataset-field-admission-door.test.tsbuilt its expression fixtures withDatasetSchema.parse, which now refuses them. The fixtures are now built UNPARSED, the shape a pre-narrowing stored row has, throughstoredDatasetWith. The controls still parse. One new case asserts that the contract refuses both fixtures atdimensions.0.field/measures.0.field. All 4 provider tiers x 2 strategies of the 403 door pins are unchanged and green.restanalytics-16019-driver-declared-fault.test.ts. The route parses every dataset first, so its inline and saved expression cases now answer400 VALIDATION_FAILED, where they answered403 PERMISSION_DENIED.400, plusinvalid_formatat the path insidedetail, the driver never called, and no expression text echoed.Tests
All runs are at head
0d5e446e(after mergingorigin/mainat3ddd3d0c) unless stated otherwise. Filter direction: each package's own suite, no consumer sweep.@objectstack/specvitest run --project local: 597 files, 17468 passed, 1 todo.src/ui/dataset-field-column-reference.test.tshas 11 cases.cube-member-sql-column-reference.test.tsstays green, with its'*'-on-a-cube-dimension case unchanged.@objectstack/service-analyticsvitest run: 162 files, 3741 passed, 45 skipped.@objectstack/lintvitest run: 119 files, 5502 passed.@objectstack/metadata-protocolvitest run: 200 files passed, 3 skipped; 2973 tests passed, 19 skipped.@objectstack/restvitest run --project local: 257 files, 4858 passed, 316 skipped.pnpm --filter PKG typecheckexit 0 for@objectstack/spec(tsc+check:scripts-typecheck+check:test-typecheck),@objectstack/service-analytics(itstsconfigincludes all ofsrc, so the edited__tests__file is in the program) and@objectstack/rest(tsc+check:test-typecheck).@objectstack/spec--project repo, the relevant files:step18-rationale-merge,conversions-major18-merge,liveness/evidence,liveness/proof-registry,retired-key-migrate-sentence,file-description,root-index,export-list,category-title,schema-tree-freshness,escape-mdxandreferences-banner. 12 files, 294 passed.eslint --no-inline-config --format jsonover the 8 changed lintable files at0d5e446egave 8 files, 0 errors, 0 warnings.eslint.config.mjs, the**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}block. The other changed files are.md,.mdxand.json.parserOptions.project, no typed rules), so this diff cannot move a verdict on an untouched file.pnpm lintis CI's.Round 2 readings (final head
fc4e91c0;origin/main3dc33b2dmerged throughos-regen-merge.sh)These readings were taken after a container restart. The restart cut a first round-2 gate run short at
2e57fa29. Every reading below was re-taken atfc4e91c0, the pushed head.turbo run buildover the closures of spec, cli, service-automation, metadata-protocol, rest and client-react: 59/59 tasks.@objectstack/specvitest run --project local: 597 files, 17465 passed, 1 todo. The counts moved withmain's merge.src/conversions/dataset-count-measure-empty-field-removed.test.ts, the table-wide fixture replay inconversions.test.tsandretired-after.census.test.ts.--project repo: the same 12 relevant files as round 1 (step18-rationale-mergeandconversions-major18-mergeamong them), 294 passed.@objectstack/climeta.report-order.test.ts(unit tier): 16 passed.@objectstack/service-automationdecision-overlapping-edge-conditions.pin.test.ts: 22 passed.@objectstack/metadata-protocolfull suite (it hosts the stored-row seam): 200 files passed and 3 skipped; 2973 tests passed and 19 skipped.rest,service-analyticsandlint: this round's diff does not reach them (spec only), and their round-1 readings stand.eslint --no-inline-config --format jsonover the 10 changed lintable files atfc4e91c0gave 10 files, 0 errors, 0 warnings. The population and invariance are as in round 1.dispatch-gates --commandsatfc4e91c0derived the same 115 families. All were run with exit codes recorded, and--ranreconciled them as "115 derived, 115 run, 0 NOT-MEASURED, 0 UNRUN".check:generated: 15/15 up to date.check:migration-registry: exit 0.check:adr-0087-registration: it readsregistered dataset-member-field-expression-refused, dataset-count-measure-empty-field-removed, both new here.check:skill-examplesandcheck:dual-build-cjs-loadsboth exited 0 this time. Both had been NOT MEASURED in round 1 for want of built packages.Ablations
Each ablation ran from committed state, disk-verified through
scripts/ablation-replace.mjs. Each restore was proven by blob hash equal to HEAD, an emptygit diff HEAD, and a clean status. The predicted direction was "turns red" in all four, and that is what was observed.fieldpattern admits anythingANALYTICS_COLUMN_REFERENCE(admits'*')'*'and the two pattern pins)ANALYTICS_COLUMN_PATHadmits anything, then@objectstack/specrebuiltablation-dist-preflight: marker in 18 built files.rest: the 2 re-pinned cases red,expected 403 to be 400(the route's parse passes the expression to the service door, the direction its docblock predicts); 6 green.service-analyticsdoor test: the contract case red, 20 greenfc4e91c0)field !== ''guard reads a value no row carries)conversions.test.tsfixture pindataset-count-measure-empty-field-removed: before → after, emits 2 notice(s)and the stored-row pin are red; the controls are green75f4166c== HEAD;git diff HEADempty; status cleanNo ablation file is left in the tree.
Gates
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands, run at0d5e446ewith no paths, derived 115 families.--ranreconciled them: "115 derived, 114 run, 1 NOT-MEASURED, 0 UNRUN", every row carrying its recorded exit code.check:dual-build-cjs-loads. Reason: it exited 3 (PREREQUISITE NOT MET) because 44 packages outside this diff's build closure have nodist/, and only a full monorepo build supplies them. This diff changes no package entry, export map or build config. CI runs it on the full build.check:skill-examplesfirst exited 3 for want of a built@objectstack/client-react. After building that package it exited 0 at0d5e446e: 259 examples type-check.check:generated: all 15 artifacts up to date against a stamp-matched dist.check:adr-0087-registration: at the first round's head it readregistered dataset-member-field-expression-refused (new here).check:changeset-no-majorandcheck:empty-changeset.check:liveness,check:migration-registryandcheck:doc-authoring.check:cross-package-test-inputsandcheck:nul-bytes.Acceptance notes
dataset.zod.ts,analytics.zod.ts"only as far as sharing needs", the ADR-0087 entry and registry, the retirement kit, pins and one changeset. Four paths go beyond that, each for the stated reason:data/analytics-column-reference.ts: the sharing change itself, which avoids a public export.content/docs/data-modeling/analytics.mdx: the skill's docs row.field. The route's existingDatasetSchema.parserefuses it first, asVALIDATION_FAILEDnaming the path. The changeset says so. The service door's 403 is unchanged.DatasetDefaultInspector.tsxat the pinned31971ff1eseeds a new dimension row as{ name: '', field: '', type: 'string' }and a new measure row as{ name: '', aggregate: 'sum', field: '' }. A plain count measure left with a blank Field box is saved asfield: ''. That parsed before. Its query answered 500 on the ObjectQL path (the SQLite native path happened to acceptCOUNT()).measures.N.field, with the prescription to omit the key. A row already stored that way is repaired on load by the D2 conversiondataset-count-measure-empty-field-removed.fieldwhen the box is blank. It is reported to the seat, not edited here.sum(or any non-count aggregate) over'*'parses on a dataset measure and answers 500 on both strategies. That is the count-only'*'boundary theanalytics_cubeledger already assigns to spec+service-analytics: retire the cube metric typesnumber/string/boolean— they existed to carry a SQL expression, which #20943 retired from a member'ssql#21000's family.fieldanswered 200 with the dimension column missing on the ObjectQL bridge. It is now refused at parse. A stored padded row would still reach that path through the build probe; no producer of one is known.Authored by
session_01UtnxvdiN376GF3sgXwAw4d(rounds 1 and 2).