fix(objectql): a write answers every stored declared column the driver did not echo as null, as the next read does (#29538)

.changeset/29538-write-answer-declared-column-null.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/objectql': patch
+---
+
+fix(objectql): a write answers every stored declared column the driver did not echo as `null`, so the insert and update answer is the record the next read returns, on every driver (#29538)
+
+Clause-②: no — nothing a request may send is refused or newly admitted, and nothing a write stores changes; measured on base and head through the engine and the protocol doors on SQLite, driver-memory and driver-mongodb (fake `Db`), every write was accepted on both. An insert or by-id update answer gains a `null` key for a stored declared column the driver did not echo, and on driver-mongodb and driver-memory a `formula` over such a column in that answer now evaluates against that `null`, as the next read and driver-sql already did.
+Metadata-protocol: none
+
+- **Before.** After #29408, `find` and `findOne` answer an unwritten declared column `null` on every driver, but the write answer still came straight from the driver. driver-sql echoed `null` (its table has the column); driver-mongodb and driver-memory echoed only the keys the write stored. So a create on mongo or memory answered `{ title: 'x' }` and the very next `GET` of the same row answered `{ title: 'x', note: null }`, and a formula reading `note` answered `null` (a fault) in the create response and its value on the `GET`.
+- **Now.** The engine answers the insert result (single, batch, and each `insertMany` outcome) and the by-id update readback on every stored declared column, `null` where the driver echoed no key, before the formulas and the `afterInsert` / `afterUpdate` hooks. Every door on those two verbs answers alike: `POST /data/:object`, `PATCH /data/:object/:id`, `createManyData`, `updateManyData`, and `batchData` create / update / both upsert arms.
+- **Never a `null` the row does not hold.** A column the insert payload, or the PATCH and then the stored row before it, carries a value for is stored, so a driver that stores it without echoing it leaves it absent from the answer, as before, rather than answered `null`. A reader that layers the payload under the answer (the record-change flow's `record`, which overlays the write result on the input) keeps the value it wrote.
+- **Unchanged.** What a write stores. A predicate update still answers its affected-row count. A column field-level security refuses the writer stays absent (the security mask still deletes it), and so do a name outside the declared set, an injected column a federated (`external`) object has no storage for, and a `created_at` / `updated_at` a federated object neither declares nor has injected — the read's own exclusions.
+- **Who sees it.** A client of an object served by driver-mongodb or driver-memory that diffs a create or update response against its own state, or renders it: a field the save never mentioned now arrives in that response as `null`, as it does on the next read, instead of appearing only on the next read. In-process readers of the write result see the same: an `afterInsert` / `afterUpdate` hook, a `data.record.created` / `data.record.updated` event payload and the webhooks that deliver it. A reader that told a missing key from `null` (`'x' in row`, `has(record.x)`) now reads what driver-sql already answered; test emptiness with `record.x != null`. A `formula` over an unwritten field in a write response on those two drivers changes value: it used to answer `null` with a fault, and now answers what the next read answers (`record.qty != null ? record.qty * 2 : -1` was `null`, now `-1`).

packages/drivers/driver-mongodb/src/mongodb-declared-column-null-on-write.test.ts

@@ -0,0 +1,170 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#29538] driver-mongodb through the engine: a write answers the record the
+ * next read answers — a declared column the write never touched is `null` in
+ * the insert and update answer, as on the following `findOne` (#29408).
+ *
+ * This driver stores no key for a field given no value (`withoutUndefinedOwnKeys`
+ * at the insert doors), its `create` / `bulkCreate` echo the document they
+ * inserted, and its `update` re-reads the document. So the row each write
+ * hands the engine is sparse. Measured on `origin/main` before the fix with
+ * this same harness: every write answer below carried no `note` / `qty` key
+ * and answered the formula `doubled` as `null` (a fault on the missing
+ * `qty`), while the next `findOne` answered `note: null`, `qty: null` and
+ * `doubled: -1`.
+ *
+ * Runs without a server, on the fake `Db` of
+ * `mongodb-declared-column-null-on-read.test.ts`, with the write calls the
+ * engine's insert and by-id update doors reach. The fake keeps each document
+ * as inserted and answers a projection as a real mongod does — the named keys
+ * the document HAS. The real-server suites are gated on `mongodb-memory-server`
+ * (#5517).
+ */
+
+import { describe, it, expect, beforeEach } from 'vitest';
+import { ObjectQL } from '@objectstack/objectql';
+import { MongoDBDriver } from './mongodb-driver.js';
+
+type Doc = Record<string, unknown>;
+
+/**
+ * A fake `Db`: one store of documents kept as inserted, served through a
+ * collection that declares ONLY the native `Collection` members the path under
+ * test calls. Two shapes, one per path: the single-row doors (`create`,
+ * `update`, `findOne`) reach `insertOne` / `updateOne` / `findOne`; the batch
+ * door (`bulkCreate`) reaches `insertMany`, and its next read is a `find`.
+ */
+function fakeStore() {
+  const docs: Doc[] = [];
+  const project = (doc: Doc, projection: Record<string, number> | undefined): Doc => {
+    const keep = Object.keys(projection ?? {}).filter((k) => projection![k] === 1);
+    const out: Doc = {};
+    for (const [k, v] of Object.entries(doc)) {
+      if (k === '_id' && projection?._id === 0) continue;
+      if (keep.length > 0 && !keep.includes(k)) continue;
+      out[k] = v;
+    }
+    return out;
+  };
+  // Equality on plain keys only — the writes and reads below address a row
+  // by `id` alone. A combinator or any other operator is REFUSED, never read
+  // as a field name.
+  const matches = (doc: Doc, filter: Doc | undefined): boolean =>
+    Object.entries(filter ?? {}).every(([k, v]) => {
+      if (k.startsWith('$')) throw new Error(`fake Db: filter operator ${k} is not implemented`);
+      if (v !== null && typeof v === 'object') {
+        const ops = Object.keys(v as Doc);
+        if (ops.length !== 1 || ops[0] !== '$eq') throw new Error(`fake Db: operator ${ops.join(',')} is not implemented`);
+        return doc[k] === (v as Doc).$eq;
+      }
+      return doc[k] === v;
+    });
+  const push = (doc: Doc) => { docs.push({ _id: `oid_${docs.length}`, ...doc }); };
+  const pick = (filter: Doc) => docs.find((x) => matches(x, filter));
+  const pickAll = (filter: Doc, options?: { projection?: Record<string, number> }) =>
+    docs.filter((x) => matches(x, filter)).map((d) => project(d, options?.projection));
+  return { docs, push, pick, pickAll, project };
+}
+
+/** The single-row doors: `create`, `update` and `findOne`. */
+function singleRowDb() {
+  const store = fakeStore();
+  const collection = {
+    insertOne: async (doc: Doc) => { store.push(doc); return {}; },
+    updateOne: async (filter: Doc, update: { $set?: Doc }) => {
+      const updateOps = Object.keys(update);
+      if (updateOps.length !== 1 || updateOps[0] !== '$set') throw new Error(`fake Db: update ${updateOps.join(',')} is not implemented`);
+      const d = store.pick(filter);
+      if (d) Object.assign(d, update.$set);
+      return { matchedCount: d ? 1 : 0 };
+    },
+    findOne: async (filter: Doc, options: { projection?: Record<string, number> }) => {
+      const d = store.pick(filter);
+      return d ? store.project(d, options?.projection) : null;
+    },
+  };
+  return { db: { collection: () => collection }, docs: store.docs };
+}
+
+/** The batch door: `bulkCreate`, and `find` for the next read. */
+function batchDb() {
+  const store = fakeStore();
+  const collection = {
+    insertMany: async (many: Doc[]) => { for (const doc of many) store.push(doc); return {}; },
+    find: (filter: Doc, options: { projection?: Record<string, number> }) => ({
+      toArray: async () => store.pickAll(filter, options),
+    }),
+  };
+  return { db: { collection: () => collection }, docs: store.docs };
+}
+
+const OBJECT = 'nk_mongo_write_item';
+
+const own = (row: unknown, key: string): boolean =>
+  row != null && typeof row === 'object' && Object.prototype.hasOwnProperty.call(row, key);
+
+describe('[#29538] driver-mongodb: a write answers the record the next read answers', () => {
+  let engine: ObjectQL;
+  let docs: Doc[];
+
+  const boot = async (fake: { db: unknown; docs: Doc[] }) => {
+    const driver = new MongoDBDriver({ url: 'mongodb://127.0.0.1:1/unused' });
+    docs = fake.docs;
+    (driver as any).db = fake.db;
+    (driver as any).connect = async () => {};
+    (driver as any).disconnect = async () => {};
+    engine = new ObjectQL();
+    engine.registerDriver(driver as never, true);
+    await engine.init();
+    engine.registry.registerObject({
+      name: OBJECT,
+      label: 'Item',
+      fields: {
+        name: { type: 'text' },
+        note: { type: 'text' },
+        qty: { type: 'number' },
+        doubled: { type: 'formula', expression: { dialect: 'cel', source: 'record.qty != null ? record.qty * 2 : -1' } },
+      },
+    } as never);
+  };
+
+  beforeEach(async () => { await boot(singleRowDb()); });
+
+  const expectAnswerIsNextRead = async (answer: Doc, next?: Doc) => {
+    expect(answer).toStrictEqual(next ?? await engine.findOne(OBJECT, { where: { id: answer.id } }));
+    expect(own(answer, 'note')).toBe(true);
+    expect(answer.note).toBeNull();
+    expect(own(answer, 'qty')).toBe(true);
+    expect(answer.qty).toBeNull();
+    // FORMULA CONTROL: evaluated against the `null`, as the read evaluates it.
+    expect(answer.doubled).toBe(-1);
+  };
+
+  it('insert: the answer deep-equals the next findOne', async () => {
+    const answer = await engine.insert(OBJECT, { name: 'Ada' }) as Doc;
+    expect(own(docs[0], 'note')).toBe(false);
+    await expectAnswerIsNextRead(answer);
+  });
+
+  it('batch insert (bulkCreate): every answered row deep-equals its next read', async () => {
+    await boot(batchDb());
+    const answers = await engine.insert(OBJECT, [{ name: 'Ada' }, { name: 'Bo' }]) as Doc[];
+    expect(answers).toHaveLength(2);
+    expect(own(docs[0], 'note')).toBe(false);
+    for (const answer of answers) {
+      const [next] = await engine.find(OBJECT, { where: { id: answer.id } }) as Doc[];
+      await expectAnswerIsNextRead(answer, next);
+    }
+  });
+
+  it('update by id: the answer deep-equals the next findOne', async () => {
+    const created = await engine.insert(OBJECT, { name: 'Ada' }) as Doc;
+    const answer = await engine.update(OBJECT, { id: created.id, name: 'Ada L.' }) as Doc;
+    expect(answer.name).toBe('Ada L.');
+    await expectAnswerIsNextRead(answer);
+    // The stored document still carries no key for the column: only the
+    // answer is shaped, never what the write stores.
+    expect(own(docs[0], 'note')).toBe(false);
+  });
+});

packages/objectql/src/declared-read-columns.ts

@@ -159,7 +159,13 @@ export function rowsWithDeclaredColumnsOnly<T>(rows: T[], declared: ReadonlySet<
  *   engine's own widening (formula inputs, a file field's owner key, each cut
  *   back off the answer before the verb returns) are what count;
  * - no projection (a whole-record GET, a list read) selects every declared
- *   column.
+ *   column;
+ * - [#29538] a write's answer (the insert result, single or batch, and the
+ *   by-id update readback) names no projection either, so it is answered on
+ *   the whole stored record: the write answer equals the next read of the row.
+ *   Less the columns the write itself carried a value for
+ *   ({@link storedColumnsLeftUnwritten}): those are stored, so a driver that
+ *   did not echo one leaves it absent rather than answered `null`.
  *
  * ## What stays absent, and why
  *
@@ -233,6 +239,38 @@ export function withSelectedColumnsAnswered<T>(row: T, selected: ReadonlySet<str
   return (shaped ?? row) as unknown as T;
 }
 
+/**
+ * [#29538] The {@link selectedStoredColumns} a write's answer may fill with
+ * `null`: those minus every column the write's images say holds a value. A
+ * write image is what the engine knows the row stores without the driver's
+ * echo, newest first — the payload it handed the driver and, on an update, the
+ * stored row it read before the write. The first image that names a column
+ * decides it: a non-null value IS stored, so a driver that stored it and did
+ * not echo it leaves it absent from the answer (unknown), never answered
+ * `null` (a value the row does not hold) — a reader that layers the payload
+ * under the answer (`{ ...input, ...result }`) would otherwise lose the value
+ * it wrote. A `null` there (a PATCH clearing a value the prior row held, too)
+ * is a stored `null` and stays answerable; an `undefined` names nothing. Same
+ * set reference when nothing is removed.
+ */
+export function storedColumnsLeftUnwritten(
+  selected: ReadonlySet<string> | undefined,
+  ...writeImages: unknown[]
+): ReadonlySet<string> | undefined {
+  if (!selected || selected.size === 0) return selected;
+  const images = writeImages.filter(
+    (image): image is Record<string, unknown> => !!image && typeof image === 'object' && !Array.isArray(image),
+  );
+  let left: Set<string> | undefined;
+  for (const name of selected) {
+    const decider = images.find((image) => image[name] !== undefined);
+    if (decider === undefined || decider[name] === null) continue;
+    left ??= new Set(selected);
+    left.delete(name);
+  }
+  return left ?? selected;
+}
+
 /** {@link withSelectedColumnsAnswered} over a driver's result page. */
 export function rowsWithSelectedColumnsAnswered<T>(rows: T[], selected: ReadonlySet<string> | undefined): T[] {
   if (!selected || selected.size === 0) return rows;

packages/objectql/src/engine-default-value-tokens.test.ts

@@ -103,8 +103,11 @@ describe('[#4560] the `current_user` defaultValue token is engine-owned', () =>
     // decision a column DEFAULT used to override, writing the literal string
     // `current_user` into a lookup('sys_user') column (#4560).
     const row: any = await engine.insert('tok_doc', { title: 'B' }, { context: { isSystem: true } } as any);
-    expect(row.owner).toBeUndefined();
-    expect(row.owner).not.toBe('current_user');
+    // [#29538] The answer is the next read's record: a column nothing was
+    // stored in reads `null`. "Unset" is the STORED row's own: no key at all.
+    expect(row.owner).toBeNull();
+    const stored = await engine.getDriverForObject('tok_doc')!.findOne('tok_doc', { where: { id: row.id } });
+    expect(stored).not.toHaveProperty('owner');
   });
 
   it('an explicit null is treated as "not supplied" and still resolves the token (#2706)', async () => {

packages/objectql/src/engine-lookup-referential-integrity.test.ts

@@ -370,7 +370,11 @@ describe('[#4441] a lookup id that resolves to nothing is refused', () => {
     const row: any = await engine.insert(
       'ref_audit_note', { note: 'n', recorded_by: 'ps_does_not_exist' }, { context: userCtx } as any,
     );
-    expect(row.recorded_by, 'stripped before the reference check, not refused by it').toBeUndefined();
+    // [#29538] The answer reads the unstored column `null`, as the next read
+    // does; the strip is the STORED row's own: no key at all.
+    expect(row.recorded_by, 'stripped before the reference check, not refused by it').toBeNull();
+    const stored = await engine.getDriverForObject('ref_audit_note')!.findOne('ref_audit_note', { where: { id: row.id } });
+    expect(stored, 'stripped before the reference check, not refused by it').not.toHaveProperty('recorded_by');
   });
 
   it('…and the issue\'s own fields are NOT readonly, so they stay enforced', () => {

packages/objectql/src/engine-select-option-default.test.ts

@@ -72,6 +72,10 @@ function makeStubDriver() {
 
 const sys = { context: { isSystem: true } } as any;
 
+/** The row as the stub driver stored it — what the write persisted, before any engine shaping. */
+const storedRow = (engine: ObjectQL, object: string, id: unknown) =>
+  engine.getDriverForObject(object)!.findOne(object, { where: { id } });
+
 describe('[#7246] the option marked `default: true` is the field default', () => {
   let engine: ObjectQL;
 
@@ -132,7 +136,10 @@ describe('[#7246] the option marked `default: true` is the field default', () =>
       },
     } as any, 'test.issue7246');
     const row: any = await engine.insert('opt_unmarked', { title: 'D' }, sys);
-    expect(row.status).toBeUndefined();
+    // [#29538] The answer reads the unstored column `null`, as the next read
+    // does; "left alone" is the STORED row's own: no key at all.
+    expect(row.status).toBeNull();
+    expect(await storedRow(engine, 'opt_unmarked', row.id)).not.toHaveProperty('status');
   });
 });
 
@@ -348,8 +355,11 @@ describe('[#7246] the default follows the FIELD shape, not the option count', ()
       { label: 'Quick', value: 'quick' },
     ]) as any, 'test.issue7246');
     const row: any = await engine.insert('opt_valueless', {}, sys);
-    expect(row.tags).toBeUndefined();
-    expect(row.tags).not.toBeNull();
+    // [#29538] The answer reads the unstored column `null`, as the next read
+    // does; "left unset" (no value, not even a stored `null`) is the STORED
+    // row's own: no key at all.
+    expect(row.tags).toBeNull();
+    expect(await storedRow(engine, 'opt_valueless', row.id)).not.toHaveProperty('tags');
   });
 });
 
@@ -400,6 +410,8 @@ describe('[#7246] the lint heuristic and the engine agree BY CONSTRUCTION', () =
       },
     } as any, 'test.issue7246');
     const row: any = await engine.insert('opt_alias', {}, sys);
-    expect(row.status).toBeUndefined();
+    // [#29538] The answer reads the unstored column `null`; the STORED row has no key.
+    expect(row.status).toBeNull();
+    expect(await storedRow(engine, 'opt_alias', row.id)).not.toHaveProperty('status');
   });
 });

packages/objectql/src/engine-write-formula-hydration.test.ts

@@ -557,8 +557,12 @@ describe('#5504 — cost threshold: same gate the read path uses', () => {
   it('an object declaring NO formula evaluates nothing on write', async () => {
     const created = await rig.protocol.createData({ object: 'wf_plain', data: { name: 'nothing derived' } });
     expect(evaluate).not.toHaveBeenCalled();
-    // …and the response carries exactly the stored keys — no phantom columns.
-    expect(Object.keys(created.record as Rec).sort()).toEqual(['id', 'name']);
+    // …and the response carries exactly the stored values — no phantom columns.
+    // [#29538] Every other stored declared column is answered `null`, as the
+    // next read answers it: the response IS that read's record.
+    const record = created.record as Rec;
+    expect(Object.keys(record).filter((k) => record[k] !== null).sort()).toEqual(['id', 'name']);
+    expect(record).toStrictEqual(await rig.engine.findOne('wf_plain', { where: { id: record.id } }));
   });
 
   it('an object declaring formulas evaluates once per formula field per row', async () => {