diff --git a/.changeset/27193-recompute-home-amounts.md b/.changeset/27193-recompute-home-amounts.md new file mode 100644 index 00000000000..c68142aa0a7 --- /dev/null +++ b/.changeset/27193-recompute-home-amounts.md @@ -0,0 +1,13 @@ +--- +'@objectstack/objectql': patch +'@objectstack/cli': patch +--- + +feat(objectql,cli): `os migrate recompute-home-amounts` — recompute stored home-currency amounts at the rates in force now, report first, rows without a rate listed and skipped + +Clause-②: yes (widening) +Metadata-protocol: none + +- **`@objectstack/objectql`: `recomputeHomeAmounts(engine, logger, { object, field?, since?, apply?, maxRecords?, maxListed?, source? })`**, with `assertHomeAmountRecomputeSince` (the pure `since` check a door can run before it opens a database), `formatHomeAmountRecomputeReport`, `homeAmountRecomputeComplete` and `HOME_AMOUNT_SKIP_REASONS`. It walks every stored row of an object whose `record`-mode amounts declare `currencyConfig.homeAmount` and projects each through the write-time projection itself (the same rate row, division, single rounding, same-currency rule and organization-timezone day), in the row's own organization, so each organization converts into its own home currency at its own rates and is counted apart. A dry run (the default) reports what would change and writes nothing; `apply` writes only the rows that differ, through `engine.update` with the home field in the payload, so the value stored is the one the write itself computes, and only the home and rate fields move (`updated_at` is kept, so a consumer syncing by `updated_at` does not see the recompute). Being a write, it moves what reads the home field as a fresh write would: stored formulas, parent summaries, and the per-row audit rows where audit capture is installed. Rows that cannot be converted are listed by reason and never written: `rate_missing` (with the currency and date), `code_blank` (an amount with no currency code; the remedy is `os migrate record-currency-codes`), `document_date_blank`, `home_currency_unset` and `unconvertible`. `since` filters on the document date (`homeAmount.rateDate`). One run-level `sys_audit_log` entry records each applied run (counts per organization and reason, and the skipped rows), also when an unclassified error stops the run partway (marked `aborted`, then the error is thrown on). A walk cut short by `maxRecords` is `truncated` and not complete. Refused before any row is read: an unknown object (`OBJECT_NOT_FOUND`), an object or `field` declaring no home amount (`INVALID_FIELD`), a `since` that is not `YYYY-MM-DD` (`VALIDATION_FAILED`). +- **`@objectstack/objectql`: `ObjectQL.homeAmountReadsFor(context?)`**, the reads the write-time projection makes (its serving, and the organization a write converts a row under), for a caller that projects a stored row as a write would. +- **`@objectstack/cli`: `os migrate recompute-home-amounts --object [--field ] [--since YYYY-MM-DD] [--apply] [--yes] [--json] [--force] [--max-records] [--database-url]`**, the shell of `os migrate record-currency-codes`, booting the settings service (each organization's `localization.currency` and `.timezone`) and the audit ledger. It exits 1 while any row is listed and skipped, on a dry run as on an apply, because a total over the home field silently leaves such a row out, and when `--max-records` cut the walk short. diff --git a/content/docs/data-modeling/currency.mdx b/content/docs/data-modeling/currency.mdx index 9700c8942a0..4c39d34c694 100644 --- a/content/docs/data-modeling/currency.mdx +++ b/content/docs/data-modeling/currency.mdx @@ -317,7 +317,8 @@ same path as a single write. A rate row corrected later does not rewrite the amounts already converted with it: an update that does not touch the amount, its code or its document date leaves the stored -home amount and rate as they are. +home amount and rate as they are. To bring stored rows up to the rates in force now, run +the [recompute](#recomputing-stored-home-amounts). An update by filter (`multi: true`) writes one patch to every matched row, so it is refused, with `VALIDATION_FAILED`, when the patch would convert the rows to different home @@ -336,6 +337,36 @@ Each refusal is answered before anything is written, and the row is not stored: The `validate()` preview, and an import's dry run, answer the same refusals for the same rows. +### Recomputing stored home amounts + +`os migrate recompute-home-amounts` brings the stored home amounts of an object up to the +rates in force now: after a rate row is corrected or added, or after a home amount is +declared on an object that already holds rows. + +```bash +os migrate recompute-home-amounts --object invoice # Report: what would change, what cannot be converted +os migrate recompute-home-amounts --object invoice --since 2026-01-01 # Only documents dated on or after January 1 +os migrate recompute-home-amounts --object invoice --apply # Write the changes (prompts) +``` + +Each row is converted exactly as a write converts it, in its own organization's home +currency at that organization's rates, and an applied row holds what a fresh write of it +would store, so what reads the home field (formulas, roll-ups, the audit trail) follows as +it does for a fresh write. The row's `updated_at` is kept: an integration that syncs by +`updated_at` does not see the recomputed value until it resyncs the object. `--since` compares the **document date** (the field `homeAmount.rateDate` +names), read as the day the write reads it. The report counts each organization apart and +lists every row that changes, with its stored and new values. + +**A row that cannot be converted is listed and left as stored**, never filled with a rate +of 1, a later rate or a zero. The report names each one and why: no rate in force on its +document date (add the rate), an amount with no currency code (run +[`os migrate record-currency-codes`](#switching-a-field-that-already-holds-rows) first: no +rate can help a row that names no currency; it writes its one code into every +organization's codeless rows, so check the per-organization count first), no document date, or no home currency. Until +a re-run converts them, a total over the home field leaves those rows out, so the command +keeps exiting 1 while any is listed, and the applied run's audit entry records them. The +full reference is [`os migrate recompute-home-amounts`](/docs/deployment/cli#os-migrate-recompute-home-amounts). + ### Totals in the home currency The home field is an ordinary stored `currency` field, so every aggregate reads it as it diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index b0acd520d19..3ab16296211 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1307,6 +1307,7 @@ where the data lives. | `os migrate option-codes` | Rewrite option values still stored as a number, a boolean or SQLite `"N.0"` text (rows written before option fields stored the code string) to that code string, so filters, reads and unique keys see one value. Values it cannot decide are listed, never rewritten. No flag; idempotent. See [below](#os-migrate-option-codes) | | `os migrate text-scalars` | Rewrite numbers and booleans still stored raw on text fields (rows written before text fields stored the string) to that string, so filters, reads and unique keys see one value. SQLite `"N.0"` text it cannot decide is listed, never rewritten. No flag; idempotent. See [below](#os-migrate-text-scalars) | | `os migrate record-currency-codes` | Write the currency code you give into the rows of a record-mode currency field that hold an amount and no code, the rows stored before the field was switched to `currencyMode: 'record'`. The code is never derived. No flag; idempotent. See [below](#os-migrate-record-currency-codes) | +| `os migrate recompute-home-amounts` | Recompute the home-currency amounts a record-mode currency field stores, at the exchange rates in force now, after a rate is corrected or added. Reports the rows that change and lists, without writing, the rows it cannot convert. No flag; idempotent. See [below](#os-migrate-recompute-home-amounts) | | `os migrate comment-reactions` | Carry the emoji reactions a 17.x deployment stored in the retired `sys_comment.reactions` column into `sys_comment_reaction` records, list every stored value that does not read, and record the verified run that lets `os migrate apply --allow-destructive` drop the column | | `os migrate organization-ownership` | The v18 ownership ceremony (ADR-0131 D10): write a plan of what each table's rows will become, then `--apply` that reviewed plan. It upgrades a database 17.x or an 18.0 pre-release (`18.0.0-next.*`) created; a v18 server refuses to start on one until it has run. See [below](#os-migrate-organization-ownership) | | `os migrate activity-changed-fields` | Stamp the two read-rule columns (`changed_fields`, `milestone_key`) on the `sys_activity` rows written before they existed. Until it runs, those rows are withheld from every reader but the system, so run it at once after upgrading. No flag; idempotent | @@ -1621,6 +1622,68 @@ every write turns a blank code into a code, so a second run finds nothing to wri Re-running until the report says zero *is* the verification, which is why this command records no flag. +#### `os migrate recompute-home-amounts` + +A record-mode amount that declares a home amount is converted into the organization's home +currency when it is written, and stored with the rate it used +([Currency fields](/docs/data-modeling/currency#converting-into-the-home-currency)). A rate +row corrected or added later does not rewrite what was stored. This command does, after a +report. + +```bash +os migrate recompute-home-amounts --object invoice # Dry run: reports, writes none +os migrate recompute-home-amounts --object invoice --since 2026-01-01 # Documents dated on or after Jan 1 only +os migrate recompute-home-amounts --object invoice --field amount --apply # Write (prompts) +os migrate recompute-home-amounts --object invoice --apply --yes --json # CI / scripts +``` + +**The write's own numbers.** Each stored row is projected exactly as a write projects it: +the rate row with the latest effective date on or before the document date, in the row's +own organization; the amount divided by the rate in exact decimal arithmetic, rounded once +to the home currency's decimal places; rate `1` for an amount in the home currency; and a +`datetime` document date read as its day in the organization's timezone. An applied row is +written through the engine, which converts it again, so it holds what a fresh write of the +row would store. Each organization converts into its own home currency at its own rates, +and the report counts each organization apart. + +**What is written.** Only rows whose stored home amount or rate differs, and in them only +the home and rate fields: `updated_at` keeps its stored value. Because each one is an +ordinary write of the home field, whatever reads that field moves with it, as it would for +a fresh write: stored formulas, parent summaries, and, where audit capture is installed +(it is in the command's own boot), one per-row `sys_audit_log` `update` row for each row +rewritten. ⚠️ Since `updated_at` does not move, a consumer that syncs incrementally by +`updated_at` does not see a recomputed row; resync the object, or the rows the report +lists. (`os migrate record-currency-codes` does move `updated_at`.) `--field` limits the run to +one amount; without it, every amount of the object that declares a home amount is +recomputed. `--since YYYY-MM-DD` keeps the rows whose **document date** (the field +`homeAmount.rateDate` names, the date that picks the rate) is on or after that day, a +`datetime` read as its day in the organization's timezone; a row with no document date is +outside every `--since` window. A malformed `--since` is refused with `VALIDATION_FAILED` / +400 before the database is opened; an object the deployment does not declare +(`OBJECT_NOT_FOUND`), or one or a `--field` that declares no home amount (`INVALID_FIELD`), +is refused after the boot and before any row is read. + +**Rows it cannot convert are listed and skipped**, each with its reason, and never written: + +| Reason | The row | Fix, then re-run | +|:---|:---|:---| +| `rate_missing` | No rate for its currency is in force on its document date (the report names both). | Add the rate. | +| `code_blank` | An amount with no currency code: stored before the field switched to record mode. | [`os migrate record-currency-codes`](#os-migrate-record-currency-codes). It writes its one code into the codeless rows of **every** organization, so check the report's `code_blank` count per organization first. | +| `document_date_blank` | A foreign-currency amount with no document date. | Fill the document date. | +| `home_currency_unset` | Its organization has no `localization.currency`. | Set **Localization → Default currency**. | +| `unconvertible` | A code, amount or document date the write itself refuses. | Correct the value. | + +A skipped row keeps whatever it stored, so a total over the home field leaves it out (a +`sum` skips an empty home amount silently). That gap stays visible: the command exits 1, +on a dry run and on an apply, for as long as any row is listed, and an applied run's +run-level `sys_audit_log` entry (one per run beside the per-row rows, no record, +`action: update`, `metadata.event: home_amount.recompute`) records the counts per +organization and reason and every skipped row. A run that an unexpected error stops +partway still writes that entry, marked `aborted` with the counts it reached, before the +command fails. A walk cut short by `--max-records` also exits 1: rows past the bound were +not looked at. Idempotent: re-running until it lists nothing and changes nothing *is* the +verification, which is why it records no flag. + #### `os migrate comment-reactions` v18 retired `sys_comment.reactions`, the JSON column that held every emoji diff --git a/packages/cli/src/commands/migrate/recompute-home-amounts.test.ts b/packages/cli/src/commands/migrate/recompute-home-amounts.test.ts new file mode 100644 index 00000000000..83f1ce2d3a1 --- /dev/null +++ b/packages/cli/src/commands/migrate/recompute-home-amounts.test.ts @@ -0,0 +1,205 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `os migrate recompute-home-amounts` command shape (ADR-0144 D5, D7 + * parameter 1, Open item 8). + * + * The recompute itself is proven in `@objectstack/objectql`'s + * `engine-home-amount-projection.test.ts` (in memory) and in + * `packages/qa/dogfood/test/home-amount-projection.test.ts` (SQLite and + * PostgreSQL). What is pinned here is what those cannot see: the command is a + * dry run by default, its flags reach `recomputeHomeAmounts` as given, a + * malformed `--since` is refused before the probe, the prompt and the boot, + * the boot carries the settings service and the audit ledger, a listed and + * skipped row keeps the exit code at 1 on a dry run as on an apply run, and a + * refusal reaches the `--json` envelope with its code. The seams that would + * boot a database or walk a real engine are replaced; the command's own parse + * and control flow run for real, and the report's formatter is the real one. + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import MigrateRecomputeHomeAmounts from './recompute-home-amounts.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; +import { isExitSignal } from '../../utils/format.js'; +import { recomputeHomeAmounts } from '@objectstack/objectql'; +import { assertEngineUpdateDispatch } from '@objectstack/metadata-core'; + +vi.mock('../../utils/schema-migrate.js', () => ({ bootSchemaStack: vi.fn() })); +vi.mock('../../utils/migrate-occupancy-gate.js', () => ({ + OCCUPANCY_HINT: 'occupancy hint', + probeMigrationTarget: vi.fn(), +})); +vi.mock('../../utils/data-migration-plugins.js', () => ({ buildDataMigrationPlugins: vi.fn(async () => []) })); +vi.mock('@objectstack/objectql', async (importOriginal) => { + const real = await importOriginal(); + return { + recomputeHomeAmounts: vi.fn(), + // The `--since` check is the recompute's own, called by the command before its probe. + assertHomeAmountRecomputeSince: real.assertHomeAmountRecomputeSince, + formatHomeAmountRecomputeReport: real.formatHomeAmountRecomputeReport, + }; +}); + +const HERE = dirname(fileURLToPath(import.meta.url)); +const CLI_ROOT = resolve(HERE, '..', '..', '..'); +/** oclif builds its whole command table on the first `run()` in a process. */ +const RUN_TIMEOUT = 60_000; + +const engine = { + getObject: () => ({ fields: {} }), + find: async () => [], + update: async (data: Record, options?: Record) => { + assertEngineUpdateDispatch(data, options); + return {}; + }, + insert: async () => ({}), + homeAmountReadsFor: () => ({}), + getConfigs: () => ({ invoice: {}, sys_user: {} }), +}; + +const zero = { rate_missing: 0, code_blank: 0, document_date_blank: 0, home_currency_unset: 0, unconvertible: 0 }; +const report = (over: Record = {}) => ({ + object: 'invoice', + fields: [{ field: 'amount', codeField: 'currency_code', homeField: 'amount_home', rateField: 'amount_rate', rateDateField: 'invoice_date' }], + since: null, applied: false, scannedRecords: 2, amounts: 2, unchanged: 1, changed: 1, written: 0, outsideSince: 0, + skippedTotal: 0, skippedByReason: { ...zero }, + organizations: [{ organizationId: 'org_a', homeCurrency: 'CNY', amounts: 2, unchanged: 1, changed: 1, written: 0, skipped: { ...zero }, outsideSince: 0 }], + changes: [{ recordId: 'i1', organizationId: 'org_a', field: 'amount', from: { home: 71994.24, rate: 0.1389 }, to: { home: 80000, rate: 0.125 } }], + changesTruncated: false, skipped: [], skippedTruncated: false, failures: [], truncated: false, + audit: { written: false, reason: 'dry_run' }, + ...over, +}); +const SKIPPED = { + skippedTotal: 1, + skippedByReason: { ...zero, code_blank: 1 }, + skipped: [{ recordId: 'i2', organizationId: 'org_a', field: 'amount', reason: 'code_blank' }], +}; + +let stdout: ReturnType; +let log: ReturnType; +beforeEach(() => { + vi.mocked(probeMigrationTarget).mockReset(); + vi.mocked(probeMigrationTarget).mockResolvedValue({ status: 'free' } as any); + vi.mocked(bootSchemaStack).mockReset(); + vi.mocked(bootSchemaStack).mockResolvedValue({ + kernel: { getService: () => engine }, + dbLabel: 'test database', + shutdown: vi.fn(async () => {}), + } as any); + vi.mocked(buildDataMigrationPlugins).mockClear(); + vi.mocked(recomputeHomeAmounts).mockReset(); + vi.mocked(recomputeHomeAmounts).mockResolvedValue(report() as any); + // `emitJson` awaits the write's drain callback, so the double invokes it. + stdout = vi.spyOn(process.stdout, 'write').mockImplementation(((chunk: unknown, enc?: unknown, cb?: unknown) => { + const done = typeof enc === 'function' ? enc : cb; + if (typeof done === 'function') done(); + return true; + }) as typeof process.stdout.write); + log = vi.spyOn(console, 'log').mockImplementation(() => {}); +}); +afterEach(() => { + stdout.mockRestore(); + log.mockRestore(); +}); + +const json = () => JSON.parse(stdout.mock.calls.map((c: unknown[]) => String(c[0])).join('')); +const exitOf = (err: unknown) => (err as { oclif?: { exit?: number } } | undefined)?.oclif?.exit; +const printed = () => log.mock.calls.map((c: unknown[]) => c.map(String).join(' ')).join('\n'); + +describe('os migrate recompute-home-amounts', () => { + it('is a dry run by default, --yes is opt-in, and only --object is required', () => { + expect(MigrateRecomputeHomeAmounts.flags.apply.default).toBe(false); + expect(MigrateRecomputeHomeAmounts.flags.yes.default).toBe(false); + expect(MigrateRecomputeHomeAmounts.flags.object.required).toBe(true); + expect(MigrateRecomputeHomeAmounts.flags.field.required).toBeFalsy(); + expect(MigrateRecomputeHomeAmounts.flags.since.required).toBeFalsy(); + }); + + it('hands the flags to recomputeHomeAmounts as given, and boots the settings service and the audit ledger', async () => { + await MigrateRecomputeHomeAmounts.run( + ['--json', '--object', 'invoice', '--field', 'amount', '--since', '2026-01-01'], + { root: CLI_ROOT }, + ); + expect(vi.mocked(recomputeHomeAmounts).mock.calls[0]![2]).toEqual({ + object: 'invoice', field: 'amount', since: '2026-01-01', apply: false, maxRecords: undefined, + source: 'os migrate recompute-home-amounts', + }); + expect(vi.mocked(buildDataMigrationPlugins)).toHaveBeenCalledWith({ settings: true, audit: true }); + expect(vi.mocked(bootSchemaStack).mock.calls[0]![0]).toMatchObject({ deferSchemaDdl: true, readOnlyProbe: true }); + expect(json()).toMatchObject({ apply: false, report: { changed: 1, written: 0 } }); + }, RUN_TIMEOUT); + + it('--apply --yes writes, on the plain boot', async () => { + await MigrateRecomputeHomeAmounts.run(['--json', '--apply', '--yes', '--object', 'invoice'], { root: CLI_ROOT }); + expect(vi.mocked(recomputeHomeAmounts).mock.calls[0]![2]).toMatchObject({ apply: true }); + expect(vi.mocked(bootSchemaStack).mock.calls[0]![0]).not.toHaveProperty('deferSchemaDdl'); + }, RUN_TIMEOUT); + + it('a malformed --since is refused VALIDATION_FAILED / 400 before the probe, the prompt and the boot', async () => { + const err = await MigrateRecomputeHomeAmounts.run( + ['--json', '--apply', '--object', 'invoice', '--since', '2026-02-30'], + { root: CLI_ROOT }, + ).catch((e: unknown) => e); + expect(exitOf(err)).toBe(1); + expect(json()).toMatchObject({ code: 'VALIDATION_FAILED', httpStatus: 400 }); + expect(json().error).toContain("since '2026-02-30' is not a calendar date"); + expect(vi.mocked(probeMigrationTarget)).not.toHaveBeenCalled(); + expect(vi.mocked(bootSchemaStack)).not.toHaveBeenCalled(); + expect(vi.mocked(recomputeHomeAmounts)).not.toHaveBeenCalled(); + }, RUN_TIMEOUT); + + it('an object the deployment does not declare is refused before the recompute is called', async () => { + const err = await MigrateRecomputeHomeAmounts.run(['--json', '--object', 'nope'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(exitOf(err)).toBe(1); + expect(vi.mocked(recomputeHomeAmounts)).not.toHaveBeenCalled(); + // The shared refusal's envelope carries the code alone (its key set is pinned in + // object-scope.integration.test.ts); the exit code is asserted above. + expect(json()).toMatchObject({ code: 'OBJECT_NOT_FOUND' }); + }, RUN_TIMEOUT); + + it('a listed and skipped row keeps the exit code at 1, on a dry run and on an apply run (Open item 8)', async () => { + vi.mocked(recomputeHomeAmounts).mockResolvedValue(report(SKIPPED) as any); + const dry = await MigrateRecomputeHomeAmounts.run(['--object', 'invoice'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(isExitSignal(dry)).toBe(true); + expect(exitOf(dry)).toBe(1); + expect(printed()).toContain('os migrate record-currency-codes --object invoice'); + vi.mocked(recomputeHomeAmounts).mockResolvedValue(report({ ...SKIPPED, applied: true, audit: { written: true } }) as any); + const applied = await MigrateRecomputeHomeAmounts.run(['--json', '--apply', '--yes', '--object', 'invoice'], { root: CLI_ROOT }) + .catch((e: unknown) => e); + expect(exitOf(applied)).toBe(1); + expect(json()).toMatchObject({ report: { skippedTotal: 1 } }); + }, RUN_TIMEOUT); + + it('a walk --max-records cut short exits 1: rows past the bound were not looked at', async () => { + vi.mocked(recomputeHomeAmounts).mockResolvedValue(report({ truncated: true }) as any); + const dry = await MigrateRecomputeHomeAmounts.run(['--object', 'invoice', '--max-records', '2'], { root: CLI_ROOT }) + .catch((e: unknown) => e); + expect(exitOf(dry)).toBe(1); + expect(printed()).toContain('--max-records'); + expect(vi.mocked(recomputeHomeAmounts).mock.calls[0]![2]).toMatchObject({ maxRecords: 2 }); + vi.mocked(recomputeHomeAmounts).mockResolvedValue(report({ truncated: true, applied: true, written: 1 }) as any); + const applied = await MigrateRecomputeHomeAmounts.run(['--json', '--apply', '--yes', '--object', 'invoice'], { root: CLI_ROOT }) + .catch((e: unknown) => e); + expect(exitOf(applied)).toBe(1); + expect(json()).toMatchObject({ report: { truncated: true } }); + }, RUN_TIMEOUT); + + it('a run with nothing skipped exits 0', async () => { + await expect(MigrateRecomputeHomeAmounts.run(['--object', 'invoice'], { root: CLI_ROOT })).resolves.toBeUndefined(); + expect(printed()).toContain('i1 amount: 71994.24 @ 0.1389 → 80000 @ 0.125'); + }, RUN_TIMEOUT); + + it('a refusal from the recompute reaches the --json envelope with its code and httpStatus', async () => { + const real = await vi.importActual('@objectstack/objectql'); + const refusal = await real.recomputeHomeAmounts(engine as never, { info() {}, warn() {} }, { object: 'invoice' }) + .then(() => { throw new Error('expected the object to be refused'); }, (e: unknown) => e as Error); + vi.mocked(recomputeHomeAmounts).mockRejectedValue(refusal); + const err = await MigrateRecomputeHomeAmounts.run(['--json', '--object', 'invoice'], { root: CLI_ROOT }).catch((e: unknown) => e); + expect(exitOf(err)).toBe(1); + expect(json()).toMatchObject({ code: 'INVALID_FIELD', httpStatus: 400, error: refusal.message }); + }, RUN_TIMEOUT); +}); diff --git a/packages/cli/src/commands/migrate/recompute-home-amounts.ts b/packages/cli/src/commands/migrate/recompute-home-amounts.ts new file mode 100644 index 00000000000..b37238e18b2 --- /dev/null +++ b/packages/cli/src/commands/migrate/recompute-home-amounts.ts @@ -0,0 +1,297 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import chalk from 'chalk'; +import { createInterface } from 'node:readline'; +import { + printHeader, + printSuccess, + printWarning, + printError, + printInfo, + printStep, + createTimer, + emitJson, + isExitSignal, + errorCodeFields, +} from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { describeOccupancy, JOURNAL_MODE_SWITCH_NOTE } from '../../utils/sqlite-occupancy.js'; +import { describeDriverSqliteFile } from '../../utils/connection-display.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; +import { refuseUndeclaredObjects } from '../../utils/migrate-object-scope.js'; +// Type-only: the engine package is imported dynamically inside run(), first for the `--since` +// check (before the probe, the prompt and the boot), never at module load, so `os --help` and +// every other command pay nothing for it. +import type { HomeAmountRecomputeEngine } from '@objectstack/objectql'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((resolve) => rl.question(question, resolve)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} + +/** + * `os migrate recompute-home-amounts` — the explicit recompute of the home + * amounts a `record`-mode amount stores (ADR-0144 D5, D7 parameter 1). + * + * A write stores each such amount converted into its organization's home + * currency, with the rate it used. A rate row corrected (or added) later does + * not rewrite what was stored; this command does, after a report. + * + * The command is the shell; the walk, its refusals, its report and its audit + * entry are `@objectstack/objectql`'s `recomputeHomeAmounts` — the split + * `os migrate record-currency-codes` has with `record-currency-backfill.ts`. + * The numbers are the write's: the run projects each stored row through the + * write's own projection, and an applied row is written through the engine, + * which projects it again. The boot carries the settings service (each + * organization's home currency and timezone) and the audit ledger (one run-level entry + * per applied run). + * + * Dry run by default (writes no row), `--apply` to write. Rows that cannot be + * converted (no rate in force on the document date, an amount with no + * currency code, no document date, no home currency) are listed and skipped, + * and keep the command's exit code at 1 until a re-run converts them + * (ADR-0144 Open item 8). Idempotent: a re-run with nothing listed is the + * verification. No `sys_migration` flag is recorded: nothing gates on the run. + */ +export default class MigrateRecomputeHomeAmounts extends Command { + static override description = + 'Recompute the home-currency amounts that record-mode currency fields store, at the exchange rates in force ' + + 'now — after a rate is corrected or added. Reports what would change and the rows that cannot be converted ' + + '(listed and skipped); dry run by default, --apply writes.'; + + static override examples = [ + '$ os migrate recompute-home-amounts --object invoice', + '$ os migrate recompute-home-amounts --object invoice --since 2026-01-01', + '$ os migrate recompute-home-amounts --object invoice --field amount --apply', + '$ os migrate recompute-home-amounts --object invoice --apply --yes --json', + ]; + + static override flags = { + 'database-url': Flags.string({ + description: 'Database URL to migrate (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + object: Flags.string({ description: 'The object whose stored home amounts are recomputed', required: true }), + field: Flags.string({ + description: 'One record-mode amount declaring homeAmount (default: every one the object declares)', + }), + since: Flags.string({ + description: + 'Only rows whose DOCUMENT DATE (homeAmount.rateDate, the date that picks the rate) is on or after this ' + + 'date, YYYY-MM-DD. A datetime document date is read as its calendar day in the organization\'s timezone; ' + + 'a row with no document date is outside every --since window', + }), + apply: Flags.boolean({ + description: 'Write the recomputed home amounts and rates (default is a read-only dry run)', + default: false, + }), + yes: Flags.boolean({ char: 'y', description: 'Skip the --apply confirmation prompt', default: false }), + force: Flags.boolean({ + description: 'Apply even when another process is using the database (SQLite occupancy check)', + default: false, + }), + 'max-records': Flags.integer({ + description: 'Safety bound on rows read — exceeding it truncates the walk', + }), + json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to apply)' }), + }; + + async run(): Promise { + const { flags } = await this.parse(MigrateRecomputeHomeAmounts); + const timer = createTimer(); + const apply = flags.apply; + + if (!flags.json) printHeader('Migrate · recompute-home-amounts'); + + // `--since` first: before the occupancy probe, the `--apply` prompt and the boot. The check + // and its envelope are the recompute's own, so the command keeps no copy of either. + try { + const { assertHomeAmountRecomputeSince } = await import('@objectstack/objectql'); + assertHomeAmountRecomputeSince(flags.since); + } catch (error: any) { + if (flags.json) { + await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); + this.exit(1); + return; + } + printError(error.message); + this.exit(1); + return; + } + + // Occupancy gate, as `record-currency-codes`: probed before boot and before the prompt. + const occupancy = await probeMigrationTarget(flags['database-url']); + if (occupancy.status === 'busy' && apply && !flags.force) { + if (flags.json) { + await emitJson({ + error: 'database_busy', + database: occupancy.filename, + signal: occupancy.signal, + detail: occupancy.detail, + hint: OCCUPANCY_HINT, + }, 0, { compact: true }); + this.exit(1); + return; + } + printError(describeOccupancy(occupancy)); + printWarning(OCCUPANCY_HINT); + this.exit(1); + return; + } + if (occupancy.status === 'busy' && !flags.json) { + printWarning(apply + ? `--force: ${describeOccupancy(occupancy)} Recomputing anyway — the live process may write rows mid-walk.` + : `${describeOccupancy(occupancy)} The dry run below writes no row or table, but its counts may shift while that process is running. ${JOURNAL_MODE_SWITCH_NOTE}`); + } + if (occupancy.status === 'unknown' && !flags.json) { + printWarning(`Could not check whether the database is in use — ${occupancy.detail}`); + } + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning('Apply mode rewrites record data. Re-run with --yes to confirm, or run without --apply to preview.'); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold( + `\nRewrite the stored home amounts of ${flags.object}${flags.field ? `.${flags.field}` : ''} at the rates ` + + 'in force now, on this database? [y/N] ', + ), + ); + if (!ok) { + printInfo('Aborted — no changes made.'); + return; + } + } + + if (!flags.json) { + printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (dry run)…'); + } + + let stack; + try { + // The dry run boots read-only, as `record-currency-codes` does; `--apply` + // keeps the plain boot, because the tables must exist before it writes. + stack = await bootSchemaStack({ + jsonOutput: flags.json, + databaseUrl: flags['database-url'], + extraPlugins: await buildDataMigrationPlugins({ settings: true, audit: true }), + ...(apply ? {} : { deferSchemaDdl: true, readOnlyProbe: true }), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + return; + } + + try { + const engine: HomeAmountRecomputeEngine & { getConfigs?(): Record } = + stack.kernel.getService('objectql'); + if (typeof engine?.getObject !== 'function' || typeof engine?.homeAmountReadsFor !== 'function') { + throw new Error('No ObjectQL engine on this stack — cannot read the object declaration.'); + } + // A stack with no app metadata loaded would answer "object not found" + // for a real object: say what is missing instead. + const loadedObjects: string[] = + typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []; + if (!loadedObjects.some((name) => !name.startsWith('sys_'))) { + throw new Error( + 'No app objects are loaded, so the walk would examine nothing. ' + + 'Run "os build" in your project root first (the migration reads dist/objectstack.json), then re-run.', + ); + } + refuseUndeclaredObjects([flags.object], loadedObjects); + + const { recomputeHomeAmounts, formatHomeAmountRecomputeReport } = await import('@objectstack/objectql'); + + // In JSON mode keep stdout parseable — route warnings to stderr. + const logger = flags.json + ? { info: (m: string) => console.error(m), warn: (m: string) => console.error(m) } + : { info: (m: string) => printInfo(m), warn: (m: string) => printWarning(m) }; + + const report = await recomputeHomeAmounts(engine, logger, { + object: flags.object, + field: flags.field, + since: flags.since, + apply, + maxRecords: flags['max-records'], + source: 'os migrate recompute-home-amounts', + }); + // [ADR-0144 Open item 8] A listed-and-skipped row is a gap a total over + // the home field silently leaves out, so the run is not done while one + // remains: exit 1, on a dry run as on an apply run. A walk `--max-records` + // cut short has not looked at every row, so it is not done either. + const gap = report.failures.length > 0 || report.skippedTotal > 0 || report.truncated; + + if (flags.json) { + await emitJson({ database: stack.dbLabel, apply, report, duration: timer.elapsed() }); + if (gap) this.exit(1); + return; + } + + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + console.log(''); + // The report's dry-run line claims no row or table; on a SQLite file the + // read-only boot's connect may still have switched the journal mode (its + // header, no row), so the line is followed by the note naming the switch. + const journalNote = !report.applied && report.changed > 0 + && describeDriverSqliteFile(stack.driver?.config) !== undefined + ? ` ${JOURNAL_MODE_SWITCH_NOTE}` + : ''; + console.log(`${formatHomeAmountRecomputeReport(report).join('\n')}${journalNote}`); + console.log(''); + + if (report.failures.length > 0) { + printError(`${report.failures.length} row(s) could not be written — re-run to finish them.`); + } else if (report.skippedTotal > 0) { + printError( + `${report.skippedTotal} amount(s) could not be converted and were left as stored — totals over the home ` + + 'field leave them out. Fix each reason listed above, then re-run.', + ); + } else if (report.truncated) { + printError( + `The walk stopped at --max-records (${report.scannedRecords} row(s) read): rows past it were not looked at. ` + + 'Re-run with a higher --max-records to cover every row.', + ); + } else if (apply && report.written > 0) { + printSuccess(`Recomputed ${report.written} row(s). Re-run any time — it only rewrites rows that still differ.`); + } else if (!apply && report.changed > 0) { + printInfo(`Dry run only — ${report.changed} amount(s) would be recomputed. Re-run with --apply.`); + } else { + printSuccess(`Nothing to recompute — every stored home amount of ${report.object} is current.`); + } + console.log(chalk.dim(` ${timer.display()}`)); + console.log(''); + if (gap) this.exit(1); + } catch (error: any) { + if (isExitSignal(error)) throw error; + if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); } + // `INVALID_FIELD` is the run refusing the object or field before its walk + // ("Refused before the walk read or wrote any row."). The boot above ran + // first, so on a SQLite file the note naming a journal-mode switch follows. + const journalNote = error?.code === 'INVALID_FIELD' + && describeDriverSqliteFile(stack.driver?.config) !== undefined + ? ` ${JOURNAL_MODE_SWITCH_NOTE}` + : ''; + printError(`${error.message || String(error)}${journalNote}`); + this.exit(1); + } finally { + await stack.shutdown(); + } + } +} diff --git a/packages/cli/src/utils/data-migration-plugins.ts b/packages/cli/src/utils/data-migration-plugins.ts index f6ed6cc1245..ab801636f46 100644 --- a/packages/cli/src/utils/data-migration-plugins.ts +++ b/packages/cli/src/utils/data-migration-plugins.ts @@ -33,7 +33,7 @@ import { oneShotSettingsPlugin } from './one-shot-settings.js'; * swap the adapter out from under the root the operator named. */ export async function buildDataMigrationPlugins( - opts: { storage?: boolean; automation?: boolean; audit?: boolean } = {}, + opts: { storage?: boolean; automation?: boolean; audit?: boolean; settings?: boolean } = {}, ): Promise { const plugins: unknown[] = []; const { PlatformObjectsPlugin } = await import('@objectstack/platform-objects/plugin'); @@ -82,6 +82,14 @@ export async function buildDataMigrationPlugins( const { AutomationServicePlugin } = await import('@objectstack/service-automation'); plugins.push(new AutomationServicePlugin({ armRuntime: false, suspendedRunStore: 'memory' })); } + if (opts.settings === true && opts.storage !== true) { + // [ADR-0144 D4, D5] `os migrate recompute-home-amounts` projects stored + // rows as a write does, and a write reads each organization's home + // currency and timezone (`localization.currency` / `.timezone`) from the + // settings service; without it every organization reads as having no home + // currency. The one-shot composition, as the storage arm below takes it. + plugins.push(await oneShotSettingsPlugin()); + } if (opts.storage === true) { try { plugins.push(await oneShotSettingsPlugin()); diff --git a/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts b/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts index c4afc1457e0..9e82ff51aa5 100644 --- a/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts +++ b/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts @@ -76,6 +76,7 @@ import MigrateMultiValueColumns from '../commands/migrate/multi-value-columns.js import MigrateOrganizationOwnership from '../commands/migrate/organization-ownership.js'; import MigratePackagelessOverlays from '../commands/migrate/packageless-overlays.js'; import MigratePlan from '../commands/migrate/plan.js'; +import MigrateRecomputeHomeAmounts from '../commands/migrate/recompute-home-amounts.js'; import MigrateRecordCurrencyCodes from '../commands/migrate/record-currency-codes.js'; import MigrateRecordedBy from '../commands/migrate/recorded-by.js'; import MigrateResume from '../commands/migrate/resume.js'; @@ -301,6 +302,17 @@ const CALLERS: Record = { noWrite: [{ label: 'migrate recorded-by', argv: ['--database-url', '@DB@', '--json'] }], write: [{ label: 'migrate recorded-by --apply', argv: ['--apply', '--yes', '--database-url', '@DB@', '--json'] }], }, + 'commands/migrate/recompute-home-amounts.ts': { + run: invoke(MigrateRecomputeHomeAmounts), + // The fixture's app declares no home amount, so the recompute refuses + // `osf_lead` (INVALID_FIELD) after the boot and writes nothing. The boot + // is what these cases are about. + noWrite: [{ label: 'migrate recompute-home-amounts', argv: ['--object', 'osf_lead', '--database-url', '@DB@', '--json'] }], + write: [{ + label: 'migrate recompute-home-amounts --apply', + argv: ['--apply', '--yes', '--object', 'osf_lead', '--database-url', '@DB@', '--json'], + }], + }, 'commands/migrate/record-currency-codes.ts': { run: invoke(MigrateRecordCurrencyCodes), // The fixture's app declares no record-mode amount, so the backfill diff --git a/packages/cli/test/json-stdout-purity.e2e.test.ts b/packages/cli/test/json-stdout-purity.e2e.test.ts index 7ff6e6ce2c8..ce875583760 100644 --- a/packages/cli/test/json-stdout-purity.e2e.test.ts +++ b/packages/cli/test/json-stdout-purity.e2e.test.ts @@ -156,6 +156,10 @@ const FAMILY: Record = { // and BEFORE the `finally` shutdown, so the shutdown's diagnostics follow // the document and must stay off stdout. 'migrate record-currency-codes': ['--object', 'jp_ticket', '--field', 'title', '--code', 'USD'], + // [ADR-0144] Its sibling's shape: `--object` is required, the dry run is + // the default, and the face driven here is the same "No app objects are + // loaded" refusal after the read-only boot. + 'migrate recompute-home-amounts': ['--object', 'jp_ticket'], 'migrate recorded-by': [], 'migrate resume': [], // A preview is its DEFAULT and the only form driven here: without `--apply` diff --git a/packages/cli/test/migrate-confirmation-exit.pin.test.ts b/packages/cli/test/migrate-confirmation-exit.pin.test.ts index ea1c697a4b2..fa9abfd7049 100644 --- a/packages/cli/test/migrate-confirmation-exit.pin.test.ts +++ b/packages/cli/test/migrate-confirmation-exit.pin.test.ts @@ -279,6 +279,11 @@ const ROSTER: Record = { }, writes: () => [seams.resumeMigrationJournal], }, + 'recompute-home-amounts': { + argv: ['--apply', '--object', 'invoice'], + confirm: '--yes', + writes: BOOT, + }, 'record-currency-codes': { argv: ['--apply', '--object', 'opportunity', '--field', 'amount', '--code', 'USD'], confirm: '--yes', diff --git a/packages/objectql/src/engine-home-amount-projection.test.ts b/packages/objectql/src/engine-home-amount-projection.test.ts index 1e5c5bd3744..ec11a23613a 100644 --- a/packages/objectql/src/engine-home-amount-projection.test.ts +++ b/packages/objectql/src/engine-home-amount-projection.test.ts @@ -35,6 +35,13 @@ import { describe, it, expect, beforeEach } from 'vitest'; import type { EngineUpdateOptions } from '@objectstack/spec/data'; import { ObjectQL } from './engine.js'; import { applyDriverQueryInMemory } from './driver-query-in-memory.js'; +import { + recomputeHomeAmounts, + assertHomeAmountRecomputeSince, + homeAmountRecomputeComplete, + formatHomeAmountRecomputeReport, + type HomeAmountRecomputeOptions, +} from './home-amount-recompute.js'; function makeDriver() { const stores = new Map>>(); @@ -422,3 +429,275 @@ describe('validate(): the preview refuses what the write refuses', () => { expect((await t.engine.validate('fx_invoice', { id: row.id, title: 'x' }, { mode: 'update' })).valid).toBe(true); }); }); + +// ── The explicit recompute (`recomputeHomeAmounts`, `os migrate recompute-home-amounts`) ── +// +// [ADR-0144 D5, D7 parameter 1, Open item 8] A stored home amount moves only +// through this run. Pinned on the same harness, so every number it reports and +// writes is checked against what the write stores for the same row. + +/** The audit ledger's shape (`sys_audit_log`, owned by plugin-audit; a stand-in here). */ +const AUDIT_OBJECT = { + name: 'sys_audit_log', + fields: { action: { type: 'text' }, object_name: { type: 'text' }, record_id: { type: 'text' }, metadata: { type: 'textarea' } }, +}; + +const noopLogger = { info: () => {}, warn: () => {} }; +const recompute = (t: Awaited>, options: Partial = {}) => + recomputeHomeAmounts(t.engine as never, noopLogger, { object: 'fx_invoice', ...options }); +const auditRows = (t: Awaited>) => + [...t.storeFor('sys_audit_log').values()].map((r) => ({ ...r, metadata: JSON.parse(String(r.metadata)) })); + +describe('recompute: a corrected rate reaches stored rows through an explicit, reported run', () => { + let t: Awaited>; + let id: string; + const stored = () => t.storeFor('fx_invoice').get(id)!; + beforeEach(async () => { + t = await boot(); + t.engine.registry.registerObject(AUDIT_OBJECT as any); + t.rate('USD', '2026-01-01', 0.1389); + ({ id } = await t.engine.insert('fx_invoice', { title: 'i1', currency_code: 'USD', invoice_date: '2026-03-15', amount: 10000 })); + t.rate('USD', '2026-01-01', 0.125); // the correction: 10,000 ÷ 0.125 = 80,000 + }); + + it('the dry run lists the row with its stored and recomputed values, and writes nothing', async () => { + const before = { ...stored() }; + const report = await recompute(t); + expect(report).toMatchObject({ applied: false, amounts: 1, changed: 1, unchanged: 0, written: 0, skippedTotal: 0 }); + expect(report.changes).toEqual([ + { recordId: id, organizationId: null, field: 'amount', from: { home: 71994.24, rate: 0.1389 }, to: { home: 80000, rate: 0.125 } }, + ]); + expect(stored()).toEqual(before); + expect(t.storeFor('sys_audit_log').size).toBe(0); + expect(homeAmountRecomputeComplete(report)).toBe(false); + }); + + it('the apply writes only the home and rate fields, and a second run finds nothing', async () => { + const before = { ...stored() }; + const applied = await recompute(t, { apply: true }); + expect(applied).toMatchObject({ applied: true, changed: 1, written: 1, failures: [], audit: { written: true } }); + const after = stored(); + expect(after).toMatchObject({ amount_home: 80000, amount_rate: 0.125 }); + const moved = Object.keys(after).filter((k) => !Object.is(after[k], before[k])).sort(); + expect(moved).toEqual(['amount_home', 'amount_rate']); + const again = await recompute(t, { apply: true }); + expect(again).toMatchObject({ changed: 0, unchanged: 1, written: 0 }); + expect(homeAmountRecomputeComplete(again)).toBe(true); + }); + + it('a recomputed row equals what a fresh write of the same row stores', async () => { + await recompute(t, { apply: true }); + const fresh = await t.engine.insert('fx_invoice', { title: 'fresh', currency_code: 'USD', invoice_date: '2026-03-15', amount: 10000 }); + const pick = (r: Record) => [r.amount_home, r.amount_rate]; + expect(pick(stored())).toEqual(pick(t.storeFor('fx_invoice').get(fresh.id)!)); + }); + + it('one audit entry per applied run, carrying its counts per organization; none for a dry run', async () => { + await recompute(t); + await recompute(t, { apply: true, source: 'test' }); + const rows = auditRows(t); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ action: 'update', object_name: 'fx_invoice' }); + expect(rows[0]!.metadata).toMatchObject({ + event: 'home_amount.recompute', source: 'test', object: 'fx_invoice', fields: ['amount'], since: null, + changed: 1, written: 1, skipped: [], + organizations: [{ organizationId: null, homeCurrency: 'CNY', amounts: 1, changed: 1, written: 1 }], + }); + }); + + it('without the audit ledger in the composition the run says so instead of failing', async () => { + const bare = await boot(); + bare.rate('CNY', '2026-01-01', 1); + await bare.engine.insert('fx_invoice', { title: 'i', currency_code: 'CNY', amount: 5 }); + bare.storeFor('fx_invoice').forEach((r) => { r.amount_home = 4; }); + const report = await recompute(bare, { apply: true }); + expect(report).toMatchObject({ written: 1, audit: { written: false, reason: 'ledger_not_mounted' } }); + }); + + it('a run an unclassified error stops partway still writes its run entry, marked aborted, then throws', async () => { + t.rate('EUR', '2026-01-01', 0.128); + const second = await t.engine.insert('fx_invoice', { title: 'i2', currency_code: 'EUR', invoice_date: '2026-03-15', amount: 128 }); + t.rate('EUR', '2026-01-01', 0.125); // both rows now differ from what a write stores + // The EUR rate read fails outright: no class names it, so it must end the walk. + const find = t.driver.find; + t.driver.find = async (object: string, ast: any) => { + if (object === 'sys_exchange_rate' && JSON.stringify(ast).includes('EUR')) throw new Error('rate store unreachable'); + return find.call(t.driver, object, ast); + }; + const thrown = await refusalOf(recompute(t, { apply: true, source: 'test' })); + expect(thrown?.message).toBe('rate store unreachable'); + // The row before the failure landed, the one at it did not… + expect(stored()).toMatchObject({ amount_home: 80000, amount_rate: 0.125 }); + expect(t.storeFor('fx_invoice').get(second.id)).toMatchObject({ amount_home: 1000, amount_rate: 0.128 }); + // …and the ledger says so: one run entry, aborted, with the counts it reached. + const rows = auditRows(t); + expect(rows).toHaveLength(1); + expect(rows[0]!.metadata).toMatchObject({ + event: 'home_amount.recompute', source: 'test', written: 1, aborted: 'rate store unreachable', + organizations: [{ organizationId: null, written: 1 }], + }); + }); + + it('a dry run an unclassified error stops writes nothing, the ledger included', async () => { + const find = t.driver.find; + t.driver.find = async (object: string, ast: any) => { + if (object === 'sys_exchange_rate') throw new Error('rate store unreachable'); + return find.call(t.driver, object, ast); + }; + expect((await refusalOf(recompute(t)))?.message).toBe('rate store unreachable'); + expect(t.storeFor('sys_audit_log').size).toBe(0); + }); +}); + +describe('recompute: rows that cannot be converted are listed, skipped and never written (parameter 1, Open item 8)', () => { + let t: Awaited>; + beforeEach(async () => { + t = await boot(); + t.engine.registry.registerObject(AUDIT_OBJECT as any); + t.rate('USD', '2026-01-01', 0.1389); + }); + + it('a missing rate: listed with its currency and date, skipped, nothing written, the run stays incomplete', async () => { + const { id } = await t.engine.insert('fx_invoice', { title: 'i1', currency_code: 'USD', invoice_date: '2026-03-15', amount: 10000 }); + const ok = await t.engine.insert('fx_invoice', { title: 'i2', currency_code: 'CNY', amount: 3 }); + t.storeFor('fx_invoice').get(ok.id)!.amount_home = 2; // stale, recomputable + t.storeFor('sys_exchange_rate').clear(); // the USD rate row is gone + const before = { ...t.storeFor('fx_invoice').get(id)! }; + const report = await recompute(t, { apply: true }); + expect(report.skipped).toEqual([{ recordId: id, organizationId: null, field: 'amount', reason: 'rate_missing', currency: 'USD', date: '2026-03-15' }]); + expect(report).toMatchObject({ skippedTotal: 1, changed: 1, written: 1 }); + expect(report.skippedByReason).toMatchObject({ rate_missing: 1, code_blank: 0 }); + expect(t.storeFor('fx_invoice').get(id)).toEqual(before); + expect(t.storeFor('fx_invoice').get(ok.id)).toMatchObject({ amount_home: 3, amount_rate: 1 }); + // The gap outlives the report: the run stays incomplete and its audit entry names the row. + expect(homeAmountRecomputeComplete(report)).toBe(false); + expect(auditRows(t)[0]!.metadata.skipped).toEqual(report.skipped); + expect(auditRows(t)[0]!.metadata.skippedByReason).toMatchObject({ rate_missing: 1 }); + expect(formatHomeAmountRecomputeReport(report).join('\n')).toContain('no USD rate on or before 2026-03-15'); + }); + + it('a home field added over stored rows: rows without a rate stay blank, never 0, the rest are filled', async () => { + // Rows stored before the declaration: an amount, its code, a document date, no home amount. + const legacy = (id: string, code: string, date: string) => + t.storeFor('fx_invoice').set(id, { id, title: id, currency_code: code, invoice_date: date, amount: 1389, amount_home: null, amount_rate: null }); + legacy('old_usd', 'USD', '2026-03-15'); + legacy('old_jpy', 'JPY', '2026-03-15'); + const report = await recompute(t, { apply: true }); + expect(report).toMatchObject({ changed: 1, written: 1, skippedTotal: 1 }); + expect(t.storeFor('fx_invoice').get('old_usd')).toMatchObject({ amount_home: 10000, amount_rate: 0.1389 }); + expect(t.storeFor('fx_invoice').get('old_jpy')).toMatchObject({ amount_home: null, amount_rate: null }); + }); + + it('a blank code beside an amount is its own class, apart from a missing rate, with its own remedy', async () => { + t.storeFor('fx_invoice').set('codeless', { id: 'codeless', title: 'c', invoice_date: '2026-03-15', amount: 50, amount_home: null, amount_rate: null }); + const report = await recompute(t, { apply: true }); + expect(report.skipped).toEqual([{ recordId: 'codeless', organizationId: null, field: 'amount', reason: 'code_blank' }]); + expect(report.skippedByReason).toMatchObject({ code_blank: 1, rate_missing: 0 }); + expect(t.storeFor('fx_invoice').get('codeless')).toMatchObject({ amount_home: null, amount_rate: null }); + expect(formatHomeAmountRecomputeReport(report).join('\n')).toContain('os migrate record-currency-codes --object fx_invoice'); + }); + + it('a blank document date and an unset home currency are their own classes', async () => { + t.storeFor('fx_invoice').set('undated', { id: 'undated', title: 'u', currency_code: 'USD', amount: 50 }); + expect((await recompute(t)).skipped.map((s) => s.reason)).toEqual(['document_date_blank']); + const unset = await boot(invoice(), null); + unset.storeFor('fx_invoice').set('x', { id: 'x', title: 'x', currency_code: 'USD', invoice_date: '2026-03-15', amount: 50 }); + const report = await recompute(unset); + expect(report.skipped.map((s) => s.reason)).toEqual(['home_currency_unset']); + expect(report.organizations).toEqual([expect.objectContaining({ homeCurrency: null, amounts: 1 })]); + }); +}); + +describe('recompute: the write\'s own rules, per organization', () => { + it('the same currency as home is rate 1 and reads no rate', async () => { + const t = await boot(); + const { id } = await t.engine.insert('fx_invoice', { title: 'i', currency_code: 'CNY', amount: 1.005 }); + t.storeFor('fx_invoice').get(id)!.amount_home = 1; // a stale value + const before = t.finds.sys_exchange_rate ?? 0; + const report = await recompute(t, { apply: true }); + expect(report.changes[0]!.to).toEqual({ home: 1.01, rate: 1 }); + expect(t.storeFor('fx_invoice').get(id)).toMatchObject({ amount_home: 1.01, amount_rate: 1 }); + expect(t.finds.sys_exchange_rate ?? 0).toBe(before); + }); + + it('a datetime document date is dated in the organization\'s timezone, for the rate and for since', async () => { + const t = await boot(invoice({ dateType: 'datetime' })); + let timezone: string | undefined; + t.engine.setHomeLocalizationResolver(async () => ({ currency: 'CNY', timezone })); + t.rate('USD', '2026-01-01', 0.1389); + t.rate('USD', '2026-04-01', 0.15); + // 2026-03-31T17:00Z is 01:00 on 2026-04-01 in Shanghai. + const { id } = await t.engine.insert('fx_invoice', { title: 'o', currency_code: 'USD', invoice_date: '2026-03-31T17:00:00Z', amount: 1500 }); + expect(t.storeFor('fx_invoice').get(id)).toMatchObject({ amount_rate: 0.1389 }); + expect(await recompute(t, { since: '2026-04-01' })).toMatchObject({ amounts: 0, outsideSince: 1 }); + timezone = 'Asia/Shanghai'; + const report = await recompute(t, { since: '2026-04-01', apply: true }); + expect(report).toMatchObject({ amounts: 1, outsideSince: 0, written: 1 }); + expect(t.storeFor('fx_invoice').get(id)).toMatchObject({ amount_home: 10000, amount_rate: 0.15 }); + }); + + it('since filters on the document date; a row with none is outside every window', async () => { + const t = await boot(); + t.rate('USD', '2026-01-01', 0.1389); + await t.engine.insert('fx_invoice', { title: 'march', currency_code: 'USD', invoice_date: '2026-03-15', amount: 1 }); + await t.engine.insert('fx_invoice', { title: 'april', currency_code: 'USD', invoice_date: '2026-04-15', amount: 1 }); + await t.engine.insert('fx_invoice', { title: 'undated', currency_code: 'CNY', amount: 1 }); + t.rate('USD', '2026-01-01', 0.5); + const report = await recompute(t, { since: '2026-04-01' }); + expect(report).toMatchObject({ since: '2026-04-01', amounts: 1, outsideSince: 2, changed: 1 }); + const april = [...t.storeFor('fx_invoice').values()].find((r) => r.title === 'april')!; + expect(report.changes.map((c) => c.recordId)).toEqual([april.id]); + expect((await recompute(t)).changed).toBe(2); + }); + + it('two organizations convert into their own home currencies and are counted apart', async () => { + const t = await boot(); + const homes: Record = { org_a: 'CNY', org_b: 'JPY' }; + t.engine.setHomeLocalizationResolver(async (org) => ({ currency: org ? homes[org] : undefined })); + t.rate('USD', '2026-01-01', 0.1389); // org_a's rate for USD (home CNY) + t.rate('CNY', '2026-01-01', 0.05); // org_b's rate for CNY (home JPY) + const as = (org: string) => ({ context: { isSystem: true, tenantId: org } }) as any; + const a = await t.engine.insert('fx_invoice', { title: 'a', currency_code: 'USD', invoice_date: '2026-03-15', amount: 1389, organization_id: 'org_a' }, as('org_a')); + const b = await t.engine.insert('fx_invoice', { title: 'b', currency_code: 'CNY', invoice_date: '2026-03-15', amount: 721, organization_id: 'org_b' }, as('org_b')); + expect(t.storeFor('fx_invoice').get(b.id)).toMatchObject({ amount_home: 14420, amount_rate: 0.05 }); + t.rate('CNY', '2026-01-01', 0.048); // a correction that only org_b's row reads + const report = await recompute(t, { apply: true }); + expect(report.organizations).toEqual([ + expect.objectContaining({ organizationId: 'org_a', homeCurrency: 'CNY', amounts: 1, unchanged: 1, changed: 0 }), + expect.objectContaining({ organizationId: 'org_b', homeCurrency: 'JPY', amounts: 1, unchanged: 0, changed: 1, written: 1 }), + ]); + expect(t.storeFor('fx_invoice').get(a.id)).toMatchObject({ amount_home: 10000, amount_rate: 0.1389 }); + // 721 ÷ 0.048 = 15020.83…, whole yen: JPY has no fraction digits — in the report and in the row. + expect(report.changes.map((c) => c.to)).toEqual([{ home: 15021, rate: 0.048 }]); + expect(t.storeFor('fx_invoice').get(b.id)).toMatchObject({ amount_home: 15021, amount_rate: 0.048 }); + }); +}); + +describe('recompute: refused before any row is read', () => { + it('an unknown object, an object with no home amount, a field without one, and a malformed since', async () => { + const t = await boot(); + t.engine.registry.registerObject({ name: 'plain', fields: { title: { type: 'text' } } } as any); + const before = t.finds.fx_invoice ?? 0; + const codes = await Promise.all([ + refusalOf(recompute(t, { object: 'nope' })), + refusalOf(recompute(t, { object: 'plain' })), + refusalOf(recompute(t, { field: 'title' })), + refusalOf(recompute(t, { since: '2026-02-30' })), + refusalOf(recompute(t, { since: '15/03/2026' })), + ]); + expect(codes.map((e) => [e?.code, e?.status])).toEqual([ + ['OBJECT_NOT_FOUND', 404], ['INVALID_FIELD', 400], ['INVALID_FIELD', 400], ['VALIDATION_FAILED', 400], ['VALIDATION_FAILED', 400], + ]); + expect(codes[2]?.message).toContain("'amount'"); + expect(t.finds.fx_invoice ?? 0).toBe(before); + }); + it('the since check is one pure function the doors call before they open a database, with the run\'s own envelope', async () => { + expect(() => assertHomeAmountRecomputeSince(undefined)).not.toThrow(); + expect(() => assertHomeAmountRecomputeSince('2026-01-01')).not.toThrow(); + const pure = (() => { try { assertHomeAmountRecomputeSince('2026-02-30'); return null; } catch (e) { return e as Thrown; } })(); + expect([pure?.code, pure?.status, (pure as { httpStatus?: number } | null)?.httpStatus]).toEqual(['VALIDATION_FAILED', 400, 400]); + const t = await boot(); + const run = await refusalOf(recompute(t, { since: '2026-02-30' })); + expect(run?.message).toBe(pure?.message); + }); +}); diff --git a/packages/objectql/src/engine-references-and-master-detail.ts b/packages/objectql/src/engine-references-and-master-detail.ts index 3d4876065ba..77913761e38 100644 --- a/packages/objectql/src/engine-references-and-master-detail.ts +++ b/packages/objectql/src/engine-references-and-master-detail.ts @@ -42,7 +42,7 @@ import type { IRealtimeService } from '@objectstack/spec/contracts'; import type { ExecutionContext } from '@objectstack/spec/kernel'; import { homeAmountsTouched, planHomeAmounts, projectHomeAmounts } from './home-amount-projection.js'; -import type { HomeAmountPlanEntry, HomeAmountServing, HomeLocalizationResolver } from './home-amount-projection.js'; +import type { HomeAmountPlanEntry, HomeAmountReads, HomeAmountServing, HomeLocalizationResolver } from './home-amount-projection.js'; import { materializeDeclaredFields } from './declared-fields.js'; import { collectCascadeDeleteObjects } from './engine-delete-cascade.js'; import { ObjectQLStatics as ObjectQL } from './engine.js'; @@ -109,6 +109,14 @@ export class ReferencesAndMasterDetailFacet { return derived ?? undefined; } + /** The body of `ObjectQL#homeAmountReadsFor`; the method's docblock there is its contract. */ + static homeAmountReadsFor(this: Pick, context: ExecutionContext | undefined): HomeAmountReads { + return { + serving: this.homeAmountServing(context), + organizationOf: (row) => ObjectQL.homeAmountOrganizationOf(row, context), + }; + } + /** * [ADR-0144 D5] The `validate()` preview's run of the home projection: the * write's own computation over the preview's image (the insert row, or the diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index a08721baf39..e7f68d2a7da 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -263,6 +263,7 @@ import { projectHomeAmounts, settleBulkHomeAmounts, type HomeAmountPlanEntry, + type HomeAmountReads, type HomeAmountServing, type HomeLocalization, type HomeLocalizationResolver, @@ -3539,6 +3540,21 @@ export class ObjectQL implements IObjectQLEngine { private homeLocalizationResolver?: HomeLocalizationResolver; + /** + * [ADR-0144 D5, D7 parameter 1] What the write-time home projection reads, + * for a caller that projects a STORED row as a write would: the serving a + * write under `context` reads each organization's home currency and + * timezone and the rate rows through (memoized for the serving's life), and + * the organization a write converts a row under. The explicit recompute + * (`recomputeHomeAmounts`, `os migrate recompute-home-amounts`) projects + * stored rows with these and `projectHomeAmounts`, so its preview is the + * write's own computation, never a copy of it. + */ + homeAmountReadsFor(context?: ExecutionContext): HomeAmountReads { + // [ADR-0144] The body lives in `engine-references-and-master-detail.ts` (see `ReferencesAndMasterDetailHost`). + return ReferencesAndMasterDetailFacet.homeAmountReadsFor.call(referencesAndMasterDetailHost.of(this), context); + } + /** * [ADR-0144 D5, Open item 2] What one write's home projection reads, memoized * for that write: each organization's home currency once, and each diff --git a/packages/objectql/src/federated-injected-column-readers.test.ts b/packages/objectql/src/federated-injected-column-readers.test.ts index ea7ee72fba8..337d869d99b 100644 --- a/packages/objectql/src/federated-injected-column-readers.test.ts +++ b/packages/objectql/src/federated-injected-column-readers.test.ts @@ -365,6 +365,18 @@ const READERS: Record = { 'engine-compatibility-api.ts#syncObjectSchema :: isFederatedObject()': { disposition: 'not-a-read', why: SYNC_ROUTE }, 'engine-compatibility-api.ts#syncSchemas :: isFederatedObject()': { disposition: 'not-a-read', why: SYNC_ROUTE }, 'engine-secret-internal-fields.ts#encryptSecretFields :: created_at': { disposition: 'not-a-read', why: PLATFORM_ROW }, + 'home-amount-recompute.ts#recomputeHomeAmounts :: isFederatedUnprovisionedInjectedColumn()': { + disposition: 'skips', + why: 'the recompute never selects an injected column a federated object does not provision', + }, + 'home-amount-recompute.ts#recomputeHomeAmounts :: organization_id': { + disposition: 'skips', + why: 'the recompute reads the organization a write converts a row under, and leaves the column out of its read on a federated object', + }, + 'home-amount-recompute.ts#recomputeHomeAmounts :: updated_at': { + disposition: 'skips', + why: 'the recompute reads the stamp its apply keeps, and leaves the column out of its read (and its write) on a federated object', + }, 'engine-migration-state.ts#recordObservedDeviation :: updated_at': { disposition: 'not-a-read', why: PLATFORM_ROW }, 'engine-migration-state.ts#retractCreationAttestation :: updated_at': { disposition: 'not-a-read', why: PLATFORM_ROW }, 'plugin.ts#reconcileFederatedBindings :: isFederatedObject()': { disposition: 'not-a-read', why: SYNC_ROUTE }, @@ -691,7 +703,7 @@ describe('[#21918] every engine reader of an injected column has a disposition t } }, SCAN_BUDGET_MS); - it('the sites that skip are exactly the cascade scan, its plan, the organization-delete count, and the lifecycle tenant partition', () => { + it('the sites that skip are exactly the cascade scan, its plan, the organization-delete count, the lifecycle tenant partition, and the home-amount recompute', () => { const skipping = new Set( Object.entries(READERS) .filter(([, row]) => row.disposition === 'skips') @@ -701,6 +713,7 @@ describe('[#21918] every engine reader of an injected column has a disposition t 'engine-cascade-planning.ts#cascadeRelationBehavior', 'engine-cascade-planning.ts#organizationOwnedRowCounts', 'engine-cascade-planning.ts#planCascadeAtomicity', + 'home-amount-recompute.ts#recomputeHomeAmounts', 'lifecycle/lifecycle-service.ts#tenantWindowsFor', ]); }); diff --git a/packages/objectql/src/home-amount-projection.ts b/packages/objectql/src/home-amount-projection.ts index 61216f0c62f..8eb82c5ee2d 100644 --- a/packages/objectql/src/home-amount-projection.ts +++ b/packages/objectql/src/home-amount-projection.ts @@ -301,6 +301,17 @@ export interface HomeLocalization { /** Reads an organization's {@link HomeLocalization}; the engine's host bridge to the settings service. */ export type HomeLocalizationResolver = (organizationId: string | undefined) => Promise; +/** + * What the write-time projection reads, handed to a caller that projects a + * STORED row as a write would (the explicit recompute): the serving a write + * reads its home currency, timezone and rate rows through, and the + * organization a write converts a given row under. + */ +export interface HomeAmountReads { + readonly serving: HomeAmountServing; + organizationOf(row: Record): string | undefined; +} + /** Wrap a serving so each (organization, currency, date) and each organization's home currency and timezone are read once. */ export function memoizeHomeAmountServing(serving: HomeAmountServing): HomeAmountServing { const homes = new Map>(); diff --git a/packages/objectql/src/home-amount-recompute.ts b/packages/objectql/src/home-amount-recompute.ts new file mode 100644 index 00000000000..51eeff665ba --- /dev/null +++ b/packages/objectql/src/home-amount-recompute.ts @@ -0,0 +1,725 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [ADR-0144 D5, D7 parameter 1, Open item 8] The explicit recompute of stored + * home amounts — `os migrate recompute-home-amounts`. + * + * ## Why it exists + * + * A write stores a `record`-mode amount converted into the organization's home + * currency, with the rate it used (`home-amount-projection.ts`). A corrected + * rate row does not, on its own, rewrite what was stored: D5 says it reaches + * stored rows only through "an explicit recompute operation". This is that + * operation. It also fills a home field added to an object that already holds + * rows, whose stored rows were never converted. + * + * ## The write's own computation, never a copy + * + * Every number here comes from the code path a write runs: + * + * - The preview projects each stored row with {@link projectHomeAmounts} over + * the engine's own reads ({@link HomeAmountRecomputeEngine.homeAmountReadsFor}): + * the same rate-row rule (the latest `effective_date` on or before the + * document date, in the row's organization), the same exact division and + * single rounding, the same same-currency rule, and the same calendar day of + * a `datetime` document date in the organization's `localization.timezone`. + * - The apply writes through `engine.update` with the home field in the + * payload. A payload that touches the home field makes the write project + * the row again, over the stored row, and the value the write computes is + * the one stored — so a recomputed row holds exactly what a fresh write of + * the same row would store. + * + * Each organization is converted into its OWN home currency at its OWN rates, + * because the organization is the row's (`organization_id`), and the report + * counts per organization. + * + * ## Which rows it lists and skips (parameter 1) + * + * Parameter 1 reports and skips a row whose historical rate is missing, and + * applies the rest. A skipped row is never written: its home amount and rate + * stay as stored, and no fallback (a rate of 1, a later rate, a zero) is put in + * their place. Five reasons, each its own class with its own remedy: + * + * - `rate_missing` — no rate row for the currency is in force on the document + * date. Remedy: add the rate, re-run. + * - `code_blank` — the amount holds a value beside a blank currency code (rows + * stored before the amount was switched to `record` mode; ruling 6492000994, + * letter B). There is no currency to convert from, so no rate can help. + * Remedy: `os migrate record-currency-codes`, then re-run. That command + * writes its one code into every organization's codeless rows (Open item + * 13), so the report's per-organization `code_blank` count is the check + * before it is given one. + * - `document_date_blank` — a foreign amount with no document date. + * - `home_currency_unset` — the organization has no `localization.currency`. + * - `unconvertible` — the code is not ISO 4217, or the amount or document + * date does not parse; the write's own validator refuses such a row. + * + * ## Open item 8: how a skipped row's gap stays visible + * + * `sum` over a home field silently ignores an empty home amount, so a skipped + * row would make a consolidated total low with no sign of it once the report + * scrolled away. This run keeps the gap visible three ways: + * + * 1. **The run is not complete while any row is skipped** (or the walk was + * truncated). {@link homeAmountRecomputeComplete} answers false, and the + * command exits non-zero, on a dry run as on an apply run, until a re-run + * skips nothing. + * 2. **The applied run's audit entry carries the skipped rows**: their count + * per organization and reason, and each skipped row (record, field, + * reason, currency and date), so the gap outlives the terminal. + * 3. **A skipped row is never written**, so the run's report is reproducible: + * a dry run lists the same rows again until they are fixed, and the + * stored home amount of a never-converted row stays blank, never 0. + * + * ## `--since` + * + * `since` filters on the DOCUMENT DATE (`homeAmount.rateDate`), the date that + * picks the rate, read as the calendar day the write reads it as (a `datetime` + * in the organization's timezone): a row is in the run when that day is on or + * after `since`. A row with no document date is outside every `since` window. + * + * ## Writes + * + * Dry run unless `apply`. An apply writes only the rows whose stored home + * amount or rate differs from the projection, and its payload carries only the + * home field (the write sets the rate beside it): under `preserveAudit` with + * the row's stored `updated_at`, as the other `os migrate` data runs that + * rewrite a column do. Because it IS a write, what reads the home field moves + * with it exactly as for a fresh write: stored formulas, parent summaries and, + * where the audit writers are installed, one per-row `sys_audit_log` `update` + * row for each row rewritten. ⚠️ `updated_at` does not move, so a consumer that + * syncs incrementally by `updated_at` does not see a recomputed row + * (`os migrate record-currency-codes` does move it). + * + * One run-level `sys_audit_log` entry records each applied run, beside those + * per-row rows. A run an unclassified error ends (a page read or a rate read + * that fails) still writes it, marked `aborted` with the counts it reached, + * before the error is thrown on. A walk cut short by `maxRecords` is + * `truncated`, and a truncated run is not complete. Idempotent: a second run + * finds nothing to change. + */ + +import { keysetWalk, type KeysetPageQuery } from '@objectstack/types'; +import { objectNotFoundError, withTransientRetry, type RetryOptions } from '@objectstack/core'; + +import { + documentDateOf, + isHomeAmountRefusal, + planHomeAmounts, + projectHomeAmounts, + toScaledDecimal, + EXCHANGE_RATE_MISSING_CODE, + type HomeAmountPlanEntry, + type HomeAmountReads, + type HomeAmountServing, +} from './home-amount-projection.js'; +import { isRecordCurrencyBlank } from './validation/record-currency-code.js'; +import { isFederatedUnprovisionedInjectedColumn } from './federated-object.js'; +import { VALIDATION_FAILED_CODE } from './validation/record-validator.js'; + +/** Why a stored amount was listed and not written. */ +export type HomeAmountSkipReason = + | 'rate_missing' + | 'code_blank' + | 'document_date_blank' + | 'home_currency_unset' + | 'unconvertible'; + +/** Every skip reason, in report order. */ +export const HOME_AMOUNT_SKIP_REASONS: readonly HomeAmountSkipReason[] = [ + 'rate_missing', + 'code_blank', + 'document_date_blank', + 'home_currency_unset', + 'unconvertible', +]; + +/** One stored amount the run listed and did not write. */ +export interface HomeAmountRecomputeSkip { + recordId: string; + organizationId: string | null; + /** The `record`-mode amount field. */ + field: string; + reason: HomeAmountSkipReason; + /** The amount's currency, where it has one. */ + currency?: string; + /** The document date read (`YYYY-MM-DD`), for `rate_missing`. */ + date?: string; +} + +/** The stored and the recomputed values of one amount's two projected fields. */ +export interface HomeAmountRecomputeChange { + recordId: string; + organizationId: string | null; + field: string; + from: { home: unknown; rate: unknown }; + to: { home: unknown; rate: unknown }; +} + +/** The counts of one organization. Each count is of amounts (a row times a declaration). */ +export interface HomeAmountRecomputeOrganization { + organizationId: string | null; + /** The organization's home currency as the write reads it, or null when unset. */ + homeCurrency: string | null; + /** Amounts examined: in the `since` window. */ + amounts: number; + /** Stored home amount and rate already equal the projection. */ + unchanged: number; + /** Stored home amount or rate differs from the projection. */ + changed: number; + /** Rows written (an apply run). */ + written: number; + skipped: Record; + /** Amounts left out because their document date is before `since`, or blank. */ + outsideSince: number; +} + +export interface HomeAmountRecomputeFailure { + recordId: string; + error: string; +} + +/** Where the applied run's audit entry went. */ +export type HomeAmountRecomputeAudit = + | { written: true } + | { written: false; reason: 'dry_run' | 'ledger_not_mounted' | 'refused'; error?: string }; + +export interface HomeAmountRecomputeReport { + object: string; + /** The declarations recomputed: the amount, its code, the home and rate fields and the document date. */ + fields: Array<{ field: string; codeField: string; homeField: string; rateField: string; rateDateField: string }>; + /** The document-date floor, `YYYY-MM-DD`, or null for every row. */ + since: string | null; + applied: boolean; + scannedRecords: number; + amounts: number; + unchanged: number; + changed: number; + written: number; + outsideSince: number; + skippedTotal: number; + skippedByReason: Record; + /** One entry per organization the walk met, ordered by id (null last). */ + organizations: HomeAmountRecomputeOrganization[]; + /** The amounts that change, up to `maxListed`. */ + changes: HomeAmountRecomputeChange[]; + changesTruncated: boolean; + /** The amounts listed and skipped, up to `maxListed`. */ + skipped: HomeAmountRecomputeSkip[]; + skippedTruncated: boolean; + /** Rows whose write failed after retries; the run continues past them. */ + failures: HomeAmountRecomputeFailure[]; + /** The `maxRecords` cap cut the walk short. */ + truncated: boolean; + audit: HomeAmountRecomputeAudit; +} + +/** + * The engine surface the recompute needs: the declaration, a read, the update + * every write goes through, an insert for the audit entry, and the serving the + * write-time projection reads through. + */ +export interface HomeAmountRecomputeEngine { + getObject(name: string): { fields?: Record } | undefined; + find(object: string, options: Record): Promise>>; + update(object: string, data: Record, options?: any): Promise; + insert(object: string, data: Record, options?: any): Promise; + homeAmountReadsFor(context?: any): HomeAmountReads; +} + +export interface HomeAmountRecomputeLogger { + info(msg: string, meta?: unknown): void; + warn(msg: string, meta?: unknown): void; +} + +export interface HomeAmountRecomputeOptions { + /** The object whose stored rows are recomputed. */ + object: string; + /** One `record`-mode amount declaring `homeAmount`; omitted, every declaration of the object. */ + field?: string; + /** Only rows whose document date is on or after this calendar date (`YYYY-MM-DD`). */ + since?: string; + /** Write the changes. Omit for a read-only dry run. */ + apply?: boolean; + /** Safety bound on rows read. */ + maxRecords?: number; + /** How many changes and how many skipped amounts the report lists (each); the counts are always whole. */ + maxListed?: number; + /** Names the caller in the audit entry (`os migrate recompute-home-amounts`). */ + source?: string; + /** Retry policy for one row's update; defaults to the transient defaults. */ + retry?: RetryOptions; +} + +const SCAN_PAGE_SIZE = 500; +const DEFAULT_MAX_LISTED = 100; +/** How many skipped rows and failures the audit entry carries, each. */ +const AUDIT_MAX_LISTED = 1000; +const AUDIT_OBJECT = 'sys_audit_log'; +/** Unscoped by design, the context every `os migrate` data run reads with: every organization's rows. */ +const SYSTEM_CTX = { isSystem: true } as const; +/** The write keeps the row's own `updated_at`: the run changes only the two projected fields. */ +const SYSTEM_PRESERVING_CTX = { isSystem: true, preserveAudit: true } as const; +const CALENDAR_DATE = /^\d{4}-\d{2}-\d{2}$/; + +type RefusalError = Error & { code: string; status: number; httpStatus: number; field?: string }; + +function refusal(code: string, message: string, field?: string): RefusalError { + const err = new Error(message) as RefusalError; + err.code = code; + err.status = 400; + // …and `httpStatus`, the same number under ADR-0112 D5's spelling (the CLI `--json` envelope reads it). + err.httpStatus = 400; + if (field !== undefined) err.field = field; + return err; +} + +const zeroReasons = (): Record => ({ + rate_missing: 0, + code_blank: 0, + document_date_blank: 0, + home_currency_unset: 0, + unconvertible: 0, +}); + +/** Is `text` a real calendar date spelled `YYYY-MM-DD`? */ +function isCalendarDate(text: string): boolean { + if (!CALENDAR_DATE.test(text)) return false; + const ms = Date.parse(`${text}T00:00:00Z`); + return !Number.isNaN(ms) && new Date(ms).toISOString().slice(0, 10) === text; +} + +/** + * The `since` refusal on its own: a value that is not a real `YYYY-MM-DD` + * date throws `VALIDATION_FAILED` / 400. Pure — it reads no engine and no + * row — so a door can judge `since` before it opens a database (the CLI does, + * before its occupancy probe, its `--apply` prompt and its boot), and + * {@link recomputeHomeAmounts} judges it again for every other caller. One + * check, one envelope, owned here. + */ +export function assertHomeAmountRecomputeSince(since: string | undefined): void { + if (since === undefined || isCalendarDate(since)) return; + throw refusal( + VALIDATION_FAILED_CODE, + `[home-amount-recompute] since '${since}' is not a calendar date: give it as YYYY-MM-DD (for example 2026-01-01). ` + + 'It is compared with each row\'s document date. Refused before any row was read or written.', + ); +} + +/** + * The declarations to recompute, judged before any row is read: an unknown + * object, an object declaring no home amount, a `field` that declares none, + * and a malformed `since` are refused as a whole, on a dry run as on an apply. + */ +function resolvePlan(engine: HomeAmountRecomputeEngine, options: HomeAmountRecomputeOptions): HomeAmountPlanEntry[] { + const { object, field, since } = options; + const schema = engine.getObject(object); + if (!schema) throw objectNotFoundError(object); + assertHomeAmountRecomputeSince(since); + const plan = planHomeAmounts(schema); + if (plan.length === 0) { + throw refusal( + 'INVALID_FIELD', + `[home-amount-recompute] '${object}' declares no home amount: none of its currency fields declares ` + + `currencyConfig: { currencyMode: 'record', homeAmount: { field, rateDate, rateField } }, so it stores no ` + + 'converted amount to recompute. Refused before the walk read or wrote any row.', + ); + } + if (field === undefined) return plan; + const entry = plan.find((e) => e.amountField === field); + if (!entry) { + throw refusal( + 'INVALID_FIELD', + `[home-amount-recompute] '${object}.${field}' declares no home amount. The amounts of '${object}' that do: ` + + `${plan.map((e) => `'${e.amountField}'`).join(', ')}. Refused before the walk read or wrote any row.`, + field, + ); + } + return [entry]; +} + +/** Do two stored numbers spell the same decimal (`71994.24`, `'71994.2400'`)? Both blank is the same. */ +function sameStoredNumber(a: unknown, b: unknown): boolean { + const blankA = a === undefined || a === null || a === ''; + const blankB = b === undefined || b === null || b === ''; + if (blankA || blankB) return blankA && blankB; + const x = toScaledDecimal(a); + const y = toScaledDecimal(b); + if (x === undefined || y === undefined) return Object.is(a, b); + const scale = Math.max(x.scale, y.scale); + return x.digits * 10n ** BigInt(scale - x.scale) === y.digits * 10n ** BigInt(scale - y.scale); +} + +/** + * Recompute the stored home amounts of `options.object`. + * + * Dry run unless `options.apply` — a dry run writes nothing at all, so the + * report can be reviewed before any row changes. Refusals before the walk: + * an unknown object (`OBJECT_NOT_FOUND`), no home amount to recompute + * (`INVALID_FIELD`), and a `since` that is not `YYYY-MM-DD` + * (`VALIDATION_FAILED`). + */ +export async function recomputeHomeAmounts( + engine: HomeAmountRecomputeEngine, + logger: HomeAmountRecomputeLogger, + options: HomeAmountRecomputeOptions, +): Promise { + const apply = options.apply === true; + const { object } = options; + const plan = resolvePlan(engine, options); + const since = options.since ?? null; + const maxListed = options.maxListed ?? DEFAULT_MAX_LISTED; + const schema = engine.getObject(object); + const declared = schema?.fields ?? {}; + + const report: HomeAmountRecomputeReport = { + object, + fields: plan.map((e) => ({ + field: e.amountField, + codeField: e.codeField, + homeField: e.homeField, + rateField: e.rateField, + rateDateField: e.rateDateField, + })), + since, + applied: apply, + scannedRecords: 0, + amounts: 0, + unchanged: 0, + changed: 0, + written: 0, + outsideSince: 0, + skippedTotal: 0, + skippedByReason: zeroReasons(), + organizations: [], + changes: [], + changesTruncated: false, + skipped: [], + skippedTruncated: false, + failures: [], + truncated: false, + audit: { written: false, reason: 'dry_run' }, + }; + // Every skipped amount, for the audit entry (capped there), kept apart from the report's own cap. + const auditSkips: HomeAmountRecomputeSkip[] = []; + const organizations = new Map(); + + const columns = new Set(['id']); + // The organization a write converts the row under, and the stamp the apply keeps. A federated + // object's injected columns are not provisioned, so neither is read there: its rows convert + // under the caller's organization, as its writes do, and the apply sends no stamp. + for (const name of ['organization_id', 'updated_at']) { + if (name in declared && !isFederatedUnprovisionedInjectedColumn(schema, name)) columns.add(name); + } + for (const e of plan) for (const name of [e.amountField, e.codeField, e.rateDateField, e.homeField, e.rateField]) columns.add(name); + + // Seek by `id`: this run's claim is about EVERY stored row, which an offset walk cannot promise. + const walk = keysetWalk>( + (q: KeysetPageQuery) => engine.find(object, { ...q, fields: [...columns], context: { ...SYSTEM_CTX } }), + { pageSize: SCAN_PAGE_SIZE, max: options.maxRecords ?? 100_000 }, + ); + + // An error no class above names (a page read or a rate read that fails) ends the walk. Rows an + // apply already wrote stay written, so the run is still recorded — as aborted, with the + // counts it reached — before the error goes on to the caller. + try { + for await (const page of walk.pages()) { + // The write's own reads, memoized across this page as one write memoizes across its batch. + const reads = engine.homeAmountReadsFor({ ...SYSTEM_CTX }); + const serving = reads.serving; + for (const row of page) { + report.scannedRecords++; + const id = row.id; + if (id == null || id === '') continue; + const recordId = String(id); + // The organization the write converts this row under: the write's own reading. + const organizationId = reads.organizationOf(row) ?? null; + const org = await organizationEntry(organizations, serving, organizationId); + const patch: Record = {}; + + for (const entry of plan) { + if (since !== null) { + const day = isRecordCurrencyBlank(row[entry.rateDateField]) + ? undefined + : documentDateOf(row[entry.rateDateField], entry.rateDateIsInstant ? await serving.timezone(organizationId ?? undefined) : undefined); + if (day === undefined || day < since) { + report.outsideSince++; + org.outsideSince++; + continue; + } + } + report.amounts++; + org.amounts++; + const skip = (reason: HomeAmountSkipReason, extra: { currency?: string; date?: string } = {}) => { + report.skippedTotal++; + report.skippedByReason[reason]++; + org.skipped[reason]++; + const item: HomeAmountRecomputeSkip = { recordId, organizationId, field: entry.amountField, reason, ...extra }; + if (report.skipped.length < maxListed) report.skipped.push(item); + else report.skippedTruncated = true; + if (auditSkips.length < AUDIT_MAX_LISTED) auditSkips.push(item); + }; + + // A blank code beside an amount is its own class: there is no currency to convert from + // (the projection leaves such an entry out, so the code-blank refusal can answer instead). + if (!isRecordCurrencyBlank(row[entry.amountField]) && isRecordCurrencyBlank(row[entry.codeField])) { + skip('code_blank'); + continue; + } + + let projected: Record; + try { + projected = await projectHomeAmounts([entry], row, serving, { + object, + operation: 'update', + organizationId: organizationId ?? undefined, + recordId, + }); + } catch (err) { + const code = (err as { code?: unknown })?.code; + const currency = typeof row[entry.codeField] === 'string' ? (row[entry.codeField] as string).trim() : undefined; + if (isHomeAmountRefusal(err)) { + if (code === EXCHANGE_RATE_MISSING_CODE) { + skip('rate_missing', { currency, date: (err as { date: string }).date }); + } else { + skip('home_currency_unset', { currency }); + } + continue; + } + if (code === VALIDATION_FAILED_CODE) { + skip('document_date_blank', { currency }); + continue; + } + throw err; + } + // The projection leaves out an entry whose inputs the write's validator refuses. + if (!(entry.homeField in projected)) { + skip('unconvertible', typeof row[entry.codeField] === 'string' ? { currency: String(row[entry.codeField]) } : {}); + continue; + } + + const to = { home: projected[entry.homeField], rate: projected[entry.rateField] }; + const from = { home: row[entry.homeField] ?? null, rate: row[entry.rateField] ?? null }; + if (sameStoredNumber(from.home, to.home) && sameStoredNumber(from.rate, to.rate)) { + report.unchanged++; + org.unchanged++; + continue; + } + report.changed++; + org.changed++; + if (report.changes.length < maxListed) report.changes.push({ recordId, organizationId, field: entry.amountField, from, to }); + else report.changesTruncated = true; + // The write recomputes over whatever the payload holds for the home field: the + // projection the write runs is the value stored, never this preview's. + patch[entry.homeField] = to.home; + } + + if (!apply || Object.keys(patch).length === 0) continue; + try { + await withTransientRetry( + () => engine.update( + object, + { + id, + ...patch, + ...(row.updated_at !== undefined && row.updated_at !== null ? { updated_at: row.updated_at } : {}), + }, + { context: { ...SYSTEM_PRESERVING_CTX } }, + ), + options.retry ?? {}, + ); + // Bumped AFTER the retry settles, never inside the retried closure. + report.written++; + org.written++; + } catch (err) { + // One row's failure must not abort the rest; a re-run revisits it, because it still differs. + const message = (err as Error)?.message ?? String(err); + logger.warn(`[home-amount-recompute] ${object} record ${recordId}: ${message}`); + report.failures.push({ recordId, error: message }); + } + } + } + } catch (err) { + if (apply) { + finishOrganizations(report, organizations); + const message = (err as Error)?.message ?? String(err); + logger.warn( + `[home-amount-recompute] the run over ${object} stopped after ${report.written} row(s) were rewritten: ${message}`, + ); + report.audit = await writeAuditEntry(engine, logger, report, auditSkips, options.source, message); + } + throw err; + } + if (walk.truncated) report.truncated = true; + finishOrganizations(report, organizations); + + if (apply) report.audit = await writeAuditEntry(engine, logger, report, auditSkips, options.source); + return report; +} + +function finishOrganizations( + report: HomeAmountRecomputeReport, + organizations: Map, +): void { + report.organizations = [...organizations.values()].sort((a, b) => + a.organizationId === b.organizationId ? 0 : a.organizationId === null ? 1 : b.organizationId === null ? -1 : a.organizationId < b.organizationId ? -1 : 1, + ); +} + +async function organizationEntry( + organizations: Map, + serving: HomeAmountServing, + organizationId: string | null, +): Promise { + const key = organizationId ?? '\u0000'; + let org = organizations.get(key); + if (!org) { + const home = await serving.homeCurrency(organizationId ?? undefined); + org = { + organizationId, + homeCurrency: typeof home === 'string' && home.trim() !== '' ? home.trim() : null, + amounts: 0, + unchanged: 0, + changed: 0, + written: 0, + skipped: zeroReasons(), + outsideSince: 0, + }; + organizations.set(key, org); + } + return org; +} + +/** + * The applied run's one `sys_audit_log` entry: a deployment-level `update` + * (no record; the run spans organizations) whose `metadata` carries the run — + * its scope, its counts per organization and reason, the skipped rows and the + * failures (each capped at {@link AUDIT_MAX_LISTED}). Asks the registry first: + * a composition without the audit ledger has no table to write, and says so + * through the report instead of a failed insert. A refused write does not undo + * the run, which already landed: it is warned and reported. + */ +async function writeAuditEntry( + engine: HomeAmountRecomputeEngine, + logger: HomeAmountRecomputeLogger, + report: HomeAmountRecomputeReport, + skips: HomeAmountRecomputeSkip[], + source: string | undefined, + aborted?: string, +): Promise { + if (!engine.getObject(AUDIT_OBJECT)) return { written: false, reason: 'ledger_not_mounted' }; + const metadata = { + event: 'home_amount.recompute', + source: source ?? null, + object: report.object, + fields: report.fields.map((f) => f.field), + since: report.since, + scannedRecords: report.scannedRecords, + amounts: report.amounts, + unchanged: report.unchanged, + changed: report.changed, + written: report.written, + outsideSince: report.outsideSince, + skippedByReason: report.skippedByReason, + organizations: report.organizations, + skipped: skips, + skippedListed: skips.length < report.skippedTotal ? `${skips.length} of ${report.skippedTotal}` : undefined, + failures: report.failures.slice(0, AUDIT_MAX_LISTED), + truncated: report.truncated, + // Set only when an error ended the walk: the counts above are as far as it got. + ...(aborted !== undefined ? { aborted } : {}), + }; + try { + await engine.insert( + AUDIT_OBJECT, + { action: 'update', object_name: report.object, metadata: JSON.stringify(metadata) }, + { context: { ...SYSTEM_CTX } }, + ); + return { written: true }; + } catch (err) { + const message = (err as Error)?.message ?? String(err); + logger.warn( + `[home-amount-recompute] the audit entry for this run was NOT written: the run itself landed ` + + `(${report.written} row(s) of ${report.object} rewritten), but ${AUDIT_OBJECT} refused the record of it, ` + + `so nothing in the ledger says who recomputed what. Cause: ${message}`, + ); + return { written: false, reason: 'refused', error: message }; + } +} + +/** Human-readable report body, shared by the CLI's dry-run and apply output. */ +export function formatHomeAmountRecomputeReport(report: HomeAmountRecomputeReport): string[] { + const lines: string[] = []; + const fields = report.fields.map((f) => `${f.field} → ${f.homeField}`).join(', '); + lines.push( + `Scanned ${report.scannedRecords} row(s) of ${report.object} (${fields})` + + (report.since ? `; document date on or after ${report.since}: ${report.amounts} amount(s), ${report.outsideSince} outside.` : `: ${report.amounts} amount(s).`), + ); + for (const org of report.organizations) { + const skipped = HOME_AMOUNT_SKIP_REASONS.filter((r) => org.skipped[r] > 0).map((r) => `${org.skipped[r]} ${r}`); + lines.push( + ` organization ${org.organizationId ?? '(none)'} · home ${org.homeCurrency ?? '(unset)'}: ` + + `${org.amounts} amount(s), ${org.unchanged} unchanged, ${org.changed} ${report.applied ? `changed (${org.written} row(s) written)` : 'would change'}` + + (skipped.length > 0 ? `, skipped ${skipped.join(', ')}` : ''), + ); + } + if (report.changed === 0) { + lines.push('✓ Every stored home amount already equals what a write would store now — nothing to recompute.'); + } else { + lines.push(report.applied ? `Rewrote ${report.written} row(s) (${report.changed} amount(s)):` : `Would rewrite ${report.changed} amount(s):`); + for (const c of report.changes) { + lines.push(` • ${c.recordId} ${c.field}: ${String(c.from.home)} @ ${String(c.from.rate)} → ${String(c.to.home)} @ ${String(c.to.rate)}`); + } + if (report.changesTruncated) lines.push(` … and ${report.changed - report.changes.length} more`); + } + if (report.skippedTotal > 0) { + lines.push(`✗ ${report.skippedTotal} amount(s) listed and NOT converted — a total over the home field leaves them out until fixed:`); + for (const s of report.skipped) { + const detail = s.reason === 'rate_missing' + ? `no ${s.currency} rate on or before ${s.date}` + : s.reason === 'code_blank' ? 'amount without a currency code' : s.reason.replace(/_/g, ' '); + lines.push(` • ${s.recordId} ${s.field} (organization ${s.organizationId ?? '(none)'}): ${detail}`); + } + if (report.skippedTruncated) lines.push(` … and ${report.skippedTotal - report.skipped.length} more`); + const remedies: string[] = []; + if (report.skippedByReason.rate_missing > 0) remedies.push('rate_missing: add the exchange rate (Setup → Exchange Rates) and re-run'); + if (report.skippedByReason.code_blank > 0) { + remedies.push( + `code_blank: write the codes with \`os migrate record-currency-codes --object ${report.object} --field --code \`, then re-run` + + ' — that command writes its one code into the codeless rows of EVERY organization (ADR-0144 Open item 13), so check the' + + ' code_blank count per organization above before you give it one', + ); + } + if (report.skippedByReason.document_date_blank > 0) remedies.push('document_date_blank: fill the document date, then re-run'); + if (report.skippedByReason.home_currency_unset > 0) remedies.push('home_currency_unset: set Localization → Default currency for the organization, then re-run'); + if (report.skippedByReason.unconvertible > 0) remedies.push('unconvertible: correct the code, amount or document date the write refuses, then re-run'); + for (const r of remedies) lines.push(` → ${r}`); + } + if (report.failures.length > 0) { + lines.push(`✗ ${report.failures.length} row(s) could not be written:`); + for (const f of report.failures.slice(0, 5)) lines.push(` • ${f.recordId}: ${f.error}`); + if (report.failures.length > 5) lines.push(` … and ${report.failures.length - 5} more`); + lines.push('Re-running is safe: a row is only rewritten while its stored home amount differs.'); + } + if (report.truncated) lines.push('⚠ Walk incomplete — some rows were not examined; re-run without the cap to finish.'); + if (report.applied) { + lines.push(report.audit.written + ? 'Recorded in the audit log (sys_audit_log, one entry for this run).' + : `⚠ No audit entry for this run: ${report.audit.reason === 'ledger_not_mounted' ? 'the audit ledger (sys_audit_log) is not part of this composition' : `the audit ledger refused it (${report.audit.error ?? 'unknown'})`}.`); + } else if (report.changed > 0) { + lines.push('Dry run — no row or table was written. Re-run with --apply to recompute.'); + } + return lines; +} + +/** + * Did this run leave every stored home amount equal to what a write would store + * now? False while any amount is skipped (Open item 8: the gap keeps the run + * open), any write failed, the walk was cut short, or a dry run found changes. + */ +export function homeAmountRecomputeComplete(report: HomeAmountRecomputeReport): boolean { + return report.skippedTotal === 0 + && report.failures.length === 0 + && !report.truncated + && (report.applied || report.changed === 0); +} diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index c65ba17d008..237f6a00e84 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -546,6 +546,29 @@ export type { RecordCurrencyBackfillLogger, RecordCurrencyBackfillOptions, } from './record-currency-backfill.js'; +// [ADR-0144 D5, D7 parameter 1, Open item 8] The explicit recompute of stored +// home amounts behind `os migrate recompute-home-amounts`: the write's own +// projection over stored rows. Same split as the two backfills above. +export { + recomputeHomeAmounts, + assertHomeAmountRecomputeSince, + formatHomeAmountRecomputeReport, + homeAmountRecomputeComplete, + HOME_AMOUNT_SKIP_REASONS, +} from './home-amount-recompute.js'; +export type { + HomeAmountRecomputeReport, + HomeAmountRecomputeChange, + HomeAmountRecomputeSkip, + HomeAmountSkipReason, + HomeAmountRecomputeOrganization, + HomeAmountRecomputeFailure, + HomeAmountRecomputeAudit, + HomeAmountRecomputeEngine, + HomeAmountRecomputeLogger, + HomeAmountRecomputeOptions, +} from './home-amount-recompute.js'; +export type { HomeAmountReads, HomeAmountServing, ExchangeRateRow } from './home-amount-projection.js'; export { summaryEmptySetValue, summaryNullIsBackfillable, aggregateSummaryValue } from './summary-aggregate.js'; export type { SummaryDescriptor, SummaryAggregateEngine } from './summary-aggregate.js'; export { evaluateValidationRules, needsPriorRecord, legalNextStates } from './validation/rule-validator.js'; diff --git a/packages/plugins/plugin-audit/src/home-amount-recompute-audit.integration.test.ts b/packages/plugins/plugin-audit/src/home-amount-recompute-audit.integration.test.ts new file mode 100644 index 00000000000..8d7d9a1f99e --- /dev/null +++ b/packages/plugins/plugin-audit/src/home-amount-recompute-audit.integration.test.ts @@ -0,0 +1,101 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#27193, ADR-0144 D5] What `os migrate recompute-home-amounts --apply` leaves + * in `sys_audit_log` when the REAL audit writers are installed, as they are in + * the command's own boot (`buildDataMigrationPlugins({ audit: true })` mounts + * AuditPlugin). + * + * The recompute writes each row it changes through `engine.update`, so it is a + * write like any other: the shipped record-level writer records one `update` + * row per rewritten record, and the run adds its own deployment-level entry + * (no record, `metadata.event: home_amount.recompute`). The objectql pins see + * only the run entry, because their composition installs no writers; this is + * the pin with them, so "one entry per run" can never again be read as "one + * row in the ledger". + */ + +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import { ObjectQL, recomputeHomeAmounts } from '@objectstack/objectql'; +import { SqliteWasmDriver } from '@objectstack/driver-sqlite-wasm'; +import { SysExchangeRate } from '@objectstack/platform-objects/system'; + +import { installAuditWriters } from './audit-writers.js'; +import { SysActivity, SysAuditLog } from './objects/index.js'; +import { installDefaultOrganization } from './default-organization.testkit.js'; + +const PKG = 'com.objectstack.audit.test.home-amount-recompute'; +const SYS = { context: { isSystem: true } } as const; +const INVOICE = 'hr_invoice'; + +const invoiceObject = { + name: INVOICE, + label: 'Invoice', + fields: { + name: { name: 'name', label: 'Name', type: 'text' }, + currency_code: { name: 'currency_code', label: 'Currency', type: 'text', valueDomain: 'iso_4217_currency' }, + invoice_date: { name: 'invoice_date', label: 'Date', type: 'date' }, + amount: { + name: 'amount', + label: 'Amount', + type: 'currency', + currencyConfig: { + currencyMode: 'record', + currencyField: 'currency_code', + homeAmount: { field: 'amount_home', rateDate: 'invoice_date', rateField: 'amount_rate' }, + }, + }, + amount_home: { name: 'amount_home', label: 'Amount (home)', type: 'currency', readonly: true }, + amount_rate: { name: 'amount_rate', label: 'Rate', type: 'number', readonly: true }, + }, +}; + +describe('[#27193] an applied recompute, under the real audit writers', () => { + let driver: SqliteWasmDriver; + let engine: ObjectQL; + + beforeAll(async () => { + engine = new ObjectQL(); + driver = new SqliteWasmDriver({ filename: ':memory:' }); + await driver.connect(); + engine.registerDriver(driver, true); + await engine.init(); + for (const o of [invoiceObject, SysExchangeRate, SysAuditLog, SysActivity]) engine.registry.registerObject(o as any, PKG); + await engine.syncSchemas(); + await installDefaultOrganization(engine as never); + engine.setHomeLocalizationResolver(async () => ({ currency: 'CNY' })); + // The REAL shipped record-level writer, not a stand-in for it. + installAuditWriters(engine, PKG); + }); + + afterAll(async () => { + await engine?.destroy(); + }); + + it('records one per-row update for each rewritten row, plus the one run-level entry', async () => { + const rate = await engine.insert('sys_exchange_rate', { currency: 'USD', effective_date: '2026-01-01', rate: 0.1389, source: 'manual' }, SYS); + const ids: string[] = []; + for (const name of ['a', 'b', 'c']) { + ids.push((await engine.insert(INVOICE, { name, currency_code: 'USD', invoice_date: '2026-03-15', amount: 10000 }, SYS)).id); + } + await engine.update('sys_exchange_rate', { id: rate.id, rate: 0.125 }, SYS); + const ledger = async () => + (await engine.find('sys_audit_log', { where: { object_name: INVOICE }, context: SYS.context } as never)) as Array>; + // The inserts' own `create` rows land first; the recompute adds only `update` rows. + await vi.waitFor(async () => expect((await ledger()).filter((r) => r.action === 'create')).toHaveLength(3)); + expect((await ledger()).filter((r) => r.action === 'update')).toEqual([]); + + const report = await recomputeHomeAmounts(engine as never, { info() {}, warn() {} }, { object: INVOICE, apply: true, source: 'test' }); + expect(report).toMatchObject({ changed: 3, written: 3, audit: { written: true } }); + + const updates = async () => (await ledger()).filter((r) => r.action === 'update'); + await vi.waitFor(async () => expect(await updates()).toHaveLength(4)); + const added = await updates(); + const perRow = added.filter((r) => r.record_id != null); + const runLevel = added.filter((r) => r.record_id == null); + expect(perRow.map((r) => r.record_id).sort()).toEqual([...ids].sort()); + expect(runLevel).toHaveLength(1); + expect(runLevel[0]).toMatchObject({ action: 'update', object_name: INVOICE }); + expect(JSON.parse(String(runLevel[0]!.metadata))).toMatchObject({ event: 'home_amount.recompute', written: 3 }); + }); +}); diff --git a/packages/qa/dogfood/test/home-amount-projection.test.ts b/packages/qa/dogfood/test/home-amount-projection.test.ts index 666bd1d3233..098518bdf65 100644 --- a/packages/qa/dogfood/test/home-amount-projection.test.ts +++ b/packages/qa/dogfood/test/home-amount-projection.test.ts @@ -14,11 +14,17 @@ // rate row is read in the written row's organization only. The in-memory face of the same pins (and the strips, defaults, // preview and predicate-update cells) is packages/objectql's // `engine-home-amount-projection.test.ts`. +// +// The second half drives the explicit recompute (`recomputeHomeAmounts`, +// `os migrate recompute-home-amounts`) over the same cells: a corrected rate, +// a missing one, a blank code, the same currency, a datetime document date +// across the organization's timezone boundary, and two organizations with +// different home currencies. import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import { basename } from 'node:path'; import { fileURLToPath } from 'node:url'; -import { ObjectQL } from '@objectstack/objectql'; +import { ObjectQL, recomputeHomeAmounts, homeAmountRecomputeComplete } from '@objectstack/objectql'; import { SqlDriver } from '@objectstack/driver-sql'; import { SqliteWasmDriver } from '@objectstack/driver-sqlite-wasm'; import { SysExchangeRate } from '@objectstack/platform-objects/system'; @@ -276,3 +282,178 @@ for (const cell of CELLS) { }, ); } + +// ── The explicit recompute on the real drivers ──────────────────────────── +// +// [ADR-0144 D5, D7 parameter 1, Open item 8] `recomputeHomeAmounts` (behind +// `os migrate recompute-home-amounts`) over rows stored through each dialect: +// the stored value it compares is read back from the column, the rate row is +// each dialect's pick, and the value an apply stores is the write's own. One +// object per pin, so each pin's counts are its own. The in-memory face (the +// audit entry, the refusals, the report text) is packages/objectql's +// `engine-home-amount-projection.test.ts`. + +const recomputeObject = (name: string, dateType: 'date' | 'datetime' = 'date') => ({ + ...FX_INVOICE, + name, + label: name, + fields: { ...FX_INVOICE.fields, invoice_date: { name: 'invoice_date', type: dateType } }, +}); +const RC = { + corrected: 'rc_corrected', + missing: 'rc_missing', + codeless: 'rc_codeless', + same: 'rc_same', + zone: 'rc_zone', + orgs: 'rc_orgs', +} as const; +const quiet = { info: () => {}, warn: () => {} }; + +for (const cell of CELLS) { + describe.skipIf(!cell.url)( + `the explicit recompute of stored home amounts — ${cell.id}${cell.url ? '' : ` (skipped: set ${cell.env} to run this cell)`}`, + () => { + let driver: SqlDriver; + let engine: ObjectQL; + let homes: Record = {}; + let timezone: string | undefined; + + beforeAll(async () => { + driver = await cell.make(); + engine = new ObjectQL(); + engine.registerDriver(driver, true); + await engine.init(); + engine.registerApp(appWith([ + SysExchangeRate, + ...Object.values(RC).map((name) => recomputeObject(name, name === RC.zone ? 'datetime' : 'date')), + ])); + await engine.syncSchemas(); + await installDefaultOrganization(engine); + engine.setHomeLocalizationResolver(async (org) => ({ currency: homes[org ?? ''] ?? 'CNY', timezone })); + for (const [currency, effective_date, rate] of [['USD', '2026-01-01', 0.1389], ['USD', '2026-04-01', 0.15]] as const) { + await engine.insert('sys_exchange_rate', { currency, effective_date, rate, source: 'manual' }, SYS); + } + }); + + afterAll(async () => { + await engine?.destroy(); + await cell.teardown(); + }); + + const run = (object: string, options: { apply?: boolean; since?: string } = {}) => + recomputeHomeAmounts(engine as never, quiet, { object, ...options }); + async function stored(object: string, id: unknown) { + const back = await engine.findOne(object, { where: { id }, context: SYS.context } as never) as Record; + return { home: back.amount_home == null ? null : Number(back.amount_home), rate: back.amount_rate == null ? null : Number(back.amount_rate) }; + } + /** A row stored before its home field was declared (or its code backfilled): written at the driver. */ + async function legacy(object: string, id: string, row: Record) { + await engine.getDriverForObject(object)!.create(object, { id, name: id, organization_id: DEFAULT_ORGANIZATION_ID, ...row } as never); + } + + it('a corrected rate: the dry run reports and writes nothing, the apply stores what a fresh write stores, a re-run is clean', async () => { + const { id } = await engine.insert(RC.corrected, { name: 'c', currency_code: 'USD', invoice_date: '2026-03-15', amount: 10000 }, SYS); + expect(await stored(RC.corrected, id)).toEqual({ home: 71994.24, rate: 0.1389 }); + const [rateRow] = await engine.find('sys_exchange_rate', { where: { currency: 'USD', effective_date: '2026-01-01' }, context: SYS.context } as never) as Array<{ id: string }>; + await engine.update('sys_exchange_rate', { id: rateRow!.id, rate: 0.125 }, SYS); + try { + const stamp = async () => (await engine.findOne(RC.corrected, { where: { id }, context: SYS.context } as never) as Record).updated_at; + const stampBefore = await stamp(); + expect(stampBefore).toBeTruthy(); + const dry = await run(RC.corrected); + expect(dry).toMatchObject({ amounts: 1, changed: 1, written: 0, skippedTotal: 0 }); + expect(dry.changes[0]!.to).toEqual({ home: 80000, rate: 0.125 }); + expect(await stored(RC.corrected, id)).toEqual({ home: 71994.24, rate: 0.1389 }); + + await new Promise((resolve) => setTimeout(resolve, 20)); // a stamp taken now would differ + const applied = await run(RC.corrected, { apply: true }); + expect(applied).toMatchObject({ changed: 1, written: 1, failures: [] }); + // Only the two projected fields move: the row keeps its own updated_at. + expect(String(await stamp())).toBe(String(stampBefore)); + const fresh = await engine.insert(RC.corrected, { name: 'fresh', currency_code: 'USD', invoice_date: '2026-03-15', amount: 10000 }, SYS); + expect(await stored(RC.corrected, id)).toEqual(await stored(RC.corrected, fresh.id)); + expect(await stored(RC.corrected, id)).toEqual({ home: 80000, rate: 0.125 }); + + const again = await run(RC.corrected, { apply: true }); + expect(again).toMatchObject({ amounts: 2, unchanged: 2, changed: 0, written: 0 }); + expect(homeAmountRecomputeComplete(again)).toBe(true); + } finally { + await engine.update('sys_exchange_rate', { id: rateRow!.id, rate: 0.1389 }, SYS); + } + }); + + it('a missing rate is listed with its currency and date and skipped; the rest are filled; nothing else is written', async () => { + await legacy(RC.missing, 'old_usd', { currency_code: 'USD', invoice_date: '2026-03-15', amount: 1389 }); + await legacy(RC.missing, 'old_sek', { currency_code: 'SEK', invoice_date: '2026-03-15', amount: 500 }); + const report = await run(RC.missing, { apply: true }); + expect(report).toMatchObject({ amounts: 2, changed: 1, written: 1, skippedTotal: 1 }); + expect(report.skipped).toEqual([ + { recordId: 'old_sek', organizationId: DEFAULT_ORGANIZATION_ID, field: 'amount', reason: 'rate_missing', currency: 'SEK', date: '2026-03-15' }, + ]); + expect(await stored(RC.missing, 'old_usd')).toEqual({ home: 10000, rate: 0.1389 }); + expect(await stored(RC.missing, 'old_sek')).toEqual({ home: null, rate: null }); + expect(homeAmountRecomputeComplete(report)).toBe(false); + }); + + it('a blank code is its own class, apart from a missing rate, and is never written', async () => { + await legacy(RC.codeless, 'codeless', { invoice_date: '2026-03-15', amount: 50 }); + const report = await run(RC.codeless, { apply: true }); + expect(report.skipped.map((s) => s.reason)).toEqual(['code_blank']); + expect(report.skippedByReason).toMatchObject({ code_blank: 1, rate_missing: 0 }); + expect(report.written).toBe(0); + expect(await stored(RC.codeless, 'codeless')).toEqual({ home: null, rate: null }); + }); + + it('the same currency as home converts at 1, rounded once in decimal', async () => { + await legacy(RC.same, 'same', { currency_code: 'CNY', amount: 1.005, amount_home: 1, amount_rate: 1 }); + const report = await run(RC.same, { apply: true }); + expect(report).toMatchObject({ changed: 1, written: 1 }); + expect(await stored(RC.same, 'same')).toEqual({ home: 1.01, rate: 1 }); + }); + + it('a datetime document date is dated in the organization\'s timezone, through the column, for the rate and for since', async () => { + // 2026-03-31T17:00Z is 01:00 on 2026-04-01 in Shanghai; USD has a new rate from April 1. + const { id } = await engine.insert(RC.zone, { name: 'z', currency_code: 'USD', invoice_date: '2026-03-31T17:00:00Z', amount: 1500 }, SYS); + expect(await stored(RC.zone, id)).toMatchObject({ rate: 0.1389 }); + try { + expect(await run(RC.zone, { since: '2026-04-01' })).toMatchObject({ amounts: 0, outsideSince: 1 }); + timezone = 'Asia/Shanghai'; + expect(await run(RC.zone, { since: '2026-04-01', apply: true })).toMatchObject({ amounts: 1, written: 1 }); + expect(await stored(RC.zone, id)).toEqual({ home: 10000, rate: 0.15 }); + } finally { + timezone = undefined; + } + }); + + it('two organizations: each converts into its own home currency at its own rates, counted apart', async () => { + engine.setTenancyPostureProvider(() => 'isolated'); + try { + const orgDriver = engine.getDriverForObject('sys_organization')!; + for (const id of ['org_rc_a', 'org_rc_b']) { + if (!(await orgDriver.findOne('sys_organization', { where: { id } } as never))) await orgDriver.create('sys_organization', { id, name: id }); + } + homes = { org_rc_a: 'CNY', org_rc_b: 'USD' }; + const as = (org: string) => ({ context: { isSystem: true, tenantId: org } }); + await engine.insert('sys_exchange_rate', { currency: 'EUR', effective_date: '2026-01-01', rate: 0.128, source: 'manual' }, as('org_rc_a')); + const eurB = await engine.insert('sys_exchange_rate', { currency: 'EUR', effective_date: '2026-01-01', rate: 0.9, source: 'manual' }, as('org_rc_b')); + const a = await engine.insert(RC.orgs, { name: 'a', currency_code: 'EUR', invoice_date: '2026-03-15', amount: 64 }, as('org_rc_a')); + const b = await engine.insert(RC.orgs, { name: 'b', currency_code: 'EUR', invoice_date: '2026-03-15', amount: 90 }, as('org_rc_b')); + expect(await stored(RC.orgs, a.id)).toEqual({ home: 500, rate: 0.128 }); + expect(await stored(RC.orgs, b.id)).toEqual({ home: 100, rate: 0.9 }); + // Organization B corrects its EUR rate; A's row reads A's rate and stays. + await engine.update('sys_exchange_rate', { id: eurB.id, rate: 0.8 }, as('org_rc_b')); + const report = await run(RC.orgs, { apply: true }); + expect(report.organizations).toEqual([ + expect.objectContaining({ organizationId: 'org_rc_a', homeCurrency: 'CNY', amounts: 1, unchanged: 1, changed: 0 }), + expect.objectContaining({ organizationId: 'org_rc_b', homeCurrency: 'USD', amounts: 1, unchanged: 0, changed: 1, written: 1 }), + ]); + expect(await stored(RC.orgs, a.id)).toEqual({ home: 500, rate: 0.128 }); + expect(await stored(RC.orgs, b.id)).toEqual({ home: 112.5, rate: 0.8 }); + } finally { + homes = {}; + engine.setTenancyPostureProvider(() => undefined); + } + }); + }, + ); +} diff --git a/scripts/adr-anchors/packages__objectql__src__home-amount-recompute.ts.json b/scripts/adr-anchors/packages__objectql__src__home-amount-recompute.ts.json new file mode 100644 index 00000000000..f4b41495412 --- /dev/null +++ b/scripts/adr-anchors/packages__objectql__src__home-amount-recompute.ts.json @@ -0,0 +1,7 @@ +{ + "file": "packages/objectql/src/home-amount-recompute.ts", + "adrs": [ + "ADR-0144" + ], + "invariant": "ADR-0144 D5 and D7 parameter 1 — the explicit recompute of stored home amounts (`recomputeHomeAmounts`, `os migrate recompute-home-amounts`) computes NOTHING itself: its preview projects each stored row with the write's own `projectHomeAmounts` over the engine's own reads (`ObjectQL.homeAmountReadsFor`: the rate row in force on the document date in the row's organization, that organization's home currency and timezone), and its apply writes through `engine.update` with the home field in the payload, so the write projects the row again and the value stored is the write's. Do NOT add a second copy of the division, the rounding, the rate-row rule, the same-currency rule or the timezone day here, and do NOT write the preview's value at the driver: a recomputed row must hold what a fresh write of it stores. Parameter 1: a row whose rate is missing is listed and skipped and never written (no rate of 1, no later rate, no zero), and a row with an amount beside a blank currency code is its own class (`code_blank`, remedy `os migrate record-currency-codes`), never folded into `rate_missing`. Open item 8: a skipped row keeps the run incomplete (`homeAmountRecomputeComplete`, the command's exit 1) and is recorded in the applied run's audit entry, because `sum` over the home field silently leaves an empty home amount out." +} diff --git a/scripts/engine-double-contract.pinned.json b/scripts/engine-double-contract.pinned.json index 566599e5eec..eacc92da399 100644 --- a/scripts/engine-double-contract.pinned.json +++ b/scripts/engine-double-contract.pinned.json @@ -1,6 +1,11 @@ { "$comment": "GENERATED — the RETAINED ledger of check-engine-double-contract.mjs (#9680). Regenerate with `node scripts/check-engine-double-contract.mjs --write`; never hand-edit. Each row is one (file, verb) whose engine double routes through the producer's dispatch predicate. This is the OPPOSITE polarity to engine-double-contract.baseline.json: that ledger records DEBT and may only shrink, this one records COVERAGE and may only grow. A row that disappears is a pinned double that left the population — the blind spot #9680 measured, where deleting a double's delete() member took 319 pinned to 318 with the gate green. Read a removal in this file's diff as a coverage loss and check it was intended.", "entries": [ + { + "file": "packages/cli/src/commands/migrate/recompute-home-amounts.test.ts", + "verb": "update", + "pinned": 1 + }, { "file": "packages/cli/src/commands/serve-datasource-secret-resolver.test.ts", "verb": "delete",