Repository navigation
fix(example): bound projects.list reads to the page, not the table - #452
Conversation
`projects.list` asked "owner OR member" of `projects`, which no index spans, so
it walked the table from the newest end and post-filtered in JS. A viewer with
nothing to see drained the whole table, and every scanned row joined the
subscription read set. Measured 40 scanned / 0 returned at 40 rows.
Two more defects were hiding behind it:
- `.filter()` hands the callback the raw Convex row, whose id field is `_id`, so
`memberProjectIds.has(project.id)` was `has(undefined)` and always false.
Projects visible only through a membership never appeared in the list at all.
- `hasAnyProject` was unbounded too: `{ ownerId, archived }` ties `index('archived')`
and `index('ownerId')` at the same planner score, declaration order wins, and the
loser becomes a post-filter over the whole partition. 40 and 43 reads against
bounds of 2 and 3.
Add `projectAccess`, one row per (user, project), indexed
`(userId, archived, projectCreatedAt)`. `_helpers/project_access.ts` owns every
grant/revoke and is the single owner of "which projects can this user see", used
by `list`, `listForDropdown` and `hasAny`. 4 reads at 10 noise rows and 4 at 400.
Maintained by explicit mutation helpers, not schema triggers: async cascade
continuation workers build their ORM from the raw `ctx.db` and drop triggers past
the first batch, so a trigger-maintained copy would drift. Deletion needs neither,
both FKs cascade and cascades still run in those batches. `revokeProjectAccess`
re-derives entitlement rather than deleting blindly, and `grantProjectAccess`
derives `isOwner` from the project so a caller cannot mislabel a row.
Also repair the read-bound harness, which was measuring nothing:
- `countDocumentReads` patched `db.get` as a one-arg forwarder while the lifecycle
writer only calls the two-arg overload, so `oldDoc`/`newDoc` came back null and
every trigger was silently skipped.
- The example read-bound tests installed the counter after the ORM was built, so
they measured a constant 0. Fixing placement exposed the `hasAnyProject` scan and
a tag-merge bound that was really 16, not 12 (same at 40 and 120 target rows, so
the property it pins still holds).
Bounds are asserted as equalities so a dead counter fails loudly.
Add compound `isPublic_archived` for the unauthenticated branch, which had the
same planner tie and read every archived public project.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f0275f179
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // One index walk over the viewer's own access rows. The previous read model | ||
| // scanned `projects` and post-filtered owner-or-member in JS, so it cost | ||
| // O(table) per page and every scanned row joined this subscription. | ||
| const results = await listAccessibleProjects(ctx, { |
There was a problem hiding this comment.
Populate access rows before switching reads
When this schema is deployed over existing project data, projectAccess starts empty, so this call makes projects.list return no projects; listForDropdown and hasAny fail similarly. A repo-wide search shows backfillProjectAccess is referenced only by its test, while migrations/manifest.ts remains empty, so there is no callable migration to populate existing rows. Wire and run the backfill before switching these read paths to the derived table.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Populate access rows before switching reads
The migrations are registered, but the activation-order defect is still present at 81b7f8e3cc2323296960f56f985f6817b0d835c1. A fresh public-helper replay with an existing owned project and an existing shared project returns an empty list, an empty dropdown and hasAny: false before backfill. The deploy command activates the backend before running migrations. Keeping this P1 open; a staged schema/write, backfill, then read-cutover rollout is required to preserve visibility without a transitional scan. No deployment or data reset was performed.
There was a problem hiding this comment.
Populate access rows before switching reads
Preparation is split into #487. This activation PR now targets its preparation branch and is draft. The preparation revision retains canonical public readers while owner/member backfills run. No target deployment or completed-backfill receipts have been supplied, so this finding remains open. Post-push source proof at58ff9132ed7de370d0fc688a2ff32049dc161ee3 passed all22 focused tests; that does not substitute for real rollout receipts.
`projectAccess` started empty on any deployment that already held projects, so switching `list`, `listForDropdown` and `hasAny` onto it made all three return nothing. `backfillProjectAccess` existed but was referenced only by its test and `migrations/manifest.ts` was empty, so there was no callable path to populate it. Add `20260906_022807_backfill_project_access`, scaffolded with `kitcn migrate create` and registered in the manifest. It walks `projects` in batches and re-derives each project's rows; `grantProjectAccess` updates rather than inserts, so re-running is safe. `down` clears them. The runner pages `ctx.db.query(table)`, so `migrateOne` receives the RAW Convex document -- `_id`, not `id`, and no hydrated `createdAt`. Same trap that made the old `.filter()` membership check dead. Re-read the row through the ORM instead of handing that shape to a helper expecting the public one. Extract `syncProjectAccessForProject` as the single owner of "what should this project's access rows look like", shared by the migration and `backfillProjectAccess`, so a deployment repaired by the migration and one rebuilt in a test cannot drift. This also fixes a latent bug in the old backfill, which capped memberships globally at 1000 rather than per project. Test drives the migration's own `migrateOne` over raw docs: reads are empty before, correct after, re-running does not duplicate, `down` empties. Verified armed by neutering the migration and watching it fail.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0376934447
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The backfill read each project's memberships in one `limit: 1000` call. Nothing caps a project's member count -- `addMember` only rejects the owner, a duplicate, and a non-owner caller -- and `findMany` truncates silently rather than throwing (probe: 5 rows, `limit: 3`, returned 3, no error). Members past the bound would get no access row, and `list`, `hasAny` and `listForDropdown` would hide a project they belong to, permanently and with the migration reporting success. Split the backfill into one migration per source table: owners over `projects`, memberships over `projectMembers`. The runner pages each table itself, so neither is bounded by a limit this code picks. That shape is also what keeps it correct at runtime. A cursor loop inside `migrateOne` would issue a second `.paginate()` inside the runner's own paginated read, which real Convex rejects with `MultiplePaginatedDatabaseQueries` while convex-test does not model it -- it would have passed tests and failed in prod. `backfillProjectAccess` is now explicitly test-only and reads `limit + 1` so it throws on truncation instead of quietly backfilling a prefix. Tests drive both migrations over raw `ctx.db` docs and cover every member row; both were verified armed by neutering each migration in turn.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1387b4ccc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // One index walk over the viewer's own access rows. The previous read model | ||
| // scanned `projects` and post-filtered owner-or-member in JS, so it cost | ||
| // O(table) per page and every scanned row joined this subscription. | ||
| const results = await listAccessibleProjects(ctx, { |
There was a problem hiding this comment.
Backfill access before activating derived reads
On deployments containing existing projects, authenticated reads switch to projectAccess while it is still empty or partially populated. Fresh evidence after the earlier comment is the ordering in packages/kitcn/src/cli/commands/deploy.ts: the backend deployment completes at lines 50-62, and only afterward does runMigrationFlow start at lines 74-82. Consequently users see no or incomplete projects throughout both backfill migrations; use an expand/backfill/contract rollout or retain a fallback read until backfill completion.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Backfill access before activating derived reads
Confirmed at published head 81b7f8e3cc2323296960f56f985f6817b0d835c1. The pre-backfill owned/shared-project replay fails: list and dropdown are empty and hasAny is false. This is not resolved by registering the migrations, because backend activation still precedes their execution. Keeping the P1 open and the PR unmerged; preserving visibility and the indexed-read bound requires a staged cutover, not accepting the empty-result window.
There was a problem hiding this comment.
Backfill access before activating derived reads
The maintainer approved separate deployable preparation #487 and dependent activation #452. This PR stays draft until every target records the exact preparation deployment OID and both owner/member migrations as applied/completed. Those receipts are still absent, so this P1 remains open. Full bun check and post-push22 focused tests pass; no fallback reader, readiness flag or empty-window waiver is used.
| await ctx.orm | ||
| .update(projectAccessTable) | ||
| .set({ archived: input.archived }) | ||
| .where(eq(projectAccessTable.projectId, input.projectId)); |
There was a problem hiding this comment.
Bound archive synchronization to the supported row count
When a project has more than 10,000 access rows, archiving or restoring it always fails because this synchronous update collects every match and the ORM throws above its default mutationMaxRows of 10,000 (mutation-utils.ts:746,952-963). Contrary to the comment above this helper, projects.addMember imposes no member-count cap, so such projects are reachable through the public mutation path; either enforce a compatible cap or perform this synchronization in bounded batches.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Bound archive synchronization to the supported row count
Deferred under the user's explicit P2 scope. The current synchronous archive update retains the 10,000-row ceiling and fails loudly rather than synchronizing only a prefix. No member cap or batched-sync contract was added, and this thread remains open.
There was a problem hiding this comment.
Bound archive synchronization to the supported row count
The maintainer explicitly deferred this P2 on2026-10-14 in the current review session. Archive/restore still synchronously mirror access rows and can exceed the10000-row mutation limit. No membership cap or asynchronous archive contract was added. This thread stays open, and both task plans record the deferral.
… bounds docs Four review findings on #452, all verified against source. Member-migration rollback deleted owner-derived rows. An owner can also hold a `projectMembers` row -- `aggregateDemo` seeds exactly that -- so `migrate down --steps 1` removed the access row the still-applied owner migration created, and the owner lost their own project from every read. The member `down` now leaves the row alone when the user is the project's current owner. Dropdown lost capacity. `listProjectsForDropdown` bounds the union with one limit, where the read it replaced bounded each side separately: up to 1000 owned plus up to 1000 member. A user with 750 of each silently lost 500 from an unpaginated dropdown. The caller now passes the combined figure. `syncProjectArchived`'s comment was wrong. It claimed the update was "bounded by the project's member count, which addMember already gates"; `addMember` caps nothing. The real ceiling is the ORM's `mutationMaxRows` (10,000), above which the update throws. Comment corrected to state that; a member cap or a batched sync is a product decision, not one to make here. Rollout window documented. `kitcn deploy` pushes the backend and only then runs migrations, so a deployment holding existing projects shows empty lists until both backfills finish. Acceptable for a re-seedable example app; the owner migration now says so and points at expand/backfill/contract for apps where it is not. Both new tests verified armed by reverting the fix under them.
|
Blocked activation review receipt for Cleanup is pushed and source-verified. This is not a clean activation/merge receipt. PR452 is draft and depends on preparation #487 atb555b70406ffb79e486d24e50d36b596f805e6b1. The maintainer approved the split and holding activation until target receipts. No real remote deployment or merge performed. Full bun check passed, including fixtures/verify/runtime. Three independent source lenses found no new source defect. Committed-branch autoreview at58ff9132 against preparation exited0 without actionable P0 findings. After the final material plan push, all22 focused tests passed, including resolved/outdated backfill P1 replay, fractional ordering,1500-project dropdown and registered authenticated lifecycle. Generated artifacts come from real isolated CLI codegen. Two actionable rollout P1 findings remain OPEN. Every target still needs the exact preparation deployment OID plus both owner/member migrations applied=true and status=completed. No target receipts have been supplied. The completion checker intentionally fails on the outstanding receipt checkbox/open Closeout phase. Required checks are not reported for the dependent base; hosted CI is tracked separately. The archive>10000-row P2 is explicitly deferred by the maintainer on2026-10-14 and stays OPEN. The dropdown capacity P2 is source-fixed, post-push-proven and RESOLVED. Raw inventory before this receipt: 2 top-level comments, 13 reviews, 6 threads;3 unresolved (2 rolloutP1 plus1 deferredP2). Resolved/outdated threads included. Feedback ledger:
Separate inherited ORM cursor-protocol follow-up: #488. |
* fix(example): bound projects.list reads to the page, not the table
`projects.list` asked "owner OR member" of `projects`, which no index spans, so
it walked the table from the newest end and post-filtered in JS. A viewer with
nothing to see drained the whole table, and every scanned row joined the
subscription read set. Measured 40 scanned / 0 returned at 40 rows.
Two more defects were hiding behind it:
- `.filter()` hands the callback the raw Convex row, whose id field is `_id`, so
`memberProjectIds.has(project.id)` was `has(undefined)` and always false.
Projects visible only through a membership never appeared in the list at all.
- `hasAnyProject` was unbounded too: `{ ownerId, archived }` ties `index('archived')`
and `index('ownerId')` at the same planner score, declaration order wins, and the
loser becomes a post-filter over the whole partition. 40 and 43 reads against
bounds of 2 and 3.
Add `projectAccess`, one row per (user, project), indexed
`(userId, archived, projectCreatedAt)`. `_helpers/project_access.ts` owns every
grant/revoke and is the single owner of "which projects can this user see", used
by `list`, `listForDropdown` and `hasAny`. 4 reads at 10 noise rows and 4 at 400.
Maintained by explicit mutation helpers, not schema triggers: async cascade
continuation workers build their ORM from the raw `ctx.db` and drop triggers past
the first batch, so a trigger-maintained copy would drift. Deletion needs neither,
both FKs cascade and cascades still run in those batches. `revokeProjectAccess`
re-derives entitlement rather than deleting blindly, and `grantProjectAccess`
derives `isOwner` from the project so a caller cannot mislabel a row.
Also repair the read-bound harness, which was measuring nothing:
- `countDocumentReads` patched `db.get` as a one-arg forwarder while the lifecycle
writer only calls the two-arg overload, so `oldDoc`/`newDoc` came back null and
every trigger was silently skipped.
- The example read-bound tests installed the counter after the ORM was built, so
they measured a constant 0. Fixing placement exposed the `hasAnyProject` scan and
a tag-merge bound that was really 16, not 12 (same at 40 and 120 target rows, so
the property it pins still holds).
Bounds are asserted as equalities so a dead counter fails loudly.
Add compound `isPublic_archived` for the unauthenticated branch, which had the
same planner tie and read every archived public project.
* docs(plans): record PR #452 in the 443 task plan
* fix(example): ship the projectAccess backfill as a real migration
`projectAccess` started empty on any deployment that already held projects, so
switching `list`, `listForDropdown` and `hasAny` onto it made all three return
nothing. `backfillProjectAccess` existed but was referenced only by its test and
`migrations/manifest.ts` was empty, so there was no callable path to populate it.
Add `20260906_022807_backfill_project_access`, scaffolded with `kitcn migrate
create` and registered in the manifest. It walks `projects` in batches and
re-derives each project's rows; `grantProjectAccess` updates rather than inserts,
so re-running is safe. `down` clears them.
The runner pages `ctx.db.query(table)`, so `migrateOne` receives the RAW Convex
document -- `_id`, not `id`, and no hydrated `createdAt`. Same trap that made the
old `.filter()` membership check dead. Re-read the row through the ORM instead of
handing that shape to a helper expecting the public one.
Extract `syncProjectAccessForProject` as the single owner of "what should this
project's access rows look like", shared by the migration and
`backfillProjectAccess`, so a deployment repaired by the migration and one
rebuilt in a test cannot drift. This also fixes a latent bug in the old backfill,
which capped memberships globally at 1000 rather than per project.
Test drives the migration's own `migrateOne` over raw docs: reads are empty
before, correct after, re-running does not duplicate, `down` empties. Verified
armed by neutering the migration and watching it fail.
* fix(example): back fill project access from both source tables
The backfill read each project's memberships in one `limit: 1000` call. Nothing
caps a project's member count -- `addMember` only rejects the owner, a duplicate,
and a non-owner caller -- and `findMany` truncates silently rather than throwing
(probe: 5 rows, `limit: 3`, returned 3, no error). Members past the bound would
get no access row, and `list`, `hasAny` and `listForDropdown` would hide a
project they belong to, permanently and with the migration reporting success.
Split the backfill into one migration per source table: owners over `projects`,
memberships over `projectMembers`. The runner pages each table itself, so neither
is bounded by a limit this code picks.
That shape is also what keeps it correct at runtime. A cursor loop inside
`migrateOne` would issue a second `.paginate()` inside the runner's own paginated
read, which real Convex rejects with `MultiplePaginatedDatabaseQueries` while
convex-test does not model it -- it would have passed tests and failed in prod.
`backfillProjectAccess` is now explicitly test-only and reads `limit + 1` so it
throws on truncation instead of quietly backfilling a prefix.
Tests drive both migrations over raw `ctx.db` docs and cover every member row;
both were verified armed by neutering each migration in turn.
* fix(example): correct project access rollback, dropdown capacity, and bounds docs
Four review findings on #452, all verified against source.
Member-migration rollback deleted owner-derived rows. An owner can also hold a
`projectMembers` row -- `aggregateDemo` seeds exactly that -- so `migrate down
--steps 1` removed the access row the still-applied owner migration created, and
the owner lost their own project from every read. The member `down` now leaves
the row alone when the user is the project's current owner.
Dropdown lost capacity. `listProjectsForDropdown` bounds the union with one
limit, where the read it replaced bounded each side separately: up to 1000 owned
plus up to 1000 member. A user with 750 of each silently lost 500 from an
unpaginated dropdown. The caller now passes the combined figure.
`syncProjectArchived`'s comment was wrong. It claimed the update was "bounded by
the project's member count, which addMember already gates"; `addMember` caps
nothing. The real ceiling is the ORM's `mutationMaxRows` (10,000), above which
the update throws. Comment corrected to state that; a member cap or a batched
sync is a product decision, not one to make here.
Rollout window documented. `kitcn deploy` pushes the backend and only then runs
migrations, so a deployment holding existing projects shows empty lists until
both backfills finish. Acceptable for a re-seedable example app; the owner
migration now says so and points at expand/backfill/contract for apps where it
is not.
Both new tests verified armed by reverting the fix under them.
* Prepare project access rollout; retain canonical reads
* Record preparation PR ownership and proof
---------
Co-authored-by: MikeyZhang75 <[email protected]>
PR body check passesThe body now has its plan page, or its Why and Verification sections and a |
🐛 Fixes #443, after preparation #487.
🧭 Task plan: docs/plans/2026-09-05-443-projects-list-read-bounds.md
Why
projects.listwalked the wholeprojectstable and kept owner-or-member rows in JS. A viewer with few or only old projects drained the table, and every scanned row joined the subscription's read set. #487 added the indexedprojectAccessrows and their owner and member backfills. This PR switches the reads over to them:projects.listpages through the viewer's access rows (listAccessibleProjects). It keeps the archived-only filter, creation order and cursor pagination.listForDropdownandhasAnyread the same rows. Theproject_existencehelper and its test go away.No workflow or Vercel project deploys the example's Convex backend, so no deployed backend can serve these reads before its backfill finishes. The local dev backend runs the registered migrations on
kitcn dev.Verification
bunx vitest run convex/orm/example-project-access-reads.test.ts convex/orm/example-project-lifecycle.test.ts convex/orm/example-tag-merge-reads.test.ts: 22 passed. The suites run the real example ORM and the registered authenticated handlers.bun lint,bun typecheck: exit 0.kitcn codegenregeneratedprocedure-names.gen.ts.Beyond the ask: none