Skip to content

fix(example): bound projects.list reads to the page, not the table - #452

Merged
zbeyens merged 11 commits into
mainfrom
fix/project-list-read-bounds
Nov 12, 2026
Merged

zbeyens merged 11 commits into
mainfrom
fix/project-list-read-bounds

Conversation

@MikeyZhang75

@MikeyZhang75 MikeyZhang75 commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator

🐛 Fixes #443, after preparation #487.

🧭 Task plan: docs/plans/2026-09-05-443-projects-list-read-bounds.md

Why

projects.list walked the whole projects table and kept owner-or-member rows in JS. A viewer with few or only old projects drained the table, and every scanned row joined the subscription's read set. #487 added the indexed projectAccess rows and their owner and member backfills. This PR switches the reads over to them:

  • projects.list pages through the viewer's access rows (listAccessibleProjects). It keeps the archived-only filter, creation order and cursor pagination.
  • listForDropdown and hasAny read the same rows. The project_existence helper and its test go away.

No workflow or Vercel project deploys the example's Convex backend, so no deployed backend can serve these reads before its backfill finishes. The local dev backend runs the registered migrations on kitcn dev.

Verification

  • bunx vitest run convex/orm/example-project-access-reads.test.ts convex/orm/example-project-lifecycle.test.ts convex/orm/example-tag-merge-reads.test.ts: 22 passed. The suites run the real example ORM and the registered authenticated handlers.
  • bun lint, bun typecheck: exit 0. kitcn codegen regenerated procedure-names.gen.ts.

Beyond the ask: none

`projects.list` asked "owner OR member" of `projects`, which no index spans, so
it walked the table from the newest end and post-filtered in JS. A viewer with
nothing to see drained the whole table, and every scanned row joined the
subscription read set. Measured 40 scanned / 0 returned at 40 rows.

Two more defects were hiding behind it:

- `.filter()` hands the callback the raw Convex row, whose id field is `_id`, so
  `memberProjectIds.has(project.id)` was `has(undefined)` and always false.
  Projects visible only through a membership never appeared in the list at all.
- `hasAnyProject` was unbounded too: `{ ownerId, archived }` ties `index('archived')`
  and `index('ownerId')` at the same planner score, declaration order wins, and the
  loser becomes a post-filter over the whole partition. 40 and 43 reads against
  bounds of 2 and 3.

Add `projectAccess`, one row per (user, project), indexed
`(userId, archived, projectCreatedAt)`. `_helpers/project_access.ts` owns every
grant/revoke and is the single owner of "which projects can this user see", used
by `list`, `listForDropdown` and `hasAny`. 4 reads at 10 noise rows and 4 at 400.

Maintained by explicit mutation helpers, not schema triggers: async cascade
continuation workers build their ORM from the raw `ctx.db` and drop triggers past
the first batch, so a trigger-maintained copy would drift. Deletion needs neither,
both FKs cascade and cascades still run in those batches. `revokeProjectAccess`
re-derives entitlement rather than deleting blindly, and `grantProjectAccess`
derives `isOwner` from the project so a caller cannot mislabel a row.

Also repair the read-bound harness, which was measuring nothing:

- `countDocumentReads` patched `db.get` as a one-arg forwarder while the lifecycle
  writer only calls the two-arg overload, so `oldDoc`/`newDoc` came back null and
  every trigger was silently skipped.
- The example read-bound tests installed the counter after the ORM was built, so
  they measured a constant 0. Fixing placement exposed the `hasAnyProject` scan and
  a tag-merge bound that was really 16, not 12 (same at 40 and 120 target rows, so
  the property it pins still holds).

Bounds are asserted as equalities so a dead counter fails loudly.

Add compound `isPublic_archived` for the unauthenticated branch, which had the
same planner tie and read every archived public project.
@vercel

vercel Bot commented Sep 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
better-convex Error Error Nov 12, 2026 12:55pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Sep 6, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: ebcdd8b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f0275f179

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// One index walk over the viewer's own access rows. The previous read model
// scanned `projects` and post-filtered owner-or-member in JS, so it cost
// O(table) per page and every scanned row joined this subscription.
const results = await listAccessibleProjects(ctx, {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Populate access rows before switching reads

When this schema is deployed over existing project data, projectAccess starts empty, so this call makes projects.list return no projects; listForDropdown and hasAny fail similarly. A repo-wide search shows backfillProjectAccess is referenced only by its test, while migrations/manifest.ts remains empty, so there is no callable migration to populate existing rows. Wire and run the backfill before switching these read paths to the derived table.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Populate access rows before switching reads

The migrations are registered, but the activation-order defect is still present at 81b7f8e3cc2323296960f56f985f6817b0d835c1. A fresh public-helper replay with an existing owned project and an existing shared project returns an empty list, an empty dropdown and hasAny: false before backfill. The deploy command activates the backend before running migrations. Keeping this P1 open; a staged schema/write, backfill, then read-cutover rollout is required to preserve visibility without a transitional scan. No deployment or data reset was performed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Populate access rows before switching reads

Preparation is split into #487. This activation PR now targets its preparation branch and is draft. The preparation revision retains canonical public readers while owner/member backfills run. No target deployment or completed-backfill receipts have been supplied, so this finding remains open. Post-push source proof at58ff9132ed7de370d0fc688a2ff32049dc161ee3 passed all22 focused tests; that does not substitute for real rollout receipts.

`projectAccess` started empty on any deployment that already held projects, so
switching `list`, `listForDropdown` and `hasAny` onto it made all three return
nothing. `backfillProjectAccess` existed but was referenced only by its test and
`migrations/manifest.ts` was empty, so there was no callable path to populate it.

Add `20260906_022807_backfill_project_access`, scaffolded with `kitcn migrate
create` and registered in the manifest. It walks `projects` in batches and
re-derives each project's rows; `grantProjectAccess` updates rather than inserts,
so re-running is safe. `down` clears them.

The runner pages `ctx.db.query(table)`, so `migrateOne` receives the RAW Convex
document -- `_id`, not `id`, and no hydrated `createdAt`. Same trap that made the
old `.filter()` membership check dead. Re-read the row through the ORM instead of
handing that shape to a helper expecting the public one.

Extract `syncProjectAccessForProject` as the single owner of "what should this
project's access rows look like", shared by the migration and
`backfillProjectAccess`, so a deployment repaired by the migration and one
rebuilt in a test cannot drift. This also fixes a latent bug in the old backfill,
which capped memberships globally at 1000 rather than per project.

Test drives the migration's own `migrateOne` over raw docs: reads are empty
before, correct after, re-running does not duplicate, `down` empties. Verified
armed by neutering the migration and watching it fail.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0376934447

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The backfill read each project's memberships in one `limit: 1000` call. Nothing
caps a project's member count -- `addMember` only rejects the owner, a duplicate,
and a non-owner caller -- and `findMany` truncates silently rather than throwing
(probe: 5 rows, `limit: 3`, returned 3, no error). Members past the bound would
get no access row, and `list`, `hasAny` and `listForDropdown` would hide a
project they belong to, permanently and with the migration reporting success.

Split the backfill into one migration per source table: owners over `projects`,
memberships over `projectMembers`. The runner pages each table itself, so neither
is bounded by a limit this code picks.

That shape is also what keeps it correct at runtime. A cursor loop inside
`migrateOne` would issue a second `.paginate()` inside the runner's own paginated
read, which real Convex rejects with `MultiplePaginatedDatabaseQueries` while
convex-test does not model it -- it would have passed tests and failed in prod.

`backfillProjectAccess` is now explicitly test-only and reads `limit + 1` so it
throws on truncation instead of quietly backfilling a prefix.

Tests drive both migrations over raw `ctx.db` docs and cover every member row;
both were verified armed by neutering each migration in turn.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1387b4ccc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// One index walk over the viewer's own access rows. The previous read model
// scanned `projects` and post-filtered owner-or-member in JS, so it cost
// O(table) per page and every scanned row joined this subscription.
const results = await listAccessibleProjects(ctx, {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Backfill access before activating derived reads

On deployments containing existing projects, authenticated reads switch to projectAccess while it is still empty or partially populated. Fresh evidence after the earlier comment is the ordering in packages/kitcn/src/cli/commands/deploy.ts: the backend deployment completes at lines 50-62, and only afterward does runMigrationFlow start at lines 74-82. Consequently users see no or incomplete projects throughout both backfill migrations; use an expand/backfill/contract rollout or retain a fallback read until backfill completion.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backfill access before activating derived reads

Confirmed at published head 81b7f8e3cc2323296960f56f985f6817b0d835c1. The pre-backfill owned/shared-project replay fails: list and dropdown are empty and hasAny is false. This is not resolved by registering the migrations, because backend activation still precedes their execution. Keeping the P1 open and the PR unmerged; preserving visibility and the indexed-read bound requires a staged cutover, not accepting the empty-result window.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backfill access before activating derived reads

The maintainer approved separate deployable preparation #487 and dependent activation #452. This PR stays draft until every target records the exact preparation deployment OID and both owner/member migrations as applied/completed. Those receipts are still absent, so this P1 remains open. Full bun check and post-push22 focused tests pass; no fallback reader, readiness flag or empty-window waiver is used.

Comment on lines +140 to +143
await ctx.orm
.update(projectAccessTable)
.set({ archived: input.archived })
.where(eq(projectAccessTable.projectId, input.projectId));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bound archive synchronization to the supported row count

When a project has more than 10,000 access rows, archiving or restoring it always fails because this synchronous update collects every match and the ORM throws above its default mutationMaxRows of 10,000 (mutation-utils.ts:746,952-963). Contrary to the comment above this helper, projects.addMember imposes no member-count cap, so such projects are reachable through the public mutation path; either enforce a compatible cap or perform this synchronization in bounded batches.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bound archive synchronization to the supported row count

Deferred under the user's explicit P2 scope. The current synchronous archive update retains the 10,000-row ceiling and fails loudly rather than synchronizing only a prefix. No member cap or batched-sync contract was added, and this thread remains open.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bound archive synchronization to the supported row count

The maintainer explicitly deferred this P2 on2026-10-14 in the current review session. Archive/restore still synchronously mirror access rows and can exceed the10000-row mutation limit. No membership cap or asynchronous archive contract was added. This thread stays open, and both task plans record the deferral.

Comment thread example/convex/functions/projects.ts Resolved
… bounds docs

Four review findings on #452, all verified against source.

Member-migration rollback deleted owner-derived rows. An owner can also hold a
`projectMembers` row -- `aggregateDemo` seeds exactly that -- so `migrate down
--steps 1` removed the access row the still-applied owner migration created, and
the owner lost their own project from every read. The member `down` now leaves
the row alone when the user is the project's current owner.

Dropdown lost capacity. `listProjectsForDropdown` bounds the union with one
limit, where the read it replaced bounded each side separately: up to 1000 owned
plus up to 1000 member. A user with 750 of each silently lost 500 from an
unpaginated dropdown. The caller now passes the combined figure.

`syncProjectArchived`'s comment was wrong. It claimed the update was "bounded by
the project's member count, which addMember already gates"; `addMember` caps
nothing. The real ceiling is the ORM's `mutationMaxRows` (10,000), above which
the update throws. Comment corrected to state that; a member cap or a batched
sync is a product decision, not one to make here.

Rollout window documented. `kitcn deploy` pushes the backend and only then runs
migrations, so a deployment holding existing projects shows empty lists until
both backfills finish. Acceptable for a re-seedable example app; the owner
migration now says so and points at expand/backfill/contract for apps where it
is not.

Both new tests verified armed by reverting the fix under them.
@zbeyens
zbeyens marked this pull request as draft October 14, 2026 13:17
@zbeyens
zbeyens changed the base branch from main to codex/project-access-preparation October 14, 2026 13:17
@zbeyens

zbeyens commented Oct 14, 2026

Copy link
Copy Markdown
Member

Blocked activation review receipt for f8a2d41d595a74a8399aaf976905cc8c18e01e96.

Cleanup is pushed and source-verified. This is not a clean activation/merge receipt. PR452 is draft and depends on preparation #487 atb555b70406ffb79e486d24e50d36b596f805e6b1. The maintainer approved the split and holding activation until target receipts. No real remote deployment or merge performed.

Full bun check passed, including fixtures/verify/runtime. Three independent source lenses found no new source defect. Committed-branch autoreview at58ff9132 against preparation exited0 without actionable P0 findings. After the final material plan push, all22 focused tests passed, including resolved/outdated backfill P1 replay, fractional ordering,1500-project dropdown and registered authenticated lifecycle. Generated artifacts come from real isolated CLI codegen.

Two actionable rollout P1 findings remain OPEN. Every target still needs the exact preparation deployment OID plus both owner/member migrations applied=true and status=completed. No target receipts have been supplied. The completion checker intentionally fails on the outstanding receipt checkbox/open Closeout phase. Required checks are not reported for the dependent base; hosted CI is tracked separately.

The archive>10000-row P2 is explicitly deferred by the maintainer on2026-10-14 and stays OPEN. The dropdown capacity P2 is source-fixed, post-push-proven and RESOLVED.

Raw inventory before this receipt: 2 top-level comments, 13 reviews, 6 threads;3 unresolved (2 rolloutP1 plus1 deferredP2). Resolved/outdated threads included.

Feedback ledger:

Separate inherited ORM cursor-protocol follow-up: #488.

zbeyens added a commit that referenced this pull request Oct 14, 2026
* fix(example): bound projects.list reads to the page, not the table

`projects.list` asked "owner OR member" of `projects`, which no index spans, so
it walked the table from the newest end and post-filtered in JS. A viewer with
nothing to see drained the whole table, and every scanned row joined the
subscription read set. Measured 40 scanned / 0 returned at 40 rows.

Two more defects were hiding behind it:

- `.filter()` hands the callback the raw Convex row, whose id field is `_id`, so
  `memberProjectIds.has(project.id)` was `has(undefined)` and always false.
  Projects visible only through a membership never appeared in the list at all.
- `hasAnyProject` was unbounded too: `{ ownerId, archived }` ties `index('archived')`
  and `index('ownerId')` at the same planner score, declaration order wins, and the
  loser becomes a post-filter over the whole partition. 40 and 43 reads against
  bounds of 2 and 3.

Add `projectAccess`, one row per (user, project), indexed
`(userId, archived, projectCreatedAt)`. `_helpers/project_access.ts` owns every
grant/revoke and is the single owner of "which projects can this user see", used
by `list`, `listForDropdown` and `hasAny`. 4 reads at 10 noise rows and 4 at 400.

Maintained by explicit mutation helpers, not schema triggers: async cascade
continuation workers build their ORM from the raw `ctx.db` and drop triggers past
the first batch, so a trigger-maintained copy would drift. Deletion needs neither,
both FKs cascade and cascades still run in those batches. `revokeProjectAccess`
re-derives entitlement rather than deleting blindly, and `grantProjectAccess`
derives `isOwner` from the project so a caller cannot mislabel a row.

Also repair the read-bound harness, which was measuring nothing:

- `countDocumentReads` patched `db.get` as a one-arg forwarder while the lifecycle
  writer only calls the two-arg overload, so `oldDoc`/`newDoc` came back null and
  every trigger was silently skipped.
- The example read-bound tests installed the counter after the ORM was built, so
  they measured a constant 0. Fixing placement exposed the `hasAnyProject` scan and
  a tag-merge bound that was really 16, not 12 (same at 40 and 120 target rows, so
  the property it pins still holds).

Bounds are asserted as equalities so a dead counter fails loudly.

Add compound `isPublic_archived` for the unauthenticated branch, which had the
same planner tie and read every archived public project.

* docs(plans): record PR #452 in the 443 task plan

* fix(example): ship the projectAccess backfill as a real migration

`projectAccess` started empty on any deployment that already held projects, so
switching `list`, `listForDropdown` and `hasAny` onto it made all three return
nothing. `backfillProjectAccess` existed but was referenced only by its test and
`migrations/manifest.ts` was empty, so there was no callable path to populate it.

Add `20260906_022807_backfill_project_access`, scaffolded with `kitcn migrate
create` and registered in the manifest. It walks `projects` in batches and
re-derives each project's rows; `grantProjectAccess` updates rather than inserts,
so re-running is safe. `down` clears them.

The runner pages `ctx.db.query(table)`, so `migrateOne` receives the RAW Convex
document -- `_id`, not `id`, and no hydrated `createdAt`. Same trap that made the
old `.filter()` membership check dead. Re-read the row through the ORM instead of
handing that shape to a helper expecting the public one.

Extract `syncProjectAccessForProject` as the single owner of "what should this
project's access rows look like", shared by the migration and
`backfillProjectAccess`, so a deployment repaired by the migration and one
rebuilt in a test cannot drift. This also fixes a latent bug in the old backfill,
which capped memberships globally at 1000 rather than per project.

Test drives the migration's own `migrateOne` over raw docs: reads are empty
before, correct after, re-running does not duplicate, `down` empties. Verified
armed by neutering the migration and watching it fail.

* fix(example): back fill project access from both source tables

The backfill read each project's memberships in one `limit: 1000` call. Nothing
caps a project's member count -- `addMember` only rejects the owner, a duplicate,
and a non-owner caller -- and `findMany` truncates silently rather than throwing
(probe: 5 rows, `limit: 3`, returned 3, no error). Members past the bound would
get no access row, and `list`, `hasAny` and `listForDropdown` would hide a
project they belong to, permanently and with the migration reporting success.

Split the backfill into one migration per source table: owners over `projects`,
memberships over `projectMembers`. The runner pages each table itself, so neither
is bounded by a limit this code picks.

That shape is also what keeps it correct at runtime. A cursor loop inside
`migrateOne` would issue a second `.paginate()` inside the runner's own paginated
read, which real Convex rejects with `MultiplePaginatedDatabaseQueries` while
convex-test does not model it -- it would have passed tests and failed in prod.

`backfillProjectAccess` is now explicitly test-only and reads `limit + 1` so it
throws on truncation instead of quietly backfilling a prefix.

Tests drive both migrations over raw `ctx.db` docs and cover every member row;
both were verified armed by neutering each migration in turn.

* fix(example): correct project access rollback, dropdown capacity, and bounds docs

Four review findings on #452, all verified against source.

Member-migration rollback deleted owner-derived rows. An owner can also hold a
`projectMembers` row -- `aggregateDemo` seeds exactly that -- so `migrate down
--steps 1` removed the access row the still-applied owner migration created, and
the owner lost their own project from every read. The member `down` now leaves
the row alone when the user is the project's current owner.

Dropdown lost capacity. `listProjectsForDropdown` bounds the union with one
limit, where the read it replaced bounded each side separately: up to 1000 owned
plus up to 1000 member. A user with 750 of each silently lost 500 from an
unpaginated dropdown. The caller now passes the combined figure.

`syncProjectArchived`'s comment was wrong. It claimed the update was "bounded by
the project's member count, which addMember already gates"; `addMember` caps
nothing. The real ceiling is the ORM's `mutationMaxRows` (10,000), above which
the update throws. Comment corrected to state that; a member cap or a batched
sync is a product decision, not one to make here.

Rollout window documented. `kitcn deploy` pushes the backend and only then runs
migrations, so a deployment holding existing projects shows empty lists until
both backfills finish. Acceptable for a re-seedable example app; the owner
migration now says so and points at expand/backfill/contract for apps where it
is not.

Both new tests verified armed by reverting the fix under them.

* Prepare project access rollout; retain canonical reads

* Record preparation PR ownership and proof

---------

Co-authored-by: MikeyZhang75 <[email protected]>
@zbeyens
zbeyens changed the base branch from codex/project-access-preparation to main November 12, 2026 12:55
@zbeyens
zbeyens marked this pull request as ready for review November 12, 2026 12:55
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

PR body check passes

The body now has its plan page, or its Why and Verification sections and a Beyond the ask: line.

@zbeyens
zbeyens merged commit 3c4c7b6 into main Nov 12, 2026
11 of 13 checks passed

This branch had an error being deployed

1 failed deployment
Preview — ebcdd8bb Deployed Nov 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example: projects.list scans the whole table to authorize owner-or-member, and every scanned row joins the read set

2 participants