[kriszyp]

⊙ Problem

Since August, 87 of 124 merged documentation PRs were companions to a harper or harper-pro PR, about 90% of them to harper. Each user-facing change takes two PRs, a companion-check gate, and a second review.

🤔 Should the v5 reference and release notes live in the code repos?

Proposed yes: a code PR then carries its own docs and gets one review. Reverting is three PR reverts.

💡 Solution

• The v5 reference and the 5.4 release notes leave this repo for docs/ in harper and harper-pro.
• A prebuild step overlays both repos' docs/reference/ and builds the 5.4 page from one file per change; the sidebar is generated and edit links open the owning repo.
• The site publishes harper-pro's latest published release, with harper at its core pointer (main until 5.4.0 ships docs/); a docs/<release>-<n> tag publishes a correction sooner. Either one triggers a redeploy.
• learn/, fabric/, the v4 reference and release notes before 5.4 stay here.

flowchart LR
  P["harper-pro docs/ @ published release<br/>or docs/ tag"] --> A[assemble-content.mjs]
  P -- "core pointer" --> H["harper docs/"] --> A
  A --> R["reference/ + 5.4 page"] --> B[docusaurus build]
  S["this repo: learn/, fabric/, v4, older notes"] --> B
  X["harper-pro: release published,<br/>docs/ tag pushed"] -- repository_dispatch --> C
  C["content-check (also hourly)"] -- "docs/ tree changed" --> D[deploy Pages + Fabric]

Loading

⚖️ Alternatives

• Everything in harper — harper-pro changes (about 10%) would still need companions.
• Poll only — no credential, but a release waits for the next poll; the dispatch reuses harper-pro's HarperFast AI App token.
• Release tag — pushed before approval (v5.3.5: 54 minutes early).
• Release branch — can show a fix's docs before it ships; docs/ tags are the opt-in version.
• Code repos as submodules here — docs lag until the pointer is bumped.

🔧 Changes

• Assembly — assemble-content.mjs reads each repo from a local checkout or a sparse fetch of harper-pro's latest published release or its highest docs/ tag (main until a release contains docs/, but never for a docs/ tag), with harper at harper-pro's core pointer or its local core/, and fails when two repos provide the same page. It collects _redirects.json from the code repos and builds each open minor's page from fragments, refusing a minor also committed here. It writes nothing unless both parts assemble, and only changed files, so the dev server reloads only edited pages. It publishes content-sources.json, refuses a checkout that still tracks reference/, and in --watch mode runs the dev server and stops it when watching fails.
• Site config — docusaurus.config.ts reads the assembly manifest for reference edit links, release-note edit links and code-repo redirects, and serves content-sources.json. sidebarsReference.ts is now one autogenerated entry.
• Deploys — content-check.yaml compares the docs/ trees the assembler would publish with the live site and dispatches both deploys, waiting while a deploy is unfinished. It runs on a docs-source-changed dispatch from harper-pro, and hourly, one run at a time. deploy.yaml builds with a read-only token and no persisted credential. deploy.yaml, deploy-fabric.yaml and pr-preview.yaml trigger on the assembler instead of reference/.
• Checks — validate.yaml adds a Site Build check and drops the example-test step. claude-review.yml and gemini-review.yml tell the reviewers where the reference now lives.
• Package — package.json runs the assembler before dev, build and start, adds js-yaml for front matter, and drops test:examples; package-lock.json records it. .gitignore ignores the generated paths.
• Contributor docs — AGENTS.md and CONTRIBUTING.md say where the reference lives, how to preview a local checkout, and where v5 redirects go.
• Removed — reference/ and release-notes/v5-lincoln/5.4.md move to harper and harper-pro (file list below). nested-relationship-examples.test.mjs, github-fixture-fetch.test.mjs and the fixture copy (github-fixture-fetch.mjs, resources.js, schema.graphql) move to harper, which checks the docs against the integration fixture directly.

Moved files (100)

• reference/analytics/operations.md
• reference/analytics/overview.md
• reference/backups/operations.md
• reference/backups/overview.md
• reference/cli/authentication.md
• reference/cli/commands.md
• reference/cli/operations-api-commands.md
• reference/cli/overview.md
• reference/components/applications.md
• reference/components/extension-api.md
• reference/components/javascript-environment.md
• reference/components/module-loading.md
• reference/components/nextjs.md
• reference/components/overview.md
• reference/components/plugin-api.md
• reference/components/scheduler.md
• reference/configuration/debugging.md
• reference/configuration/operations.md
• reference/configuration/options.md
• reference/configuration/overview.md
• reference/database/api.md
• reference/database/compaction.md
• reference/database/data-loader.md
• reference/database/jobs.md
• reference/database/overview.md
• reference/database/schema.md
• reference/database/storage-algorithm.md
• reference/database/storage-tuning.md
• reference/database/system-tables.md
• reference/database/transaction.md
• reference/environment-variables/encrypted-values.md
• reference/environment-variables/overview.md
• reference/fastify-routes/overview.md
• reference/full-text-search/configuration.md
• reference/full-text-search/overview.md
• reference/full-text-search/querying.md
• reference/graphql-querying/overview.md
• reference/http/api.md
• reference/http/configuration.md
• reference/http/overview.md
• reference/http/tls.md
• reference/index.md
• reference/legacy/cloud.md
• reference/legacy/custom-functions.md
• reference/logging/api.md
• reference/logging/configuration.md
• reference/logging/operations.md
• reference/logging/overview.md
• reference/mcp/cli.md
• reference/mcp/configuration.md
• reference/mcp/migration.md
• reference/mcp/overview.md
• reference/mcp/subscriptions.md
• reference/mcp/tool-metadata.md
• reference/mcp/tools-and-resources.md
• reference/models/analytics.md
• reference/models/api.md
• reference/models/backends.md
• reference/models/calibration.md
• reference/models/local-development.md
• reference/models/overview.md
• reference/models/routing.md
• reference/models/tool-calling.md
• reference/mqtt/configuration.md
• reference/mqtt/overview.md
• reference/operations-api/operations.md
• reference/operations-api/overview.md
• reference/operations-api/sql.md
• reference/replication/clustering.md
• reference/replication/overview.md
• reference/replication/residency.md
• reference/replication/sharding.md
• reference/resources/overview.md
• reference/resources/query-optimization.md
• reference/resources/resource-api.md
• reference/rest/content-types.md
• reference/rest/headers.md
• reference/rest/overview.md
• reference/rest/querying.md
• reference/rest/server-sent-events.md
• reference/rest/websockets.md
• reference/security/api.md
• reference/security/basic-authentication.md
• reference/security/certificate-management.md
• reference/security/certificate-verification.md
• reference/security/configuration.md
• reference/security/impersonation.md
• reference/security/jwt-authentication.md
• reference/security/mtls-authentication.md
• reference/security/overview.md
• reference/security/response-headers.md
• reference/security/secrets.md
• reference/static-files/overview.md
• reference/studio/overview.md
• reference/users-and-roles/configuration.md
• reference/users-and-roles/operations.md
• reference/users-and-roles/overview.md
• reference/web-application-firewall/configuration.md
• reference/web-application-firewall/operations.md
• reference/web-application-firewall/overview.md

✅ Verification

• Built the site from the three branches and diffed it against a build of main: all 1,806 pages match, except two code blocks in resource-api that harper's prettier 3.9 formats differently. Sidebar data, tables of contents and heading anchors are identical.
• Exercised fetch mode, release selection (v5.3.5 found through the Releases API, an unknown tag 404s, main while v5.3.5 has no docs/), the docs/ tag pick against a scratch repo (-10 over -2, v5.3.50-1 excluded), harper from the core pointer and from a local core/, dev reload of an added page, CRLF fragments, and a failing fragment that leaves the last assembly intact.
• format:check, lint, typecheck and test:workflows pass.

Cutover order

Paired with harper#3944 and harper-pro#1237.

1. Pause merges here and port the open docs PRs afterwards (22 open today).
2. Regenerate the harper and harper-pro content commits from the latest main here (the migration is scripted), and rebase the hand-written commits.
3. Merge the harper and harper-pro PRs with a merge commit, which keeps the imported history. Their Docs check fails until step 4, on purpose.
4. Merge this PR. Its Site Build check passes once step 3 is on both mains.

_{🤖 Generated by Claude Code (Claude Opus 5.5); posted via @kriszyp.}

Related PRs: #568 overlaps (each open docs PR here edits a page this PR moves; it is ported to harper or harper-pro after merge), #723 overlaps, #733 overlaps, #744 overlaps, #747 overlaps, #749 overlaps, #750 overlaps, #756 overlaps, #758 overlaps, #759 overlaps, #760 overlaps, #762 overlaps, #772 overlaps, #777 overlaps, #778 overlaps, #779 overlaps, #783 overlaps, #784 overlaps, #786 overlaps, #787 overlaps, #789 overlaps, #780 overlaps (merged; its changes are in the import), #794 overlaps, #795 overlaps, #790 overlaps (merged after the import; the cutover re-import carries it), #791 overlaps (merged after the import; the cutover re-import carries it), #793 overlaps (merged after the import; the cutover re-import carries it)
Complexity: medium

Deferred findings

• 06b36a0f00f2bf93 major reliability — scripts/assemble-content.mjs — default builds throw until upstream docs/ trees land — NOT FILED
• 8b912b9ef6495ad2 minor reliability — .github/workflows/content-check.yaml — failed live-manifest fetch is treated as changed and forces a redeploy — NOT FILED
• e7033c0d32a1d458 minor test-coverage — .github/workflows/validate.yaml — documented-example and fixture-parity tests deleted from this repo — NOT FILED
• 0f199c8b8bae7b2f minor reliability — .github/workflows/content-check.yaml — scheduled workflow auto-disables after 60 days of inactivity — NOT FILED
• 187f9da91d997905 nit maintainability — scripts/assemble-content.mjs — _redirects.json entries are not shape-validated at assembly — NOT FILED
• 824a7ddfe2e22248 nit reliability — scripts/assemble-content.mjs — Windows shell:true kill leaves docusaurus child orphaned — NOT FILED
• f8af58306f549685 nit documentation — docusaurus.config.ts — edit links target main while the page shows a release tag — NOT FILED

🤔 defer this PR's major reliability finding 06b36a0f00f2bf93 (default builds throw until upstream docs/ trees land) to a separate issue instead of fixing it here?

Late Grok findings

17 surviving (highest major) — Grok reviewed 92e1979, ruled against 3ba8455

• major — .github/workflows/deploy.yaml:149 — Concurrency is split across the build and deploy jobs, both with cancel-in-progress: true. Either of two triggers drops newer content: an older run's deploy cancels a newer build, or the next 15-minute schedule tick reads the live content-sources.json before the deploy in flight updates it, (previously adjudicated)
• minor — .github/workflows/deploy.yaml:57 — When the curl fetch fails, || echo '{}' makes the step read it as a change and forces a full build, deploy and skills dispatch. A failed gh api call turns the job red (previously adjudicated)
• minor — .github/workflows/deploy-fabric.yaml:8 — The Fabric deploy has no schedule and does not watch scripts/assemble-content.mjs. Changes to docs/ in harper or harper-pro never reach the Fabric production site (previously adjudicated)
• minor — .github/workflows/deploy.yaml:69 — The build job compiles MDX fetched from harper and harper-pro main without any review in this repo. It sets no permissions: block, and checkout keeps its credential for later steps
• minor — docusaurus.config.ts:35 — The config reads .content/manifest.json unconditionally. npm run serve from scripts/preview-pr.mjs, and Docusaurus commands run without the prebuild hook, fail with ENOENT on a clean checkout (previously adjudicated)
• minor — docusaurus.config.ts:100 — The reference editUrl reads a manifest captured when the config loads. A reference page added during npm run dev causes a TypeError on reload (previously adjudicated)
• minor — CONTRIBUTING.md:68 — This line still tells contributors to run npm run test:examples and says the Lint Check job runs it. The script no longer exists (previously adjudicated)
• minor — .github/workflows/validate.yaml:44 — The documented-example and fixture-parity tests are deleted, and nothing visible replaces them (previously adjudicated)
• minor — .github/workflows/pr-preview.yaml:6 — The preview path filter leaves out scripts/assemble-content.mjs and still lists reference/**, which can no longer change (previously adjudicated)
• minor — .github/workflows/deploy.yaml:30 — GitHub disables the cron after 60 days with no repo activity. Reference deploys would then stop with no alert (previously adjudicated)
• minor — .github/workflows/validate.yaml:20 — Required PR checks never run the production build
• nit — docusaurus.config.ts:36 — Edit links always point at main. An assembled release-note page links to the _index.md of the repo that supplied the index, even for a paragraph that came from a harper-pro fragment (previously adjudicated)
• nit — package.json:10 — The dev watcher's lifecycle: & under Windows cmd, a watcher left running after Docusaurus exits, no fs.watch error listener, and assembly errors that are only logged (previously adjudicated)
• nit — scripts/assemble-content.mjs:95 — The front-matter regex only matches \n line endings. On a CRLF _index.md, the front matter goes into the page body and sections is ignored. On a CRLF fragment, the build fails with "has no section"
• nit — scripts/assemble-content.mjs:122 — Entries in _redirects.json are not shape-checked during assembly (previously adjudicated)
• nit — scripts/assemble-content.mjs:134 — syncFiles overwrites or deletes reference/ files on branches that still track that directory (previously adjudicated)
• nit — scripts/assemble-content.mjs:141 — The added map-shape comment narrates the code below it, and deploy.yaml:74 misstates when cancellation happens (previously adjudicated)

Not standing: none. Grok ran 16m (16m from launch to report); the fold adjudication (cursor-grok@92e197920343-ff2d164f, claude) took 2m.

Late Grok findings

1 surviving (highest nit), 10 previously adjudicated — Grok reviewed 67d8f6c

• major — .github/workflows/deploy-fabric.yaml:60 — upstream MDX runs during npm run build in the same job that later holds HARPER_PRODUCTION_PASSWORD (previously adjudicated)
• minor — scripts/assemble-content.mjs:83 — the release list reads only the first page of 100 releases (previously adjudicated)
• minor — scripts/assemble-content.mjs:133 — a docs/<release>-<n> correction tag with no docs/ makes the site fall back to main (previously adjudicated)
• minor — .github/workflows/content-check.yaml:45 — run history is capped at 20 before filtering, so a still-running rerun older than that is missed and the new dispatch cancels it (previously adjudicated)
• minor — .github/workflows/content-check.yaml:53 — freshness is checked only against the Pages stamp, so a failed Fabric deploy is never retried (previously adjudicated)
• minor — .github/workflows/content-check.yaml:39 — when DOCUSAURUS_URL is unset, SITE_URL is / and every tick dispatches both deploys (previously adjudicated)
• minor — .github/workflows/content-check.yaml:64 — a build that always fails is re-dispatched every hour and on every docs-source-changed event (previously adjudicated)
• nit — scripts/assemble-content.mjs:150 — fetching harper at the commit the core submodule pins fails with a raw git error if GitHub can't serve that commit
• nit — scripts/assemble-content.mjs:150 — when the core gitlink is missing, the failure is a raw git rev-parse error instead of the missing() message (previously adjudicated)
• nit — .github/workflows/deploy.yaml:60 — GH_TOKEN is in the environment of the step that runs upstream MDX (previously adjudicated)
• nit — scripts/assemble-content.mjs:280 — during npm run dev the manifest is rewritten in place, so a concurrent readContent can parse half-written JSON (previously adjudicated)

Not standing: 1 accepted-known. Grok ran 15m (15m from launch to report); the fold adjudication (cursor-grok@67d8f6c78d51-72d9b2d1, claude) took 1m.

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=9; full=3 @ 5944e16}

_{Review-Attention: deep ~35m (sensitive: content-check.yaml, deploy.yaml; decisions: tags-then-release-by-tag, harper-from-core-gitlink, publish-latest-release-plus-docs-tags, push-plus-hourly-poll, do-less-alternative, deferred-06b36a0f) @ 5944e16}

[gemini-code-assist]

Code Review

This pull request transitions the documentation repository to an assembled model, where the v5 reference and release notes are dynamically fetched and assembled from the harper and harper-pro repositories during the build process. Consequently, numerous static markdown files have been deleted, and docusaurus.config.ts has been updated to read a generated manifest to dynamically resolve edit URLs and client redirects. The code review identified two critical issues in docusaurus.config.ts where a missing or malformed manifest file, or an undefined reference lookup, could cause the Docusaurus build or startup to crash. Implementing robust parsing and optional chaining as suggested will prevent these failures.

[gemini-code-assist]

If .content/manifest.json is missing, contains invalid JSON, or lacks expected properties (like reference, releaseNotes, or redirects), readContent can throw unhandled exceptions or return undefined for properties, leading to crashes during Docusaurus startup or build. Making this helper more robust ensures smoother local development and CI runs.

const readContent = (): ContentManifest => {
	if (!existsSync('.content/manifest.json')) {
		return { reference: {}, releaseNotes: {}, redirects: [] };
	}
	try {
		const data = JSON.parse(readFileSync('.content/manifest.json', 'utf8'));
		return {
			reference: data?.reference || {},
			releaseNotes: data?.releaseNotes || {},
			redirects: data?.redirects || [],
		};
	} catch {
		return { reference: {}, releaseNotes: {}, redirects: [] };
	}
};

[gemini-code-assist]

If readContent().reference[docPath] is undefined (e.g., on a clean checkout or if the document path is not yet in the manifest), accessing .repo directly will throw a TypeError and crash the build. Using optional chaining and falling back to the default repository prevents this crash.

					const repo = readContent().reference[docPath]?.repo;
					return repo
						? contentEditUrl(repo, 'reference/' + docPath)
						: 'https://github.com/HarperFast/documentation/blob/main/reference/' + docPath;