Skip to content

Phase 1 evidence loop: SEC facts, evidence drafts, verification and review gate - #5

Merged
axlezhao merged 7 commits into
mainfrom
feat/sec-normalize
Sep 28, 2026
Merged

axlezhao merged 7 commits into
mainfrom
feat/sec-normalize

Conversation

@axlezhao

@axlezhao axlezhao commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Completes the Phase 1 gate in docs/PRODUCT_ARCHITECTURE_PLAN.md §14: a user can go from a research job to a reviewable, cited draft without any model, and nothing with an open blocking issue can be approved or published.

  1. SEC normalization (lib/research/normalize.ts): the runner used to fetch SEC XBRL companyfacts and discard it. It now keeps revenue, net income, diluted EPS, operating cash flow and shares outstanding as versioned FACT evidence, each cited to accession, form, filing date, concept, unit and period. Values filed on or after as_of are excluded; restatements, year-to-date durations and multi-class ambiguities are handled explicitly.
  2. Deterministic evidence draft (lib/research/draft.ts, POST/GET /api/v1/research/:jobId/draft): one draft per job, every number linked to an evidence_id, gaps listed as unknowns, and thesis, scenarios and falsifiers left empty for a person. Idempotent under concurrent requests.
  3. Verification (lib/research/verify.ts, migration 0008): blocking checks cover evidence existence, ownership and kind; number/unit/period/citation tie-out; per-metric units; citation completeness; and the as_of cutoff. Warnings cover EPS cross-check divergence and every draft unknown. Blocking issues close only on a passing re-check; editors acknowledge warnings with a note.
  4. Review (app/research-draft.tsx, /draft/review): a draft panel in the result view. Platform approval and publish refuse any version with an open blocking issue and any research draft that was never verified; only owners approve. Approval requests only move draft versions into review, and generic artifact.save can no longer overwrite or forge research drafts.
  5. Gate: tests/phase1-gate.test.ts runs the full loop through HTTP routes. pnpm local:verify:e2e (already in CI) now builds, verifies, reviews and publishes a draft on the real Worker and local D1.

No LLM is called anywhere.

Verification

  • pnpm run lint
  • pnpm run typecheck
  • pnpm test: 177 unit tests (130 before) plus build and SSR check
  • Relevant regression tests added: sec-normalize, research-draft, research-verify, research-review, phase1-gate. Key guards were checked by deliberately breaking them and confirming the tests fail.
  • pnpm db:local:migrate, pnpm db:local:verify-integrity (now including the two new tenant guards) and pnpm local:verify:e2e pass locally
  • Clicked through create draft → submit → approve → publish in the browser against the local fixture server

Data and security impact

  • Migration 0008_verification_issues.sql: adds the verification_issues table, research_artifact_versions.verified_at / verifier_version, and research_artifacts.research_job_id with a partial unique index. It only adds columns and tables, and it includes tenant-guard triggers for issues and draft-to-job links.
  • Behavior change to existing platform actions: approval.request now requires a draft version, approval.decide (approve) and artifact.publish go through the verification gate, and artifact.save rejects research-draft targets and evidence_draft content.
  • Timestamps: SEC facts are filtered by filing date strictly before as_of.
  • No new outbound requests, secrets or credentials. Fixture data is synthetic and uses .invalid URLs.

Documentation and limitations

  • README (EN/ZH), roadmap, architecture plan, architecture map, development, operations and fixture guides updated
  • Changelog updated
  • No credentials, private user data or restricted fixtures committed
  • Remaining gaps documented:
    • validated on synthetic fixtures only, not yet on real SEC data;
    • point-in-time filtering covers SEC facts only;
    • a clean check proves consistency with stored evidence, not source accuracy;
    • thesis and falsifier authoring and version diffs are the next step.

Rollout: apply migration 0008 before deploying. Snapshots from before this change cannot become drafts (409); re-run the job.

axlezhao and others added 7 commits September 27, 2026 23:37
The runner fetched SEC XBRL companyfacts and discarded it. It now
normalizes revenue, net income, diluted EPS, operating cash flow and
shares outstanding into facts that cite the accession, form, filing
date, concept, unit and period, excludes values filed on or after the
as_of date, and stores each fact as versioned FACT evidence. The
snapshot gains `secFacts` (facts + issues) without the raw payload.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
POST /api/v1/research/:jobId/draft turns a succeeded job's snapshot
into version 1 of a research artifact draft without any model call.
Every SEC fact links to its evidence row and citation; market data and
expectations stay in separate sections; gaps are listed as unknowns;
thesis, scenarios and falsifiers are left for a person to author.
Creation is idempotent, audited once, and refused when cited evidence
is outside the job's workspace and security. Snapshots now record the
evidence rows each job wrote.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every new draft is checked in the same batch that creates it. Blocking
checks cover missing, foreign or wrong-kind evidence, number/unit/
period/citation tie-out against the stored evidence row, units per
metric, incomplete citations, and filings or periods at or past the
as_of cutoff. Warnings cover cross-source EPS divergence and every
draft unknown. Issues are stored in the new verification_issues table
(migration 0008, tenant-guarded) with stable fingerprints, so re-checks
reopen, keep or resolve them idempotently. Blocking issues close only
on a passing re-check; editors may acknowledge warnings with a note.
Draft artifacts now link to their research job.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The research result view gains an evidence draft panel: create the
draft, see the blocking/warning gate, acknowledge warnings with a note,
re-run checks, read each SEC fact with its filing link, accession and
evidence id, and submit, approve, request changes, reject or publish.

POST /api/v1/research/:jobId/draft/review delegates to the platform
approval and publish actions, which now refuse approval or publication
while a version has an open blocking issue, and refuse a research draft
that was never verified. Approval requests only move draft versions
into review, and generic artifact saves can no longer write to or forge
research drafts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
tests/phase1-gate.test.ts drives only HTTP route handlers from a
fixture research job to a verified, fully cited draft, through human
approval to publication, and shows that a draft whose as_of predates
every filing, or whose evidence stops tying out, cannot be approved.
`pnpm local:verify:e2e` (run in CI) now performs the same review loop
against the real Worker and local D1, including refusing to publish
before approval, and the D1 integrity script checks the new draft-job
and verification-issue tenant guards.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
README (English and Chinese), roadmap, architecture plan, architecture
map, development, operations and local fixture guides now describe SEC
normalization, deterministic evidence drafts, verification issues and
the owner-approved publish gate, what they do and do not prove, and
what comes next. The changelog records steps 3 to 5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The integrity-check descriptions in both READMEs, the development guide
and the local fixture guide now include the draft-job and
verification-issue tenant guards, list the Phase 1 test files, and state
that the fixture run proves the draft review loop but not normalization
of real SEC data.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@axlezhao
axlezhao merged commit 7759dcf into main Sep 28, 2026
1 check passed
axlezhao added a commit that referenced this pull request Sep 28, 2026
Phase 1 evidence loop: SEC facts, evidence drafts, verification and review gate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant