Skip to content

docs: generate llms.txt and Markdown versions of each page - #1010

Open
jonathannorris wants to merge 2 commits into
mainfrom
docs-generate-llms-txt
Open

jonathannorris wants to merge 2 commits into
mainfrom
docs-generate-llms-txt

Conversation

@jonathannorris

@jonathannorris jonathannorris commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Adds @signalwire/docusaurus-plugin-llms-txt to generate llms.txt, llms-full.txt, and a .md copy of each doc page, so AI tools can read the docs without scraping HTML. Same approach as feat: generate llms.txt and per-page markdown open-feature/openfeature.dev#1579.
  • Adds a rehype cleanup step for the converted Markdown (heading permalinks, code block languages, tab labels, admonitions, card lists, YouTube embeds) and works around a few plugin bugs with /path/.md links and absolute links.
  • Excludes the Redoc API pages (linked as OpenAPI specs under ## Optional instead), the SDK example app pages, and the generated dvc command reference, to keep the output focused.

Examples

@dynatrace-llc-secret-protection dynatrace-llc-secret-protection Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Daphne (fueled by kingfisher) detected 6 potential secrets in files touched by this PR.

If any finding is a real credential (not a sample or placeholder), treat it as compromised. It's no longer confidential the moment it lands in git.

🔴 verified live · ⚪ unverified — the engine could not validate it, it may still be live

Found in touched files outside this PR's changes

Daphne scans each touched file in full, so a finding here may predate this PR — open blame to see who introduced the line.

Secret File
⚪ DevCycle Server SDK Key docs/cli-guides/environments.md:118
⚪ DevCycle Client SDK Key docs/cli-guides/environments.md:43
⚪ DevCycle Server SDK Key docs/cli-guides/environments.md:51
⚪ DevCycle Mobile SDK Key docs/cli-guides/environments.md:35
⚪ DevCycle Mobile SDK Key docs/cli-guides/environments.md:102
⚪ DevCycle Client SDK Key docs/cli-guides/environments.md:110

What to do

  1. No need to file a Jira ticket. If a finding is confirmed live (🔴), it's already tracked confidentially in the DTSEC Jira project. Tracked isn't rotated — the steps below are still yours.
  2. Rotate or revoke it now at the provider, and update whatever consumes it.
  3. If this branch introduced it, purge it from history. Deleting the file or force-pushing won't remove it — it stays in every clone and the reflog. Rewrite history with git filter-repo, then force-push. See GitHub: removing sensitive data.

Daphne alerts, it doesn't gate — it never blocks a merge or requests changes. Questions? → #dev-help-security

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Deploying devcycle-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 95c882e
Status: ✅  Deploy successful!
Preview URL: https://22574a59.devcycle-docs.pages.dev
Branch Preview URL: https://docs-generate-llms-txt.devcycle-docs.pages.dev

View logs

@jonathannorris
jonathannorris marked this pull request as ready for review October 15, 2026 19:40
@jonathannorris
jonathannorris requested review from cdruxerman, jsalaber, kaushalkapasi, leslie-lau and suthar26 and a balanced review from Copilot October 15, 2026 19:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The custom conversion pipeline and routing workarounds need generated-output verification before final human approval.

0 open findings

What changed in this PR

Adds AI-readable exports to the Docusaurus documentation site.

Changes:

  • Generates llms.txt, llms-full.txt, and per-page Markdown with route grouping and exclusions.
  • Cleans converted content and repairs Markdown links.
  • Corrects documentation links, formatting, and metadata.
File Description
yarn.lock Locks new plugin dependencies.
plugins/​rehype-llms-cleanup.js Normalizes converted content and links.
plugins/​llms-txt.js Post-processes generated Markdown and indexes.
package.json Adds the export plugin.
docusaurus.config.js Configures exports and improves README heading handling.
docs/​sdk/​server-side-sdks/​python/​python-gettingstarted.md Adds spacing before a wizard marker.
docs/​sdk/​server-side-sdks/​node/​node-typescript.md Corrects CLI links.
docs/​sdk/​server-side-sdks/​nestjs/​nestjs-typescript.md Corrects CLI links.
docs/​sdk/​client-side-sdks/​react/​react-typescript.md Corrects CLI links.
docs/​sdk/​client-side-sdks/​nextjs/​nextjs-typescript.md Corrects CLI links.
docs/​sdk/​client-side-sdks/​javascript/​javascript-typescript.md Corrects CLI links.
docs/​integrations/​vercel-edge-config.md Corrects SDK and quickstart links.
docs/​integrations/​index.mdx Simplifies GitHub link wording.
docs/​index.mdx Marks the hero image as decorative.
docs/​best-practices/​security/​compromised-keys.md Fixes a description typo.
.yarnrc.yml Supplies the plugin’s undeclared logger dependency.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@suthar26

Copy link
Copy Markdown
Contributor

checked agaisnt https://docs-generate-llms-txt.devcycle-docs.pages.dev/llms.txt and the output looks good

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants