Repository navigation
ci/duplicates: Add duplicated content PR check - #2076
Open
marcleblanc2 wants to merge 4 commits into
Open
marcleblanc2 wants to merge 4 commits into
marcleblanc2 wants to merge 4 commits into
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Each fact should have one home in the docs, with other pages linking to it; copies drift out of date the moment the original changes. This adds a non-blocking PR check that comments the passages a PR adds which say the same thing as a passage elsewhere under `docs/`, paraphrased or verbatim, plus an inline comment on each added passage pointing at its duplicates. `dev/check-duplicates.mjs` splits every page into passages (paragraphs, list items, tables, fenced code blocks), normalizes them, and compares word trigrams by Jaccard similarity. Auto-generated pages and SCHEMA_SYNC blocks are skipped. Like the links and redirects checks, CI records the findings on the merge base and reports only the ones the PR adds; main has hundreds of pre-existing findings. AGENTS.md gains the "One home per fact" rules so the duplication does not grow back. Amp-Thread-ID: https://ampcode.com/threads/T-01a11326-478d-72cf-bbb7-587d16b378e9 Co-authored-by: Amp <[email protected]>
A PR that edits one copy of a duplicated passage now gets a comment naming the other copies, so the author can pick the home and replace the rest with links, or make the same change there. Added duplication and edited duplication are reported in separate sections with their own advice. Also strip trailing spaces from the example-hostnames bullet in AGENTS.md. Amp-Thread-ID: https://ampcode.com/threads/T-01a11326-478d-72cf-bbb7-587d16b378e9 Co-authored-by: Amp <[email protected]>
marcleblanc2
force-pushed
the
marc/dev/check-duplicates
branch
from
October 6, 2026 22:36
45935df to
f98c857
Compare
dev/check-duplicates.mjs, check-hostnames.mjs, check-redirects.mjs, and example-hostnames.json only run in PR checks; the site build never reads them, so a change to them should not deploy a preview. Amp-Thread-ID: https://ampcode.com/threads/T-01a11326-478d-72cf-bbb7-587d16b378e9 Co-authored-by: Amp <[email protected]>
Amp-Thread-ID: https://ampcode.com/threads/T-01a11326-478d-72cf-bbb7-587d16b378e9 Co-authored-by: Amp <[email protected]>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add PR check to compare PR changes by Jaccard similarity to existing doc content, to identify if the PR is adding duplicate content.
How
dev/check-duplicates.mjs: splits every page underdocs/into passages (paragraphs, list items, tables, fenced code blocks), normalizes them (lowercase; Markdown, JSX, link targets stripped), and compares word trigrams by Jaccard similarity, so paraphrases are caught, not only verbatim copies. Reports groups of similar passages asfile:lineplus a snippet, biggest groups first.--baselineand--diffscope it to what a PR adds, same ascheck-linksandcheck-redirects. Auto-generated pages andSCHEMA_SYNCblocks are skipped; those are fixed upstream..github/workflows/check-duplicates.yml: records the findings on the merge base, then comments the passages the PR adds that duplicate another, plus an inline review comment on each added passage naming its duplicates. Advisory, never fails the PR, same as spelling and hostnames.AGENTS.md: a "One home per fact" section (link instead of copy;rgthe key underdocs/before writing a config snippet; theSCHEMA_SYNCblocks own every setting's default), and the new check listed with the others.Against main today
node dev/check-duplicates.mjsfinds 622 pairs in 291 groups across the hand-written pagescode-search/features.mdxparagraphs that restatesearch-contexts.mdxandadmin/search.mdx, which came up in docs: Fix stale experimentalFeatures settings #2074--thresholdif those are noisyTested locally
origin/mainas the baseadmin/config/settings.mdxinto another page was reported with the inline comment on its first added lineactionlintand CSpell passTest result
Throwaway PR #2077 copied a paragraph from
docs/admin/config/settings.mdxintodocs/admin/how-to/enable-experimental-feature.mdx, and changed one word indocs/code-search/features.mdx:60, which was already a duplicate ofdocs/code-search/working/search-contexts.mdx:9. The check posted:Nothing else on the base branch was reported.