Skip to content

mirror: stop re-downloading packages indexed by SHA512 only - #1665

Open
somaz94 wants to merge 2 commits into
aptly-dev:masterfrom
somaz94:fix/sha512-only-mirror-redownload
Open

somaz94 wants to merge 2 commits into
aptly-dev:masterfrom
somaz94:fix/sha512-only-mirror-redownload

Conversation

@somaz94

@somaz94 somaz94 commented Nov 11, 2026

Copy link
Copy Markdown

Fixes #1664

Requirements

All new code should be covered with tests, documentation should be updated. CI should pass.

Also, to speed up things, if you could kindly "Allow edits and access to secrets by maintainers" in the
PR settings, as this allows us to rebase the PR on master, fix conflicts, run coverage and help with
implementing code and tests.

Description of the Change

Ubuntu 26.10 indexes list packages with SHA512 only. The package pool derives its paths from SHA256 (or MD5 for legacy paths), so Verify never finds a file that was already downloaded, and every aptly mirror update fetches the whole mirror again. -skip-existing-packages does not help either: the key built from the index hashes the MD5, SHA1 and SHA256 sums, which are all empty here, so it never matches the stored key, which was computed after the download filled in the full checksums.

BuildDownloadQueue now looks up the package with the same architecture, name and version from the previous update. When a file listed by SHA512 alone has a match there with the same name, size and SHA512, its pool path and full checksums are reused and the files hash is recomputed. Verify then finds the file in the pool the same way it does for SHA256 indexes, and the key matches again for -skip-existing-packages. A file whose SHA512 changed is still downloaded.

The new TestDownloadSHA512OnlyFlat fails on master (the second update queues the package again) and passes with this change. It covers the default path, -skip-existing-packages, and an index whose SHA512 differs. go test ./deb/ ./files/ ./http/ and golangci-lint run pass locally. I did not add a system test since the change is in the queue building logic, but I can add one under t04_mirror if you prefer.

I used Claude Code to help trace the code path and write the test, and reviewed and ran everything myself.

Checklist

  • allow Maintainers to edit PR (rebase, run coverage, help with tests, ...)
  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (if applicable)
  • documentation updated
  • author name in AUTHORS

somaz94 added a commit to somaz94/somaz94 that referenced this pull request Nov 11, 2026
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.35897% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.62%. Comparing base (4663076) to head (9b5ac20).

Files with missing lines Patch % Lines
deb/remote.go 72.97% 5 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1665      +/-   ##
==========================================
- Coverage   77.94%   77.62%   -0.33%     
==========================================
  Files         165      165              
  Lines       16141    16180      +39     
==========================================
- Hits        12581    12559      -22     
- Misses       2347     2407      +60     
- Partials     1213     1214       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@neolynx neolynx added the bug label Nov 17, 2026
@neolynx neolynx self-assigned this Nov 17, 2026
@neolynx
neolynx requested a review from a team November 17, 2026 08:48
@neolynx
neolynx force-pushed the fix/sha512-only-mirror-redownload branch from 19cca50 to 77ac4b3 Compare November 17, 2026 08:48
@neolynx

neolynx commented Nov 17, 2026

Copy link
Copy Markdown
Member

rebased

@neolynx neolynx added needs review Ready for review & merge increase coverage The PR lacks test coverage labels Nov 17, 2026
@neolynx

neolynx commented Nov 17, 2026

Copy link
Copy Markdown
Member

@somaz94 since this concerns mirroring, could you add a system test, where a real mirror is created which has only sha512 ?

@somaz94

somaz94 commented Nov 18, 2026

Copy link
Copy Markdown
Author

Added it in 9b5ac20. UpdateMirror33Test serves a local repo whose Packages index has only SHA512 sums, creates the mirror and updates it once, then runs the update again and expects an empty download queue. On master the same test fails because the second update downloads the package again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug increase coverage The PR lacks test coverage needs review Ready for review & merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Updating mirrors for Ubuntu stonking redownloads existing packages

2 participants