Repository navigation
Conversation
Author
|
Forgot to say that those changes are strictly restricted to the "filesystem" storage backend and brings no changes to other backends such as Object store. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it fixes
This proposal has been heavily produced with AI assisted code, I don't know if there are any rules for or against such process. As far as I can tell, I used it in order to improve publish/drop performances which are very slow and can take hours each time I need to publish new snapshoted repositories. I strictly checked and tested the generated code and did many benchmarks before pushing it. It's up to you to decide whether or not it violates your acceptance criteria.
Description of the Change
I maintain a huge aptly mirror with plenty of snapshots that I publish regularly.
This process is very long and time consuming more specifically the "publish" part so I tried to see if there was ways to improve that on my test setup.
My test setup is a Ubuntu 22.04 KVM VM with :
Speed benchmarks were done against official aptly v1.6.3 Linux amd64 release.
Test workload
The benchmark workload publishes the four Ubuntu Noble snapshot components into a local filesystem publication (72500 packages):
The cleanup command is:
Publish snapshot optimizations
Contentsgeneration.main,restricted,universe, andmultiverseconcurrently. This is enabled by default.pgzipwithBestSpeed, and bzip2 usesbzip2 -1to favor publication latency.Publish drop optimizations
rm -rf -- <path>by default instead of Goos.RemoveAll, which was slower on the XFS test filesystem.publish droprenames deleted paths into.aptly-trash/<batch-id>/and updates the aptly DB before starting the physicalrm -rfin the background.Performance comparison
"publish" operation
publish snapshot122.43s69.63s68.48spublish snapshotbuild, best observed run49.13s52.64s"drop" operation
publish dropusing Goos.RemoveAll~9.2spublish dropusingrm -rf~7.2spublish dropvisible latency0.10sto0.14s~7sper publishThe final safe optimized
publish snapshottime is about 2.3x to 2.5x faster than the reference v1.6.3 build for this workload, depending on run-to-run variance. The asynchronous drop mode reduces user-visiblepublish droplatency from about 9 seconds to about 0.1 seconds, while the physical deletion continues in the background.Environment variables
APTLY_PARALLEL_COMPONENTS0to disable concurrent component processing.APTLY_PARALLEL_PACKAGES0to disable parallel package processing inside each component.APTLY_PARALLEL_PACKAGE_WORKERSruntime.NumCPU()APTLY_FAST_REMOVE0to use Goos.RemoveAllinstead of externalrm -rffor local filesystem cleanup.APTLY_ASYNC_REMOVE0to disable asynchronous local filesystem removals and run the physical cleanup synchronously. Inpublish drop, physical deletion is committed only after the DB update succeeds.The default asynchronous remove behavior changes the operational semantics of
publish drop: after the command returns, the published repository is already removed from aptly metadata and its paths have been moved out of the served location, but disk space is reclaimed only after the backgroundrm -rfcompletes. SetAPTLY_ASYNC_REMOVE=0to restore synchronous cleanup.Checklist
AUTHORS