Skip to content

Optimized publish and drop operations for local filesystem storage - #1670

Open
Hybrid512 wants to merge 3 commits into
aptly-dev:masterfrom
Hybrid512:feature/optimized_publish_drop
Open

Hybrid512 wants to merge 3 commits into
aptly-dev:masterfrom
Hybrid512:feature/optimized_publish_drop

Conversation

@Hybrid512

Copy link
Copy Markdown

What it fixes

⚠️ AI warning ⚠️
This proposal has been heavily produced with AI assisted code, I don't know if there are any rules for or against such process. As far as I can tell, I used it in order to improve publish/drop performances which are very slow and can take hours each time I need to publish new snapshoted repositories. I strictly checked and tested the generated code and did many benchmarks before pushing it. It's up to you to decide whether or not it violates your acceptance criteria.

Description of the Change

I maintain a huge aptly mirror with plenty of snapshots that I publish regularly.
This process is very long and time consuming more specifically the "publish" part so I tried to see if there was ways to improve that on my test setup.

My test setup is a Ubuntu 22.04 KVM VM with :

  • 4 vCPUs (Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz as the hypervisor)
  • 8GB RAM
  • 3TB local storage (NvME underneath) with XFS

Speed benchmarks were done against official aptly v1.6.3 Linux amd64 release.

Test workload

The benchmark workload publishes the four Ubuntu Noble snapshot components into a local filesystem publication (72500 packages):

sudo /usr/local/bin/aptly publish snapshot \
  -label=Ubuntu \
  -component=main,restricted,universe,multiverse \
  -distribution=noble \
  20261022-ubuntu-noble-main \
  20261022-ubuntu-noble-restricted \
  20261022-ubuntu-noble-universe \
  20261022-ubuntu-noble-multiverse \
  filesystem:public:test-ubuntu

The cleanup command is:

sudo /usr/local/bin/aptly publish drop noble filesystem:public:test-ubuntu

Publish snapshot optimizations

  • Temporary LevelDB tuning: the temporary publish database now uses larger write buffers and cache/table settings, and disables sync for that temporary database. This reduced LevelDB write stalls during Contents generation.
  • Parallel component processing: publication metadata generation can process main, restricted, universe, and multiverse concurrently. This is enabled by default.
  • Parallel package processing inside each component: package linking, contents collection, and LevelDB batch writes are parallelized within a component. Package index stanzas are built in workers but written back sequentially to preserve the original index order.
  • Destination-level hardlink locking: parallel package processing is protected by a fine-grained lock per publish destination, avoiding races when two package records refer to the same target file.
  • Thread-safe index file registry: the shared metadata/index registry is protected so components and package workers can request indexes concurrently.
  • Parallel metadata finalization: unsigned metadata files are compressed and published in parallel. Signed release files remain serialized.
  • Faster compression path: gzip and bzip2 compression run concurrently. gzip uses pgzip with BestSpeed, and bzip2 uses bzip2 -1 to favor publication latency.

Publish drop optimizations

  • Fast synchronous filesystem removal: local filesystem removals use rm -rf -- <path> by default instead of Go os.RemoveAll, which was slower on the XFS test filesystem.
  • Asynchronous drop mode: by default, publish drop renames deleted paths into .aptly-trash/<batch-id>/ and updates the aptly DB before starting the physical rm -rf in the background.
  • DB-safe async deletion: physical deletion starts only after the DB update succeeds. If the DB update fails, the staged renames are rolled back to their original paths.
  • Batch isolation: every async drop receives its own trash batch directory, preventing overlapping drops from committing or rolling back each other's staged removals.

Performance comparison

"publish" operation

Operation / build Measured time
Reference aptly v1.6.3 publish snapshot 122.43s
Optimized temporary LevelDB tuned build 69.63s
Component-level parallelism only 68.48s
Final safe optimized publish snapshot build, best observed run 49.13s
Final cleaned build validation run 52.64s

"drop" operation

Operation / build Measured time
Reference-style publish drop using Go os.RemoveAll ~9.2s
Optimized synchronous publish drop using rm -rf ~7.2s
Async batch-isolated publish drop visible latency 0.10s to 0.14s
Async physical deletion in background ~7s per publish

The final safe optimized publish snapshot time is about 2.3x to 2.5x faster than the reference v1.6.3 build for this workload, depending on run-to-run variance. The asynchronous drop mode reduces user-visible publish drop latency from about 9 seconds to about 0.1 seconds, while the physical deletion continues in the background.

Environment variables

Variable Default Effect
APTLY_PARALLEL_COMPONENTS enabled Set to 0 to disable concurrent component processing.
APTLY_PARALLEL_PACKAGES enabled Set to 0 to disable parallel package processing inside each component.
APTLY_PARALLEL_PACKAGE_WORKERS runtime.NumCPU() Overrides the number of package workers. On the 4-vCPU reference VM, the default value of 4 was best; 8 workers was slightly slower and 3 workers was slower.
APTLY_FAST_REMOVE enabled Set to 0 to use Go os.RemoveAll instead of external rm -rf for local filesystem cleanup.
APTLY_ASYNC_REMOVE enabled Set to 0 to disable asynchronous local filesystem removals and run the physical cleanup synchronously. In publish drop, physical deletion is committed only after the DB update succeeds.

The default asynchronous remove behavior changes the operational semantics of publish drop: after the command returns, the published repository is already removed from aptly metadata and its paths have been moved out of the served location, but disk space is reclaimed only after the background rm -rf completes. Set APTLY_ASYNC_REMOVE=0 to restore synchronous cleanup.

Checklist

  • [ x ] allow Maintainers to edit PR (rebase, run coverage, help with tests, ...)
  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (if applicable)
  • [ x ] documentation updated
  • [ x ] author name in AUTHORS

@Hybrid512

Hybrid512 commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

Forgot to say that those changes are strictly restricted to the "filesystem" storage backend and brings no changes to other backends such as Object store.
This has not been tested with other filesystems than XFS or network mounts such as NFS (but I'd happy to know if there are gains on these too)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant