Repository navigation
feat(rsync): the engine decides every Content-Type - #17
Merged
Merged
Conversation
The AWS CLI stored the first half of Python's guess, so 1,440 gnu diffs and 293 nongnu files were served as text over gzip bytes, and its guesses moved with the image (tlnet holds .tar.xz under two types). - label: every upload's type from its name and first KiB; publish goes up one call per type, from trees renamed out of staging and back - relabel: once a bucket, compressed objects copied onto themselves with their type, one canary copy first; fails soft, .state/labels-v1 - smoke: one key per type read back, type against label; a warning for paths upstream lists and rsync never sends - fresh: FRESH_KEY / FRESH_HOURS fail the run when upstream's clock in that file is stale - index: pages go up as text/html explicitly
This was referenced Oct 8, 2026
jshvn
added a commit
to katoptra/gnu
that referenced
this pull request
Oct 8, 2026
Sets `FRESH_KEY: mirror-updated-timestamp.txt` for lib v2.4.0's `fresh` check (katoptra/lib#17). After every run the engine reads that file through the domain and fails the run when upstream's clock in it is 24 hours old, so a source that stops moving alerts through healthchecks.io before the upstream's own mirror monitor flags us at 28 hours. README gains the bullet and the failure mode. Merge order: inert until v2 moves to v2.4.0, and `task check` passes against today's v2. Once v2 moves I'll push this repo's render update to this branch; merge after that. Test plan: `task check` passes now; after v2.4.0, the first run's summary shows an "Upstream clock" row.
jshvn
added a commit
to katoptra/nongnu
that referenced
this pull request
Oct 8, 2026
Sets `FRESH_KEY: 00_TIME.txt` for lib v2.4.0's `fresh` check (katoptra/lib#17). After every run the engine reads that file through the domain and fails the run when upstream's clock in it is 24 hours old, so a source that stops moving alerts through healthchecks.io before the upstream's own mirror monitor flags us at 28 hours. README gains the bullet and the failure mode. Merge order: inert until v2 moves to v2.4.0, and `task check` passes against today's v2. Once v2 moves I'll push this repo's render update to this branch; merge after that. Test plan: `task check` passes now; after v2.4.0, the first run's summary shows an "Upstream clock" row.
jshvn
added a commit
to katoptra/ctan
that referenced
this pull request
Oct 8, 2026
Sets `FRESH_KEY: timestamp` and `FRESH_HOURS: 6` for lib v2.4.0's `fresh` check (katoptra/lib#17). After every run the engine reads that file through the domain and fails the run when upstream's clock in it is 6 hours old, so a source that stops moving alerts through healthchecks.io before the upstream's own mirror monitor flags us at 28 hours. README gains the bullet and the failure mode. Merge order: inert until v2 moves to v2.4.0, and `task check` passes against today's v2. Once v2 moves I'll push this repo's render update to this branch; merge after that. Test plan: `task check` passes now; after v2.4.0, the first run's summary shows an "Upstream clock" row.
jshvn
added a commit
to katoptra/tlnet
that referenced
this pull request
Oct 8, 2026
tlnet's `aws.config` had no retry settings, so the CLI used legacy mode with 5 attempts; gnu, nongnu and ctan use `retry_mode = standard` and `max_attempts = 10`. R2 answered the gnu/nongnu fills with occasional 503s, and lib v2.4.0's one-time relabel (katoptra/lib#17) issues ~15k server-side copies here in one run. Merge order: safe now. Once v2 moves to v2.4.0 I'll push the render update to this branch; merge after that, or merge now and take the render as its own PR. Test plan: `task check` passes against today's v2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A census of all four mirrors (2026-10-08, every file HEAD-read, text-labeled ones sniffed) found compressed files served under the type of what they contain: 1,440 on gnu and 293 on nongnu are labeled as something a browser draws (
text/x-diffover gzip bytes and the like), plus a handful on ctan. The cause is the AWS CLI storing only the first half of Python'sguess_type. Its guesses also move with the image: tlnet's.tar.xzfrom the August fill sayapplication/x-xz, everything since 2026-09-22 saysapplication/x-tar.What changes
labeldecides every upload's Content-Type and the CLI guesses none. Rules, in order: aCOMPRESSEDending (.gz,.tgz,.bz2,.xz,.Z,.zst,.lz,.lzma, ftp.gnu.org's types) names the type; an ending Python reads as compressed and the table lacks is never text; otherwise the image'smime.types, unless it is a type a browser draws and the first KiB disagrees (a compression signature, a NUL, a PDF without%PDF-); a name with no type is typed from its bytes. Zip-based formats (.xpi,.jar) keep their names' types.publishuploads oneaws s3 cp --recursive --content-typeper type, from trees renamed out of staging and back (no byte copied). Same PutObject count as before;timestampstill last.relabel, once a bucket, in the scheduled run: one canary copy read back through the domain, then a server-side--metadata-directive REPLACEcopy perCOMPRESSEDline, then.state/labels-v1. Directory pages whose directory is named like a tarball (gnu'slibrejs/librejs-5.1.tar.gz/) are excluded. Every failure is a warning, never the run's; no mark means the next run tries again. Markedponytail:for removal once all four buckets hold the mark.smokereads back one key of every type the run published and compares the served type with the label, and warns (Actions annotation + summary row) for every path upstream lists that rsync never sends: nongnu'stsp/prebuilt/windows/tsp-0.8.2-Windows.exeand its.sigfail every run with "Invalid cross-device link" upstream and nothing said so.fresh: withFRESH_KEYset, reads upstream's clock (Unix time or CTAN'sYYYY-MM-DD-HH-MM) through the domain and fails the run pastFRESH_HOURS(24), under the 28 h at which GNU's and CTAN's monitors call a mirror old.indexuploads pages with an explicittext/html.New verbs
label,label-trees,relabel,relabel-plan,freshare added to the reserved list intoolbox.yml. CLAUDE.md gains a verified hazard: task's shell keepsset -einside a subshell under||, unlike bash and dash.Rollout
Release as v2.4.0. Each mirror relabels itself on its next scheduled run after
v2moves. Consumer PRs follow withFRESH_KEY/FRESH_HOURS(gnu, nongnu, ctan), tlnet's retry settings, and every repo's render.Test plan
examples/rsync:task image-build,task run -- task tools,task check,task run -- task offlineall pass. Offline now coverslabelover 18 crafted names and byte prefixes,label-treesout and back (staging byte-identical after),relabel-plan(glob escaping,.zvs.Z, tarball-named directories), andfreshpassing on 1 h and 2 h clocks and failing on 48 h.examples/proton: the same four steps pass.relabelexercised inside the image against stand-inaws/curl: success marks after 7 passes; a refused canary copy, a wrong type served back, and a failed pass each stop with a warning and no mark..diff.gzon each host.