Skip to content

feat(rsync): the engine decides every Content-Type - #17

Merged
jshvn merged 1 commit into
mainfrom
josh/labels
Oct 8, 2026
Merged

jshvn merged 1 commit into
mainfrom
josh/labels

Conversation

@jshvn

@jshvn jshvn commented Oct 8, 2026

Copy link
Copy Markdown
Member

A census of all four mirrors (2026-10-08, every file HEAD-read, text-labeled ones sniffed) found compressed files served under the type of what they contain: 1,440 on gnu and 293 on nongnu are labeled as something a browser draws (text/x-diff over gzip bytes and the like), plus a handful on ctan. The cause is the AWS CLI storing only the first half of Python's guess_type. Its guesses also move with the image: tlnet's .tar.xz from the August fill say application/x-xz, everything since 2026-09-22 says application/x-tar.

What changes

  • label decides every upload's Content-Type and the CLI guesses none. Rules, in order: a COMPRESSED ending (.gz, .tgz, .bz2, .xz, .Z, .zst, .lz, .lzma, ftp.gnu.org's types) names the type; an ending Python reads as compressed and the table lacks is never text; otherwise the image's mime.types, unless it is a type a browser draws and the first KiB disagrees (a compression signature, a NUL, a PDF without %PDF-); a name with no type is typed from its bytes. Zip-based formats (.xpi, .jar) keep their names' types.
  • publish uploads one aws s3 cp --recursive --content-type per type, from trees renamed out of staging and back (no byte copied). Same PutObject count as before; timestamp still last.
  • relabel, once a bucket, in the scheduled run: one canary copy read back through the domain, then a server-side --metadata-directive REPLACE copy per COMPRESSED line, then .state/labels-v1. Directory pages whose directory is named like a tarball (gnu's librejs/librejs-5.1.tar.gz/) are excluded. Every failure is a warning, never the run's; no mark means the next run tries again. Marked ponytail: for removal once all four buckets hold the mark.
  • smoke reads back one key of every type the run published and compares the served type with the label, and warns (Actions annotation + summary row) for every path upstream lists that rsync never sends: nongnu's tsp/prebuilt/windows/tsp-0.8.2-Windows.exe and its .sig fail every run with "Invalid cross-device link" upstream and nothing said so.
  • fresh: with FRESH_KEY set, reads upstream's clock (Unix time or CTAN's YYYY-MM-DD-HH-MM) through the domain and fails the run past FRESH_HOURS (24), under the 28 h at which GNU's and CTAN's monitors call a mirror old.
  • index uploads pages with an explicit text/html.

New verbs label, label-trees, relabel, relabel-plan, fresh are added to the reserved list in toolbox.yml. CLAUDE.md gains a verified hazard: task's shell keeps set -e inside a subshell under ||, unlike bash and dash.

Rollout

Release as v2.4.0. Each mirror relabels itself on its next scheduled run after v2 moves. Consumer PRs follow with FRESH_KEY/FRESH_HOURS (gnu, nongnu, ctan), tlnet's retry settings, and every repo's render.

Test plan

  • examples/rsync: task image-build, task run -- task tools, task check, task run -- task offline all pass. Offline now covers label over 18 crafted names and byte prefixes, label-trees out and back (staging byte-identical after), relabel-plan (glob escaping, .z vs .Z, tarball-named directories), and fresh passing on 1 h and 2 h clocks and failing on 48 h.
  • examples/proton: the same four steps pass.
  • relabel exercised inside the image against stand-in aws/curl: success marks after 7 passes; a refused canary copy, a wrong type served back, and a failed pass each stop with a warning and no mark.
  • After merge and tag: dispatch nongnu first and read its run summary (Relabel row, types row), then curl a .diff.gz on each host.

The AWS CLI stored the first half of Python's guess, so 1,440 gnu
diffs and 293 nongnu files were served as text over gzip bytes, and
its guesses moved with the image (tlnet holds .tar.xz under two types).

- label: every upload's type from its name and first KiB; publish goes
  up one call per type, from trees renamed out of staging and back
- relabel: once a bucket, compressed objects copied onto themselves
  with their type, one canary copy first; fails soft, .state/labels-v1
- smoke: one key per type read back, type against label; a warning for
  paths upstream lists and rsync never sends
- fresh: FRESH_KEY / FRESH_HOURS fail the run when upstream's clock in
  that file is stale
- index: pages go up as text/html explicitly
@jshvn
jshvn merged commit 5031b90 into main Oct 8, 2026
2 checks passed
jshvn added a commit to katoptra/gnu that referenced this pull request Oct 8, 2026
Sets `FRESH_KEY: mirror-updated-timestamp.txt` for lib v2.4.0's `fresh`
check (katoptra/lib#17). After every run the engine reads that file
through the domain and fails the run when upstream's clock in it is 24
hours old, so a source that stops moving alerts through healthchecks.io
before the upstream's own mirror monitor flags us at 28 hours. README
gains the bullet and the failure mode.

Merge order: inert until v2 moves to v2.4.0, and `task check` passes
against today's v2. Once v2 moves I'll push this repo's render update to
this branch; merge after that.

Test plan: `task check` passes now; after v2.4.0, the first run's
summary shows an "Upstream clock" row.
jshvn added a commit to katoptra/nongnu that referenced this pull request Oct 8, 2026
Sets `FRESH_KEY: 00_TIME.txt` for lib v2.4.0's `fresh` check
(katoptra/lib#17). After every run the engine reads that file through
the domain and fails the run when upstream's clock in it is 24 hours
old, so a source that stops moving alerts through healthchecks.io before
the upstream's own mirror monitor flags us at 28 hours. README gains the
bullet and the failure mode.

Merge order: inert until v2 moves to v2.4.0, and `task check` passes
against today's v2. Once v2 moves I'll push this repo's render update to
this branch; merge after that.

Test plan: `task check` passes now; after v2.4.0, the first run's
summary shows an "Upstream clock" row.
jshvn added a commit to katoptra/ctan that referenced this pull request Oct 8, 2026
Sets `FRESH_KEY: timestamp` and `FRESH_HOURS: 6` for lib v2.4.0's
`fresh` check (katoptra/lib#17). After every run the engine reads that
file through the domain and fails the run when upstream's clock in it is
6 hours old, so a source that stops moving alerts through
healthchecks.io before the upstream's own mirror monitor flags us at 28
hours. README gains the bullet and the failure mode.

Merge order: inert until v2 moves to v2.4.0, and `task check` passes
against today's v2. Once v2 moves I'll push this repo's render update to
this branch; merge after that.

Test plan: `task check` passes now; after v2.4.0, the first run's
summary shows an "Upstream clock" row.
jshvn added a commit to katoptra/tlnet that referenced this pull request Oct 8, 2026
tlnet's `aws.config` had no retry settings, so the CLI used legacy mode
with 5 attempts; gnu, nongnu and ctan use `retry_mode = standard` and
`max_attempts = 10`. R2 answered the gnu/nongnu fills with occasional
503s, and lib v2.4.0's one-time relabel (katoptra/lib#17) issues ~15k
server-side copies here in one run.

Merge order: safe now. Once v2 moves to v2.4.0 I'll push the render
update to this branch; merge after that, or merge now and take the
render as its own PR.

Test plan: `task check` passes against today's v2.
@jshvn
jshvn deleted the josh/labels branch October 9, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant