Skip to content

Cache a screen-sized copy of TIFF/HEIC images served by /images/ - #423

Merged
lstein merged 4 commits into
masterfrom
lstein/feature/cache-converted-images
Oct 6, 2026
Merged

lstein merged 4 commits into
masterfrom
lstein/feature/cache-converted-images

Conversation

@lstein

@lstein lstein commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Fixes #414.

Summary

/images/ used to re-encode every TIFF/HEIC as a full-resolution PNG on every request, with no validators. It now serves a JPEG capped at 4096 px on the long edge, or a WebP when the image has real transparency. The copy is cached in the album's thumbnail directory and served with a content ETag (304 on revalidation). Downloads are unchanged because they use ?original=1.

Measured on a 6000×4000 LZW TIFF (97 MB):

time bytes
before (every request) 2.55 s 53.5 MB PNG
after, cold 0.67 s 3.9 MB JPEG
after, warm ~1 ms 3.9 MB JPEG

How it works

  • Naming: <tile digest>_display_<album tag>_<source stamp>.<jpg|webp>.
    • The tile digest is the same one keep_hashes_for produces. That lets the post-index sweep and the per-image delete (discard) reclaim display copies with no second keep-set.
    • The album tag stops two albums whose index files share a directory from serving each other's same-named files.
    • The source stamp is a digest of the source's mtime_ns and size, taken before decoding. A file replaced with an older mtime (cp -p, a backup restore) is therefore reconverted. The stat is repeated after encoding, and if the source changed in between, the result is served but not cached.
  • Relative path: taken from the resolved file, the way the index stores it. Before this, an album root reached through a symlink or a mapped Windows drive fell back to the bare filename, so same-named images collided.
  • Concurrency: a per-stem lock plus write-then-rename. Concurrent requests (swiper preloads) convert once and never read a partial file. After each write, older copies of the same file are removed. A copy written after its source was deleted mid-conversion is removed as well.
  • Mode handling: _png_safe's 16-bit, 32-bit, float and CMYK/ICC handling still applies. 16-bit is then reduced to 8-bit for JPEG. An alpha channel that is fully opaque is dropped, and an ICC profile is dropped whenever the mode changes.
  • Failure handling: if the cache can't be written (e.g. a read-only index directory), the image is still served and the failure is logged once.
  • Sweep safety: prune/discard now delete only the exact filename shapes the routes write. A hash-named user photo in a thumbnails/ folder next to a custom index path is left alone.

Review

Two rounds of adversarial fresh-context review. Every confirmed finding has a regression test that fails on the earlier code.

Known and deliberately out of scope, because each predates this change:

  • When a custom index path lies inside an image root, cached files in thumbnails/ are indexed as photos. The default photomap_index/ is excluded from scanning.
  • Deleting a directory album doesn't remove its thumbnails/ folder.
  • On case-insensitive filesystems, a hand-typed URL with different capitalization creates extra copies that the per-image delete misses. They are reclaimed at the next index save. Tiles already behave this way.

Test plan

  • pytest tests: 1308 passed
  • ruff check photomap tests
  • Manual: browse a folder of large TIFF/HEIC scans; check that revisits are instant and that the download button still saves the original

🤖 Generated with Claude Code

lstein and others added 4 commits October 4, 2026 18:55
/images/ re-encoded every TIFF/HEIC to a full-resolution PNG on every
request, with no validators, so each view and slideshow preload paid for
the decode and a 30-70 MB encode again. It now serves a JPEG (WebP when
the image has transparency) capped at 4096px on the long edge, cached in
the album's thumbnail directory and served with a content ETag/304.

- Named with the tile digest, so the reindex sweep and the per-image
  delete reclaim display copies with no second keep-set.
- Per-file lock plus write-then-rename: concurrent requests convert once
  and never read a partial file. Abandoned temporaries are swept.
- Rebuilt when the source is newer; a failed cache write still serves.
- 16-bit scans are reduced to 8-bit after _png_safe's rescale.

On a 6000x4000 LZW TIFF: 2.55 s / 53 MB before, 0.67 s / 3.9 MB cold,
~1 ms warm. Downloads keep using ?original=1.

Fixes #414

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Adversarial review found the cached copy could show the wrong picture:
same-named files under a symlinked root (or a mapped Windows drive)
shared one copy, as did albums whose indexes share a directory, and a
source replaced with an older mtime, or edited mid-conversion, kept its
old copy. The copy's name now carries the album and a digest of the
source's mtime+size taken before decoding; the relative path is taken
from the resolved file, as the index stores it. Also: a copy written
after its source was deleted is removed, a greyscale profile is not
carried onto RGBA pixels, and an unwritable cache warns once.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
From a second adversarial review:
- prune/discard matched any <32 hex>.jpg/.webp/.png, so a hash-named
  user photo beside a custom index path could be deleted. Both now match
  only the exact names the routes write.
- A source edited between the up-front stat and the decode was cached
  under the old stamp; restoring the original then showed the edit.
  Re-stat after encoding and serve without caching if it moved.
- A temporary swept mid-write marked the directory unwritable for good,
  silencing real failures; one undeletable sibling skipped the
  vanished-source check. Each step now handles its own errors.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@lstein
lstein enabled auto-merge (squash) October 6, 2026 18:55
@lstein
lstein merged commit d9da26b into master Oct 6, 2026
10 checks passed
@lstein
lstein deleted the lstein/feature/cache-converted-images branch October 6, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache (and downscale) TIFF/HEIC conversions served by /images/

1 participant