Skip to content

fix: preserve colormap state through reversed(), with_extremes(), pickle, and as_dict() - #155

Merged
tlambert03 merged 15 commits into
pyapp-kit:mainfrom
matthiasschabel:fix/state-preservation
Oct 20, 2026
Merged

tlambert03 merged 15 commits into
pyapp-kit:mainfrom
matthiasschabel:fix/state-preservation

Conversation

@matthiasschabel

@matthiasschabel matthiasschabel commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

None of the four channels preserved the extreme colors, and two of them dropped the interpolation
mode as well. Verified on main before this change:

  • reversed() passes only stops, name, and category, so a nearest colormap comes back linear
    with no extremes.
  • with_extremes() clears anything not repeated in the call.
  • __reduce__ carries only color_stops, so pickle.loads(pickle.dumps(cm)) == cm is already
    False for any colormap with bad set.
  • as_dict() has no keys for interpolation or the extremes, and it is what both the pydantic
    serializer and _json_encode emit.

Following matplotlib 3.11 where it has a position:

  • reversed() swaps under/over and neg_inf/pos_inf, which name the ends they extend, and
    preserves the rest.
  • with_extremes() preserves anything not passed. Clearing a single color now means constructing
    a new Colormap, the same limitation matplotlib has. This is the one backward-incompatible
    change here.
  • __reduce__ carries the full constructor state.
  • as_dict() gains optional keys, written only when they hold non-default state, so existing
    payloads are unchanged.

Two changes that are more than preservation:

  • Colormap("x_r") now swaps the catalog record's under/over, so it agrees with
    Colormap("x").reversed(). napari:HiLo is the only affected entry. An explicit
    under=/over= argument still lands on the end the caller named.
  • The pydantic serializer emitted a catalog colormap's qualified name unconditionally, so
    Colormap("viridis", under="red") came back as plain viridis, and Colormap("viridis_r") came
    back unreversed because info is looked up under the stripped name. It now emits the name only
    when the name alone rebuilds the same as_dict(). __eq__ is not usable as that guard: it
    ignores name, identifier, and category, and ColorStops.__eq__ compares with np.allclose, so
    a cmap_kwargs={"start": 0.500001} cubehelix compares equal to the default one.

What I left alone, and why:

  • Pickle passes color_stops as the object rather than as_dict()'s samples, so a colormap
    backed by a lut function keeps the function. Routing pickle through as_dict() shifted
    lut(17, gamma=2) of a parametrized cubehelix by 2e-5.
  • A round trip through the dict form does not restore info, which is set only from a string or
    another Colormap, so an unpickled catalog colormap serializes as a dict rather than as its
    name. Fixing that means putting the qualified name in value, which would change existing
    payloads for every catalog colormap.
  • identifier follows the name: reversed() and shifted() rename, so it re-derives;
    with_extremes() does not, so it is preserved. One consequence: cm.reversed().reversed()
    restores the name but not an explicitly supplied identifier. Say if you would rather it were
    carried through.
  • The converters still have no representation for the finer classes.

Depends on #164 to fix CI/pydantic issues.

cmap colors three exceptional classes: under, over, and bad. Floating point
data has more. Negative and positive infinity are indistinguishable from
ordinary out-of-range values, and NaN is indistinguishable from a masked entry.

Adds neg_inf, pos_inf, nan, and masked. Each falls back to the color its class
uses now: neg_inf to under, pos_inf to over, nan and masked to bad. bad is kept
as the joint fallback for both of its children, so code that sets it is
unaffected and either child may be set alone.

Routing appends four fallback-resolved rows to a call-local copy of the
over/under LUT, so a class with no color of its own lands on exactly the row it
lands on now. Colormap.lut() is unchanged.

The infinity masks are taken before the input is scaled by N: that multiply
overflows large finite values to infinity (float16 65504 does it), and those
are out of range rather than infinite.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reviewed-By: Codex (gpt-5.6-sol, reasoning effort xhigh)
@codecov

codecov Bot commented Aug 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.90%. Comparing base (0ac9baa) to head (7d0f11e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #155      +/-   ##
==========================================
+ Coverage   95.87%   95.90%   +0.03%     
==========================================
  Files         168      168              
  Lines        2228     2248      +20     
==========================================
+ Hits         2136     2156      +20     
  Misses         92       92              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@matthiasschabel
matthiasschabel marked this pull request as ready for review August 14, 2026 19:39
matthiasschabel and others added 2 commits August 24, 2026 16:51
Keep the proposal scoped to negative infinity, NaN, and positive infinity in addition to the existing under and over behavior. Masked arrays continue to use the existing bad color, including when they hide an infinity or NaN.
…kle, and as_dict()

None of the four channels carried the extreme colors, and reversed() and
with_extremes() dropped the interpolation mode as well. Following matplotlib
where it has a position: reversed() swaps the directional colors and keeps the
rest, with_extremes() keeps anything not passed, __reduce__ carries the
constructor state alongside the stops, and as_dict() gains optional keys that
are written only when set, so existing payloads are unchanged.

The pydantic serializer emitted a catalog colormap's qualified name
unconditionally, which discarded any added extremes and turned "viridis_r"
back into plain viridis. It now emits the name only when the name alone
rebuilds the same colormap.

Colormap("x_r") swaps the catalog record's under and over, so it agrees with
Colormap("x").reversed().

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reviewed-By: Codex (gpt-5.6-sol, reasoning effort xhigh)
@tlambert03

Copy link
Copy Markdown
Member

#151 is merged now! can you clean up the conflicts and merge in main?

Resolves the conflicts left by pyapp-kit#151 landing as a squash: every hunk outside
with_extremes() takes main's side, since it was this branch's stale copy of
pyapp-kit#151 before review. with_extremes() keeps this branch's behavior of carrying
over colors that are not passed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@matthiasschabel

Copy link
Copy Markdown
Contributor Author

OK, I think this should be merged and ready to go...

@matthiasschabel

Copy link
Copy Markdown
Contributor Author

@tlambert03 It looks like the codecov failure is related to the pydantic serializer? I can look into it some more later...

tlambert03 and others added 10 commits October 19, 2026 15:57
Resolve conflict in tests/test_model_fields.py: keep main's pydantic v2
forms and port the new state-preservation tests to them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… serialization

An unmodified catalog colormap now serializes and pickles as its qualified
name (keeping `_r`), which restores `info` after pickle/copy and makes
serialization as cheap as on main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@tlambert03

tlambert03 commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

hey @matthiasschabel, I pushed a bunch of stuff directly to your branch. here's what changed:

  • merged main.
  • ColormapDict: swapped the _RequiredColormapDict base-class pattern for a single TypedDict with NotRequired keys.
  • serializer perf: the as_dict() comparison at serialization time was ~20-50x slower than main for catalog colormaps. now __init__ records a _catalog_name when the colormap comes from a catalog name with no other arguments, and the serializer just emits that (or falls back to as_dict()). that's back to about main's speed, and _is_identical is gone.
  • __reduce__: unmodified catalog colormaps now pickle as just their name, which also restores info. everything else still pickles its full state.
  • with_extremes() docstring: dropped "To clear one, construct a new Colormap", since Colormap(cm) currently drops all the extremes (a pre-existing bug, I'll follow up separately).
  • separately, fix: keep the _r suffix in the name of a reversed catalog colormap #167 made Colormap("viridis_r").name come out as bids:viridis_r (it used to lose the _r).

trade-offs I'm ok with for now:

  • pickled catalog colormaps now depend on the name still existing in the catalog
  • Colormap(Colormap("viridis")) serializes as a dict
  • any explicit argument counts as an override, even one equal to the default

@tlambert03
tlambert03 merged commit 0e5d91e into pyapp-kit:main Oct 20, 2026
23 checks passed
@matthiasschabel
matthiasschabel deleted the fix/state-preservation branch October 21, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants