feat: colors for +inf, -inf, NaN, and masked values

[matthiasschabel]

Closes #144.

Adds four optional colors so exceptional float values can be told apart: neg_inf,
pos_inf, nan, and masked. Today -inf is indistinguishable from any other under-range
value, and NaN is indistinguishable from a masked entry. Log transformed signal data and
saturated logistic regression both produce infinities worth marking rather than blending
into the ends of the scale.

Each new color falls back to the one its class uses now, so nothing changes for an existing
colormap:

Class	Color	Falls back to
negative infinity	neg_inf	under, then the first ramp color
positive infinity	pos_inf	over, then the last ramp color
NaN	nan	bad, then transparent
masked	masked	bad, then transparent

bad stays the joint fallback for nan and masked rather than being replaced by them, so
code that sets it is unaffected and either child can be set alone. A masked entry takes the
masked color whatever value it hides, as it does now.

Two things the diff does not show:

• The infinity masks are taken before xa *= N. That multiply overflows large finite values
  to infinity (float16 65504 does at N=256), and those are out of range, not infinite.
  There is a test for it, because classifying after the multiply looks right and is not.

to_napari now prefers nan_color over bad_color for napari's nan_color, since it is
the one converter target that represents the class. matplotlib's bad covers NaN and masked
together, so bad is still what goes there.

Depends on #150, which this branches from. Only the last commit is mine; the first two are
#150's.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.78%. Comparing base (f0a4aec) to head (9380611).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #151      +/-   ##
==========================================
+ Coverage   95.72%   95.78%   +0.06%     
==========================================
  Files         168      168              
  Lines        2197     2231      +34     
==========================================
+ Hits         2103     2137      +34     
  Misses         94       94              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tlambert03]

@matthiasschabel I'm ok with this if it enables you to do things that regularly pop up in your work. My only objection would be the cognitive burden on the more casual user who doesn't particularly care to discriminate between "over" and "positive infinity". So, as long as a user not providing these values has behavior that just degrades to the old behavior, and as long as they are documented well (they are), then I'm fine with this. 👍

I think i'm most concerned with now having bad_color, nan_color, and masked_color. Are all three definitely needed? Or is nan vs bad just "regret" over an imprecise naming decision with "bad"? (i.e. what remains for bad_color to claim if nan and masked have been used?) @jni do you have thoughts here?

A couple questions. (some of these might be broken before this PR, but now is as good a time as any to address them). Do all of these over/under/inf/neg-inf values get preserved in:

• cmap.reversed
• cmap.with_extremes (should previously configured exceptional colors be lost unless they are repeated?)
• pickle/unpickle
• as_dict() and pydantic serialization?

(it occurs to me that some of that might not be tested)

[matthiasschabel]

@matthiasschabel I'm ok with this if it enables you to do things that regularly pop up in your work. My only objection would be the cognitive burden on the more casual user who doesn't particularly care to discriminate between "over" and "positive infinity". So, as long as a user not providing these values has behavior that just degrades to the old behavior, and as long as they are documented well (they are), then I'm fine with this. 👍

Thanks for the close read and for being agreeable! I do think this proposal addresses legitimate use cases and closes the circle on handling special/exceptional values in cmap. The implementation should be purely additive and takes care to degrade in the predictable/expected backward-compatible path, so transparent to current users and hopefully the cognitive overhead is minimal.

When new classes are unused the routing appends fallback-resolved rows to a call-local copy of the LUT, so a class with no color of its own lands on exactly the row it lands on today. test_exceptional_colors_fall_back_to_the_legacy_extremes pins the four legacy destinations, and I also confirmed that outputs against main for data containing both infinities, NaN, and a masked entry, with and without under/over/bad set were identical.

I think i'm most concerned with now having bad_color, nan_color, and masked_color. Are all three definitely needed? Or is nan vs bad just "regret" over an imprecise naming decision with "bad"? (i.e. what remains for bad_color to claim if nan and masked have been used?) @jni do you have thoughts here?

Here's my argument in favor (in addition to the completeness argument):bad is not naming regret; it is the umbrella tier of a two-level hierarchy, parallel to under/over above neg_inf/pos_inf. What remains for bad to claim is the common (and matplotlib compatible) case: one color for "invalid, I don't care why". nan and masked are for when "measured but undefined" and "deliberately excluded" have to read differently. masked can mark values that are in range, which none of the other classes handle. Masking by a predicate gives you a cheap contour band or a polarity change overlay that stays put with changes in window level/width without touching the values.

A couple questions. (some of these might be broken before this PR, but now is as good a time as any to address them). Do all of these over/under/inf/neg-inf values get preserved in:

• cmap.reversed
• cmap.with_extremes (should previously configured exceptional colors be lost unless they are repeated?)
• pickle/unpickle
• as_dict() and pydantic serialization?

(it occurs to me that some of that might not be tested)

None of those four channels preserve the existing under/over/bad either.

• reversed() passes only stops, name, and category, so it drops every extreme color and the interpolation mode (a nearest colormap comes back linear).
• with_extremes() clears anything not repeated. This deviates from matplotlib's behavior, which preserves existing values that are not passed.
• __reduce__ carries only color_stops, so a pickle round trip loses name, category, interpolation, and all extreme colors. pickle.loads(pickle.dumps(cm)) == cm is already False for any colormap with bad set.
• as_dict() has no extreme keys, and it is what the pydantic serializer and _json_encode emit, so a model round trip silently strips them. A catalog colormap serializes as just its qualified name, so Colormap("viridis", under="red") comes back as plain viridis.

I think we should conform to matplotlib wherever it has a position:

• reversed(): swap under and over, preserve bad, as matplotlib does; neg_inf and pos_inf swap along with the ends they extend, and nan/masked are preserved with bad. Interpolation carries through.
• with_extremes(): preserve anything not passed, as matplotlib does. Clearing a single color then means constructing a fresh Colormap, the same limitation matplotlib has.
• pickle: carry the full constructor state through __reduce__.
• as_dict(): optional keys for interpolation and the extreme colors, emitted only when set, so existing payloads are byte-for-byte unchanged; the pydantic serializer falls back to the dict form when a catalog colormap carries extremes. Deserialization already works, since _validate calls the constructor and it accepts all of these as kwargs.

Would you prefer that these get broken up into more granular, separate PRs or lumped in with this one?

[tlambert03]

Thank you for the thorough response! Yeah I thought that might be the case (that it was broken before this PR). And I agree that we should mirror mpl where there is prior art.

I suppose we should go ahead and split that fix out into a new PR. And it needn't hold this one up either.

I would like to get a quick opinion from @jni on the API addition, but then this is all good by me. Thanks again!