Model the distinction between a missing unit and a unit whose value is null

[reckart]

Is your feature request related to a problem? Please describe.

The library has no consistent, queryable notion that distinguishes two genuinely different situations:

1. A missing unit - a rater produced no annotation for a given position/item (an absence).
2. A present unit with a null value - a rater did produce a unit, but its category/label is deliberately null (present but unlabeled).

Each of the three study families models this differently (or not at all), so the capability API added in #16 (IMissingValueSupport / IAgreementMeasure.canHandle) can only express the coding interpretation:

Family	What null means today	missing ≠ null-value?	Detector	Queryable via IMissingValueSupport?
Coding	A missing value: a rater left an item unrated (hole in the item × rater grid), represented as a null category.	n/a (only one notion)	ICodingAnnotationStudy.hasMissingValues()	Yes - canHandle consults it for coding studies.
Unitizing	The gap itself. Per UnitizingAnnotationStudy.addUnit(...) Javadoc, a unit whose category is null is a gap; callers are told to use a dummy category (e.g. "X") for a real-but-unlabeled unit. The two concepts are fused.	Fused - no separate null-value state exists.	none	No (and vacuous as modelled - every unitizing measure must handle gaps).
Aligning	A genuine label value, distinct from an empty unit. In NominalFeatureDissimilarity.dissimilarity(...), aUnit == null is the empty/padding unit (fixed cost), whereas a present unit with getType() == null falls through to Objects.equals(type1, type2) - so two null-labeled units match and a null-labeled unit differs from an "X"-labeled one.	Yes, distinct	none (no study-level detector)	No - canHandle skips non-coding studies, so Gamma's real null-label tolerance is not expressible.

Consequences:

• "Does this measure support missing / null-valued annotations?" is discriminating for coding (some measures do, some warn) and for aligning (Gamma supports null labels distinctly from gaps), but vacuous for unitizing as currently modelled (null = gap).
• Because the distinction is conflated/omitted outside coding, IMissingValueSupport had to be scoped to coding studies. Marking an aligning measure such as GammaAgreement would make the instanceof IMissingValueSupport query semantically true, but canHandle has no aligning-side detector to back it - a half-wired capability.
• Unitizing offers callers no way to say "this unit exists but its label is unknown" separate from "there is no unit here," even though that is a meaningful annotation state.
• Even within coding, the current IMissingValueSupport assignments have not been audited against each measure's actual arithmetic. WeightedKappaAgreement is marked as supporting missing values, but - like Cohen's kappa, which it generalizes and which is (correctly) not marked - it normalizes over a complete rater×item grid (calculateObservedDisagreement divides by itemCount * raterCount * (raterCount - 1); calculateExpectedDisagreement divides by itemCount² over all rater pairs). With missing values these denominators over-count, so the coefficient is skewed. The marker therefore makes canHandle return true and suppresses the "does not support missing values" warning for a measure whose result is in fact biased - the opposite of what the capability query promises. Any cross-family rework must include a pass over the existing coding markers, not just add new ones.

Describe the solution you'd like

1. Introduce an explicit, first-class distinction in the data model between unit absence (gap in unitizing, empty/padding unit in aligning, unrated item in coding) and unit present with a null (missing) value/label.
2. Give each family a consistent detector analogous to ICodingAnnotationStudy.hasMissingValues() - a study-level "contains units with a null/missing value" predicate for unitizing and aligning studies.
3. Decouple the unitizing gap encoding from null so a present-but-unlabeled unit can be represented without being reinterpreted as a gap (today the Javadoc forces a dummy category). This is the most invasive part and needs a migration/compatibility plan, since existing callers rely on null == gap.
4. Extend IAgreementMeasure.canHandle to consult the per-family detector, so IMissingValueSupport means the same thing across coding, unitizing, and aligning and the query is backed by an actual check rather than being coding-only.
5. Audit which measures declare IMissingValueSupport, in both directions:
   • Add the marker where a measure genuinely tolerates missing/null values and a detector can back it (e.g. GammaAgreement becomes a legitimate, fully-wired candidate once aligning has a detector).
   • Remove the marker where a measure's arithmetic contradicts the claim. In particular, re-examine WeightedKappaAgreement (complete-grid normalization, same chance-correction structure as the unmarked CohenKappaAgreement) and any other coding measure whose expected-agreement estimate depends on complete marginals.

Describe alternatives you've considered

• Leave IMissingValueSupport coding-only (status quo). Simple and honest, but the concept stays inexpressible for aligning even though Gamma genuinely supports null labels, unitizing keeps conflating the two states, and the un-audited coding markers (e.g. WeightedKappaAgreement) keep giving callers a wrong canHandle answer.
• Mark aligning measures (e.g. GammaAgreement) IMissingValueSupport without a detector. Makes the instanceof query semantically correct, but canHandle cannot verify or enforce anything for an aligning study - a half-wired capability that can mislead callers.
• Do nothing. Callers keep working around the missing distinction (e.g. dummy "X" categories in unitizing), the capability query stays coding-scoped, and existing mis-markings persist.

Acceptance criteria

• A documented, consistent definition of "missing unit" vs "unit with null value" across all three study families.
• A study-level detector for the null-value case in each family (mirroring hasMissingValues()).
• canHandle gives correct, detector-backed answers for coding, unitizing, and aligning studies.
• Unitizing can represent a present-but-unlabeled unit distinctly from a gap (or an explicit, documented decision that it deliberately does not, with rationale).
• Every measure's IMissingValueSupport marker matches its actual behavior under missing/null values - verified against the arithmetic, with WeightedKappaAgreement explicitly resolved (marker removed, or its normalization fixed to tolerate incomplete grids and a test added to justify keeping it).
• No regression in existing measure results; a migration note for any change to the null == gap unitizing convention.

Additional context

Follow-up to #16 (Asking a measure whether it supports null values), which introduced IMissingValueSupport, IMultiRaterAgreement, and IAgreementMeasure.canHandle scoped to coding studies. This issue is the principled next step: make the underlying missing-vs-null concept cross-family and detector-backed, and audit the markers already assigned within coding.

Relevant code:

• org.dkpro.statistics.agreement.coding.CodingAnnotationStudy#hasMissingValues()
• org.dkpro.statistics.agreement.coding.WeightedKappaAgreement (marked IMissingValueSupport, but normalizes over a complete rater×item grid)
• org.dkpro.statistics.agreement.coding.CohenKappaAgreement (same chance-correction structure, correctly not marked)
• org.dkpro.statistics.agreement.unitizing.UnitizingAnnotationStudy#addUnit(...) (Javadoc equating a null category with a gap)
• org.dkpro.statistics.agreement.aligning.dissimilarity.NominalFeatureDissimilarity#dissimilarity(...) (empty unit vs null-typed unit handling)