feat(u32): document compressed boundary

[brenorb]

Summary

• document and catalog the existing u32_compress() / u32_uncompress() ScriptNum boundary
• add boundary tests covering zero, sign-byte transitions, 0x80000000, and u32::MAX
• reject wider-than-five-byte decoder inputs and add generated script/witness/stack metrics

Research framing

Question: what is the exact executable contract and cost of the existing u32-to-ScriptNum boundary?

Result: four byte items compress to one minimal ScriptNum, using a fifth sign byte when needed; decompression expands an at-most-five-byte ScriptNum back to four byte items. The representative fragments are 76 and 413 script bytes, with 9 and 7 serialized witness bytes and 7-item strict peaks.

Execution is locally reproduced and remains unclassified for deployment. The helpers do not certify canonical byte limbs or the unsigned range; callers must enforce those protocol boundaries. Six-byte inputs fail; the complete catalog entries exclude input pushes and output checks.

Validation

• cargo fmt --all -- --check
• cargo test --locked compressed_u32
• cargo test --locked uncompress_rejects_scriptnums_wider_than_five_bytes
• cargo test --locked --test primitive_metrics u32_compression_metrics_are_current
• python3 tools/kb.py validate
• git diff --check

The full repository test suite was intentionally not run; field-arithmetic tests remain skipped by default.

[RobinLinus]

Review recommendation: fix before merging.

The new representation documentation does not match u32_compress(). It describes unsigned minimal ScriptNum encoding, but the implementation maps the u32 through signed two's-complement values. For example, 0xffffffff compresses to -1 (one ScriptNum byte), rather than an unsigned five-byte encoding; 0x80000000 is the special five-byte boundary.

Correct the contract, examples and serialized witness maximum accordingly. Align the decoder's caller obligations with the explicit canonical boundary in #30. Tests passing for the existing implementation do not validate this new prose. Evidence for the mismatch: inspected.

September 11 local validation (unchanged PR head): python3 tools/kb.py validate passed; 3 selected library tests passed. These are targeted local checks, not a full-suite or complete-spend validation. Results using a stack-limit-disabled execution helper remain research-unlimited.

Integration: rebase on current main, resolve any catalog/NR/OP ID collisions, and run the affected correctness tests and focused metric checks.

Reviewed commit: 055b0b0b7a814cf909dd13c7e82a32b4f507803d.

[brenorb]

Updated in place and rebased onto current main. Corrected the contract from unsigned encoding to signed two's-complement ScriptNum encoding: 0xffffffff is -1 (81), 0x80000000 is -2^31 (00 00 00 80 80), and every five-byte input takes the unchecked decoder sentinel branch. Added direct byte-for-byte compression assertions, invalid five-byte/six-byte decoder coverage, surrounding canonical-wrapper coverage, and corrected metrics/catalog witness bounds (9/13-byte compression witness; 7-byte decompression maximum).

Checks: focused boundary tests and metrics passed; cargo test --locked --test primitive_metrics (106 passed, 1 ignored); cargo fmt --all -- --check; python3 tools/kb.py validate (101 records, 246 configurations); git diff --cached --check.

[RobinLinus]

Review of d58f3093cde492b937ec674838187f660ce1f281:

The signed-compression documentation now correctly explains 0xffffffff -> -1 and the five-byte sentinel, addressing the earlier representation error. Preserve current catalog and metric additions during the rebase, then rerun the signed-encoding/decoder boundary tests and named compression metrics.

Current integration conflicts: knowledge/catalog.json, knowledge/primitives/u32.md, tests/primitive_metrics.rs. These must be resolved and the integrated result validated before merging.

Validation scope: source/diff and existing CI review; no new full-repository or field-arithmetic test run was requested.

[brenorb]

Addressed in 565e698557065859c31643d12cb188e69b45b5d2, integrating current main (3d1001fc617eacd7920200b5c87ed7be25a2269c, which since the review has absorbed 27 more PRs including #84 signed less-than, #45 fixed rotations/equality, and #40 the reverse-byte adapter) while preserving every catalog record/configuration, every metric test, and every documentation bullet from both sides.

• Documentation claim re-checked after the merge. u32_compress() maps value as i32 before minimal ScriptNum serialization: 0xffffffff still encodes to -1 (bytes [0x81]) and 0x80000000 still takes the five-byte -2^31 sentinel (bytes [0x00, 0x00, 0x00, 0x80, 0x80]), verified by the existing arithmetic::u32::stack::tests::compress_emits_signed_scriptnum_encodings test, which passes on the merged tree. The decoder side is exercised by uncompress_treats_any_five_byte_input_as_the_signed_boundary (any 5-byte input takes the -2^31 branch) and uncompress_rejects_scriptnums_wider_than_five_bytes (6-byte input errors ScriptIntNumericOverflow); both pass.

• Conflicted files: actual vs. reported. Robin's list (relative to the pre-Integrate 27 reviewed PRs with corrected validation and Taproot preflight #244 main) was knowledge/catalog.json, knowledge/primitives/u32.md, tests/primitive_metrics.rs. Against the current origin/main, git merge --no-commit --no-ff origin/main reproduced exactly those three plus one more: src/arithmetic/u4/mod.rs (pure pub mod list churn — main added cyclic_equality, equality, msb, mul, odd_inverse, threshold, transition_count, trichotomy modules the PR predates; no semantic conflict). src/arithmetic/u32/README.md and src/arithmetic/u32/stack.rs — both touched by the PR — auto-merged cleanly with no conflict markers.

• knowledge/catalog.json: both sides' records and configurations kept. The arithmetic/u32 record's "tests" array now lists both primitive_metrics::u32_compression_metrics_are_current (PR) and primitive_metrics::u32_reverse_byte_adapter_metrics_are_current (main); the "security"/"stack_contract" prose was merged so both sides' sentences survive (main's superset security text, plus the PR's u32_compress/u32_uncompress sentence folded into main's fuller stack_contract). Proof by set comparison: main has 117 records / 283 configurations, the PR has 101 records / 246 configurations, the merged file has 117 records / 285 configurations. main_record_ids ⊆ merged, pr_record_ids ⊆ merged (both True), likewise for (record, configuration) id pairs, with zero duplicate ids in the merged file and zero ids outside the union of both sides (285 = 283 + the 2 PR-only configurations compress-unchecked/uncompress-unchecked). python3 tools/kb.py validate confirms knowledge base valid: 117 records, 285 configurations.

• knowledge/primitives/u32.md: both sides' bullets kept. Main's 43 top-level bullets and the PR's 40 are all present as a plain concatenation in the merged 45-bullet list (main_bullets ⊆ merged and pr_bullets ⊆ merged, both confirmed by set difference against the merged file).

• tests/primitive_metrics.rs: every #[test] fn and metric key from both sides kept. Main has 206 #[test] fns, the PR has 169 (168 shared with main), the merged file has 207 — exactly the union, with no duplicates. Every metric key: "..." string from both sides is present in the merged file (checked by set containment). The conflict had cut across five functions (u32_compression_metrics_are_current from the PR; u32_reverse_byte_adapter_metrics_are_current, u32_equality_metrics_are_current, u32_fixed_rotation_metrics_are_current, u32_byte_planes_metrics_are_current from main); each was reassembled as a complete, independent function and all five compile and pass.

• src/arithmetic/u4/mod.rs: resolved as the alphabetically-sorted union of both sides' pub mod declarations (46 lines, no module dropped from either side).

• Negative-test meaningfulness (mutant proof). On the integrated tree, the three signed-encoding/decoder-boundary tests pass: compress_emits_signed_scriptnum_encodings, uncompress_treats_any_five_byte_input_as_the_signed_boundary, uncompress_rejects_scriptnums_wider_than_five_bytes (part of 29/29 passing in arithmetic::u32::stack::tests). I then applied a temporary, minimal mutant to u32_uncompress() in src/arithmetic/u32/stack.rs, widening the five-byte sentinel check to six bytes (OP_SIZE OP_5 OP_EQUAL → OP_SIZE OP_6 OP_EQUAL), reintroducing exactly the kind of decoder-boundary defect Robin's comment addresses (a five/six-byte value the decoder must reject is instead accepted as the sentinel). Under the mutant:

  • uncompress_rejects_scriptnums_wider_than_five_bytes FAILED: assertion left == right failed\n left: None\n right: Some(ScriptIntNumericOverflow) — the 6-byte oversized input that must error was silently accepted.
  • uncompress_treats_any_five_byte_input_as_the_signed_boundary FAILED: panicked ... unexpected five-byte behavior: ... Error: ScriptIntNumericOverflow — genuine 5-byte sentinel inputs no longer matched the (now 6-byte) size check and fell through to the arithmetic-comparison branch, overflowing.
  • canonical_uncompress_accepts_signed_boundaries also failed as a side effect (same underlying fragment), confirming the mutant's blast radius was real, not vacuous.
    I reverted the mutant (git checkout -- src/arithmetic/u32/stack.rs) and confirmed git diff was empty before re-running the three tests, which passed again (29/29 in the module), before committing.

• Validation on the committed HEAD 565e698557065859c31643d12cb188e69b45b5d2 (clean tree, git status --porcelain empty):

  • cargo test --locked --lib arithmetic::u32::stack::tests:: -- --skip fields:: → 29 passed, 0 failed
  • cargo test --locked --test primitive_metrics u32 -- --skip fields:: → 49 passed, 0 failed (includes u32_compression_metrics_are_current and all other u32-family metric tests)
  • python3 tools/kb.py validate → knowledge base valid: 117 records, 285 configurations
  • python3 -m unittest discover -s tools -p 'test_*.py' → 49 passed
  • cargo fmt --all -- --check → clean (no diff)
  • cargo test --locked --no-run → whole crate + all test binaries compile (no errors)
  • No field-arithmetic tests were run at any point.

• Metric changes: none. u32_compression_metrics_are_current passed against the existing fixture values (76/9/13/7 for compress; 413/7/7/7 for uncompress) with no refresh; the merge did not touch src/arithmetic/u32/stack.rs's u32_compress/u32_uncompress implementations (they auto-merged with zero conflict), so no metric drift was expected or observed.

• Push/PR state: pushed fast-forward (d58f309..565e698) to brenorb/bitcoin-scripts:feat/u32-compressed-boundary. gh pr view 42 --repo solving-bitcoin/bitcoin-scripts --json headRefOid,mergeable reports headRefOid: 565e698557065859c31643d12cb188e69b45b5d2, mergeable: MERGEABLE.

Evidence throughout is local (locally-reproduced), execution class unclassified per crate::support::execution defaults (tapscript context, stack limit enforced) — no Bitcoin Core or relay-policy validation is claimed.