Skip to content

research: add evidence-bound transformer lifecycle benchmark - #2857

Draft
Tristan-Stoltz-ERC wants to merge 117 commits into
mainfrom
research/pwr-tx-bench-001
Draft

Tristan-Stoltz-ERC wants to merge 117 commits into
mainfrom
research/pwr-tx-bench-001

Conversation

@Tristan-Stoltz-ERC

@Tristan-Stoltz-ERC Tristan-Stoltz-ERC commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Tracks PWR-TX-BENCH-001 #2855, PWR-TX-PROFILE-001 #2858, PWR-TX-SOTA-001 #2859, PWR-TX-GLOBAL-001 #2872, and PWR-TX-TRACE-001 #2873.

What this adds

  • A standalone, dependency-free Rust screening harness under research/transformer-bench/, isolated as a nested Cargo workspace.
  • A first-order loss model, P_loss = P0 + Pk × mean_square_current_pu2, expected/worst-case discounted loss economics, and separate economics-only versus manifest-complete Pareto frontiers.
  • Separate interval mean-current and mean-square-current fields to prevent profile aggregation from silently understating quadratic load losses.
  • A second binary, surrogate-eval, reporting relative-error distributions, Spearman ranking, top-k recall, best-candidate regret, prediction-interval coverage/width, and caller-declared full-pool FEA-call reduction.
  • Explicit frequency (Hz) and excitation class. Metrics are stratified by frequency, waveform, geometry family, material and combined operating domain; ranking fields are blank for pooled incomparable frequency/excitation domains.
  • A third binary, global-trace-validate, which validates canonical UTC calendar-year profiles, checks exact timestamp chronology and interval coverage, supports common and leap years, rejects gaps/overlaps/duplicate intervals, verifies current-moment inequality, and requires consistent tariff currency/year/REAL-vs-NOMINAL metadata. Source-local calendar-year boundaries are not accepted directly yet; audited upstream re-bounding/resampling is required.
  • Synthetic fixtures for 50/60 Hz and waveform strata plus complete 8,760-hour and 8,784-hour annual traces. Fixtures test code behavior only and are not physical evidence.
  • GLOBAL_APPLICABILITY.md, a global physical/evidence contract, a versioned JSON region-profile schema, and DRAFT regional seeds for the EU, United States, India and South Africa. All have zero reviewers and are explicitly unverified.
  • CI coverage for cargo test --manifest-path research/transformer-bench/Cargo.toml --locked.

Research direction

Issue #2859 defines a fair comparison against published geometry-aware core-loss surrogate work, with matched query budgets, held-out geometry families, ranking/regret and uncertainty metrics. Issue #2872 defines global portability as one canonical physical/evidence contract with many explicit regional profiles. Issue #2873 adds a runnable annual trace-integrity gate so time-series input is not accepted based only on an 8,760-hour sum.

Important boundaries

This remains analytic loss screening plus input/metric validation—not an electromagnetic or thermal design solver. The manifest checker validates digest syntax only; it does not authenticate evidence. The surrogate evaluator cannot authenticate dataset splits or query-count ledgers. The trace validator checks timestamp and metadata integrity, but it does not prove meter calibration, source-trace authenticity, representative-year quality, ISO currency registry membership, tariff legality, or correct local-time-to-UTC conversion. Draft regional profiles do not certify any local law, standard or utility specification.

The loss formula still omits reference-temperature correction, harmonic/stray-loss components, voltage variation, coupled thermal state, insulation fields and short-circuit mechanics. No synthetic fixture is an FEA result or physical test.

Validation status

I checked synthetic fixture columns and interval durations independently in Python and confirmed the regional profile files parse as JSON. Those checks do not compile or execute the Rust code, validate JSON Schema conformance, or establish current legal applicability. This environment has no Rust/Cargo toolchain; CI is configured to run the tests, but no pass is claimed until a run completes successfully. Please review this as a research starter, not a production capability.

Added in follow-up research iteration

  • Added split-audit (research/transformer-bench/src/bin/split-audit.rs) to enforce split integrity for held-out geometry families, manufacturers, material lots, and measurement campaigns. It requires the held-out group dimension to be disjoint across TRAIN, VALIDATION, and TEST, rejects duplicate observation IDs, and rejects incomplete partitions. Random-row splits are explicitly labeled as baseline-only, not transfer evidence.
  • Added fixtures/split-audit-synthetic.csv, unit tests including leakage negative controls for each held-out dimension, and documentation in README.md and GLOBAL_APPLICABILITY.md.
  • This new work is committed but not Rust-compiled in the current environment because Cargo/rustc are unavailable. No test pass is claimed; the draft PR should remain unmerged until its exact head compiles and all tests pass.

Evidence integrity follow-up

  • Added evidence-receipt (research/transformer-bench/src/bin/evidence-receipt.rs) to hash actual artifact bytes against a declared manifest, restrict paths to an artifact root, reject traversal/symlink escape and duplicate IDs/paths, and emit a deterministic receipt bound to the manifest SHA-256.
  • Added a synthetic known-answer payload/manifest plus tests for standard SHA-256 vectors, tampering, malformed digests, duplicate IDs/paths, traversal, and Unix symlink escape.
  • Updated README and global applicability contract to distinguish byte integrity from provenance/authenticity and scientific validity.
  • Latest local review was limited to source/fixture consistency. No Rust compiler or Cargo is installed here; the new code and all existing Rust tests remain unverified until CI completes successfully on the exact head. Keep this PR draft/unmerged until then.

Next research work item

Created PWR-TX-HST-001 #2899 and added THERMAL_LIFETIME_BENCHMARK.md. This spec uses recent published transformer thermal-surrogate/RUL work to define fair comparisons against IEC thermal references, 3D CFD/CHT, thermal networks and ROMs; it adds held-out transfer, transient hotspot error, ageing-integral error, false-safe rate and total compute cost. It explicitly separates solver-relative accuracy from field-observed remaining-useful-life validation. This is a research protocol, not an implemented thermal solver or achieved benchmark result.

Matched-budget verification-policy benchmark

  • Added acquisition-bench plus a 24-row synthetic candidate pool to compare RANDOM, MIN_PREDICTED, MAX_UNCERTAINTY, LOWER_CONFIDENCE_BOUND, EXPECTED_IMPROVEMENT and DIVERSE policies under equal budgets and paired deterministic seeds.
  • Reports oracle-relative best-found regret, oracle top-k recall, the selected candidate IDs and prediction error on queried points.
  • Added BUDGETED_VERIFICATION_BENCHMARK.md and issue PWR-TX-ACQ-001 #2907.
  • Critical boundary: this is an offline replay over a frozen synthetic pool. It does not run FEA or retrain a surrogate and cannot establish active-learning gains or compute savings. Next work is real traceable pools, confidence-interval aggregation and eventually a pinned iterative solver/retraining interface.
  • No Cargo toolchain was available in the authoring environment. Exact-head Rust test status remains unverified.

Matched-budget summary and paired comparison

  • Added acquisition-summary (research/transformer-bench/src/bin/acquisition-summary.rs) to parse raw policy trials, enforce paired seeds and identical query budgets/oracle baseline, compute mean/median regret and top-k recall, and compare each policy against RANDOM, MIN_PREDICTED and DIVERSE with paired seed-level bootstrap intervals.
  • Added tests for summary determinism, known paired improvements, complete input parsing and missing-policy/unequal-budget rejection; documented run commands and interpretation.
  • Critical limitation: these are seed-level intervals on one pool, not uncertainty over independent transformers/datasets. Deterministic policies can have degenerate seed intervals. A superiority claim still requires multiple real, traceable pools and group-held-out evaluation.
  • Cargo/rustc are unavailable locally; no Rust compilation/test pass is claimed.

Uncertainty-aware robust design frontier

  • Added robust-frontier (research/transformer-bench/src/bin/robust-frontier.rs) across six minimized objectives: lifecycle cost, no-load loss, load loss, maximum hotspot temperature, mass and embodied carbon.
  • Enforces a common profile, currency, price year/basis and carbon boundary for all candidates; validates finite ordered bounds and unique IDs.
  • Reports nominal midpoint Pareto dominance separately from certain interval dominance and possible interval dominance. A candidate is removed from the robust frontier only if another is guaranteed no worse across every declared objective bound and strictly better on at least one.
  • Added a synthetic fixture with certain dominance and an intentionally overlapping-bound case where midpoint dominance is not enough to remove a candidate; added tests for trade-offs, possible-vs-certain dominance, metadata/bounds errors, duplicate IDs and permutation invariance.
  • Added ROBUST_PARETO_FRONTIER.md and issue PWR-TX-RPF-001 #2917.
  • This is a screening primitive, not a solver or qualification gate. Bounds are caller-declared and treated as independent interval boxes; correlated scenario uncertainty is a future extension. No physical/SOTA improvement is claimed.
  • Rust/Cargo is unavailable locally; no compilation/test pass is claimed until exact-head CI completes.

Joint-scenario risk-aware Pareto frontier

  • Added scenario-frontier (research/transformer-bench/src/bin/scenario-frontier.rs) to preserve paired objective vectors across common named scenarios and weights.
  • Computes expected, objective-wise CVaR95 (exact fractional worst 5% probability tail), worst-case and scenario-wise robust Pareto fronts across lifecycle cost, no-load/load losses, max hotspot temperature, mass and embodied carbon.
  • Requires common scenario sets/weights for every candidate and common profile/currency/price-basis/carbon-boundary metadata.
  • Added four-candidate/four-scenario synthetic fixture, hand-checkable CVaR tests, scenario-wise dominance and opposing trade-off tests, plus parser tests for missing scenarios/inconsistent weights.
  • Added JOINT_SCENARIO_FRONTIER.md and issue PWR-TX-SCEN-001 #2919.
  • This preserves scenario pairing but only guarantees properties over a finite caller-declared ensemble. Scenario weights are not validated; objective-wise CVaR is not a multivariate risk measure. No solver, physics model or achieved saving is claimed.
  • Cargo/rustc are unavailable locally; exact-head Rust tests remain unverified.

Numerical integrity follow-up

  • Scenario aggregation now fails closed if expected, CVaR95 or worst-case derived objectives become non-finite, even if individual inputs were finite.
  • The synthetic joint-scenario fixture was independently re-checked for all 16 rows, 4 candidates, 4 common scenarios, consistent regional/economic/carbon metadata and total scenario weight 1.0. The hand CVaR95 case (0.88×10, 0.10×20, 0.02×30) returns 24.0. These are independent structural/arithmetic checks, not Rust test execution.

Temporal operating-twin input contract

  • Added temporal-profile (research/transformer-bench/src/bin/temporal-profile.rs) for synchronized annual scenario inputs: current mean and mean-square, ambient temperature, EV/PV context, tariff basis, carbon intensity and source-trace identity.
  • Requires a complete UTC calendar year, exact shared interval boundaries across all scenarios, gapless/non-overlapping intervals, consistent scenario weights/metadata and the current-moment inequality. Supports 8,760/8,784-hour years.
  • Added aligned synthetic 2024 leap-year fixture and negative control for a shifted scenario grid; docs in TEMPORAL_OPERATING_TWIN.md, issue PWR-TX-TEMP-001 #2934.
  • This validates time-series integrity only. It does not calculate EV/PV net load, local-time/DST conversion, electrical network state, thermal dynamics, ageing or source provenance.

Manufacturing digital thread for transformers and electrical equipment

  • Added manufacturing-ledger (research/transformer-bench/src/bin/manufacturing-ledger.rs) for product/serial/component/lot genealogy, process/inspection/test/rework/NCR/packaging/final-release events, recipe hashes, evidence references, event chronology, energy and scrap.
  • Added synthetic power-transformer, switchgear and magnetic-component traces; tests cover genealogy cycles/missing parents, event sequence/overlap, evidence hash syntax and family reuse.
  • Added MANUFACTURING_DIGITAL_THREAD.md and issue PWR-TX-MFG-001 #2933, with process-quality benchmark targets and a federated product/process/test digital-thread design.
  • This is trace integrity, not product acceptance: it does not check mandatory route/test-plan completeness, validate test measurements against approved limits, verify evidence files, or authorize release. Those are explicit next work packages.

These extensions apply one evidence architecture across transformer operation, production execution and other electrical-equipment families, while retaining separate product-specific route and qualification profiles.

Provenance coverage refinement

The temporal profile summary now reports unique interval-ID and source-trace counts per scenario in addition to its weighted operating descriptors. UTC chronology and joint grid checks remain separate from source authenticity: IDs are recorded, not authenticated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant