Repository navigation
research: add evidence-bound transformer lifecycle benchmark - #2857
Draft
Tristan-Stoltz-ERC wants to merge 117 commits into
Draft
Tristan-Stoltz-ERC wants to merge 117 commits into
Tristan-Stoltz-ERC wants to merge 117 commits into
Conversation
This was referenced Oct 29, 2026
7 tasks
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracks PWR-TX-BENCH-001 #2855, PWR-TX-PROFILE-001 #2858, PWR-TX-SOTA-001 #2859, PWR-TX-GLOBAL-001 #2872, and PWR-TX-TRACE-001 #2873.
What this adds
research/transformer-bench/, isolated as a nested Cargo workspace.P_loss = P0 + Pk × mean_square_current_pu2, expected/worst-case discounted loss economics, and separate economics-only versus manifest-complete Pareto frontiers.surrogate-eval, reporting relative-error distributions, Spearman ranking, top-k recall, best-candidate regret, prediction-interval coverage/width, and caller-declared full-pool FEA-call reduction.global-trace-validate, which validates canonical UTC calendar-year profiles, checks exact timestamp chronology and interval coverage, supports common and leap years, rejects gaps/overlaps/duplicate intervals, verifies current-moment inequality, and requires consistent tariff currency/year/REAL-vs-NOMINAL metadata. Source-local calendar-year boundaries are not accepted directly yet; audited upstream re-bounding/resampling is required.GLOBAL_APPLICABILITY.md, a global physical/evidence contract, a versioned JSON region-profile schema, and DRAFT regional seeds for the EU, United States, India and South Africa. All have zero reviewers and are explicitly unverified.cargo test --manifest-path research/transformer-bench/Cargo.toml --locked.Research direction
Issue #2859 defines a fair comparison against published geometry-aware core-loss surrogate work, with matched query budgets, held-out geometry families, ranking/regret and uncertainty metrics. Issue #2872 defines global portability as one canonical physical/evidence contract with many explicit regional profiles. Issue #2873 adds a runnable annual trace-integrity gate so time-series input is not accepted based only on an 8,760-hour sum.
Important boundaries
This remains analytic loss screening plus input/metric validation—not an electromagnetic or thermal design solver. The manifest checker validates digest syntax only; it does not authenticate evidence. The surrogate evaluator cannot authenticate dataset splits or query-count ledgers. The trace validator checks timestamp and metadata integrity, but it does not prove meter calibration, source-trace authenticity, representative-year quality, ISO currency registry membership, tariff legality, or correct local-time-to-UTC conversion. Draft regional profiles do not certify any local law, standard or utility specification.
The loss formula still omits reference-temperature correction, harmonic/stray-loss components, voltage variation, coupled thermal state, insulation fields and short-circuit mechanics. No synthetic fixture is an FEA result or physical test.
Validation status
I checked synthetic fixture columns and interval durations independently in Python and confirmed the regional profile files parse as JSON. Those checks do not compile or execute the Rust code, validate JSON Schema conformance, or establish current legal applicability. This environment has no Rust/Cargo toolchain; CI is configured to run the tests, but no pass is claimed until a run completes successfully. Please review this as a research starter, not a production capability.
Added in follow-up research iteration
split-audit(research/transformer-bench/src/bin/split-audit.rs) to enforce split integrity for held-out geometry families, manufacturers, material lots, and measurement campaigns. It requires the held-out group dimension to be disjoint across TRAIN, VALIDATION, and TEST, rejects duplicate observation IDs, and rejects incomplete partitions. Random-row splits are explicitly labeled as baseline-only, not transfer evidence.fixtures/split-audit-synthetic.csv, unit tests including leakage negative controls for each held-out dimension, and documentation inREADME.mdandGLOBAL_APPLICABILITY.md.Evidence integrity follow-up
evidence-receipt(research/transformer-bench/src/bin/evidence-receipt.rs) to hash actual artifact bytes against a declared manifest, restrict paths to an artifact root, reject traversal/symlink escape and duplicate IDs/paths, and emit a deterministic receipt bound to the manifest SHA-256.Next research work item
Created PWR-TX-HST-001 #2899 and added THERMAL_LIFETIME_BENCHMARK.md. This spec uses recent published transformer thermal-surrogate/RUL work to define fair comparisons against IEC thermal references, 3D CFD/CHT, thermal networks and ROMs; it adds held-out transfer, transient hotspot error, ageing-integral error, false-safe rate and total compute cost. It explicitly separates solver-relative accuracy from field-observed remaining-useful-life validation. This is a research protocol, not an implemented thermal solver or achieved benchmark result.
Matched-budget verification-policy benchmark
acquisition-benchplus a 24-row synthetic candidate pool to compare RANDOM, MIN_PREDICTED, MAX_UNCERTAINTY, LOWER_CONFIDENCE_BOUND, EXPECTED_IMPROVEMENT and DIVERSE policies under equal budgets and paired deterministic seeds.Matched-budget summary and paired comparison
acquisition-summary(research/transformer-bench/src/bin/acquisition-summary.rs) to parse raw policy trials, enforce paired seeds and identical query budgets/oracle baseline, compute mean/median regret and top-k recall, and compare each policy against RANDOM, MIN_PREDICTED and DIVERSE with paired seed-level bootstrap intervals.Uncertainty-aware robust design frontier
robust-frontier(research/transformer-bench/src/bin/robust-frontier.rs) across six minimized objectives: lifecycle cost, no-load loss, load loss, maximum hotspot temperature, mass and embodied carbon.Joint-scenario risk-aware Pareto frontier
scenario-frontier(research/transformer-bench/src/bin/scenario-frontier.rs) to preserve paired objective vectors across common named scenarios and weights.Numerical integrity follow-up
Temporal operating-twin input contract
temporal-profile(research/transformer-bench/src/bin/temporal-profile.rs) for synchronized annual scenario inputs: current mean and mean-square, ambient temperature, EV/PV context, tariff basis, carbon intensity and source-trace identity.Manufacturing digital thread for transformers and electrical equipment
manufacturing-ledger(research/transformer-bench/src/bin/manufacturing-ledger.rs) for product/serial/component/lot genealogy, process/inspection/test/rework/NCR/packaging/final-release events, recipe hashes, evidence references, event chronology, energy and scrap.These extensions apply one evidence architecture across transformer operation, production execution and other electrical-equipment families, while retaining separate product-specific route and qualification profiles.
Provenance coverage refinement
The temporal profile summary now reports unique interval-ID and source-trace counts per scenario in addition to its weighted operating descriptors. UTC chronology and joint grid checks remain separate from source authenticity: IDs are recorded, not authenticated.