Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 703 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 703 commits into
sbryngelson wants to merge 703 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 61.67% 62.31% +0.63%
==========================================
Files 84 96 +12
Lines 21619 28763 +7144
Branches 3196 4719 +1523
==========================================
+ Hits 13334 17924 +4590
- Misses 6093 7707 +1614
- Partials 2192 3132 +940 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
Removing the multi-level subcycle PROHIBIT left twelve lines of explanation sitting in a list of checks with nothing after them - a comment justifying why something is NOT checked, in the file whose job is checking. The content is already in the two places it belongs: s_amr_advance_children's docstring (the level-wide walk and the interposed halo) and the C45DBB52 test comment (the amr_cpat_mar margin argument).
be94db3 shipped the per-level cap saying it was verified by reading the slot arithmetic, not by running, because no golden covers amr_max_level >= 3. It is now verified by running, and the evidence belongs where the next reader will be standing. m = 255, amr_max_level = 3, amr_buf = 48, np = 1: the fixed /2 keeps ONE oversized level-3 box where the per-level cap keeps 2, and the run aborts in s_amr_free_slot with a Fortran 'Invalid descriptor' and a core dump; the per-level cap exits 0. Two earlier counterfactuals produced byte-identical output and proved nothing - m = 63 was too small, and scaling the grid without scaling amr_buf left the boxes tracking the feature rather than the domain, far below either cap. A635AA56's comment already says it is the BUFFER that widens the box; applying that is what made the cap bind.
…nchmarking The balancer only becomes observable when a level holds many boxes, and stripe cases collapse to ~2 boxes per level by np=8 - at which point every owner mapping produces an identical assignment. Scattered blobs give many disjoint tag clusters, hence many boxes per level. Hardcoded rather than an analytic patch on purpose: analytic expressions compile into case.fpp, so every variation costs a rebuild, whereas a(2:5) (count, seed, radius, amplitude) are read at run time and sweep freely against one binary. Centres come from an additive-irrational (Weyl) sequence, NOT random_number: each rank fills only its own cells, so the IC must be a pure function of position or ranks disagree at the subdomain seam and the decomposition stops being exact. Purely additive - a new case in a select, unreachable unless a case sets hcid = 299 - so no existing golden can enter it. Verified: all four targets build (the file compiles into pre_process via m_icpp_patches and simulation via m_ib_patches), pre_process runs it clean at 511x255 on 2 ranks. Also revises the plan doc: step 5 landed, acceptance criterion 2 promoted to a runnable step, and a retraction of this file's own earlier claim that m_rank_timing was unimplemented - it is fully wired at eight call sites.
…uption) s_amr_regrid's 'brand-new region' branch - a level>=2 child whose region has no old fine data to cluster - emitted its box straight into boxes() without ever consulting amr_maxc_fit. It is the ONLY box-emitting path that skips s_amr_tile_box. The inset bounds the child as a FRACTION of its parent (span/4 each side), but the slot coord arrays are allocated ONCE to amr_ref_ratio*amr_maxc_fit, so the child must be bounded in ABSOLUTE cells. s_amr_build_block_coords then sizes fcb from the block's TRUE extent and writes past x_cb. Under -O2 that silently scribbles past a heap allocation every regrid and surfaces much later as 'corrupted size vs. prev_size' inside an unrelated free() - the crash site is nowhere near the bug site, which is why three passes of reading the seam/overlap/tiling code cleared every candidate. Integer division is the trigger: s_amr_tile_box splits a wide region into 63 and 64, not 64 and 64, and a parent of span 63 gives ins = 63/4 = 15 and a child of span 33 against a level-2 cap of 32, while span 64 gives exactly 32 and is fine. A one-cell difference in the parent flips it. Introduced by me in cfdd284 (step 4): s_amr_build_block_coords replaced a parent-relative computation with a global-extent one so a rank could build coords for a block whose parent it does not own, and the destination sizing did not follow. VERIFIED: bounds-checked CPU build (gfortran --no-gpu --debug, the ONLY configuration with -fcheck - cmake/GPU.cmake gives LLVMFlang none) traps at m_amr.fpp:584 'Index 128 of dimension 1 of array fcb above upper bound of 127' without this change, and with it the span-33 box becomes 17+16 and the run exits 0.
amr_max_grid_size had ZERO test coverage - not one golden or example case set it, only the schema, validator and definitions - despite being the mechanism the rank-independent-cap work rests on. Any defect reachable only through the pinned-cap path was invisible to CI, and one was: the brand-new-region box emitter skipped s_amr_tile_box and corrupted the heap (cfbaceb). VERIFIED to fail without that fix: 8 over-cap boxes and, under a bounds-checked build, 'Index 32 of dimension 1 of array fcb above upper bound of 31' (the trap index tracks the cap, max_f1 = 2*amr_max_grid_size - 1). Both knobs are load-bearing and narrow. amr_max_grid_size = 16 makes s_amr_tile_box split a wide region into 16 and 15, and span 15 gives ins = 15/4 = 3 and a child of 9 against a level-2 cap of 8; INTEGER DIVISION is the trigger, so a cap of 20 splits evenly (20/20 -> child 10, cap 10) and does not reproduce. The blob IC is equally load-bearing - a Sod-like patch grows level-2 regions that already have fine data, so the brand-new-region branch never fires and the bug is unreachable however the cap is set; three separate small 1D Sod candidates passed for exactly that reason. Also records the caveat in the plan doc: step 2's measured speedup was collected on this path, so it is no longer a clean measurement. 72/72 AMR goldens, precheck 7/7.
Two additions, both from today's heap-corruption hunt (cfbaceb). (1) s_amr_check_box_caps aborts if any level>=1 box exceeds amr_maxc_fit/amr_ref_ratio**(lev-1), checked once where the box set is final rather than trusting each emitter to have routed through s_amr_tile_box. A violation is otherwise silent and catastrophic - the coord arrays are sized to the cap, so an over-cap box writes past x_cb and corrupts the heap every regrid, surfacing much later inside an unrelated free(). VERIFIED BOTH WAYS: on the unfixed reproducer it prints '[amr] box cap violated: level 2 dim 1 span 9 > cap 8' and aborts by name instead of corrupting memory; and 71/71 existing AMR cases pass with it compiled in, zero false aborts. The false-positive hazard was real - in 1D/2D amr_maxc_fit(2:3) is 1, so the level-2 cap divides to 0 and a naive check would abort every 1D golden; it mirrors s_amr_tile_box's own max(tc,1) floor. (2) [amr-balance] now prints boxes_max/mean beside the weight imbalance, because measured per-rank time tracks BOX COUNT rather than cells: at np=8 the cell weight sat at 1.050 while box count was 1.308 and rank_time 1.259. Printing both makes that falsifiable in one run. 72/72 AMR goldens, precheck 7/7.
2 tasks
s_amr_sfc_cut advances a rank when cum >= tgt, where cum is an n-term ACCUMULATION and tgt = (r+1)*total/num_procs is CLOSED FORM over another n-term sum. At an exact share boundary the two differ by rounding rather than by intent, so the comparison turns on 1 ULP. This has been correct only by luck: every cost term to date is integer-valued (footprint cells, K_ib = 2, K_pc = 3 x integer iteration counts), so 4w and 32w/8 agree bit for bit. ANY fractional cost term breaks it, silently - ranks come out unbalanced with no error and no diagnostic. MEASURED with a fractional term in cost(k): 32 IDENTICAL level-1 weights (9.038222222E+03, verified by printing them) over 8 ranks split 5/3 instead of 4/4, reporting max/mean 1.250 where the same case with integer weights reports exactly 1.000. Adding a tolerance of O(n) ULP of the target - the accumulated rounding bound - restores 1.000 while leaving the genuine imbalance at the other level untouched (1.106, unchanged to three decimals). Far from a share boundary the tolerance is ~1e-12 relative and the greedy is unchanged, which is what 72/72 AMR goldens confirm: no existing case moves. Found while adding a per-box cost term; that term is NOT in this commit - the fragility predates it and is worth fixing on its own.
Fine blocks occupy slots (l0_slot_off, amr_num_blocks]; slots [1, l0_slot_off] are the L0 TILE prefix. At init s_amr_assign_block_owners runs BEFORE s_l0_tiles_init, so those prefix slots are still uninitialized - amr_block_level reads 1 and amr_region_lo is all zeros, i.e. Morton key 0. The assigner looped from slot 1 and therefore fed phantom key-0 'level-1 blocks' into the level-1 cut and split them across ranks. A key-0 block can only ever resolve to rank 0 (the cut is non-decreasing and the search returns the first r with key <= cut(r)), so any phantom placed on a higher rank is unrecoverable and s_amr_validate_owner aborts with 'SFC cut-point owner disagrees with amr_block_owner'. Observed directly by dumping the block table in a coexist np=2 run: fine_cut(:,1) = [0, 33792], owner_cut = [-1, -1] (tile cut not yet built), blocks 1-8 all lev 1 / key 0 / lo (0,0,0), blocks 9-10 the real fine blocks at keys 5120 and 33792. Blocks 7 and 8 were stored as rank 1 while f_amr_owner returned 0. LATENT, NOT NEW: with the existing weights every phantom happened to land on rank 0 and the validator agreed by luck. It surfaced only when an experimental per-box cost term moved the cut - the same shape as the ULP tie in 2051aa1, a fragility surviving on arithmetic that happened to be benign. Fix: the assigner and the validator both restrict to (l0_slot_off, amr_num_blocks], the validator skipping the prefix only while amr_owner_cut is unbuilt (the tile-init call site populates both cuts and validates there). Also records in the plan doc that s_amr_validate_owner - marked TRANSITIONAL in-source - is load-bearing and caught this immediately. 72/72 AMR goldens, precheck 7/7.
K_box entered m_constants.fpp as part of an experimental fixed per-box cost term. The consuming code in m_amr.fpp was reverted, but the constant lived in a different file and a git add -A swept it into c3364a5, leaving a parameter with no consumers and a comment citing a calibration sweep that nothing acts on. The experiment itself did not survive scrutiny: at a properly loaded size (75.5M cells, 512 level-1 boxes over 8 ranks = 64 boxes/rank) K_box=8 and K_box=0 measure 1.035 and 1.052 rank-time imbalance, a difference inside the run-to-run variance. The earlier effects were measured on cases small enough that the granularity floor made the distribution lumpy, and on an IC whose blob centres come from a low-discrepancy Weyl sequence - evenly spread by construction, so the workload is inherently balanced and cannot discriminate between cost models. What the experiment DID produce is two latent partitioner bugs it exposed by perturbing the weights, both fixed independently: the ULP-fragile SFC boundary (2051aa1) and the phantom tile-prefix slots (c3364a5).
… questions Three claims in this file were false after the day's measurements. (1) Step 5's conclusion 'box supply binds, not the owner mapping' was inferred from box counts and model imbalance on a starved case with no measured counterpart - retracted. (2) Step 7 was listed as 'measured good, blocked by a coexist regression'; the regression is fixed (c3364a5) and K_box was rejected for an entirely different reason - it delivers ~2% at production scale, inside this machine's run-to-run variance. (3) The K_box table quoted starved-GPU numbers as though they were the result. Replaced with the regime measurements: which metric predicts runtime depends on how much work a box carries, and at 75.5M the two converge (weight 1.058, box count 1.121) so the objectives barely conflict. Includes a prediction of mine that FAILED - I expected K_box to make balance worse at scale by steering toward box count, and it improved it slightly - kept because it constrains the model. Requeues the open questions around what is actually unknown: whether the 45% np=8 ceiling is MFC's or this branch's (portable uniform case on master vs branch, gen_uniform.py), what AMR costs relative to uniform at a loaded size (the AMR np=1 point exceeded the harness cap, so there is no baseline), and a note that the per-box-overhead argument for the batching arc rests on efficiency figures later found contaminated by starved GPUs and run_time_info serialisation.
… accordingly Paired experiment on the identical portable uniform case (gen_uniform.py, standard geometry patches, no hcid, no AMR in the path), MFC master bfdc8f5 vs this branch, 3 reps alternating between trees so drift hits both arms equally: branch/master median ratio 1.022 at np=4 and 1.023 at np=8, against within-tree run-to-run spreads of 6.9% and 11.1%. The branch costs ~2% on a uniform problem, indistinguishable from noise, and uniform efficiency is ~60% at np=8 on BOTH trees. An earlier single-run comparison suggested 15% at np=4; three reps put it at 2%, so that was an artifact. Two consequences. First, no AMR-side work can move the np=8 ceiling - it belongs to the uniform solver on this machine. Second, the batching arc's premise (per-block overhead dominates) was argued from efficiency figures since found contaminated by starved GPUs and by run_time_info forcing a device sync every step; against a 60% ceiling the recoverable headroom is smaller than assumed, and it should not be treated as established. The queue is reordered around what is actually unknown: what AMR costs relative to uniform at a loaded size (the np=1 AMR point blew the harness cap, so there is no baseline), and multi-node scale invariance, which is the real exascale question and which single-node np<=8 structurally cannot answer. Also records the measurement standard this session earned: at 11% within-tree spread, any A/B claiming less than ~10% needs reps.
…with the GLOBAL box set
This was known and never written down as a work item. It appeared only as one clause inside 'The problem, measured' ('every per-box collective in the regrid rebuild loop runs over all boxes on all ranks') with no design entry, no queue entry and no owner, so it was invisible when planning. Audited against the code and quantified. Three costs grow with the global box count while the whole strategy demands boxes_per_level >> num_procs: (1) ONE GLOBAL COLLECTIVE PER BOX PER REGRID - s_set_amr_fine_geometry ends in s_mpi_allreduce_integer_max and is called from do k = 1, nboxes in s_amr_regrid_rebuild_slots; 512 collectives today, 10^5 at the target, and amr_block_batching already measured it at 7.4-13 ms per call because it absorbs the spread in the owner-only work before it, so the cost grows with rank count too. (2) the SFC cut's insertion sort is O(n^2) with a comment saying 'n small' - 5e9 ops at 10^5 boxes. (3) six metadata arrays sized amr_max_blocks are replicated on every rank. Limit 1 is trivially removable: every per-box answer is immediately OR-ed into any_xchg and only the accumulator survives (m_amr_regrid.fpp:1456), so the reduction hoists out of the loop for ~10 lines. Also records which half of the AMReX model is adopted - ownership, cap, per-box scratch, per-level mapping, redundant communication-free mapping and P2P gather are ALL in place; the per-box reduction is the single gap, and it was never a design decision. Queue reordered with the hoist as item 8. None of this is reachable on a single node: 512 boxes is three orders of magnitude below where it binds, so measurement could not have found it and a code audit did.
…rement sections The file had grown to 612 lines and 27 sections by accretion, with the primary open work item buried in the middle and THREE overlapping cost-model sections, two of which carried numbers already retracted elsewhere in the same document. Reorganised to 511 lines and 12 sections. What actually blocks exascale (the per-box collective, the O(n^2) cut, the replicated metadata) now comes immediately after Status instead of sitting between two design sections. Three sections were replaced by a single note recording why they are not evidence: 'Balance is no longer the limiter' quoted 62%/38% efficiency figures produced with run_time_info on, which forces a device sync every step and measured 6% on a uniform control; 'The cost model weighs the wrong quantity' reported measured time tracking box count, which is real but confined to the starved regime; and the step-4 A/B ran at a size where the uniform control itself reached only 18% efficiency. Kept rather than deleted so a reader meeting them in git history knows not to trust them. Added a reading guide and fixed two cross-references left dangling by the removals.
s_set_amr_fine_geometry ended in a global s_mpi_allreduce_integer_max and is called from do k = 1, nboxes in the regrid rebuild loop, so a regrid cost nboxes global collectives. Every per-box answer was immediately OR-ed into a single accumulator and only the accumulator survived (m_amr_regrid.fpp:1456), so the reduction had no reason to be inside the loop. It now ORs into a module accumulator amr_xchg_bad and s_amr_reduce_xchg_flag performs ONE allreduce to close the scan; all five call sites close their scan explicitly. This also FIXES A LATENT BUG at two of them: the L0-tile loop in s_initialize_amr_module and both restart loops kept only the LAST block's answer rather than the OR, so a block needing the coarse-ghost exchange could be masked by a later one that did not, silently skipping an exchange the fine advance depends on. The accumulator makes that impossible by construction. EXPECT NO SPEEDUP AT CURRENT SIZES, and that is the point: amr_block_batching measured that batching these allreduces buys nothing at 14-21 boxes because the 7.4-13 ms per call is absorbing load-imbalance spread, not reduction cost. That measurement is regime-limited - with nboxes collectives the cost has a floor of nboxes x latency independent of any imbalance, ~0.5 s per regrid at 1e5 boxes, and the design targets boxes_per_level >> num_procs. Both docs now state the reconciliation rather than contradicting each other. Also fixes the Documentation CI failure: amr_per_level_distribution.md was the only AMR doc without a @page declaration, so an earlier workaround that referenced it as a backticked filename produced a dead auto-generated link from amr_block_batching.html. 72/72 AMR goldens, precheck 7/7.
fahnab666
added a commit
to fahnab666/mfc-nabid
that referenced
this pull request
Aug 29, 2026
…isting lint debt Merges the 20 commits added to sbryngelson/up-mega since ee7758b: the CCE descriptor and GPU_DECLARE fixes for the Frontier AMR abort, the NVHPC declare-ordering fix, restart format v2 (per-block owner/extents), amr_blocking_factor defaulting to 4, and the level-order clustering reductions. The delta is confined to AMR: m_amr, m_amr_regrid, m_amr_restart, m_constants, m_global_parameters, cmake/GPU.cmake, 14 AMR goldens, and two AMR-only toolchain hunks. No JWL, IBM, Riemann, or EOS source file is touched and no non-AMR golden changes. Merge was conflict-free; the local static-AMR GPU allocation cap for IBM survives intact. Also clears lint debt that predates this merge and blocked any commit: formatter line-wrapping in six JWL/IBM files, the duplicated rxn_val PROHIBITs in pre_process/m_checker.fpp (already enforced in case_validator.check_stiffened_eos, as its own docstring states), the stale hyperelasticity row in case.md (removed upstream), the missing sec-mhd anchor, and m_jwl/m_jwl_sources missing from module_categories.json.
…ntier CCE gpu-acc) A declare-create on a module allocatable binds a present-table entry to the descriptor at program init; the move_alloc at the allocation site then swaps that descriptor out, so every later kernel lookup misses. All 36 AMR tests on the CCE gpu-acc lane died with 'find_in_present_table failed' for amr_cons_br (432 hits) and amr_scr_prim (9). amr_cg has always used move_alloc + GPU_ENTER_DATA with no declare and works on every lane; this makes the other two match. The declares were added for the lib-4425 uninitialized-descriptor abort, but the move_alloc is what actually fixed that, so they were redundant as well as harmful. amdflang's OpenMP runtime creates the mapping implicitly, which is why no local gate could see this.
s_populate_variables_buffers sits inside s_compute_rhs, which PH_COARSE wraps whole, so a uniform (amr=F) run reported 95% 'coarse' and no communication at all. That made the one comparison that matters impossible: how much of AMR's ~31% communication share is AMR's own versus the solver's baseline.
Halo probe counts distinct block slots this rank's metadata reads touch between regrids, reduced with MPI_MAX over ranks rather than reported from rank 0 (which owns a domain corner and is the least connected rank). The reduction is entered by EVERY rank: an earlier draft placed it inside the proc_rank == 0 guard, so one rank called ALLREDUCE while the rest ran ahead into other collectives and the job died with MPI_ERR_TRUNCATE. Grid efficiency reports tagged versus covered coarse cells, which prices amr_tag_eps and amr_buf and bounds the quoted payoff.
…reshold A histogram of the tagging ratio g/(2*r0) over ~49 regrids of this case shows the distribution is continuous from 0 to 0.5 with no gap, and amr_tag_eps = 0.05 already sits at a local minimum, so there is nowhere better to move it. Near the threshold the density is ~9,400 cells per unit ratio per regrid, so one flip across 49 regrids needs a perturbation of ~2e-6; roundoff is 1e-12 to 1e-14 and cannot reach that. The flipped tag is therefore a consequence of divergence, not its cause: roundoff is amplified by a chaotic configuration until it reaches ~1e-6, and only then do tags flip. t_step_stop 100 -> 25 keeps 12 regrids and the store still grows 25 -> 36 -> 49 blocks with the cap never binding, while giving roundoff far less room to grow. override_tol = 1e-11 covers the residual toolchain roundoff (CCE measured 1.06e-12) without being loose enough to hide a mesh divergence.
The allocation is guarded by hypo_nc_mode == hypo_nc_mode_dual_pass alone, but the teardown sat nested inside if (.not. igr), so any configuration with igr and dual_pass together leaked flux_n, flux_src_n and flux_gsrc_n along with their %sf payloads and device maps. Nothing couples igr to hypo_nc_mode in the validator, so the combination is reachable. Hoisted the teardown block out of the igr guard so the two are textually identical.
Nothing in the tree writes this array, but s_finalize_riemann_solver_hatR copies all of 1:sys_size out of it into flux_gsrc_n(id), which m_rhs folds into the RHS. Under cyl_coord with the dual-pass path that fed uninitialized memory into the solution. Mirrors the zeroing its twin flux_gsrc_rsx_vf already gets for the same reason.
s_compute_load_weight reads it on the first s_write_data_files, which for a run saving at t_step_start precedes any relaxation sweep, so the first load-weight diagnostic and load_weight.*.dat came from uninitialized memory. The device copy is the one that matters: the reader is a GPU_PARALLEL_LOOP and the only writer sits inside a device region, so a host-only assignment would have reached neither.
s_write_amr_restart marks format v2 with a NEGATIVE rank count, and the simulation reader was taught both formats, but the post_process reader was not. Its rank-count check therefore fires unconditionally (a negative can never equal num_procs) and reports a rank-count mismatch even when the counts match, which sends the reader after the wrong problem; it also still reads the 3*num_procs extents record that v2 does not write. Detect the negative marker in both the serial and MPI branches and abort saying what is actually wrong. This is a stopgap: reading v2 in post needs the whole-block ownership layout (one contiguous chunk per block, ownership from the file's 4-int record rather than derived geometrically), which is a separate change. Nothing in the default suite runs AMR with post_process as a target, which is why the break shipped unnoticed.
Only the parallel_io branch of s_write_amr_restart writes the negative rank count that marks format v2; the serial branch still writes a positive num_procs to p_all/p<rank>/<t_step>/amr_fine.dat. The guard added to the serial reader could therefore never fire, and its comment claimed a failure mode that path does not have. Verified against a real 8-rank AMR run: the parallel file's header is (-8, 224, 6), so the parallel guard is the one that matters.
The simulation path raises MFCException on a non-zero return code; the --test-all post_process path captured stdout and ignored the result entirely (common.system runs with check=False). post_process could abort, segfault or fail outright and the test still reported PASS as long as the simulation goldens matched. That is how a total break of the AMR post-process reader shipped unnoticed: --test-all ran post_process on every AMR case and threw the answer away. Verified both directions: an AMR test now fails where it previously passed, and non-AMR cases are unaffected.
Two defects, both in the AMR fine-block reader, both invisible until the test suite started checking post_process's exit code. First: format v2 was never taught to post. The writer marks v2 with a negative rank count, the simulation reader handles both formats, and post did not, so its rank-count check fired unconditionally (a negative can never equal num_procs) and blamed a rank-count mismatch on runs that used the right number of ranks. v2 stores one contiguous chunk per block written by that block's single owner, plus a 4-int (owner + 1, m, n, p) record. Ownership now comes from that record rather than from the reader's own geometric intersection, which also fixes a case where a rank overlapping a block it does not own would read a -1 extent and abort. The whole chunk is read and this rank's intersection sub-box copied out, so s_setup_amr_block still reconstructs coordinates from a valid local coarse index. v1 files stay readable. Second: post runs with a LARGER sys_size than the simulation for 5eq Lagrange bubbles, because it appends beta_idx as a post-only output slot. Comparing the file's conserved-variable count against post's inflated sys_size rejected every valid AMR + Lagrange-bubbles file. Size and read by the count the writer actually used.
[amr-grideff] printed amr_n_tagged, which counts THIS rank's local tag_grid, next to amr_n_covered, which comes from the global accepted box list (s_amr_cluster runs with reduce = .true., so every rank clusters the same global tag set and already holds the same volume). Printing the unreduced pair from rank 0 made the ratio wrong by about num_procs. Sum the numerator across ranks and leave the denominator alone. Every rank enters the collective; only rank 0 prints. Still approximate and marked as such in the source: the numerator mixes the level-1 and level-2 index spaces, and it is accumulated before the amr_buf pad and the box merge, so real over-coverage is worse than the printed ratio.
With l0_ntile > 0 the L0 tiles occupy slots 1..l0_slot_off of the SAME pool as the fine blocks and amr_num_blocks counts them, so s_write_amr_restart emits level-0 tile records into the AMR fine-block file. All three reader paths rejected them twice over: lvl == 0 fails the lvl < 1 test, and a tile's refinement ratio of 1 fails rr < 2. Every AMR + L0-tiles case therefore died in post_process with 'malformed fine-block header (level/extent inconsistent); the writer and reader header layouts have drifted' -- a message that blamed a drift which had not happened. A tile's data is the base grid re-tiled and is already in the level-0 restart file, so the overlay skips it while still advancing the file offset past the record. Reproduced and fixed on a CPU build, which is where CI sees it: all nine cases fail with the old reader and pass with the new one. They pass on amdflang GPU either way, so no local GPU gate could have caught this.
amr_n_shaped counts the coarse volume the FINAL boxes occupy, after s_amr_regrid_shape_boxes has padded by amr_buf, clamped, size-capped, clipped and tiled. amr_n_covered is taken before all of that, so it cannot see the pad, which is exactly where amr_buf acts: across an amr_buf 4 -> 1 sweep covered moved 0.4% while the refined volume moved 9%. [amr-grideff] now prints tagged, covered and shaped together. Also warn when the global accepted-box count exceeds amr_max_blocks: boxes contributed by other ranks were discarded silently, so the refined mesh stopped matching the tag field with no diagnostic. The existing capped flag covers only the tree bisection, not this truncation.
The runtime banner and the per-level distribution table both asserted 1.55x as a measured subcycle speedup. It is a phase-share model and was never measured. One matched-resolution arm pair (equal physical time, level 2 at the same dt in both) gave 2.84x, but that T arm's phase table omits rhs, seam, reflux, gather and rk entirely because the subcycle advance path carries no PH_* brackets, so it is a wall ratio with no accounting behind it. Both figures are now labelled for what they are. Separately, m_time_steppers said multi-level reflux under subcycling is future work; s_amr_advance_children has been calling s_amr_reflux_to_parent(dt_sub, .true.) for some time.
Records what the reviews established: the np8-512 weak-scaling ladder already existed and had never been analysed (rg:clus is O(P^3) and 75% of wall at np=512; rb:topo is exactly O(P^2)); the emitted box set is a single bounding box re-tiled, so amr_cluster_eff cannot affect it and the measured 2.20x over-coverage is this clusterer's best case on its most flattering geometry; and the per-step global scan work this plan had as item 1 is 2.5% of wall at np=512, measured, against a P^2 and a P^3 term beside it.
Under cfl_dt the simulation names each save by save_count = int(mytime/t_save) (m_start_up.fpp:798), so when adaptive dt grows enough for one step to cross TWO t_save boundaries the index skips and no file is written for the intervening value. The post loop walks indices 0..n_save-1 and the reader aborts on the first absent one, so every CFL-driven case died in post_process with 'File restart_data/lustre_N.dat is missing'. The IGR_triple_point case writes 60 saves at indices 0, 1, 3, 5, 7 and upward, and post demanded index 2. A skipped index is legitimate output, not a fault. f_save_exists lives beside the reader that owns the filename convention and only answers for the shared-file layout, since with file_per_process each rank holds a different file and the answer would not be rank-uniform; that path keeps the original fail-closed behaviour. Reproduced and fixed on a CPU build: the case fails before and passes after, and 12 non-CFL cases still pass.
…FlowCode#1795 4C751DAF aborts on the Frontier CCE cpu lane with 'No Lagrangian bubbles remain in the domain', and passes on amdflang GPU and gfortran CPU. Measured on gfortran over the full run, from the case's own lag_bubble_evol output: the single bubble moves 0.5 -> 0.5000076, its radius goes 0.008 -> 0.0079987 (it shrinks), and the void fraction in its cell peaks at 0.0335 against a valmaxvoid threshold of 0.99 -- a 3.1x margin. Neither removal criterion is reachable, so the count reaching zero on CCE is a symptom of the bubble state degrading there, not a physical outcome. The case is therefore NOT loosened to dodge the abort: with that much margin on working toolchains, any such change would hide whatever actually goes wrong on CCE. Only the --test-all re-run trips it, so the primary run and the golden comparison still cover the case fully. This is present on master too, invisible because the suite discarded the --test-all exit code.
The subcycle driver, the shared per-block advance and the children recursion carried no PH_* markers, so a subcycle run's phase table omitted rhs, rk, gather, seam and halo entirely (zero-time phases are dropped from the report) and the measured 2.84x speedup was a wall ratio with no accounting behind it. Bracketed: the coarse cons halos (halo), the setup and lerp loops (gather), both fine-fine halos (seam), s_compute_rhs and the RK update inside s_amr_subtree_stage_advance (rhs, rk -- one site covers level 1 and the children), and the children's restrict + reflux-to-parent (rs:rfp). Same ids as the lock-step path so the two budgets read side by side. Validated: a subcycle case now reports rhs 31.9%, gather 7.7%, rk 3.1% where it reported none of them; 66/66 AMR gate green.
BASE_CFG hardcodes precision = 2 (double post_process output), which case_validator rightly prohibits on a --single build. Every post_process case on the single-precision CI lane has been failing this validation invisibly -- the suite discarded post's exit code until 68bcaf1 -- and surfaced as 'Case validation failed for post_process' across the lane once the check landed. get_post_process_mods now requests precision = 1 when the build is single. Post output files are not golden-compared, and the lane's post step never succeeded before, so no goldens move.
…it-identical The min-separation merge restarted an all-pairs scan after every fusion and removed the absorbed box by an O(n) shift; at np=128 the gathered list is ~12,400 leaves collapsing through ~11,300 fusions per call, and rg:clus grows to 75% of wall at np=512. Survivors now live on a doubly-linked list (O(1) unlink) and each pass finds its fusion through a uniform bin grid over the LIVE boxes' bounding box, width max(ext_max + thr, clamped so nb_tot stays O(n)) -- sound because a tooclose pair is within ext_max + thr - 1 per dimension and enlarging cellw only widens the candidate set; bit-identical because for each i in list order the minimum surviving candidate index equals the first tooclose j the linear walk meets. Also fixes: the nacc == 0 walk read an unwritten nxt(1); the [amr-merge] fusion counter was declared and printed but never incremented (its 'zero fusions' reading was an artifact); and this rebuild restores the truncation warning, amr_n_shaped accumulation and grideff 'shaped' field that the first version of this commit reverted through a stale-base file copy (caught in review). Measured on mi2104x pinned-binary A/B: rg:clus 138.6->83.8 (np32), 1074.9->335.7 (np64), 8462.1->1076.1 ms/call (np128).
s_amr_st_reserve chose device-native vs host-staged growth by COLUMN COUNT (32), but a column is ~33 MB at the default 64^3 block cap and ~1.8 GB on the 57.3 GiB run that motivated the guard -- the same count means wildly different bytes. The store-capacity ratchet pushes production runs to 43-81 columns, sending every later growth on the full-store host PCIe round trip (measured 4.4 s/regrid vs 0.55 under the guard). The guard now budgets the transient itself: stage on-device while the extra copy (oldcap * column bytes, which is exactly the store size regardless of factorization) stays under 4 GiB, so near-limit stores of ANY column shape still take the OOM-safe host path. Review-noted exposure on 16 GB cards is recorded at the constant; the registers site keeps its count guard (per-array transient, two orders smaller) with its stale cross-reference corrected.
The previous fix (4d5f6ee) patched get_post_process_mods, but only coverage_build.py calls that function -- the test suite embeds POST_PROCESS_OUTPUT_PARAMS directly into the generated case program, so the single-precision CI lane kept failing every post_process validation unchanged. The guard now lives in the generated program itself, keyed on ARGS['mfc']['single'], which run/input.py already serializes. Validated end-to-end by regenerating a case through the harness and executing it both ways: single=true yields precision 1, single=false yields 2. The earlier fix was committed validated only by construction; its local check had failed on environment and was not repeated -- the same never-observed-effect class as the dead counters.
…(47) glob_bytes counted 18 metadata ints per block (72 B) and ignored the amr_slots struct array, whose descriptors dominate at ~1 kB/slot -- a 10-20x under-report that silently propped up the metadata-distribution-deferred decision and would have steered the production-size rung. Ledger entry 47 records the landing, the toolchain-lock incident, and the forensic corrections.
… verdicts The [amr-mig] counters counted only blocks rank 0 itself sends and printed them unreduced -- the zero-cost audit read a frozen 141 MB startup transient at every P off what looked like machine-wide migration. Now SUM-reduced across ranks (ml_g, mirroring the [amr-halo] reduce just above; rank0_bytes kept for cross-run continuity). Ledger 48 records: the rhs growth above np128 is ranks-per-node density (memory-bandwidth contention), not P, pending the pre-registered np1024 confirmation; the dirty-box payoff is 4/np independent of bin occupancy (127x at np512); ladder migration is a frozen startup transient so keep-tol requires a moving-feature deck; escaped is 0 through np512; and the memory probe measured 201 MB/rank replicated metadata plus a ~3.3 GB/rank fixed footprint that OOMs dense packings.
Lines of Code
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.