Skip to content

Add a cuDSS GPU solver for Maxwell frequency sweeps - #77

Open
xtalax-droid wants to merge 13 commits into
maxwell/3-sweep-abcfrom
maxwell/6-cudss
Open

xtalax-droid wants to merge 13 commits into
maxwell/3-sweep-abcfrom
maxwell/6-cudss

Conversation

@xtalax-droid

Copy link
Copy Markdown
Collaborator

Depends on #75 (this PR targets its branch, so the diff shows only this layer; it will be retargeted once the layers below merge).

Adds CuDSSSolver(), a solver for solve_sweep that runs the sparse complex factorisation and the multi-port solve on an NVIDIA GPU through cuDSS, in a new package extension FiniteElementMethodCUDSSExt loaded by using CUDA, CUDSS.

using CUDA, CUDSS
sweep = solve_sweep(model, ωs; solver = CuDSSSolver())

Untested on real hardware, compiles: loads, precompiles, JET and CSR-layout tests only. No cuDSS analysis, factorisation or solve has been run (the development machine has no NVIDIA GPU). To run the GPU tests on a CUDA machine, from the repository root:

GROUP=MaxwellGPU julia --project=. -e 'using Pkg; Pkg.test()'

How it works

  • Assembly stays on the CPU.
  • First frequency: the operator is uploaded once as a CuSparseMatrixCSR{T, Int32}; the CSR layout and a CSC-to-CSR value permutation are computed once on the host; cuDSS runs analysis and factorization.
  • Later frequencies: the new values are gathered on the host through that permutation, only nzVal is copied to the device, and cuDSS runs refactorization, reusing the analysis.
  • All port columns of B are solved in one solve call and returned to the host.
  • structure = "G" (general LU) is the default. structure = "S" selects cuDSS's complex symmetric LDLᵀ, which fits the complex symmetric (not Hermitian) Maxwell operator; CUDSS.jl documents it for complex values only from 1.0.

Dependencies

Tests

  • Maxwell group (CPU, 145/145): csr_layout and csr_values! against an independently built CSR, on a nonsymmetric matrix with a stored zero and on the WR-90 operator across two frequencies; error messages without the extension.
  • New opt-in MaxwellGPU group (test/MaxwellGPU/Project.toml, not part of All, not run in CI; README explains the manual run): extension loading, JET @test_opt/@test_call on the solver, concrete return-type inference for the first and later frequencies, and a three-frequency WR-90 sweep compared with UMFPACK (S matrix and fields, rtol 1e-8) for "G", plus "S" on CUDSS ≥ 1, skipped unless CUDA.functional(). Here: 6 passed, GPU solve skipped, with CUDA 6.4.2 and CUDSS 0.8.1.
  • QA 21/21, Runic clean. A deliberate mutation confirmed the JET tests catch a mistyped cuDSS call.

What the first CUDA run should check

  • refactorization after the nzVal copy and cudss_update picks up the new values (the three frequencies' S matrices differ, and each matches UMFPACK).
  • Accuracy against UMFPACK within rtol 1e-8 under cuDSS's default pivoting.
  • "S" on CUDSS 1.x.
  • Device memory on a large mesh (B, X and the factors are resident together).

🤖 Posted by an AI agent · harness: Claude Code 2.1.318 · model: claude-opus-5-5
Review: unreviewed
Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5

t and others added 7 commits November 5, 2026 22:39
CuDSSSolver() is a solve_sweep solver that factorises the sweep operator on an NVIDIA GPU through the FiniteElementMethodCUDSSExt extension (using CUDA, CUDSS). The first frequency uploads A in CSR layout and runs analysis and factorisation; later frequencies gather the new values through a CSR permutation computed once on the host, copy only nzval to the device and refactorise. All port columns are solved in one call. The host-side CSR layout and value update are tested on the CPU in the Maxwell group; a new opt-in GPU group checks that the extension loads and infers (JET) and compares the GPU sweep with UMFPACK only when CUDA is functional. CUDSS compat is 0.8, 1 because ArrayInterface and LinearSolve cap CUDSS below 1 today.

Agent-Harness: Claude Code 2.1.318
Agent-Model: claude-opus-5-5
Agent-Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Agent-Harness: Claude Code 2.1.318
Agent-Model: claude-opus-5-5
Agent-Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Another expansion owns test/GPU. The group declares no self-hosted runner: it stays out of All and CI, and test/MaxwellGPU/README.md documents the manual run on a CUDA machine (GROUP=MaxwellGPU julia --project=. -e 'using Pkg; Pkg.test()').

Agent-Harness: Claude Code 2.1.318
Agent-Model: claude-opus-5-5
Agent-Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Agent-Harness: Claude Code 2.1.318
Agent-Model: claude-opus-5-5
Agent-Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Agent-Harness: Claude Code 2.1.318
Agent-Model: claude-opus-5-5
Agent-Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Both are internal host-side helpers of the cuDSS solver, so Documenter's checkdocs = :all failed the docs build with :missing_docs because neither is in the manual. Internal helpers carry a plain comment instead.

Agent-Harness: Claude Code 2.1.318
Agent-Model: claude-opus-5-5
Agent-Session: ee6a670e-e4ec-446c-b941-3ebb999c45c5
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Agent-Harness: Claude Code 2.1.319
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Benchmark Results (Julia v1)

Time benchmarks
master 5a94689... master / 5a94689...
construction/heat P1 10x10 9.52 ± 0.47 ms 9.72 ± 0.69 ms 0.979 ± 0.084
construction/heat P1 20x20 12.8 ± 0.46 ms 12.5 ± 0.5 ms 1.02 ± 0.055
time_to_load 6.62 ± 0.14 s 6.52 ± 0.029 s 1.02 ± 0.022
Memory benchmarks
master 5a94689... master / 5a94689...
construction/heat P1 10x10 0.0397 M allocs: 2.13 MB 0.0397 M allocs: 2.13 MB 1
construction/heat P1 20x20 0.0705 M allocs: 4.75 MB 0.0705 M allocs: 4.75 MB 1
time_to_load 0.2 k allocs: 11.8 kB 0.2 k allocs: 11.8 kB 1

Agent-Harness: Claude Code 2.1.322
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Independent review (Codex CLI v0.164.0 / gpt-6.1-sol), reviewer session 01a1c8d0-d885-7ce2-ac99-0f6893e44c3d, reviewed head 5c20966

  • Verdict: CHANGES
  • Risk assessment: medium-low (ML7: additive Maxwell public API and optional solvers; existing entry points and defaults are unchanged).
  • Round / slot: 1 / 1

This is a blind code-only review by a session other than the PR author's, copied below from the reviewer's report with only the grading metadata removed. The posting agent did not re-run the reviewer's commands. A MERGE verdict is advice; merging is left to the maintainers.

Risk

medium-low (ML7: additive Maxwell public API and optional solvers; existing entry points and defaults are unchanged).

Blocking findings

  • Verdict: CHANGES
  • Round: 1
  • Blocking priorities: P2, P2
  • Deferred P2 issues: None supplied; no issues were inspected or filed.
  1. [P2] src/maxwell/ports.jl:248 — lossy ports do not have the advertised power normalization. Confirmed by running. The documentation promises power waves, and WavePort accepts complex dielectric permittivity. For a TE10 electric amplitude c, real power is proportional to norm * Re(β/μr) * |c|²; the complex square root used here instead scales squared amplitudes by norm * |β/μr|. This computes pseudo waves, whose squared magnitudes need not be power fractions. With normalized units, a 1 × 0.5 guide, vacuum input, output eps_r = 2 - 4im, mu_r = 1, and ω = 5, a unit output mode gives |S21|² = 2.668622288174558 versus the analytic real-power ratio 2.210638826960549. An actual two-section solve also overstates transmitted real power by 20.717% (4.7519934489868385e-5 versus 3.936466119743859e-5). Document the pseudo-wave convention and its lossy-port power interpretation, restrict power-normalized ports to lossless media, or implement consistent real-power normalization. Add a lossy-port regression.

  2. [P2] src/maxwell/ports.jl:102 — zero frequency poisons the entire absorbing operator with NaNs. Confirmed by running:

    m = MaxwellModel(generate_grid(Tetrahedron, (1, 1, 1));
        absorbing = ["top"], eps0 = 1.0, mu0 = 1.0)
    all(isfinite, maxwell_operator(m, 0.0).nzval) # false

    All 163 stored entries are non-finite, including for vacuum with zero conductivity, because the coefficient evaluates 0 / 0 before multiplication by ω. Its boundary coefficient has a zero limit. Either evaluate that limit safely or explicitly reject and document zero frequency at the public boundary. Add a regression; silently returning a NaN matrix is unsuitable for an unsupported domain.

Non-blocking findings

  • docs/src/maxwell.md:73, test/MaxwellGPU/cudss.jl:25: the recommended CUDSS.jl 1 configuration cannot resolve. Pkg.add(PackageSpec(name="CUDSS", version="1")) failed: CUDSS 1.0.0 requires ArrayInterface ≤7.10.0, whereas the required DiffEqBase 7.21 requires ≥7.28.0. The documented symmetric configuration therefore cannot be installed with this dependency graph. My direct symmetric test passed on CUDSS 0.8.1. Align the instructions and test-version gate with a supported, resolvable configuration.
  • src/maxwell/sweep.jl:46: an empty frequency vector throws BoundsError from first(omegas). Confirmed by running. Define the empty-input contract with an empty result or an explanatory ArgumentError.

What I ran

AI reviewer: codex-cli 0.164.0, model gpt-6.1-sol; local session identified above (no session URL). No author rating was read.

Tests used an exact git archive HEAD under scratch/checkout; the supplied checkout remained clean. Julia commands retained the runtime's layered JULIA_DEPOT_PATH and used the prescribed TMPDIR, a 12 GiB memory scope, four Julia threads and four precompile tasks. Probe/GPU runs used one BLAS thread. Paths below are relative to this review job; execution used their absolute equivalents.

  • git diff origin/master...HEAD, affected source/tests/docs, root instruction-file checks (both absent), and git diff --check origin/master...HEAD (clean).
  • julia --version: 1.13.2. No repository Manifest or interpreter lock was present.
  • uv run --locked python --version: Python 3.15.0, with --locked has no effect when used outside of a project. There is no Python project or uv.lock; this does not establish a project-locked Python interpreter.
  • GROUP=Maxwell julia --project=scratch/checkout -e 'using Pkg; Pkg.test()': 145/145 passed (CSR 12, manufactured 4, model 80, ports 49). Manufactured H(curl) rates were 0.982/0.997 lossless and 0.967/0.993 lossy.
  • julia --project=scratch/checkout scratch/probes-aligned.jl: 7/7 passed for dielectric/permeability-step convergence, reciprocity, flux, unit scaling and port orientation. Analytic interface error decreased from 0.023397 to 0.005713 (rate 2.034); flux was 0.999905. The initial scaling probe failed because its exact-threshold predicate assigned 24 different material cells after coordinate scaling. A coordinate-roundoff allowance corrected that input; the assertion retained rtol=1e-9.
  • julia --project=scratch/checkout scratch/regressions.jl: two failed assertions, reproducing the two blocking findings. The power reference uses the analytic TE10 propagation constants, not stored solver output.
  • NVIDIA inventory, CUDA.functional() == true, and a CUDA array arithmetic check on GPU 0, NVIDIA RTX PRO 6000 Blackwell Max-Q.
  • Copied test/MaxwellGPU/Project.toml into scratch/gpu, developed the archived head, and ran the unchanged test/MaxwellGPU/cudss.jl: CUDA.jl 6.4.2, CUDSS.jl 0.8.1, 11/11 passed, including JET/inference and general-solver sweeps against UMFPACK.
  • julia --project=scratch/gpu scratch/gpu-symmetric.jl: 4/4 passed for symmetric-solver scattering matrices and fields at three frequencies against UMFPACK.
  • A separate environment with explicit Pkg.add(PackageSpec(name="CUDSS", version="1")): dependency-resolution failure described above.
  • Final git status --porcelain=v1: empty; reviewed head unchanged. All started commands completed. No commits, pushes or GitHub posts.

What I did not verify

The full existing suite, QA group, documentation build, other Julia versions/platforms, CUDSS 1 execution, and performance/storage claims. A project-locked Python interpreter is unavailable because this is a Julia project without Python environment files. PR bodies, comments, reviews, earlier review artifacts and author risk ratings were not inspected.


🤖 Posted by an AI agent — harness: Claude Code 2.1.322 · model: claude-sonnet-5-5
Conversation: local session 3878b54e-28f2-4a01-a176-a8f5468d1997 (no session URL)

t and others added 4 commits November 11, 2026 23:00
CUDSS.jl 1.0 cannot be installed alongside DiffEqBase (ArrayInterface bounds), so the docs point to 0.8 and the general LU; the test gate already skips the symmetric structure there.

Agent-Harness: Claude Code 2.1.322
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
docs/src/maxwell.md conflict resolved keeping both sections.

Agent-Harness: Claude Code 2.1.322
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Agent-Harness: Claude Code 2.1.322
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
Agent-Harness: Claude Code 2.1.322
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
@xtalax-droid

Copy link
Copy Markdown
Collaborator Author

Review feedback on this PR has been addressed on head 6047b84. Master was merged in first (merge commits only, no rebase or force-push), then each finding was fixed with a regression test, and a Codex review of the branch approved with no open findings. All named CI checks are green on this head, including the Documentation job, which had one re-run after the gh-pages deploy push was rejected by a concurrent job (the docs build itself succeeded).

Stacked on #75.

Items deliberately left as they are: reaction coefficients that are subnormal but representable are still rejected (conservative); coefficients that cancel to zero only up to roundoff are rejected rather than compared with a tolerance; the absorbing-boundary adjoint derivative uses 0 for the sigma term at omega = 0 although the true limit is finite and nonzero.

🤖 Posted by an AI agent · harness: Claude Code 2.1.322 · model: claude-sonnet-5-5
Review: unreviewed
Session: 43e03fde-3384-4386-bae9-03e81f19ba7c

@ChrisRackauckas

Copy link
Copy Markdown
Member

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Independent review (Codex CLI v0.164.0 / gpt-6.1-sol), reviewer session 01a1ceeb-2c9f-7700-8292-8531720ad788, reviewed head 6047b84

  • Verdict: CHANGES
  • Risk assessment: medium-low (ML7: additive Maxwell public API and optional GPU solver; existing entry points and defaults are unchanged).
  • Round / slot: 2 / 1

This is a blind code-only review by a session other than the PR author's, copied below from the reviewer's report with only the grading metadata removed. The posting agent did not re-run the reviewer's commands. A MERGE verdict is advice; merging is left to the maintainers.

CI: green
Session: 01a1ceeb-2c9f-7700-8292-8531720ad788

Risk

medium-low (ML7: additive Maxwell public API and optional GPU solver; existing entry points and defaults are unchanged).

Blocking findings

  • Verdict: CHANGES
  • Round: 2
  • Blocking priorities: P2
  • Deferred P2 issues: None supplied for this finding.
  1. [P2] src/maxwell/model.jl:107 — overlap validation consumes one-pass port iterators before assembly. Confirmed by running the same probe on both heads. The new comprehension exhausts Iterators.Stateful; the assembly comprehension at line 136 consequently receives no ports. This silently omits the requested port boundary terms. A vector containing the same ports works on both heads.

    using FiniteElementMethod, Ferrite
    grid = generate_grid(Tetrahedron, (2, 1, 2), Vec(0., 0., 0.), Vec(2., 1., 2.))
    ports = [WavePort("bottom"; a_direction=(1,0,0)), WavePort("top"; a_direction=(1,0,0))]
    model = MaxwellModel(grid; pec=["left","right","front","back"],
        ports=Iterators.Stateful(ports), eps0=1., mu0=1.)
    length(model.ports)

    Observed 0 on this head versus 2 on 5c209662ffba98a16db2c4dff86d7bd623bd66b9. The assertion that both supplied ports survive fails on this head and passes on the previous head. Materialize ports once before validation and reuse that collection; add this regression to the suite. No follow-up issue evidence was supplied, so round-two P2 deferral is not established.

Non-blocking findings

None. Both prior blockers are resolved: lossy port materials are rejected, and zero-frequency absorbing operators have the correct finite limit.

What I ran

AI reviewer: codex-cli 0.164.0, model gpt-6.1-sol; local session identified above (no session URL). Risk was recorded independently; no author risk rating or same-head review was read.

Commands below used absolute equivalents of the job-relative paths shown. Tests ran against exact Git archives under scratch/checkout and scratch/before, preserving the runtime's layered JULIA_DEPOT_PATH and prescribed TMPDIR. Runs used 12 GiB scopes, four Julia threads, four precompile tasks and one BLAS thread; test batches ran sequentially.

  • Whole PR diff, supplied delta, affected source/tests/docs, and supplied previous-head review. The delta matched git diff 5c209662ffba98a16db2c4dff86d7bd623bd66b9 HEAD. Root AGENTS.md and CLAUDE.md are absent. git diff --check origin/master...HEAD was clean.
  • julia --version: 1.13.2. No repository Manifest or interpreter pin was present; dependency resolution was confined to scratch environments and the writable job depot.
  • uv run --locked python --version: Python 3.15.0, with warning that --locked has no effect outside a project. No Python project or uv.lock exists; this is not a project-locked Python interpreter.
  • GROUP=Maxwell julia --project=scratch/checkout -e 'using Pkg; Pkg.test()': 163/163 passed (CSR 12, manufactured 4, model 84, ports 63). Manufactured H(curl) rates were 0.982/0.997 lossless and 0.967/0.993 lossy.
  • julia --project=scratch/checkout scratch/probes.jl: 40/40 passed, covering lossy-port rejection, the analytic absorbing-coefficient limit, overlap/partial-aperture guards, cutoff at the second port, empty sweeps, saved-buffer isolation and point-location fallback. The initial run had 39 passes and one error in my assertion: Ferrite's complex-vector norm returned a complex value. Converting its components to an array corrected the probe; tolerances were unchanged.
  • scratch/prior-blockers.jl under each archived environment: three failed assertions on the previous head, 3/3 passed on the current head. Failures were lossy-port acceptance and NaN zero-frequency operators for vacuum and conducting material.
  • NVIDIA inventory; GPU 0, RTX PRO 6000 Blackwell Max-Q. CUDA.functional() was true, and a CUDA array-arithmetic check passed. julia --project=scratch/gpu scratch/gpu-run.jl included the unchanged GPU test file: 11/11 passed, including JET/inference and general cuDSS sweeps/fields against UMFPACK. CUDA 6.4.2, CUDSS 0.8.1.
  • Explicit Pkg.add(PackageSpec(name="CUDSS", version="1")) in a separate scratch environment failed resolution: CUDSS requires ArrayInterface ≤7.10, while DiffEqBase requires ≥7.28. This confirms the revised installation caveat.
  • scratch/iterator.jl under both environments: current head 1 passed, 1 failed (exit 1); previous head 2/2 passed (exit 0), confirming the delta regression above.
  • Final supplied-checkout status was clean and HEAD remained the reviewed SHA. All started commands finished. No source edits, commits, pushes or GitHub posts.

What I did not verify

The full existing suite, QA, documentation build, other Julia versions/platforms, symmetric cuDSS execution, and performance/storage claims. A project-locked Python environment is unavailable. CI green is supplied launch metadata, not independently queried. Apart from the supplied previous-head review, PR bodies, comments, reviews and deferred-issue metadata were not inspected.


🤖 Posted by an AI agent — harness: Claude Code 2.1.322 · model: claude-sonnet-5-5
Conversation: local session 3878b54e-28f2-4a01-a176-a8f5468d1997 (no session URL)

Agent-Harness: Claude Code 2.1.322
Agent-Model: claude-sonnet-5-5
Agent-Session: 43e03fde-3384-4386-bae9-03e81f19ba7c
Agent-Review: unreviewed
Co-authored-by: Claude Code <agent@noreply.juliahub.com>
@xtalax-droid

Copy link
Copy Markdown
Collaborator Author

Follow-up to the latest review on head 5a94689: this branch now includes the fix that makes MaxwellModel collect ports once before overlap validation, so Iterators.Stateful ports are no longer consumed (the model previously ended up with zero ports); regression test added on the base branch.

Codex review of the branch approved with no open findings, and all named CI checks are green on this head. Merge commits only; no rebase or force-push.

🤖 Posted by an AI agent · harness: Claude Code 2.1.322 · model: claude-sonnet-5-5
Review: unreviewed
Session: 43e03fde-3384-4386-bae9-03e81f19ba7c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants