Repository navigation
Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498
Description
Summary
Prerequisite for #173 (CD) — specifically for precise, per-check test-failure surfacing in GitHub Actions.
scripts/Invoke-UnitTests.ps1 today only exposes:
- An exit code (
0pass /1test failures /2harness error). - A human-readable log file + stdout containing per-check
[PASS]/[FAIL] <check>lines and a
per-module[SUMMARY] Passed: N Failed: N Total: Nline. - A final per-module summary table; the orchestrator keeps only per-module counts in memory
($moduleResults) — there is no machine-readable artifact and no structured per-check result.
This makes it hard for the CD workflow to pinpoint the exact failing assertion: it would have to
regex-parse free-text log lines (brittle).
Goal
Emit a stable, machine-readable results file so CI/CD can render per-check failures robustly
(GitHub Job Summary table, ::error:: annotations, test-reporter actions).
Acceptance criteria
- Optional output path parameter(s) (e.g.
-ResultsJsonPathand/or-JUnitXmlPath) that, when
supplied, write structured results without changing default behavior. - JSON output capturing, at minimum: overall pass/fail + counts, and per-module results with
a per-check breakdown (check name, status, and message/detail for failures). - JUnit XML output (testsuites/testcase, with
<failure>for failed checks) so standard
GitHub test-reporter actions can consume it. - Per-check detail is captured at the orchestrator level (not just per-module counts) — the
per-module test scripts surface individual check results to the orchestrator. - Existing exit codes, log file, and console output remain unchanged (backward compatible).
- README / header comments document the new outputs.
Notes
- Consumed by the Implement a GitHub action to provision a new sandbox environment #173 CD workflow's
workflow-testsstep to build a$GITHUB_STEP_SUMMARYtable
and emit annotations for each failed check, plus upload the JSON/JUnit as artifacts.
Activity
doherty100 commentedon Sep 17, 2026
Terminology note: "CD" as referenced here (and throughout #173's sub-issues) means an ephemeral plan/apply validation gate for the
vnext → mainmerge — it provisions a throwaway sandbox, runs tests, and is normally torn down on success. It is not a pipeline that keeps a persistent environment continuously up to date. See the terminology clarification on #173 for the full definition: #173 (comment)doherty100 commentedon Sep 21, 2026
CD architecture clarification
The full test suite continues to run on the managed-identity self-hosted
jumplinux2runner. GitHub-hosted OIDC only starts that VM; it is not the Terraform/test execution environment because the required vWAN/P2S integration test needs post-apply Key Vault access unavailable from the hosted runner. Emit results for the self-hosted workflow.doherty100 commentedon Sep 22, 2026
Authentication boundary refinement
The test-results workflow runs on
jumplinux2using only its existing system-assigned managed identity. User-assigned identities are out of scope;arm_client_idremains exclusive to the manual service-principal-secret path.doherty100 commentedon Sep 22, 2026
Superseding MSI identity decision
Terraform and test orchestration use a pre-provisioned UAMI attached to the Terraform execution environment; in #173 this is
jumplinux2.arm_client_idselects that UAMI in MSI mode.jumpwin1retains its separate system identity only for Azure SQL bootstrap.doherty100 commentedon Oct 5, 2026
Implemented in #790 (merged to
vnexton 2026-10-05). All acceptance criteria met: optional-ResultsJsonPath/-JUnitXmlPath, JSON with per-check breakdown, JUnit XML with<failure>per failed check, per-check capture at the orchestrator level, backward-compatible defaults, and documentation in README and the script header. The PR also fixes the exit code underpwsh -File, which previously always returned 0 because of#requires -Modules. Closing manually because PRs intovnextdo not auto-close issues.