Activity

1. Roger Doherty (MSFT) (doherty100)

   self-assigned this

   on Jun 27, 2025
2. Roger Doherty (MSFT) (doherty100)

   added the

   Feature

   issue type on Jun 27, 2025
3. Roger Doherty (MSFT) (doherty100)

   added

   enhancementNew feature or requestNew feature or request

   on Jun 27, 2025
4. Roger Doherty (MSFT) (doherty100)

   added sub-issues on Jun 22, 2026

   • Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496
   • Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497
   • Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498
   • Optionally register the rg-devops-iac VM as a GitHub Actions self-hosted runner #499
   • Secretless, approval-gated GitHub OIDC access to an unattached deployment UAMI #500
   • Workflow-driven start/stop for the rg-devops-iac runner VM #501
5. doherty100 commented on Jun 22, 2026

   

   Roger Doherty (MSFT) (doherty100)

   on Jun 22, 2026

   ContributorAuthor

   More actions

   Issue #173 — CD: GitHub Action to provision an Azure Sandbox

   Problem

   Issue #173 asks for a GitHub Action that provisions a new sandbox environment (continuous
   deployment). The repo today is built for interactive/manual execution only. Several aspects
   of the current architecture block a headless CD process and must be resolved first.

   Approach (agreed with user)

   Build a GitHub Actions workflow that automates the non-interactive
   equivalent of the copilot-instructions Scenario 1 (fresh provision) plus the post-apply
   VM-start gate + Invoke-UnitTests.ps1, and gates the run on the unit-test result
   (exit 0 = pass / 1 = fail). It is the final release gate on the vnext → main PR — it runs
   automatically on pull_request targeting main (plus workflow_dispatch for manual re-runs) and
   publishes a required status check in main branch protection, so the merge into main fails
   unless CD passes. It runs on a self-hosted runner on the rg-devops-iac Linux VM
   using that VM's managed identity (no stored secrets). Terraform state defaults to local
   .tfstate persisted on that runner VM (under the repo root), with an opt-in Azure Storage
   remote backend (the rg-devops-iac tfstate account) documented for durability.

   Decisions captured

   • State backend (NOT a hard blocker — corrected): because the runner is a persistent
     self-hosted VM (not an ephemeral GitHub-hosted runner), a local .tfstate survives between
     runs and is the default. Two caveats the workflow must handle: (a) actions/checkout
     defaults to clean: true (git clean -ffdx) which deletes untracked terraform.tfstate — use
     clean: false and/or keep state at a fixed path outside the workspace (-state=/var/lib/.../ terraform.tfstate); (b) local state has no versioning/backup/leasing, so loss of the runner VM
     orphans resources. Opt-in remote backend: a backend.tf (azurerm, use_azuread_auth = true)
     using partial-config (-backend-config inputs) can point at the rg-devops-iac tfstate storage
     account/container for durability; documented but off by default. rg-devops-iac remains a
     prerequisite (it provides the runner VM + MI; its tfstate account is only needed for the remote
     option).
   • Auth / runner: self-hosted runner on the rg-devops-iac Terraform-execution VM using its
     managed identity (Owner on the subscription + Graph perms). No long-lived secrets.
   • Dual-auth prerequisite (blocker → sub-issue Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496): providers.tf + variables.tf currently
     hardcode/require SPN client-secret auth, blocking MSI/OIDC. Filed and linked sub-issue
     Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496 to add a selectable auth mode (secret default | msi | oidc) across azurerm,
     azapi, azuread, making arm_client_secret conditional while keeping the current manual
     secret flow working unchanged. Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496 must land before the CD workflow can use managed identity.
   • Headless admin identity: user_object_id/user_name grant a human RBAC + AAD group
     membership. These are non-secret identifiers. The workflow takes the admin UPN as a
     workflow_dispatch input and resolves user_object_id at runtime via Microsoft Graph
     (az ad user show --id <upn>) using the runner's managed identity. Mirrors the "signed-in user"
     concept for "the person who creates the CD" with no secret storage.
   • Non-interactive tfvars (→ sub-issue Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497): bootstrap.sh/bootstrap.ps1 are interactive and
     JWT-decode the signed-in user, so they can't run headless and would resolve the wrong identity
     under a CD managed identity. Filed and linked sub-issue Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497 to add a headless mode to
     both scripts (all values via flags/env, user_object_id resolved from the admin UPN via
     Graph, module toggles settable, interactive default preserved). The CD workflow then calls the
     headless bootstrap as the single source of truth for terraform.tfvars — no bespoke
     generation logic duplicated in the workflow. Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497 must land before the workflow's tfvars step.
   • Public-access barrier (not invoked for Scenario 1): the barrier disables KV/Storage public
     access at the end of an apply. On a fresh Scenario-1 apply (empty state) the resources are
     created with public access enabled, written to, then disabled — all within the single apply —
     so no enable-public-access.sh step is needed. ⚠️ Caveat: because the default uses
     persistent local state, re-running the workflow against that existing state becomes a
     Scenario-2 re-apply where the prior barrier already locked down public access; that path would
     require enable-public-access.sh first. Out of scope here (Scenario 1 only) — documented as a
     known limitation / future item.
   • Post-apply tests + result surfacing (made precise): run the VM-start gate
     (scripts/manage-vms.sh start + confirm VM running) then Invoke-UnitTests.ps1 (all installed
     modules; unit+integration auto-run), authenticating via Connect-AzAccount -Identity. "Test-result
     gating" means all of the following, not merely a red ❌:
     1. Job gating — the test step fails the job on a non-zero exit (1 = test failures, 2 =
        harness error), so the run goes red and blocks.
     2. Job Summary — a step parses the runner's per-module summary + overall lines and writes a
        Markdown results table to $GITHUB_STEP_SUMMARY (✅/❌ per module, Passed/Failed counts,
        overall), rendered on the run's summary page so failures are visible without opening logs.
     3. Annotations — emit ::error:: workflow commands for each failed module (and, where
        available, each failed [FAIL] check) so they appear as inline annotations.
     4. Artifact — upload the full test log file (and structured results if produced) via
        actions/upload-artifact so a user can download complete per-check detail.
     • Precision (resolved → sub-issue Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498): per-check precision is delivered by adding
       machine-readable output (JSON/JUnit) to Invoke-UnitTests.ps1 (sub-issue Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498), which the
       workflow-tests step consumes to render per-check failures robustly (Job Summary table +
       ::error:: annotations + JSON/JUnit artifacts). Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498 is a prerequisite for the per-check
       rendering (the workflow's coarser per-module table/annotations work without it).
   • Trigger / merge gate (vnext → main release PR): the workflow triggers on
     pull_request with branches: [main] (opened/synchronize/reopened — i.e. the vnext → main
     release PR) plus workflow_dispatch for manual re-runs. It is not run on the many
     day-to-day PRs targeting vnext (too slow/costly); the real deploy event is the release PR into
     main. The CD job publishes a status check that is configured as a required status check in
     main branch protection, so a merge into main is blocked/fails unless CD passes (provision
     • tests green). An Environment approval gate still pauses the expensive run for human approval.
       Restricted to same-repo PRs (vnext→main is internal; self-hosted runner / MI must never run
       fork code). Branch protection + the Environment are repo-admin settings (documented setup, not
       code). New PR commits (synchronize) re-run the check; the single runner/state serializes runs.
   • Sandbox lifecycle / cleanup (keep_sandbox input, default false): a workflow
     input keep_sandbox controls post-run teardown, acting only on a passing run (apply + tests
     green; a failed run retains everything for debugging and goes red):
     • keep_sandbox = false (DEFAULT) → auto-clean on success: (1) delete the sandbox resource
       group via az group delete -n <sandbox-rg> --yes --no-wait — not terraform destroy (fast,
       and avoids destroy-ordering issues); the sandbox RG name is read from
       terraform output -json resource_names (resource_group) before deletion; (2) clean the
       runner working tree via ./scripts/cleanterraformtemp.sh (removes terraform.tfvars,
       terraform.tfstate*, .terraform/, lock files, SSH keys, *.pem/.pfx/.log/.tfplan), so the
       now-stale local state (referencing deleted resources) is gone and the runner is ready for the
       next run.
     • keep_sandbox = true → no cleanup; sandbox RG and runner config/state preserved (sandbox
       stays Terraform-managed for later Scenario-2 work or manual inspection).
     • State-location reconciliation: cleanterraformtemp.sh is repo-root scoped, so for it to
       fully clean the local state the CD must keep terraform.tfstate inside the workspace/repo
       root on the persistent runner (with checkout clean: false) rather than a path outside the
       repo. This supersedes the earlier "fixed out-of-workspace state path" idea — the two cleanup
       mechanisms (the script + this layout) must agree.
   • Preflight leftover-config check + clean_previous_run input: because state is persistent on
     the runner, a prior run (especially keep_sandbox = true, or an aborted/failed run) can leave
     terraform.tfvars / terraform.tfstate / .terraform/ on the runner. The preflight job
     detects leftover config and honors a clean_previous_run input (default false): when true
     it runs ./scripts/cleanterraformtemp.sh first to start fresh; when false and leftover is found
     it fails fast with guidance (so a run can't silently operate on stale state from a different
     sandbox).
   • Scope (agreed): Scenario 1 + post-apply VM-start gate + tests (gated) + keep_sandbox
     cleanup. (Scenario 2/3 reuse and failure-auto-issue are out of scope this iteration — see Open
     items.)

   Blockers (surfaced; do not start CD workflow until cleared)

   1. Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496 dual-auth refactor — REQUIRED before MSI-based CD works. Hard dependency.
   2. Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497 headless bootstrap — REQUIRED before the workflow's tfvars step. Hard dependency.
   3. Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498 machine-readable test output (JSON/JUnit) — REQUIRED before the workflow's workflow-tests
      step can render per-check failures robustly (Job Summary table + ::error:: annotations +
      JSON/JUnit artifacts). The coarser per-module surfacing degrades gracefully without it, but
      per-check precision (the agreed requirement) is blocked on Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498.
   4. Optionally register the rg-devops-iac VM as a GitHub Actions self-hosted runner #499 self-hosted runner in rg-devops-iac — register the runner declaratively as an
      opt-in part of the rg-devops-iac VM provisioning (cloud-init installs/registers the
      actions/runner agent as a service, toolchain present, uses the VM managed identity,
      token via Key Vault). Replaces the former manual "runner registration" step. Default off.
   5. Managed-identity RBAC (two selectable paths; both testable via rbac_mode input): the apply
      writes role assignments and creates/populates AAD groups, so during the apply window the
      runner identity needs Owner (or User Access Administrator + Contributor) at subscription
      scope — Reader alone cannot apply. The workflow selects between two modes:
      • Option A — standing Owner (DEFAULT, simple, large blast radius): assign the runner MI a
        permanent subscription Owner role. Lowest complexity; consistent with the manual SPN today;
        long-lived privileged CI identity. Fine for a throwaway sandbox subscription.
      • Option B — PIM JIT elevation Reader → Owner with human approval (least standing privilege;
        → sub-issue Secretless, approval-gated GitHub OIDC access to an unattached deployment UAMI #500): MI holds only standing Reader; Owner is activated just-in-time per
        run (PIM for Groups: MI is a member of a role-assignable group eligible for Owner; a workflow
        step requests activation via the Graph PIM API, an approver approves, then it deactivates).
        Requires Entra ID P2. Smallest blast radius; adds approval-gate + activate/deactivate steps.
      • Both modes are implemented and selected via an rbac_mode workflow input so the CD pipeline can
        exercise both paths; standing Owner is the default.
      • Caveat (both modes): the Graph app permissions (User.Read.All, group create/membership
        for the mssql sql_admins group) are app-role grants on the MI and remain standing,
        scoped least-privilege — not PIM-JIT'd like Azure RBAC.
   6. Workflow-driven start/stop for the rg-devops-iac runner VM #501 runner-VM auto-shutdown + workflow-driven start (chicken-and-egg handled):
      rg-devops-iac currently has no power-state lifecycle management for the runner VM, so it
      would run indefinitely and leak cost. Add an opt-in scheduled auto-shutdown (e.g.
      microsoft.devtestlab/schedules), timed outside the deployment window and accounting for the
      ~2 hour max sandbox deployment so it never cuts off an in-flight run. Critical
      consequence: once auto-shutdown deallocates the VM, the self-hosted runner agent is offline,
      so a self-hosted job cannot start its own host. The CD workflow therefore needs a preflight job
      that runs OFF the VM (a GitHub-hosted ubuntu-latest job) as its first step: it
      checks the runner VM power state, starts it (az vm start) if deallocated, waits until the VM
      is running AND the runner is online (GitHub runners API), and FAILS FAST if it doesn't come up
      within a timeout. The main self-hosted provisioning job needs: this preflight. The preflight
      auths via a minimal, narrowly-scoped OIDC identity (custom role: VM read +
      Microsoft.Compute/virtualMachines/start/action, scoped to just that VM/RG — far smaller blast
      radius than the provisioning identity, no stored secret). Runner service must auto-start on boot
      and local state must persist across shutdown/restart.

   (State backend is not a blocker: local state is the default; the remote azurerm backend is an
   optional opt-in for durability.)

   Todos

   • prereq-dual-auth — Land sub-issue Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496 (selectable secret/msi/oidc auth across all three
     providers; arm_client_secret conditional; manual secret flow preserved). Blocks everything.
   • backend-config — (opt-in) Add an optional backend.tf (azurerm, use_azuread_auth,
     partial-config via -backend-config inputs) wired to the rg-devops-iac tfstate storage
     account/container for durability; off by default (local state is the default). Document
     switching between local and remote state. No account names committed.
   • runner-setup — (via sub-issue Optionally register the rg-devops-iac VM as a GitHub Actions self-hosted runner #499) Add an opt-in self-hosted runner to the rg-devops-iac
     configuration: cloud-init installs/registers the actions/runner agent as a service on the VM,
     with the toolchain present (terraform, az, pwsh + Az modules), MI-based Azure auth, registration
     token via Key Vault, and local-state persistence in the workspace/repo root (checkout clean: false, state kept under repo root so cleanterraformtemp.sh can clean it). Declarative
     replacement for manual runner registration. Verify MI RBAC (Owner + Graph).
   • tfvars-generation — (via sub-issue Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497) Land headless mode in bootstrap.sh/bootstrap.ps1
     (all inputs via flags/env, user_object_id resolved from admin UPN via Graph, module toggles
     settable, interactive default preserved); the workflow calls headless bootstrap as the single
     source of truth — no bespoke tfvars logic in the workflow.
   • prereq-test-output — (via sub-issue Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498) Add optional JSON/JUnit machine-readable output to
     Invoke-UnitTests.ps1 with a per-check breakdown (backward compatible); consumed by workflow-tests
     to render per-check failures. Prerequisite for per-check precision.
   • workflow-preflight-start — Author the preflight job (GitHub-hosted ubuntu-latest, runs
     first, needs-ed by the provision job): OIDC login with a minimal VM-start-scoped identity →
     check runner VM power state → az vm start if deallocated → wait until VM running and the
     self-hosted runner is online (GitHub runners API) → fail fast on timeout. Then on the
     runner, detect leftover config from a previous run and honor clean_previous_run (default
     false): if true, run ./scripts/cleanterraformtemp.sh to start fresh; if false and leftover
     exists, fail fast. Solves the deallocated-runner chicken-and-egg and stale-state hazards.
   • workflow-provision — Author .github/workflows/cd-provision-sandbox.yml. Triggers:
     on: pull_request (branches: [main], types opened/synchronize/reopened) + workflow_dispatch
     (the vnext→main release-PR gate; not run on vnext PRs). Same-repo guard (skip fork PRs).
     workflow_dispatch/PR inputs (location, module toggles, admin UPN, tags, optional
     use_remote_state, rbac_mode = standing_owner (default) | pim_jit, keep_sandbox (default
     false), clean_previous_run (default false)) — PR-triggered runs use repo/Environment defaults.
     The provision job runs on the self-hosted runner, uses an Environment (approval gate), and
     needs: [workflow-preflight-start]; steps:
     checkout (clean: false) → MI login → [pim_jit only] PIM-activate Owner (request via Graph
     PIM API, await approval) → resolve oid → headless bootstrap (Add headless/non-interactive mode to bootstrap.sh and bootstrap.ps1 for CD #497) to write tfvars
     → terraform init (local by default, remote if opted in) → validate → plan → apply
     (local state under repo root) → [pim_jit only] PIM-deactivate Owner. No
     enable-public-access.sh — not needed for a fresh Scenario-1 apply (the barrier runs internally
     within the single apply).
   • merge-gate-setup — Document repo-admin setup (not code): the CD check as a required status
     check in main branch protection (so vnext→main merges fail unless CD passes), and the
     Environment (with required reviewers) used as the approval gate. Cover same-repo-only policy.
   • prereq-pim-jit — (via sub-issue Secretless, approval-gated GitHub OIDC access to an unattached deployment UAMI #500) Implement Option B PIM-for-Groups JIT elevation
     (Reader → Owner with approval): role-assignable group, MI membership, eligible Owner assignment +
     approval policy, Graph PIM API activate/deactivate steps, selectable via rbac_mode. Entra ID P2.
   • prereq-auto-shutdown — (via sub-issue Workflow-driven start/stop for the rg-devops-iac runner VM #501) Add an opt-in scheduled auto-shutdown for the
     rg-devops-iac runner VM (configurable time/zone), timed outside the deployment window and
     accounting for the ~2 h max deployment so CD runs are never cut off; runner service auto-starts on
     boot, local state persists across restart.
   • workflow-tests — Post-apply: VM-start gate (manage-vms.sh start + confirm running) →
     Connect-AzAccount -Identity → Invoke-UnitTests.ps1. Surface results precisely: fail the job
     on non-zero exit, write a per-module ✅/❌ results table to $GITHUB_STEP_SUMMARY, emit
     ::error:: annotations for failed modules/checks, and upload the test log + JSON/JUnit
     (Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498) as artifacts. Per-check precision comes from the machine-readable output added in Emit machine-readable test results (JSON/JUnit) from Invoke-UnitTests.ps1 for CI/CD #498.
   • workflow-cleanup — Post-run teardown driven by keep_sandbox (default false), running only
     on success (a failed run retains everything for debugging): when false, read the sandbox RG
     name from terraform output -json resource_names → az group delete --yes --no-wait (not
     terraform destroy) → ./scripts/cleanterraformtemp.sh to wipe the now-stale local state +
     temp files from the runner; when true, skip cleanup (sandbox + state preserved).
   • concurrency-guard — Add workflow concurrency group to prevent concurrent applies against the
     same state (mirrors "one apply at a time" hard rule).
   • docs — README/CONTRIBUTING: new "Continuous Deployment" section (prerequisites, rg-devops-iac
     dependency, runner setup, inputs, auth model, security notes); update copilot-instructions if
     needed.
   • ci-lint — Ensure new workflow passes actionlint (ci-actions) and any added HCL passes
     terraform fmt/tflint/validate (ci-terraform).

   Open items / future iterations (out of current scope)

   • Scenario 2 (enable/disable modules on an existing sandbox) and Scenario 3 (VM re-provision repro)
     as additional workflows or inputs — these would add the enable-public-access.sh step before
     plan/apply (and, for re-applies against persistent local state, even Scenario 1 re-runs would).
   • Auto-open a GitHub issue on failure (mirror the copilot-instructions error policy) via a final
     if: failure() step.
   • Scheduled teardown/destroy workflow for cost control.
   • OIDC GitHub-hosted runner variant (enabled by Support secretless managed-identity Terraform provider auth in addition to SPN client secret #496) as an alternative to the self-hosted runner.

   Notes / considerations

   • rg-devops-iac RG naming (verified — no change needed): it already uses
     module.naming.resource_group.name_unique with the identical naming-module config as the
     Azure Sandbox root (suffix = [project, environment], unique-include-numbers = true,
     unique-length = 8); there is no static/default resource_group_name variable. Each config
     therefore already gets its own random-suffixed unique RG name (stable within its own state), so
     it follows the same pattern as Azure Sandbox. No sub-issue filed for this.
   • Invoke-UnitTests.ps1 is CI-ready (exit 0/1) and auto-discovers deployed modules; -Integration
     is ignored without -Module, so an all-modules run already includes integration tests.
   • GitHub job 6-hour limit comfortably covers the 25–95 min apply; long single-resource waits
     (gateways) are healthy as long as Still creating... heartbeats advance.
   • Never read/inspect state while an apply is in flight; the concurrency guard enforces single apply.
   • backend.tf is currently absent and .gitignore excludes *.tfstate*. Default CD path keeps
     state local on the persistent runner (at a fixed out-of-workspace path so checkout can't
     wipe it); the opt-in remote backend uses partial-config inputs rather than committing account
     names. Local state has no backup/versioning — accept this tradeoff or opt into remote.
6. doherty100 commented on Sep 17, 2026

   

   Roger Doherty (MSFT) (doherty100)

   on Sep 17, 2026

   ContributorAuthor

   More actions

   Addendum — hardening the self-hosted-runner trigger (public-repo security)

   Follow-up from a design review of the self-hosted-runner approach: being the sole approver of the vnext → main merge is necessary but not sufficient to protect the self-hosted runner, since (a) a naive pull_request trigger on branches: [main] would schedule a job on the runner for any PR opened against main — including a fork PR opened directly against main — independent of whether that PR would ever be merged, and (b) the code that actually executes during CD is whatever is on vnext at merge time, so the real control point is PR review discipline on every vnext-bound change (workflows, scripts, Terraform), not just the final merge click.

   Adding two explicit requirements to this issue's scope (not a new sub-issue — these belong to the CD workflow definition itself):

   1. Explicit same-repo/head-ref guard on the provisioning job — do not rely on the trigger filter alone. Add a job-level (or step-level) condition such as:

      if: github.event.pull_request.head.repo.full_name == github.repository

      so a fork-originated PR can never schedule work onto the self-hosted runner, regardless of target branch.
   2. Mandatory Environment required-reviewer approval gate — confirm the provisioning job runs under a GitHub Environment configured with required reviewers, and that this gate cannot be bypassed via workflow_dispatch (i.e. workflow_dispatch still targets the same protected Environment). This is the standard mitigation for self-hosted runners on public repositories: the job pauses before touching the runner until a human approves, giving a last checkpoint even if a run is unexpectedly triggered.

   These should be added as explicit acceptance criteria for the CD workflow itself (in addition to the existing "Restricted to same-repo PRs" note already captured above), and verified before the self-hosted runner is enabled for real.
7. doherty100 commented on Sep 17, 2026

   

   Roger Doherty (MSFT) (doherty100)

   on Sep 17, 2026

   ContributorAuthor

   More actions

   Terminology note: qualifying "CD" for this project

   Following a design discussion, clarifying what "CD" means in this issue (and its sub-issues) so future readers don't assume a persistent, always-on azuresandbox environment is being kept up to date by this pipeline — it isn't.

   Definition used here: this pipeline is Continuous Delivery in the classic sense (every change is automatically validated as release-ready), but implemented as an ephemeral plan/apply validation gate:

   • It provisions a throwaway sandbox from vnext, runs the full unit/integration test suite against that live deployment, and reports pass/fail as a required status check.
   • A human still performs the actual release (merging vnext → main) — this pipeline never auto-deploys to a persistent target.
   • On success, the sandbox is normally torn down (keep_sandbox = false default) — there is no long-lived environment this pipeline maintains or keeps in sync.

   Shorthand for this repo: CD (ephemeral plan/apply validation gate) — i.e. CD is being used to prove "this code deploys and passes tests," not to keep an environment continuously current. Sub-issues #496–#501 reference "#173 (CD)" — that shorthand should be read with this definition in mind.
8. doherty100 commented on Sep 21, 2026

   

   Roger Doherty (MSFT) (doherty100)

   on Sep 21, 2026

   ContributorAuthor

   More actions

   Decision: OIDC is not a Terraform provider mode for the current CD gate

   The required #173 validation gate must run the complete unit and integration suite. Although a GitHub-hosted runner can authenticate to Azure with OIDC and dispatch most VM-side tests through ARM Run Command, it cannot complete the required vWAN/P2S integration test after the sandbox deployment is finalized. That test runs on the Terraform execution host and retrieves the P2S client private key from the sandbox Key Vault. The public-access barrier disables Key Vault public access at the end of apply, so a GitHub-hosted runner cannot retrieve that secret.

   Therefore, the CD architecture remains: a GitHub-hosted job may use a narrowly scoped OIDC identity only to start the runner VM; jumplinux2 then performs Terraform and the test suite using its managed identity and private state backend. #496 will support the existing secret default and managed identity, but excludes OIDC Terraform-provider authentication. A future direct GitHub-hosted Terraform design would require a secure alternative for the P2S client credential and a deliberately designed state/test-network topology.
9. doherty100 commented on Sep 22, 2026

   

   Roger Doherty (MSFT) (doherty100)

   on Sep 22, 2026

   ContributorAuthor

   More actions

   Authentication boundary refinement

   For #496 and the #173 CD workflow, Terraform supports only the system-assigned managed identity on the rg-devops-iac self-hosted runner (jumplinux2). User-assigned managed identities are intentionally out of scope.

   arm_client_id remains exclusively for the existing secret authentication mode and must not be set in MSI mode. The workflow's GitHub-hosted OIDC identity remains limited to starting the runner VM; Terraform and tests run on the runner's system-assigned identity.
10. doherty100 commented on Sep 22, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Sep 22, 2026

    ContributorAuthor

    More actions

    Superseding MSI identity decision

    The earlier system-assigned-runner identity guidance is superseded. #496 now requires a pre-provisioned UAMI attached to the Terraform execution environment; arm_client_id selects it in MSI mode. This stable principal receives the required Azure RBAC and Microsoft Graph group-management permissions before apply.

    For #173, attach the UAMI to jumplinux2. Sandbox jumpwin1 remains separate: its system-assigned identity is used only for Azure SQL database-user bootstrap and receives no Terraform UAMI control-plane or Graph permissions.
11. 1 remaining item

    Load more

12. doherty100 commented on Sep 22, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Sep 22, 2026

    ContributorAuthor

    More actions

    Dependency removed: headless bootstrap no longer required

    The CD design no longer generates terraform.tfvars from the bootstrap scripts. It will be authored manually and supplied to the workflow, either as a complete file or as discrete inputs assembled by the workflow, depending on how the remaining sub-issues are implemented.

    bootstrap.sh and bootstrap.ps1 stay interactive helpers for the manual service-principal-secret flow and are not part of the automated MSI path. #497 is closed as not planned, and blocker 2 has been struck from the list above. The issue body and todos were updated accordingly.

    The underlying data requirements are unchanged, they just live in the supplied file: arm_auth_mode = "msi", arm_client_id, the human admin user_object_id (the CD identity is not the admin, so it cannot be derived from the signed-in token), location, tags, and the module toggles.
13. doherty100 commented on Oct 5, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 5, 2026

    ContributorAuthor

    More actions

    Blocker 3 (#498, machine-readable test output) is complete and is no longer blocking this work.

    Landed in #790, merged to vnext on 2026-10-05.

    What shipped

    • Invoke-UnitTests.ps1 has new optional -ResultsJsonPath and -JUnitXmlPath parameters. Default behavior is unchanged when they are omitted.
    • JSON: run metadata (result, exitCode, timing), totals, and per-suite / per-check results, including failure detail. Pass/fail totals mirror the [SUMMARY] lines. This is the intended input for the workflow-tests step's $GITHUB_STEP_SUMMARY table and ::error:: annotations.
    • JUnit XML: one <testsuite> per module or integration suite, a <failure> per failed check, <skipped> for undeployed modules, and an <error> suite on harness errors. It can be consumed by standard test-reporter actions and uploaded as an artifact.
    • Result files are written on every exit path. Exit codes are 0 all passed, 1 test failures, 2 harness/prerequisite error.
    • Exit-code fix relevant to CD: #requires -Modules Az.* caused pwsh -File to return 0 even on failure. It is replaced by an explicit Import-Module, so the workflow can now reliably gate on the process exit code.

    Validation: run against a freshly provisioned sandbox with all base modules. A passing run returned exit 0 (109/109 checks). A run with transient environmental failures returned exit 1 (105 passed, 4 failed). In both cases the JSON and JUnit counts matched the console output.

    Remaining blockers: #499 (self-hosted runner), the RBAC option (#500), and #501 (runner-VM auto-shutdown and workflow-driven start).
14. doherty100 commented on Oct 6, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 6, 2026

    Last edited by doherty100

    ContributorAuthor

    More actions

    Design update — CD decisions ratified

    Design review while implementing #499. These decisions are now reflected in the issue body; this comment records the rationale and flags what changed.

    1. Terraform state: remote backend always (supersedes the previous local-state default)

    The issue previously specified runner-local .tfstate as the default with an opt-in remote backend. That is reversed. CD always uses the azurerm backend in the rg-devops-iac tfstate storage account (use_azuread_auth = true, use_msi = true), configured by partial-config -backend-config inputs at terraform init.

    Rationale — runner-local state on a CI host is an anti-pattern:

    • No locking. The blob backend takes a lease; a local file does not. The single-run concurrency guard is a policy control, not a technical one.
    • No durability. No versioning, soft delete or backup. Losing or rebuilding the runner VM orphans every resource in the sandbox.
    • Secrets at rest on a CI host. State contains sensitive attributes; keeping it on a long-lived machine that executes workflow code widens the blast radius.
    • No shared source of truth. A human cannot inspect or recover CD state from their own workstation.

    Knock-on effects, all simplifications:

    • actions/checkout keeps its default clean: true; the clean: false requirement is gone.
    • The "state must live inside the repo root so cleanterraformtemp.sh catches it" reconciliation is gone.
    • Optionally register the rg-devops-iac VM as a GitHub Actions self-hosted runner #499 provisions no runner-local state directory (the github_runner_state_directory input was removed before it shipped).

    Clarifying the bootstrap scenario. The one apply that legitimately uses local state is the initial apply of rg-devops-iac itself, which creates the storage account that later holds state — a genuine chicken-and-egg. That apply runs from a separate execution environment before the runner exists, so the runner can never be the thing performing it. This is a one-time bootstrap, not a CD mode.

    2. CD is from-scratch only

    CD validates that a release can build a sandbox from nothing. It is explicitly not a mechanism for incremental or in-place updates to a long-lived environment.

    • Every run starts with empty backend state and ends with the sandbox torn down and the state emptied (teardown under if: always() so a failed apply still cleans up).
    • Therefore the Scenario-2 re-apply path never occurs, and no enable-public-access.sh step is needed: on a fresh apply the barrier creates the resources with public access enabled, writes to them, and disables it within the same apply.
    • The preflight leftover check must now assert both a clean working tree and empty remote backend state.

    3. Only one CD run active at a time

    Promoted from a safety net to a hard design constraint. Workflow-level concurrency group with cancel-in-progress: false so overlapping runs queue instead of aborting an apply that may be 25–95 minutes in.

    Justification: there is exactly one self-hosted runner, one sandbox footprint and one backend state file, and a sandbox deployment is large and expensive. Cancelling mid-apply is the worst outcome — it leaves partially provisioned resources and dirty state.

    4. Self-hosted runner is mandatory, not a convenience

    Worth stating explicitly because it drives the entire security posture. A GitHub-hosted runner cannot do this job:

    • The Terraform state backend is network-isolated behind a private endpoint, reachable only from the VNet.
    • Integration tests establish point-to-site VPN connections to the vWAN P2S gateway to reach sandbox resources.

    So the hardening below is not optional risk-reduction; it is the price of the only viable architecture.

    5. Environment-specific ARM tags (new — not previously captured)

    Some subscriptions enforce organizational tagging policies, and some apply modify-effect policies that disable public network access on Key Vault and Storage. That directly breaks the public-access barrier pattern, which requires public access to remain enabled for the duration of the apply so Terraform can perform data-plane writes. An environment-specific ARM tag is required to opt out.

    Mechanism in CD: the supplied terraform.tfvars file. CD already supplies a manually authored, environment-specific terraform.tfvars to the runner (the tfvars-input todo). Any tags needed to satisfy or opt out of organizational policy simply live in that file. There is no need to plumb an additional_tags value through the workflow or onto the command line.

    The -var='additional_tags={...}' command-line pattern is for interactive and agent-driven installs, where bootstrap.sh generates terraform.tfvars and the extra tags have to be layered on per-invocation. It is already implemented and working for that case, and is not a CD concern.

    The same applies to the rg-devops-iac deployment, which is a one-time bootstrap performed from a separate execution environment with its own tfvars.

    Requirements that remain:

    • The tfvars-input design must explicitly account for these tags, so a CD run in a policy-constrained tenant is not broken by a missing tag.
    • The file is environment-specific and must never be committed — this is a public repo, and tag names and values are tenant-identifying.

    6. Security hardening is a hard requirement

    A self-hosted runner attached to a public repo is the dominant risk in this design: a malicious PR that can trigger a workflow job gets arbitrary code execution on a persistent VM holding a managed identity with elevated Azure rights.

    Threat model — the controls are not equally strong. For a pull_request trigger, GitHub runs the workflow YAML from the PR head, not from the base branch. A fork PR can therefore edit cd-provision-sandbox.yml itself — deleting the environment: key or the same-repo if: guard — or simply edit a .sh/.ps1/.tf file the workflow already executes. So any control expressed inside the workflow file is PR-editable and cannot be the outermost defense:

    Control	Where enforced	What it stops	PR-editable?
    Fork-PR approval setting	Repo/org settings	The run ever being queued	No
    Environment required reviewers	Repo settings, opted into by environment:	Execution until a human approves	Yes (the opt-in line)
    Same-repo / head-ref if: guard	Workflow YAML	Fork PRs reaching the runner	Yes
    Runner group restriction	Org settings	Unauthorized repos using the runner	No (org-level only)

    The fork-PR approval setting is the control that actually holds, because it is evaluated before any PR-supplied YAML is read. The in-workflow controls are defense-in-depth against misconfiguration and same-repo mistakes, not against a hostile fork.

    Required controls:

    • Fork-PR approval — Settings → Actions → General → Approval for running fork pull request workflows from contributors. Set to the strictest option available: Require approval for all outside collaborators at repo level (organizations also offer Require approval for all external contributors). Note this is UI-only — it cannot be set via the REST API or Terraform, so it is a documented manual setup step alongside branch protection.
    • GitHub Environment with required reviewers on the provisioning job, confirmed to apply to every trigger path including workflow_dispatch.
    • Same-repo / head-ref guard on the job (github.event.pull_request.head.repo.full_name == github.repository).
    • No pull_request_target and no fork triggers.
    • Runner group restriction where available (organization-level only — see Optionally register the rg-devops-iac VM as a GitHub Actions self-hosted runner #499 for the repository-level caveat).
    • Workspace hygiene between runs, since the runner is persistent and not ephemeral.

    These extend the existing security-guard and merge-gate-setup todos.

    In practice the intended trigger is the maintainer's own vnext -> main release PR, so the fork path is an abuse case rather than a normal one — but it is reachable by anyone on a public repo, which is why it is treated as a hard requirement.

    7. Flakiness must not erode trust in the gate

    A gate that fails for environmental reasons gets ignored, then bypassed. Mitigations:

    • ./scripts/manage-vms.sh start VM-start gate before any test run (a policy can deallocate VMs at any time; a deallocated domain controller breaks DNS, Kerberos, domain join, CIFS and SQL auth — the root cause confirmed in vm-jumpbox-linux unit tests fail: AD DNS 'mysandbox.local' does not resolve + CIFS skipped (no Kerberos TGT) [centralus, vnext] #447).
    • The environment-specific tags above.
    • Distinguish infrastructure failure from test failure in the job result, so a broken apply and a genuine regression are not reported identically.

    8. Resolved — ARM_USE_MSI in the runner environment is not a conflict

    Raised as an open question, now closed. #499 writes ARM_USE_MSI=true and ARM_TENANT_ID into the runner service environment, and the concern was whether that could interfere with a sandbox apply using SPN auth (arm_auth_mode = "secret").

    It cannot. Since #496, all three providers set the argument explicitly from the tfvars input:

    use_msi       = var.arm_auth_mode == "msi"
    client_secret = var.arm_auth_mode == "secret" ? var.arm_client_secret : null

    ARM_USE_MSI is only consulted as an environment default for an argument the provider block does not set. Because use_msi is always set explicitly, the environment variable is never reached and arm_auth_mode is authoritative.

    Corroborated empirically: cloud-init on the rg-devops-iac VM already pre-sets ARM_USE_MSI=true in the interactive shell, and sandbox deployments have been run successfully from that host. The runner service environment simply replicates a configuration already in use.

    No action required.
15. doherty100 commented on Oct 6, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 6, 2026

    ContributorAuthor

    More actions

    Runner validation findings (from testing the #499 branch end to end)

    The self-hosted runner from #499 was deployed to a rg-devops-iac environment and exercised with a real dispatched workflow job against a private test repository. Summary of what this proves for the CD design, and the one gap it uncovered.

    Verified working on the runner

    Capability	Result
    Label routing (runs-on: [self-hosted, <label>])	Job routed to the runner VM
    Job execution account	Runs as the unprivileged runner service account, not root
    Runner service lifecycle	active and enabled, so it survives the reboot implied by the auto-shutdown work in #501
    Terraform provider env (ARM_USE_MSI, ARM_TENANT_ID)	Present in the job environment
    Managed identity	Usable from a job step
    Toolchain	Terraform, Azure CLI, PowerShell 7 and the Az modules all present and on PATH
    Command-scoped sudo for the vwan P2S VPN tests	Allowlisted binaries permitted; everything else denied

    Gap found: Azure CLI / Az PowerShell are not authenticated inside jobs

    A job step running az account show fails:

    ERROR: Please run 'az login' to setup account.
    

    Cause. Cloud-init performs az login --identity as root during provisioning. The Azure CLI token cache is per-user ($HOME/.azure), and jobs run as the unprivileged runner service account, which has a different home directory and therefore no cached token. Connect-AzAccount has its own separate cache with the same problem.

    Important distinction. ARM_USE_MSI and ARM_TENANT_ID are present in the job environment, so Terraform provider authentication is unaffected - terraform plan/apply work. The gap is limited to the Azure CLI and Az PowerShell, which is precisely what Invoke-UnitTests.ps1 and manage-vms.sh depend on.

    Resolution. Fixed in the workflow, not on the VM: a baked-in token cache would be stale, is the wrong lifecycle, and would store credentials on disk unnecessarily. Each job that touches Azure should begin with an explicit login:

    - name: Azure login
      run: |
        az login --identity --output none
        pwsh -c 'Connect-AzAccount -Identity'

    The az login --identity half is confirmed working from a job step. Tracked as the new workflow-azure-login todo.

    Other observations relevant to this issue

    • Provisioning takes roughly 25 minutes, overwhelmingly the base OS package upgrade rather than runner setup (runner install and registration completed in under 30 seconds). Relevant to the workflow-preflight-start timeout budget: if the preflight job ever provisions or re-provisions the runner rather than merely starting it, allow for this.
    • Toggling the runner on or off replaces the VM, because the setting is delivered through custom_data, which forces replacement. The runner must therefore never be enabled or disabled by an apply executed on the runner itself.
    • Destroying or replacing the VM leaves a stale offline runner registered in the target repository or organization; it has to be removed out of band.
16. doherty100 commented on Oct 6, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 6, 2026

    ContributorAuthor

    More actions

    Progress update: runner-setup complete

    #499 is merged (PR #794) and closed. The runner-setup todo is marked done.

    Prerequisite status

    Todo	Status
    prereq-dual-auth	✅ DONE (#496, PR #743)
    prereq-test-output	✅ DONE (#498, PR #790)
    runner-setup	✅ DONE (#499, PR #794)
    prereq-pim-jit	open (#500)
    prereq-auto-shutdown	open (#501)

    All three prerequisites that the provisioning workflow depends on directly are now in place: selectable managed identity authentication, machine-readable test output, and a working self-hosted runner.

    What the runner now provides

    Confirmed by a real dispatched job, not just by inspection:

    • label-based routing to the runner virtual machine;
    • jobs run as an unprivileged service account, not root;
    • the runner service is active and enabled, so it survives the reboot implied by prereq-auto-shutdown (Workflow-driven start/stop for the rg-devops-iac runner VM #501);
    • managed identity usable from a job step;
    • Terraform, Azure CLI, PowerShell 7 and the Az modules present on PATH;
    • command-scoped sudo for the virtual WAN point to site VPN integration tests, verified to permit only the allowlisted binaries.

    Carried forward

    • workflow-azure-login (new) — jobs must run az login --identity and Connect-AzAccount -Identity explicitly. See the preceding findings comment for the cause.
    • backend-config is now unblocked: the runner exists and its managed identity already holds Storage Blob Data Contributor on the state storage account.
    • Runner documentation is intentionally deferred until the end to end scenario works, so docs should cover it rather than Optionally register the rg-devops-iac VM as a GitHub Actions self-hosted runner #499.

    Constraint worth restating before workflow authoring

    Enabling or disabling the runner changes custom_data, which forces replacement of the virtual machine. No workflow running on the runner may ever apply a change to enable_github_runner. Related: replacing or destroying the virtual machine leaves a stale offline runner registered that must be removed out of band.
17. Roger Doherty (MSFT) (doherty100)

    added sub-issues on Oct 8, 2026

    • Add GitHub App authentication mode for self-hosted runner registration #793
    • rg-devops-iac: optionally provision a user-assigned managed identity for MSI-mode sandbox applies #797
18. doherty100 commented on Oct 14, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 14, 2026

    ContributorAuthor

    More actions

    Progress is updated at the top of this issue, with the older design retained as history.

    • Manual deployment and release-PR gate acceptance both passed 109 checks with zero failures; the release gate is now required on main.
    • Release: self-hosted Sandbox deployment gate and dependency maintenance #848 is released as v5.14.2. Sync release history and main policy documentation into vnext #849 and Show deployed resource group and consolidate release-gate documentation #850 bring vnext current with the policy sync, complete walkthrough, and resource-group summary enhancement.
    • Current decisions are explicit: fresh runs/state keys, no supported job reruns, manual Azure cleanup, and JSON-only public evidence.

    Leaving #173 open. Bounded runner-local cleanup, #500, #501, and the documented acceptance/reporting gaps remain follow-up work.
19. Roger Doherty (MSFT) (doherty100)

    added sub-issues on Oct 15, 2026

    • Azure Sandbox deployment: allow manual deployment from vnext or main #853
    • Gate release deployment requests on ready status and passing static CI #854
20. doherty100 commented on Oct 15, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 15, 2026

    ContributorAuthor

    More actions

    Live acceptance: fresh deployment after an environmental release-gate failure (2026-10-15)

    Tested successfully: recover the release gate by deploying the unchanged candidate into a completely new sandbox, without reusing the failed deployment or bypassing the gate.

    Sequence and evidence

    1. Release PR Release: promote vnext to main #852 (vnext → main) initially triggered run 38764864917. Apply failed and the release gate did not pass. The operator identified the failure as an environmental P2S VPN gateway provisioning failure, rather than a code defect, and manually deleted the failed sandbox.
    2. The operator closed and reopened the same non-draft PR. No candidate code change was made: the head remained 113cb5bc79083eb35be8308e2b80a78e0e140e0e.
    3. Reopening triggered new run 38784611441, created at 18:19 UTC. This was a new workflow run with a new run-derived deployment ID and remote-state key—not a job rerun or recovery using the previous state.
    4. The operator approved the protected sandbox-release-gate environment. Preflight, Apply, Unit-test, and final release-status publication all succeeded. The run completed at 20:12 UTC, and sandbox-deployment-gate reported success on the unchanged candidate.
    5. PR Release: promote vnext to main #852 was merged at 20:14 UTC with merge commit e2a63ef4dc1697050f2ca78743f081a8d6fa55af; release v5.14.3 was published afterward. No release-gate bypass was needed.

    Confirmed operating path

    For an environmental failure requiring a completely fresh release-gated deployment: manually clean up the failed sandbox as appropriate, close/reopen the ready release PR, and approve the new deployment run. This preserves candidate identity while allocating independent deployment state and publishing a new gate result.

    GitHub Re-run jobs is not the supported fresh-deployment path: it retains the original run ID/state key, and preflight rejects an existing state blob. Manual workflow dispatch can create a fresh sandbox but does not publish the PR release-gate status. Automatic teardown, automatic retries, existing-state recovery, and old-state cleanup were not tested or introduced by this acceptance case.

    This supplements the earlier successful release-path acceptance with a live environmental-failure → fresh-deployment → successful-gate-recovery case. Other remaining work under this issue is unchanged.
21. doherty100 commented on Oct 17, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 17, 2026

    ContributorAuthor

    More actions

    Progress: manual deployment source choice (#853)

    Manual runs can now deploy released main code as well as vnext. The Current operating decisions section above is updated.

    Delivered

    • workflow_dispatch has a new source_branch choice: vnext (the default, so earlier behavior is unchanged) or main. Every stage (preflight, apply, unit-test) works with either source, and runs keep the sandbox-deployment environment and its approvals.
    • The hosted selection job reads the input only through an environment variable. It rejects any other value (other branches, tags, SHAs, fork references, or an empty value) before checkout or environment-secret access, then checks out refs/heads/<branch> from this repository.
    • The run name (... / <stage> / <branch>) and the selection summary show the source. Release PR runs ignore the input and still deploy the exact head SHA through sandbox-release-gate. Manual runs never publish or satisfy release-static-ci or sandbox-deployment-gate.
    • scripts/tests/Test-SandboxDeploymentSelection.sh runs the selection step against sample events. It passed 32/32 in the hosted actionlint job and failed 18 cases against the previous workflow. It also runs from ./scripts/Invoke-CIChecks.sh actions.

    Publication

    1. Allow manual sandbox deployment from vnext or main #868 → vnext (squash): the workflow, test script, CI wiring, and docs (setup guide Step 95).
    2. Register manual deployment source choice on main (#853) #869 → main (squash): an authorized maintenance PR containing only sandbox-deployment.yml, identical to the vnext version. ci-actions and ci-secrets were dispatched manually and passed. An owner bypass merged it because its source isn't vnext.
    3. Merge main into vnext to record maintenance PR #869 history #870 → vnext (regular merge): records the history of Register manual deployment source choice on main (#853) #869 with no content changes.

    The test script and documentation reach main with the next normal release.

    Live acceptance

    Manual run 39050200826 (/ apply / main) selected main and skipped every release-only job. After approval, Preflight passed on the main checkout, and Apply was still in progress when this comment was posted. The operator accepted the feature.

    #173 remains open for #500, #501, and bounded runner-local cleanup.
22. Roger Doherty (MSFT) (doherty100)

    added a sub-issue on Oct 20, 2026

    • Sandbox deployment preflight: ensure runner Terraform CLI is compatible with the deployed branch's required_version #906
23. Roger Doherty (MSFT) (doherty100)

    closed this as completedon Oct 21, 2026
24. doherty100 commented on Oct 21, 2026

    

    Roger Doherty (MSFT) (doherty100)

    on Oct 21, 2026

    ContributorAuthor

    More actions

    Closeout

    All of the CD pipeline's planned sub-issues have shipped to main and passed live acceptance.

    Current state of the Sandbox deployment workflow

    • Triggers: release PRs from vnext to main (deployment gate), plus manual dispatch with a run_through stage and a source_branch of vnext or main (Azure Sandbox deployment: allow manual deployment from vnext or main #853).
    • Static CI before approval: static CI must pass before a release deployment can be approved (Gate release deployment requests on ready status and passing static CI #854). Admins can't skip that approval.
    • Authentication: GitHub OIDC with no stored secret, using an unattached deployment identity that has standing Reader only. The operator grants time-bound roles before approving each run (Secretless, approval-gated GitHub OIDC access to an unattached deployment UAMI #500: Secretless OIDC deployment identity with just-in-time roles (#500) #903, Release: secretless OIDC deployment identity with just-in-time roles #904, Maintenance: grant the sandbox job id-token: write for OIDC (#500) #905).
    • Runner power: a GitHub-hosted start_runner job waits for the deployment approval, then starts the self-hosted runner VM with a separate power state identity. stop_runner deallocates the VM only if this run started it. A rejected run never starts the VM, and a VM that was already running is left running (Workflow-driven start/stop for the rg-devops-iac runner VM #501: Runner power state identity, runner Terraform version check, and local PSScriptAnalyzer fix (#501, #906, #902) #907, Start and deallocate the runner VM from the deployment workflow (#501) #908, Maintenance: start and deallocate the runner VM from the deployment workflow (#501) #909).
    • Runner Terraform: preflight checks the runner's Terraform against the deployed branch's required_version and installs a compatible version only when needed (Sandbox deployment preflight: ensure runner Terraform CLI is compatible with the deployed branch's required_version #906).
    • Runner host: Bastion and Entra ID SSH replace the runner VM's public IP. It has Log Analytics, AMPLS and diagnostic settings (rg-devops-iac: replace jumplinux2 public IP with Bastion Standard and Entra ID SSH #898, rg-devops-iac: Log Analytics, AMPLS and diagnostic settings (#896) #899).

    Acceptance

    • Release Release: workflow starts and deallocates the runner VM; runner Terraform version check #910 (run), starting with the VM deallocated: approval, VM start, preflight (installed the required Terraform), apply, unit tests, then the VM deallocated again.
    • Manual dispatches from main: rejected (run), where the VM was never started, and VM already running (run), where the VM was not stopped.

    Details are in #500 and #501.