From 9c41af7f5dd25fc0d25e536947c11a913f9f2b71 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Wed, 2 Sep 2026 11:56:53 -0400 Subject: [PATCH 1/6] Path B: route whole-file deletes away from the offline window; fix reclaim rationale CSS feedback on a live case (customer deleted a number of VMs and wanted the space back) was that the steps before slab consolidation are heavyweight and may not always be needed, and that the key requirement is really about active file handles on the CSV. Reviewing Path B against first-party documentation, most of that is correct, and the guide's stated reason for the offline window was wrong. Changes: - Add a no-downtime pre-branch. Deleting a whole file frees its slabs outright and ReFS returns them to the pool on its own, with no Optimize-Volume. Learn documents this as "a gradual process that can take 15 minutes or so after the files are deleted" and notes that many running workloads make it take longer rather than preventing it. Only interior fragmentation, where live data still occupies part of a slab, needs consolidation and therefore an offline window. The guide already said this in the pool-fill escalation ladder; this promotes it into Path B where the reader making the decision will see it. - Correct the reason for taking VMs offline. The guide said virtual disk file handles must be released, "required for consolidation". No first-party source states that, and FSCTL_FILE_LEVEL_TRIM exists specifically to reclaim ranges inside an open virtual-disk file. The accurate reason is that ReFS allocates on write, so a running VM keeps re-dirtying slabs that consolidation is trying to empty. The step is unchanged; only the rationale is corrected. - Note that consolidation runs at low priority by default, per Optimize-Volume (-NormalPriority: "By default, the priority is low") and defrag /h, while being explicit that this is scheduling priority and not total cost: pool disks are shared by every volume, so hours of relocation I/O can still be felt elsewhere. - Add a caution against moving VM disks between volumes to relieve pressure. For Arc-managed VMs this is storage live migration, which Learn lists among operations that "can lead to Azure Local VMs becoming unmanageable from the Azure portal", because the storage path resource keeps pointing at the old volume. Scoped to Arc VMs, since Move-VMStorage remains supported for traditional clustered Hyper-V VMs, and paired with the supported alternative. - Add the stretched-cluster caveat (TRIM is disabled, so deleted capacity is never returned) and a DisableDeleteNotify precondition check, both of which explain a reclaim that appears to do nothing. - Move the optional checkpoint merge out of the numbered sequence so it stops reading as a required first step, and renumber accordingly. Not changed: the offline window itself. Whether consolidation can run safely with live VMs is a separate question that needs measurement on physical S2D hardware, and is not loosened here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4 --- ...ot-Storage-StoragePoolCapacityThreshold.md | 210 +++++++++++++----- 1 file changed, 159 insertions(+), 51 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 7fb5b7ed..943c716e 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -74,15 +74,18 @@ Triage: minutes. Adding disks (A1) or adjusting the alert (A4/A5): low and online. Converting fixed→thin (A2) or the thin reclaim (Path B): a maintenance window; slab consolidation can take hours on large - volumes. + volumes. If whole VMs or files were simply deleted, Path B's no-downtime + pre-branch may resolve it in about 15 minutes with no window at all. Downtime / maintenance window Triage and the add-capacity / alert options (A1, A4, A5) are - online. A2 (convert, then Path B), - A3 (evacuate + recreate a volume), and Path B - require data movement or a VM-offline window; Path B's window lasts through slab - consolidation. + online. A2 (convert, then Path B) and + A3 (evacuate + recreate a volume) require data movement. + Path B is conditional: reclaiming capacity from + deleted whole files is automatic and needs + no downtime; only interior fragmentation + requires slab consolidation and a VM-offline window. @@ -110,8 +113,9 @@ Then branch on `ProvisioningType`: capacity, convert to thin, or adjust the alert. Go to [Path A](#path-a-fixed-provisioned-volumes). - **`Thin`** → capacity from deleted data can be reclaimed. Go to - [Path B](#path-b-thin-provisioned-volumes-reclaim-unused-capacity) (needs a - maintenance window). + [Path B](#path-b-thin-provisioned-volumes-reclaim-unused-capacity). If whole VMs + or files were deleted, its no-downtime pre-branch may be all you need; only + interior fragmentation needs a maintenance window. > [!NOTE] > This is the short form of @@ -452,18 +456,87 @@ Set-StoragePool -FriendlyName "" -ThinProvisioningAlertThresholds @(8 ## Path B: Thin-provisioned volumes (reclaim unused capacity) > [!IMPORTANT] -> **Ownership gate (read before starting).** This is a scheduled -> maintenance-window procedure that takes VMs offline; it is owned by the -> customer's cluster or storage administrator. If you are not that +> **Ownership gate (read before starting).** Capacity work on a production volume +> is owned by the customer's cluster or storage administrator. The read-only Quick +> triage, the [Verify](#verify) queries, and the no-downtime pre-branch below are +> always safe to run. The **numbered consolidation steps** are a scheduled +> maintenance-window procedure that takes VMs offline: if you are not that > administrator, or you are unsure whether you are authorized to take these -> workloads offline, stop here and hand off. The read-only Quick triage and the -> [Verify](#verify) queries are always safe to run; the numbered steps below -> are not. +> workloads offline, stop before them and hand off. On thin volumes, capacity that was written and later deleted can remain committed to the pool in partially used 256 MB "slabs". A slab is only returned to the pool -once all of its blocks are free. The supported procedure consolidates the live -data into fewer slabs and releases the emptied slabs back to the pool. +once all of its blocks are free. Deleting a whole file frees its slabs outright and +they are returned automatically; when live data still occupies part of a slab, +consolidation is needed to move that data into fewer slabs so the emptied ones can +be released. + +### Before you start: is consolidation even the right tool? + +Two different mechanisms return capacity to the pool, and **only one of them needs +an offline window**. Identify which case you are in before scheduling anything. + +- **Whole files were deleted** (VMs deleted, VHDX removed, ISOs purged). The slabs + those files occupied become entirely free, and ReFS returns them to the pool + **on its own**, with no `Optimize-Volume` and **no downtime**. Microsoft + documents this as a gradual process that takes *"15 minutes or so after the + files are deleted"*, and notes that *"if there are many workloads running on the + system, it may take longer for all of the space to be returned to the pool"* + ([thin provisioning FAQ][thin-prov]). Running workloads **slow this down; they + do not block it**. +- **Interior fragmentation** (data deleted from *inside* a VHDX or a guest file + system). Blocks are freed inside slabs that still hold other live data, so no + whole slab frees and automatic reclamation returns nothing. This is the only + case that needs slab consolidation, and therefore the only case that needs the + offline window. + +**If you deleted whole VMs or files, start here. This path needs no downtime:** + +1. Confirm the deletions are complete. For Arc VMs, confirm they were deleted + **through Azure** (see the caution below). +2. Wait at least 15 minutes; longer on a busy cluster. +3. Re-measure with the [Verify](#verify) queries. + +If the pool has dropped below threshold, **you are done, with no maintenance +window**. Continue to the consolidation procedure only if the pool is still above +threshold *and* the volume genuinely shows large interior free space. + +#### Preconditions for automatic reclamation + +Automatic reclamation depends on TRIM/unmap being active. Check this before +concluding that reclamation "did not work": + +```powershell +# 0 = enabled (expected). 1 means deleted data is never returned to the pool. +fsutil behavior query DisableDeleteNotify +``` + +> [!IMPORTANT] +> **Stretched clusters never reclaim deleted capacity.** *"Because TRIM is disabled +> for stretched clusters, storage isn't returned to the pool after data is +> deleted"* ([thin provisioning][thin-prov]). On a stretched cluster this reclaim +> procedure will not help; treat the shortage as a +> [Path A](#path-a-fixed-provisioned-volumes) capacity problem instead. + +> [!CAUTION] +> **Do not move VM disks to another volume as a way to relieve pressure.** For +> **Arc-managed Azure Local VMs (23H2+)**, moving a VHD/VHDX to another CSV with +> host-side tools (`Move-VMStorage`, Failover Cluster Manager, or a manual file +> move) is *storage live migration*, which Microsoft lists among operations that +> *"can lead to Azure Local VMs becoming unmanageable from the Azure portal"* +> ([unsupported VM operations][unsupported-ops]). Azure tracks each disk's location +> through a **storage path** (`Microsoft.AzureStackHCI/storagecontainers`) +> resource; a host-side move leaves that resource pointing at the old volume, and +> the VM, disk, and network-interface resources can be left stale and undeletable. +> +> There is **no supported in-place move** of an existing Arc VM disk between +> volumes: a storage path is selected at **creation** time. To place a workload on +> a different volume, create the disk or VM against a storage path on that volume +> through Azure, rather than moving files on the host. +> +> This restriction applies to **Arc-managed** VMs. For traditional (non-Arc) +> clustered Hyper-V VMs, `Move-VMStorage` with the cluster resource updated +> accordingly remains supported. > [!NOTE] > This procedure recovers capacity only when the volume genuinely holds far less @@ -472,30 +545,36 @@ data into fewer slabs and releases the emptied slabs back to the pool. > close to `Size × resiliency`). If footprint matches the data actually written, > there is nothing to reclaim. -**Procedure (requires an offline window for VMs on the affected volume; the window lasts through slab consolidation, which can take hours on large volumes):** [MEDIUM RISK] +**Procedure for interior fragmentation (requires an offline window for VMs on the affected volume; the window lasts through slab consolidation, which can take hours on large volumes):** [MEDIUM RISK] -1. *(Optional, no downtime)* Merge Hyper-V checkpoints that are no longer needed - (`Get-VM | Get-VMSnapshot`, then `Remove-VMSnapshot`). Checkpoint files pin - extra slabs and reduce what consolidation can recover. +> [!TIP] +> **Preparation (optional, no downtime, do this before the window).** Merge +> Hyper-V checkpoints that are no longer needed (`Get-VM | Get-VMSnapshot`, then +> `Remove-VMSnapshot`). Checkpoint files hold live data that pins extra slabs and +> reduces what consolidation can recover. This is preparation, not part of the +> maintenance window. -2. **Take the VMs on the affected volume offline** so their virtual disk file - handles are released (required for consolidation). First find where each VM is - running: +1. **Stop active writes to the volume by taking its VMs offline.** ReFS allocates + on write, so a VM that keeps running keeps allocating and re-dirtying slabs + while consolidation is trying to empty them. Quiescing writes is what makes a + consolidation pass deterministic and complete; a pass run under live workload + can recover little or nothing and may have to be repeated. First find where + each VM is running: ```powershell Get-ClusterGroup | Where-Object GroupType -eq 'VirtualMachine' | Select-Object Name, OwnerNode, State ``` - **Prefer a clean guest shutdown**, which releases the file handles **without** - writing a saved-state file: + **Prefer a clean guest shutdown**, which ends guest writes **without** writing + a saved-state file: ```powershell Stop-VM -Name "" # graceful guest shutdown; run on/target the owner node ``` If a guest will not shut down cleanly (hung, or no integration services), a - forced turn-off also releases the file handles **without** writing a + forced turn-off also ends guest writes **without** writing a saved-state file, but only as a last resort **[HIGH RISK]**: ```powershell @@ -509,14 +588,18 @@ data into fewer slabs and releases the emptied slabs back to the pool. > after the workload owner has approved it for that specific VM. > [!CAUTION] - > Do **not** substitute `Save-VM` (or the **Save** automatic stop action) or - > `Suspend-VM` here. **Saving** releases the handles but writes a saved-state - > file the size of the VM's memory onto the very volume you are trying to free. - > **Suspending** only *pauses* the VM, its memory stays in host RAM and its - > virtual disk handles stay **open**, so slab consolidation cannot proceed. - > Putting the cluster resource into redirected access is likewise **not** - > sufficient. The VM's file handles must actually be released, which means a - > shutdown or turn-off. + > Do **not** substitute `Save-VM`, or the **Save** automatic stop action, here. + > **Saving** writes a saved-state file roughly the size of the VM's memory onto + > the very volume you are trying to free, consuming the capacity you are trying + > to recover. + > + > `Suspend-VM` (pause) writes no state file and does halt the guest's I/O, but it + > leaves the virtual disk handles open and the guest's memory resident on the + > host, and it has **not been validated** as sufficient for a complete + > consolidation pass. Until it has been, use a clean shutdown, which is the + > configuration this procedure has been validated in. Putting the cluster + > resource into redirected access is **not** a substitute either, because the VMs + > keep running and keep writing. > [!IMPORTANT] > For **Arc-managed VMs** (Azure Local 23H2+), stop the VM from Azure (portal @@ -525,7 +608,7 @@ data into fewer slabs and releases the emptied slabs back to the pool. > agent / Arc Resource Bridge view of the VM state. Once workloads on the volume > are stopped cluster-wide, proceed with consolidation. -3. **Consolidate slabs** on the volume. Run this on the CSV **owner node**. +2. **Consolidate slabs** on the volume. Run this on the CSV **owner node**. Resolve the CSV's `C:\ClusterStorage\` path to its volume object with `Get-Volume -FilePath`, confirm it is the volume you intend, then pipe it to `Optimize-Volume`: @@ -558,6 +641,19 @@ data into fewer slabs and releases the emptied slabs back to the pool. > alone.) Slab consolidation is the time-consuming step and can take hours on > multi-terabyte volumes. + > [!NOTE] + > **Consolidation runs at low priority by default.** `Optimize-Volume` + > documents `-NormalPriority` as *"Indicates that this cmdlet the operation at + > normal priority. By default, the priority is low"* + > ([Optimize-Volume](https://learn.microsoft.com/powershell/module/storage/optimize-volume)), + > matching `defrag /h` (*"Runs the operation at normal priority (default is + > low)"*). The pass therefore yields to workload I/O rather than competing with + > it. Note that this governs *scheduling priority*, not total cost: on a + > multi-terabyte volume, consolidation still performs hours of back-end data + > relocation, and the pool's physical disks are shared by **every** volume in + > the pool, so sustained relocation I/O can be felt by workloads on other + > volumes. Prefer a low-usage window on large or busy systems. + > [!NOTE] > **Substrate matters if you are validating in a lab.** The reclaim is only > observable on **physical S2D hardware**. On a nested or VM-based cluster, @@ -567,17 +663,17 @@ data into fewer slabs and releases the emptied slabs back to the pool. > not a failure of the procedure and not a defect in the volume. Grade this > remediation only on physical S2D, never on a nested or VM cluster. -4. **Wait about 15 minutes** after consolidation completes. The capacity is +3. **Wait about 15 minutes** after consolidation completes. The capacity is returned to the pool by the **ReFS background unmap workitem**, which runs after `Optimize-Volume -SlabConsolidate` finishes. This wait, not the next step, is what releases the emptied slabs. > [!NOTE] - > VMs only need to stay offline through the consolidation in Step 3. Once - > Step 3 reports complete, you can bring the VMs back online (Step 6) and run + > VMs only need to stay offline through the consolidation in Step 2. Once + > Step 2 reports complete, you can bring the VMs back online (Step 5) and run > the remaining steps with workloads online, shortening the maintenance window. -5. **(Optional) Rebalance the pool allocation:** +4. **(Optional) Rebalance the pool allocation:** ```powershell Optimize-StoragePool -FriendlyName "" -Verbose @@ -586,13 +682,13 @@ data into fewer slabs and releases the emptied slabs back to the pool. `Optimize-StoragePool` rebalances Storage Spaces allocations across the pool; it is primarily used to spread data onto newly added drives and is a finalize step here, not the mechanism that frees the slabs (that already happened in - Step 4). Monitor with `Get-StorageJob` and wait until no `Optimize` jobs are + Step 3). Monitor with `Get-StorageJob` and wait until no `Optimize` jobs are running before re-measuring pool fill. If it finishes in seconds with no jobs, that is expected when there is nothing to rebalance; it does **not** mean reclamation failed; confirm the result with the pool fill query in [Verify](#verify). -6. **Bring the VMs back online.** For **traditional non-Arc Hyper-V VMs**, start +5. **Bring the VMs back online.** For **traditional non-Arc Hyper-V VMs**, start them on the host: ```powershell @@ -604,17 +700,21 @@ data into fewer slabs and releases the emptied slabs back to the pool. > **through Azure** (the VM resource in the portal or CLI), not with host > `Start-VM`. Driving an Arc VM's power state directly on the host bypasses the > control plane and can desynchronize the Arc agent and Arc Resource Bridge - > view of the VM state; this mirrors the stop-side boundary in Step 2. + > view of the VM state; this mirrors the stop-side boundary in Step 1. > [!NOTE] -> A consolidation pass can legitimately return little or no capacity, most often -> because the volume's footprint already matches the data actually written (there -> is nothing to reclaim; see the note at the start of Path B), or because slabs -> are still pinned by data in use (confirm every VM on the volume is stopped in -> Step 2 and that stale checkpoints were merged in Step 1). If real interior free -> space exists, all workloads were offline, and checkpoints were merged, but the -> pool still does not drop after the unmap wait (Step 4), open a Microsoft support -> case rather than repeating the procedure. +> A consolidation pass can legitimately return little or no capacity. The most +> common reasons, in order: the volume's footprint already matches the data +> actually written, so there is nothing to reclaim (see the note at the start of +> this procedure); TRIM/unmap is disabled (`DisableDeleteNotify = 1`) or the +> cluster is **stretched**, where deleted capacity is never returned at all; or +> slabs are still pinned by data in use (confirm every VM on the volume is stopped +> in Step 1 and that stale checkpoints were merged in the preparation step). Note +> that some slabs report "pinned unmovable" even on a fully quiesced volume, so a +> partial reclaim is not by itself a failure. If real interior free space exists, +> TRIM is enabled, the cluster is not stretched, all workloads were offline, and +> checkpoints were merged, but the pool still does not drop after the unmap wait +> (Step 3), open a Microsoft support case rather than repeating the procedure. ## Choose the right option @@ -625,7 +725,8 @@ data into fewer slabs and releases the emptied slabs back to the pool. | Fixed | Remove unneeded volumes | A3: shrink/remove (ReFS = evacuate + recreate) | | Fixed | Stop the alert (risk accepted) | A4: disable the Health Service alert | | Fixed | Move the alert threshold | A5: raise `ThinProvisioningAlertThresholds` | -| Thin | Return deleted-data capacity to the pool | Path B: SlabConsolidate + ReFS unmap | +| Thin | Return capacity from **deleted whole files/VMs** | Path B pre-branch: delete, then wait (no downtime) | +| Thin | Return capacity stranded by **interior fragmentation** | Path B: SlabConsolidate + ReFS unmap (offline window) | ## Verify @@ -752,7 +853,8 @@ firm conditions is met. Do not simply re-run the procedure. *operational state* is `Incomplete` / read-only from a drive-quorum loss rather than capacity, that is a separate, higher-severity problem. Escalate immediately.) - **Path B completed with every precondition met** (confirmed real interior free - space, every VM on the volume stopped, checkpoints merged) and you waited out the + space, TRIM enabled and the cluster not stretched, every VM on the volume + stopped, checkpoints merged) and you waited out the ReFS unmap, but pool `AllocatedSize` still does not drop. - The reserve-capacity fault (`InsufficientReserveCapacity`) **persists after** you have added capacity or reduced footprint. @@ -775,5 +877,11 @@ Include the data-collection output above with any Microsoft support case. - [Azure Local Health Service settings (volume and pool capacity thresholds)](https://learn.microsoft.com/azure/azure-local/manage/health-service-settings) - [Azure Local Health Service faults reference (`Get-HealthFault` fault types)](https://learn.microsoft.com/azure/azure-local/manage/health-service-faults) - [Set-VM (automatic stop action)](https://learn.microsoft.com/powershell/module/hyper-v/set-vm) +- [Storage thin provisioning in Azure Local (reclamation behavior and FAQ)](https://learn.microsoft.com/azure/azure-local/manage/manage-thin-provisioning-23h2) +- [Supported and unsupported operations for Azure Local VMs](https://learn.microsoft.com/azure/azure-local/manage/virtual-machine-operations) +- [Create a storage path for Azure Local VMs](https://learn.microsoft.com/azure/azure-local/manage/create-storage-path) + +[thin-prov]: https://learn.microsoft.com/azure/azure-local/manage/manage-thin-provisioning-23h2 +[unsupported-ops]: https://learn.microsoft.com/azure/azure-local/manage/virtual-machine-operations --- From 915048f030bdea7e99589621e023e4d55a992f02 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Wed, 2 Sep 2026 12:30:05 -0400 Subject: [PATCH 2/6] Restore the in-use/pinned-slab rationale for taking VMs offline Located the March 2026 FS engineering thread that the offline-window guidance originally came from, and it does support the mechanism the guide stated. FS engineering's position there is that "pinned unmovable" slabs are mostly caused by applications holding files pinned, that slab consolidation cannot process files that are actively in use, and that the recommended sequence is to stop workloads on the volume, consolidate, and let the ReFS unmap work items reclaim. It also notes that internal testing tended to succeed because workloads were stopped, while production workloads keep slabs pinned and reduce how much a pass recovers. The same thread independently confirms the guide's existing advice not to use -ReTrim on thin-provisioned ReFS. My earlier rewording attributed the need for a quiesced volume to ReFS allocate-on-write re-dirtying slabs. That is a real secondary effect but it is not the primary mechanism, and replacing the in-use rationale with it was wrong. Restored to the accurate mechanism: consolidation relocates live data out of partially used slabs and cannot relocate data belonging to files that are actively in use, so those slabs are reported pinned unmovable and skipped. Also adjusted the Suspend-VM note. Pausing leaves the virtual disk files open and the guest resident, so it is not a reliable substitute for a shutdown. This drops the earlier "not been validated" phrasing, which understated a mechanism that is in fact documented by the owning team. Unchanged from the previous commit: the no-downtime pre-branch for whole-file deletes, which rests on separate Microsoft documentation and is not affected by this correction; the low-priority note; the Arc disk-move caution; and the stretched-cluster and DisableDeleteNotify preconditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4 --- ...ot-Storage-StoragePoolCapacityThreshold.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 943c716e..f63db547 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -554,27 +554,28 @@ fsutil behavior query DisableDeleteNotify > reduces what consolidation can recover. This is preparation, not part of the > maintenance window. -1. **Stop active writes to the volume by taking its VMs offline.** ReFS allocates - on write, so a VM that keeps running keeps allocating and re-dirtying slabs - while consolidation is trying to empty them. Quiescing writes is what makes a - consolidation pass deterministic and complete; a pass run under live workload - can recover little or nothing and may have to be repeated. First find where - each VM is running: +1. **Take the VMs on the affected volume offline.** Consolidation works by + relocating live data out of partially used slabs so whole slabs can be freed, + and it cannot relocate data belonging to files that are actively in use. Those + slabs are reported as "pinned unmovable" and skipped, which is the main reason + a pass run against a live volume recovers far less than one run against a + quiesced volume. Stopping the workload is what makes that data movable. First + find where each VM is running: ```powershell Get-ClusterGroup | Where-Object GroupType -eq 'VirtualMachine' | Select-Object Name, OwnerNode, State ``` - **Prefer a clean guest shutdown**, which ends guest writes **without** writing - a saved-state file: + **Prefer a clean guest shutdown**, which releases the VM's files **without** + writing a saved-state file: ```powershell Stop-VM -Name "" # graceful guest shutdown; run on/target the owner node ``` If a guest will not shut down cleanly (hung, or no integration services), a - forced turn-off also ends guest writes **without** writing a + forced turn-off also releases the VM's files **without** writing a saved-state file, but only as a last resort **[HIGH RISK]**: ```powershell @@ -593,13 +594,12 @@ fsutil behavior query DisableDeleteNotify > the very volume you are trying to free, consuming the capacity you are trying > to recover. > - > `Suspend-VM` (pause) writes no state file and does halt the guest's I/O, but it - > leaves the virtual disk handles open and the guest's memory resident on the - > host, and it has **not been validated** as sufficient for a complete - > consolidation pass. Until it has been, use a clean shutdown, which is the - > configuration this procedure has been validated in. Putting the cluster - > resource into redirected access is **not** a substitute either, because the VMs - > keep running and keep writing. + > `Suspend-VM` (pause) writes no state file, but it leaves the virtual disk + > files open and the guest's memory resident on the host, so it is not a + > reliable substitute for a shutdown. Putting the cluster resource into + > redirected access is **not** a substitute either, because the VMs keep running + > and their files stay in use. To make a file's slabs movable, the workload + > holding it has to be stopped. > [!IMPORTANT] > For **Arc-managed VMs** (Azure Local 23H2+), stop the VM from Azure (portal From 6cab6283fc65b22718e58c2c83fdde79bff8fb26 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Wed, 2 Sep 2026 15:37:41 -0400 Subject: [PATCH 3/6] Address review: deleted VMs do not imply deleted disks, and fix the TRIM check Five review passes across two model families found four things worth fixing. Deleting a VM does not delete its disks. Remove-VM "deletes the virtual machine's configuration file, but does not delete any virtual hard drives", so a customer who removed VMs can still be holding every VHDX on the volume. The pre-branch previously said only to confirm the deletions were complete, which would send that customer into a 15 minute wait that cannot reclaim anything and leave them concluding the guide is wrong. It now asks them to confirm the disk files themselves are gone and gives a command to find orphaned ones. The DisableDeleteNotify check described a single value, but the command returns one line per file system and a line can read "is not currently set", which is neither 0 nor 1. Azure Local CSVs are ReFS, so the ReFS line is the one that governs and the check did not say so. It now shows the real output shape, names the file system to read, covers all three states, and says to run it on the CSV owner node. The 0 and 1 mapping itself was correct and is unchanged. The precondition is also re-scoped to gate both paths rather than only the automatic one, since consolidation depends on the same ReFS unmap to return freed slabs. The options table described the no-downtime branch as "delete, then wait" while the ownership gate called that branch always safe to run. Together those made an irreversible delete read as a safe action. The table now says confirm, wait, re-measure, and the gate is explicit that the branch asks for an irreversible delete and is not covered by the safe-to-run exemption. Trimmed the -NormalPriority quote to the clause that carries the meaning. The longer sentence is missing a verb on Learn itself, so quoting it verbatim was faithful but read as an error in this guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4 --- ...ot-Storage-StoragePoolCapacityThreshold.md | 62 ++++++++++++++----- 1 file changed, 46 insertions(+), 16 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index f63db547..4c12031c 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -458,11 +458,12 @@ Set-StoragePool -FriendlyName "" -ThinProvisioningAlertThresholds @(8 > [!IMPORTANT] > **Ownership gate (read before starting).** Capacity work on a production volume > is owned by the customer's cluster or storage administrator. The read-only Quick -> triage, the [Verify](#verify) queries, and the no-downtime pre-branch below are -> always safe to run. The **numbered consolidation steps** are a scheduled -> maintenance-window procedure that takes VMs offline: if you are not that -> administrator, or you are unsure whether you are authorized to take these -> workloads offline, stop before them and hand off. +> triage and the [Verify](#verify) queries are always safe to run. Everything else +> here changes state: the no-downtime branch below asks you to **delete orphaned +> virtual disk files**, which is irreversible, and the **numbered consolidation +> steps** are a scheduled maintenance-window procedure that takes VMs offline. If +> you are not that administrator, or you are unsure whether you are authorized to +> delete those files or take these workloads offline, stop here and hand off. On thin volumes, capacity that was written and later deleted can remain committed to the pool in partially used 256 MB "slabs". A slab is only returned to the pool @@ -492,8 +493,21 @@ an offline window**. Identify which case you are in before scheduling anything. **If you deleted whole VMs or files, start here. This path needs no downtime:** -1. Confirm the deletions are complete. For Arc VMs, confirm they were deleted - **through Azure** (see the caution below). +1. **Confirm the virtual disk files themselves are gone, not just the VMs.** Deleting + a VM does not necessarily delete its disks: `Remove-VM` "deletes the virtual + machine's configuration file, but does not delete any virtual hard drives" + ([Remove-VM][remove-vm]). Orphaned `.vhdx` / `.avhdx` files left behind still + occupy their slabs, and no amount of waiting will reclaim them. Check the volume + for disks with no owning VM, and delete the ones you have confirmed are unneeded: + + ```powershell + # Virtual disk files still present on the affected volume + Get-ChildItem "C:\ClusterStorage\" -Recurse -Include *.vhdx,*.avhdx,*.vhds | + Select-Object FullName, @{N='GB';E={[math]::Round($_.Length/1GB,1)}} + ``` + + For Arc VMs, delete them **through Azure** (see the disk-relocation caution + below, which applies to Arc-managed storage generally). 2. Wait at least 15 minutes; longer on a busy cluster. 3. Re-measure with the [Verify](#verify) queries. @@ -501,16 +515,29 @@ If the pool has dropped below threshold, **you are done, with no maintenance window**. Continue to the consolidation procedure only if the pool is still above threshold *and* the volume genuinely shows large interior free space. -#### Preconditions for automatic reclamation +#### Preconditions for reclamation (both paths) + +Both the automatic return above and the slab consolidation below depend on ReFS +returning freed slabs, so these two conditions gate the whole of Path B, not just +the no-downtime branch. -Automatic reclamation depends on TRIM/unmap being active. Check this before -concluding that reclamation "did not work": +Check TRIM/unmap on the CSV owner node. The command reports **one line per file +system**, and Azure Local CSVs are ReFS (see [Terminology](#terminology)), so the +**ReFS** line is the one that governs: ```powershell -# 0 = enabled (expected). 1 means deleted data is never returned to the pool. +# Run on the CSV owner node. Output is per file system, for example: +# NTFS DisableDeleteNotify = 0 +# ReFS DisableDeleteNotify is not currently set fsutil behavior query DisableDeleteNotify ``` +Read the **ReFS** line only. `0` means delete notification is enabled, which is +what a reclaiming cluster shows. `1` means it has been explicitly disabled, and +deleted capacity will not be returned until that is reverted. `is not currently +set` means no explicit override is present, so the platform default applies; treat +that as inconclusive rather than as a fault, and move on to the other checks. + > [!IMPORTANT] > **Stretched clusters never reclaim deleted capacity.** *"Because TRIM is disabled > for stretched clusters, storage isn't returned to the pool after data is @@ -643,8 +670,8 @@ fsutil behavior query DisableDeleteNotify > [!NOTE] > **Consolidation runs at low priority by default.** `Optimize-Volume` - > documents `-NormalPriority` as *"Indicates that this cmdlet the operation at - > normal priority. By default, the priority is low"* + > documents `-NormalPriority` as running the operation at normal priority, and + > states that *"By default, the priority is low"* > ([Optimize-Volume](https://learn.microsoft.com/powershell/module/storage/optimize-volume)), > matching `defrag /h` (*"Runs the operation at normal priority (default is > low)"*). The pass therefore yields to workload I/O rather than competing with @@ -712,7 +739,8 @@ fsutil behavior query DisableDeleteNotify > in Step 1 and that stale checkpoints were merged in the preparation step). Note > that some slabs report "pinned unmovable" even on a fully quiesced volume, so a > partial reclaim is not by itself a failure. If real interior free space exists, -> TRIM is enabled, the cluster is not stretched, all workloads were offline, and +> ReFS delete notification is not explicitly disabled, the cluster is not +> stretched, all workloads were offline, and > checkpoints were merged, but the pool still does not drop after the unmap wait > (Step 3), open a Microsoft support case rather than repeating the procedure. @@ -725,7 +753,7 @@ fsutil behavior query DisableDeleteNotify | Fixed | Remove unneeded volumes | A3: shrink/remove (ReFS = evacuate + recreate) | | Fixed | Stop the alert (risk accepted) | A4: disable the Health Service alert | | Fixed | Move the alert threshold | A5: raise `ThinProvisioningAlertThresholds` | -| Thin | Return capacity from **deleted whole files/VMs** | Path B pre-branch: delete, then wait (no downtime) | +| Thin | Return capacity from **deleted whole files/VMs** | Path B pre-branch: confirm the disk files are gone, wait, re-measure (no downtime) | | Thin | Return capacity stranded by **interior fragmentation** | Path B: SlabConsolidate + ReFS unmap (offline window) | ## Verify @@ -853,7 +881,8 @@ firm conditions is met. Do not simply re-run the procedure. *operational state* is `Incomplete` / read-only from a drive-quorum loss rather than capacity, that is a separate, higher-severity problem. Escalate immediately.) - **Path B completed with every precondition met** (confirmed real interior free - space, TRIM enabled and the cluster not stretched, every VM on the volume + space, ReFS delete notification not explicitly disabled and the cluster not + stretched, every VM on the volume stopped, checkpoints merged) and you waited out the ReFS unmap, but pool `AllocatedSize` still does not drop. - The reserve-capacity fault (`InsufficientReserveCapacity`) **persists after** @@ -883,5 +912,6 @@ Include the data-collection output above with any Microsoft support case. [thin-prov]: https://learn.microsoft.com/azure/azure-local/manage/manage-thin-provisioning-23h2 [unsupported-ops]: https://learn.microsoft.com/azure/azure-local/manage/virtual-machine-operations +[remove-vm]: https://learn.microsoft.com/powershell/module/hyper-v/remove-vm --- From 8c921513b112fbb8b1f82e4514e5e505d0e4c985 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Wed, 2 Sep 2026 16:22:54 -0400 Subject: [PATCH 4/6] Say that moving disks between volumes cannot help, and when a probe run is reasonable Two gaps surfaced by the question "why not just try slab consolidation first". The guide warned against moving VM disks to another CSV on Arc-management grounds but never said the more basic thing: it cannot work anyway. Azure Local uses one storage pool per cluster, so every CSV draws from the same pool and relocating a VHDX between them returns nothing to the pool. For Arc VMs that makes it risk for no benefit, and the same futility applies to the supported non-Arc Move-VMStorage path. Added the one-pool reason, and noted that live-migrating a VM to another node does not help either, since the CSV is cluster-shared so the file stays on the volume and stays in use. Also addressed the probe question directly rather than leaving the guide reading as though the window is mandatory before anything can be attempted. Running consolidation with the workload up to see what it recovers is reasonable: it is non-destructive, it relocates rather than deletes, and it runs at low priority. Two caveats now stated. On a multi-terabyte volume it is hours of relocation I/O felt by every other volume on the shared pool, so it is cheap in risk and not in cost. And on a nearly-full pool it deserves more care, because ReFS allocates on write and whether relocation transiently raises allocation there is not established either way, while pool exhaustion is the one failure in this article that takes VMs offline. The read-only checks above cost seconds and can make the question moot, so they come first. Also states that a probe recovering little is the expected result when the workload still holds its files, not evidence the procedure does not work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4 --- ...ot-Storage-StoragePoolCapacityThreshold.md | 58 +++++++++++++++---- 1 file changed, 48 insertions(+), 10 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index 4c12031c..f7ba969c 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -546,15 +546,21 @@ that as inconclusive rather than as a fault, and move on to the other checks. > [Path A](#path-a-fixed-provisioned-volumes) capacity problem instead. > [!CAUTION] -> **Do not move VM disks to another volume as a way to relieve pressure.** For -> **Arc-managed Azure Local VMs (23H2+)**, moving a VHD/VHDX to another CSV with -> host-side tools (`Move-VMStorage`, Failover Cluster Manager, or a manual file -> move) is *storage live migration*, which Microsoft lists among operations that -> *"can lead to Azure Local VMs becoming unmanageable from the Azure portal"* -> ([unsupported VM operations][unsupported-ops]). Azure tracks each disk's location -> through a **storage path** (`Microsoft.AzureStackHCI/storagecontainers`) -> resource; a host-side move leaves that resource pointing at the old volume, and -> the VM, disk, and network-interface resources can be left stale and undeletable. +> **Moving VM disks to another volume does not relieve pool pressure, and can +> break Arc management.** Every CSV on the cluster draws from the **same storage +> pool** (Azure Local uses [one pool per cluster][s2d-overview]), so relocating a +> VHDX from one CSV to another moves the data without returning a single byte to +> the pool. It is motion with no benefit for this problem. +> +> For **Arc-managed Azure Local VMs (23H2+)** it is also actively harmful. Moving +> a VHD/VHDX to another CSV with host-side tools (`Move-VMStorage`, Failover +> Cluster Manager, or a manual file move) is *storage live migration*, which +> Microsoft lists among operations that *"can lead to Azure Local VMs becoming +> unmanageable from the Azure portal"* ([unsupported VM operations][unsupported-ops]). +> Azure tracks each disk's location through a **storage path** +> (`Microsoft.AzureStackHCI/storagecontainers`) resource; a host-side move leaves +> that resource pointing at the old volume, and the VM, disk, and +> network-interface resources can be left stale and undeletable. > > There is **no supported in-place move** of an existing Arc VM disk between > volumes: a storage path is selected at **creation** time. To place a workload on @@ -563,7 +569,13 @@ that as inconclusive rather than as a fault, and move on to the other checks. > > This restriction applies to **Arc-managed** VMs. For traditional (non-Arc) > clustered Hyper-V VMs, `Move-VMStorage` with the cluster resource updated -> accordingly remains supported. +> accordingly remains supported, though the same one-pool point applies: it still +> will not free pool capacity. +> +> Live-migrating a VM to a **different node** does not help either. The CSV is +> cluster-shared, so the virtual disk file stays on the same volume and stays in +> use, just from another node. Stopping the workload is the only action that makes +> its slabs movable. > [!NOTE] > This procedure recovers capacity only when the volume genuinely holds far less @@ -572,6 +584,31 @@ that as inconclusive rather than as a fault, and move on to the other checks. > close to `Size × resiliency`). If footprint matches the data actually written, > there is nothing to reclaim. +> [!TIP] +> **Cheapest checks first. A consolidation pass is not a cheap probe.** The steps +> above cost seconds and can make the maintenance window unnecessary: enumerate +> the virtual disk files, confirm the provisioning type, rule out the stop +> conditions, and if whole files were deleted just wait and re-measure. +> +> Running consolidation with the workload still up, to see what it recovers before +> committing to a window, is a reasonable probe. It is non-destructive, it +> relocates data rather than deleting any, it runs at low priority, and a +> disappointing result costs time rather than data. Two things to weigh before +> doing it: +> +> - On a multi-terabyte volume it is **hours** of back-end relocation I/O, and +> because every volume shares the one pool, that load is felt by workloads on +> other volumes. It is cheap in risk, not in cost. +> - On a pool that is already **close to full**, be more careful. ReFS allocates +> on write, so relocating live data writes the new copy before releasing the +> old. Whether that transiently raises pool allocation on a nearly-full pool is +> not established here either way, and pool exhaustion is the one failure in +> this article that takes VMs offline. On a pool with comfortable headroom this +> is not a concern; near the limit, do the read-only checks above first. +> +> A probe that recovers little is not proof the procedure does not work. It is +> the expected result when the workload is still holding its files. + **Procedure for interior fragmentation (requires an offline window for VMs on the affected volume; the window lasts through slab consolidation, which can take hours on large volumes):** [MEDIUM RISK] > [!TIP] @@ -913,5 +950,6 @@ Include the data-collection output above with any Microsoft support case. [thin-prov]: https://learn.microsoft.com/azure/azure-local/manage/manage-thin-provisioning-23h2 [unsupported-ops]: https://learn.microsoft.com/azure/azure-local/manage/virtual-machine-operations [remove-vm]: https://learn.microsoft.com/powershell/module/hyper-v/remove-vm +[s2d-overview]: https://learn.microsoft.com/windows-server/storage/storage-spaces/storage-spaces-direct-overview --- From 04c2daf6ece6eb78d3af2e271ce7651386421c44 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Wed, 2 Sep 2026 16:33:16 -0400 Subject: [PATCH 5/6] Remove the stretched-cluster caveat, which does not apply to Azure Local I added this earlier in the same review cycle on the strength of a sentence in the Learn thin provisioning article: "Because TRIM is disabled for stretched clusters, storage isn't returned to the pool after data is deleted." The quote is real and it is on the 23H2-named page, which is how it got through. Stretched clusters are not a supported Azure Local 23H2 or later configuration. The stretched-clusters concept page still sits on the legacy azure-stack/hci path, and Microsoft's guidance is that 23H2 does not support them. So the sentence is stale text carried forward onto a renamed page, and repeating it here implied a deployment shape a reader cannot have, and sent them to check a condition that cannot be true on a supported system. Removed the caveat, the two escalation preconditions that asked the operator to certify the cluster is not stretched, and the mention in the closing troubleshooting note. The DisableDeleteNotify check stays, still scoped to the ReFS line and still gating both paths. Verifying that a quote exists is not the same as verifying that the configuration it describes exists on the product the guide targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4 --- ...ot-Storage-StoragePoolCapacityThreshold.md | 23 ++++++------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index f7ba969c..c9580551 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -518,8 +518,8 @@ threshold *and* the volume genuinely shows large interior free space. #### Preconditions for reclamation (both paths) Both the automatic return above and the slab consolidation below depend on ReFS -returning freed slabs, so these two conditions gate the whole of Path B, not just -the no-downtime branch. +returning freed slabs, so this check gates the whole of Path B, not just the +no-downtime branch. Check TRIM/unmap on the CSV owner node. The command reports **one line per file system**, and Azure Local CSVs are ReFS (see [Terminology](#terminology)), so the @@ -538,13 +538,6 @@ deleted capacity will not be returned until that is reverted. `is not currently set` means no explicit override is present, so the platform default applies; treat that as inconclusive rather than as a fault, and move on to the other checks. -> [!IMPORTANT] -> **Stretched clusters never reclaim deleted capacity.** *"Because TRIM is disabled -> for stretched clusters, storage isn't returned to the pool after data is -> deleted"* ([thin provisioning][thin-prov]). On a stretched cluster this reclaim -> procedure will not help; treat the shortage as a -> [Path A](#path-a-fixed-provisioned-volumes) capacity problem instead. - > [!CAUTION] > **Moving VM disks to another volume does not relieve pool pressure, and can > break Arc management.** Every CSV on the cluster draws from the **same storage @@ -770,14 +763,13 @@ that as inconclusive rather than as a fault, and move on to the other checks. > A consolidation pass can legitimately return little or no capacity. The most > common reasons, in order: the volume's footprint already matches the data > actually written, so there is nothing to reclaim (see the note at the start of -> this procedure); TRIM/unmap is disabled (`DisableDeleteNotify = 1`) or the -> cluster is **stretched**, where deleted capacity is never returned at all; or -> slabs are still pinned by data in use (confirm every VM on the volume is stopped +> this procedure); ReFS delete notification has been explicitly disabled, so +> freed slabs are never returned; or slabs are still pinned by data in use (confirm every VM on the volume is stopped > in Step 1 and that stale checkpoints were merged in the preparation step). Note > that some slabs report "pinned unmovable" even on a fully quiesced volume, so a > partial reclaim is not by itself a failure. If real interior free space exists, -> ReFS delete notification is not explicitly disabled, the cluster is not -> stretched, all workloads were offline, and +> ReFS delete notification is not explicitly disabled, all workloads were +> offline, and > checkpoints were merged, but the pool still does not drop after the unmap wait > (Step 3), open a Microsoft support case rather than repeating the procedure. @@ -918,8 +910,7 @@ firm conditions is met. Do not simply re-run the procedure. *operational state* is `Incomplete` / read-only from a drive-quorum loss rather than capacity, that is a separate, higher-severity problem. Escalate immediately.) - **Path B completed with every precondition met** (confirmed real interior free - space, ReFS delete notification not explicitly disabled and the cluster not - stretched, every VM on the volume + space, ReFS delete notification not explicitly disabled, every VM on the volume stopped, checkpoints merged) and you waited out the ReFS unmap, but pool `AllocatedSize` still does not drop. - The reserve-capacity fault (`InsufficientReserveCapacity`) **persists after** From 608ae4485d5808bd36de46732038ff53912237c9 Mon Sep 17 00:00:00 2001 From: Alex Burns Date: Fri, 4 Sep 2026 11:58:44 -0400 Subject: [PATCH 6/6] Address review: label the delete branch, check ownership properly, drop the TRIM gate Three items from review, and testing the guidance on a lab cluster disproved a fourth thing I had added myself. The orphaned-disk step is the most destructive action in Path B and carried no risk label while reversible consolidation was tagged medium. Worse, it told the operator to delete files it gave them no way to identify: Get-ChildItem lists files, it does not establish that nothing owns them. It now builds the in-use set across every node in the cluster rather than the one you are signed in to, since a VM on another node holds its disks open the same way, and it presents the result as candidates rather than as garbage. Added the three exclusions that listing cannot see: Arc-managed disks and images, which live as Azure resources and can be unattached and invisible to a host-side listing while still being live data; templates, golden images and backup targets, which legitimately have no VM; and checkpoint disks, which must be merged through Hyper-V because deleting an .avhdx directly breaks the differencing chain. Labeled HIGH RISK with a warning that an unidentifiable file should be left alone and escalated rather than deleted. Moved the reclamation precondition above the no-downtime branch. As placed it came after that branch's "you are done" exit, so an operator taking the whole-file path never reached the check it claimed to gate. Then removed that precondition entirely, because testing it showed it was wrong. On a healthy single-node Azure Local 12.2610 cluster, pool Healthy and OK at 4.2% used, fsutil reports "ReFS DisableDeleteNotify = 1". My text said 1 meant the pool would not get freed slabs back, which would have flagged every healthy cluster as faulted and sent operators to change a platform default. Learn documents 1 as the ReFS v2 default, and the setting governs device-level TRIM, notifying the underlying storage device, not the Storage Spaces slab return. Wrong layer. The same lab pass re-confirmed the guidance that stayed: Get-Volume -FilePath resolves the CSV to a CSVFS volume object, and Optimize-Volume -Path against the mount fails with exactly the documented "No MSFT_Volume objects found" error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e36f861-864e-4e08-a04c-b0d7f1b897c4 --- ...ot-Storage-StoragePoolCapacityThreshold.md | 84 +++++++++++-------- 1 file changed, 50 insertions(+), 34 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md index c9580551..2e252c8a 100644 --- a/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md +++ b/TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md @@ -491,23 +491,62 @@ an offline window**. Identify which case you are in before scheduling anything. case that needs slab consolidation, and therefore the only case that needs the offline window. -**If you deleted whole VMs or files, start here. This path needs no downtime:** +**If you deleted whole VMs or files, start here. This path needs no downtime +unless you find orphaned disks that have to be removed:** -1. **Confirm the virtual disk files themselves are gone, not just the VMs.** Deleting - a VM does not necessarily delete its disks: `Remove-VM` "deletes the virtual - machine's configuration file, but does not delete any virtual hard drives" - ([Remove-VM][remove-vm]). Orphaned `.vhdx` / `.avhdx` files left behind still - occupy their slabs, and no amount of waiting will reclaim them. Check the volume - for disks with no owning VM, and delete the ones you have confirmed are unneeded: +1. **Confirm the virtual disk files are actually gone, not just the VMs.** + `Remove-VM` "deletes the virtual machine's configuration file, but does not + delete any virtual hard drives" ([Remove-VM][remove-vm]), so disks left behind + still occupy their slabs and no amount of waiting will reclaim them. + + Build the in-use set from **every node in the cluster**, not just the one you + are signed in to. A VM running on another node holds its disks open exactly the + same way, and a single-node listing will not show that: ```powershell - # Virtual disk files still present on the affected volume + # Disks in use anywhere on the cluster, including checkpoint disks + $inUse = Get-ClusterNode | ForEach-Object { + Get-VM -ComputerName $_.Name | ForEach-Object { + $_ | Get-VMHardDiskDrive | Select-Object -ExpandProperty Path + $_ | Get-VMSnapshot | Get-VMHardDiskDrive | Select-Object -ExpandProperty Path + } + } | Sort-Object -Unique + + # Files on the volume that nothing on the cluster references Get-ChildItem "C:\ClusterStorage\" -Recurse -Include *.vhdx,*.avhdx,*.vhds | - Select-Object FullName, @{N='GB';E={[math]::Round($_.Length/1GB,1)}} + Where-Object { $_.FullName -notin $inUse } | + Select-Object FullName, @{N='GB';E={[math]::Round($_.Length/1GB,1)}}, LastWriteTime ``` - For Arc VMs, delete them **through Azure** (see the disk-relocation caution - below, which applies to Arc-managed storage generally). + That output is a list of **candidates, not a list of garbage.** Before removing + anything, rule out all three of the following: + + - **Arc-managed disks and images.** For Azure Local VMs enabled by Arc the disk + is an Azure resource, and one that exists but is not currently attached to a + VM is invisible to a host-side listing while still being live customer data. + Check Azure as well as the host: `az stack-hci-vm disk list`, + `az stack-hci-vm image list`, and `az stack-hci-vm storagepath list` for the + paths in use on this volume. + - **Templates, golden images, ISOs, and backup targets**, which legitimately + have no attached VM and are still needed. + - **Checkpoint disks.** Never delete an `.avhdx` directly. It is a differencing + disk, and removing it breaks the chain and can destroy the VM's data. Merge + checkpoints through Hyper-V instead (`Get-VM | Get-VMSnapshot`, then + `Remove-VMSnapshot`), which collapses the `.avhdx` into its parent and frees + the space properly. + + Delete only what you have positively accounted for on all three counts. + **[HIGH RISK]** + + > [!WARNING] + > **Deleting a virtual disk file is irreversible and destroys whatever it + > contains.** A file that looks unreferenced from one node may be attached on + > another node, attached in Azure, or the parent of a checkpoint chain. If you + > cannot positively account for a file, leave it in place and open a support + > case. Reclaiming capacity is never worth deleting a disk you could not + > identify. + + For **Arc VMs**, delete the VM through Azure rather than with host tools. 2. Wait at least 15 minutes; longer on a busy cluster. 3. Re-measure with the [Verify](#verify) queries. @@ -515,29 +554,6 @@ If the pool has dropped below threshold, **you are done, with no maintenance window**. Continue to the consolidation procedure only if the pool is still above threshold *and* the volume genuinely shows large interior free space. -#### Preconditions for reclamation (both paths) - -Both the automatic return above and the slab consolidation below depend on ReFS -returning freed slabs, so this check gates the whole of Path B, not just the -no-downtime branch. - -Check TRIM/unmap on the CSV owner node. The command reports **one line per file -system**, and Azure Local CSVs are ReFS (see [Terminology](#terminology)), so the -**ReFS** line is the one that governs: - -```powershell -# Run on the CSV owner node. Output is per file system, for example: -# NTFS DisableDeleteNotify = 0 -# ReFS DisableDeleteNotify is not currently set -fsutil behavior query DisableDeleteNotify -``` - -Read the **ReFS** line only. `0` means delete notification is enabled, which is -what a reclaiming cluster shows. `1` means it has been explicitly disabled, and -deleted capacity will not be returned until that is reverted. `is not currently -set` means no explicit override is present, so the platform default applies; treat -that as inconclusive rather than as a fault, and move on to the other checks. - > [!CAUTION] > **Moving VM disks to another volume does not relieve pool pressure, and can > break Arc management.** Every CSV on the cluster draws from the **same storage