fix(windows): write back edits after a blocked Cloud Files commit

changes/unreleased/windows-cloud-files-blocked-commit.md

@@ -0,0 +1,7 @@
+category: fix
+issue: none
+pull: 517
+platforms: windows
+user-facing: yes
+
+Windows Cloud Files now keeps writing back edits to a file that another application had open during an earlier upload, and marks that file as synced once the application closes it.

changes/unreleased/windows-cloud-files-dehydration.md

@@ -0,0 +1,7 @@
+category: fix
+issue: none
+pull: 518
+platforms: windows
+user-facing: yes
+
+Windows Cloud Files Free up space, automatic cleanup, and the storage summary now count and free downloaded files instead of finding nothing. Edited, pinned, and open files stay on the computer.

docs/windows-release.md

@@ -205,6 +205,18 @@ A file edited during the upload stays dirty and is uploaded again against the
 new revision, at most three times in one writeback before it reports a failed
 writeback. If the exclusive handle is unavailable, for example because another
 application has the file open, the commit is retried without uploading again.
+If recording the new revision fails after the server accepted the upload, only
+that update is retried, so the retry never uploads against the superseded
+revision. Both retries are held in memory; after a restart, a placeholder whose
+revision was not recorded is uploaded against its old revision and rejected by
+the server.
+
+Until a blocked commit completes, the placeholder identity holds the uploaded
+size, while the close-completion callback reports the size from the last
+in-sync point. The close callback therefore does not require the two sizes to
+match; hydration still does. On a local Windows 11 sync root, the close
+notification that a blocking application sends when it releases the file
+completes the pending commit.
 An application that restores an earlier last-write time while keeping the same
 size can defeat the stamp; this is an accepted limit of the guard.
 
@@ -224,13 +236,44 @@ root. They cover:
 
 - an edit made while disconnected, written back at the next start;
 - a size-changing edit made while connected;
-- an edit made during an upload.
+- an edit made during an upload;
+- a commit blocked by another process's open handle, which completes when that
+  handle closes, followed by a later edit.
 
 Each case must end with the placeholder in sync at the server's revision.
 `WindowsCloudFilesProviderTest` covers the remaining recovery and writeback
 behavior. These tests do not replace the Explorer and published-MSI acceptance
 checks above.
 
+### Dehydration and storage accounting
+
+**Free up space**, automatic cleanup, and the Virtual files storage summary read
+allocated bytes and dehydrate placeholders through
+[`WindowsCloudDehydration`](../ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudDehydration.kt).
+A connected file placeholder is a cloud reparse point, which Java NIO does not
+report as a regular file, so both operations classify the path from native
+find data with the same rule as the startup enumeration. Allocated bytes come
+from `GetCompressedFileSizeW`: a dehydrated placeholder keeps its logical size
+and allocates no bytes.
+
+Only an unpinned, in-sync file placeholder that allocates bytes is dehydrated,
+so an online-only placeholder costs no native call. As
+[`CfDehydratePlaceholder`](https://learn.microsoft.com/en-us/previous-versions/mt827480(v=vs.85))
+requires, dehydration holds an exclusive `CfOpenFileWithOplock` handle. When
+the oplock cannot be taken (`0x80070020`) or breaks first, or when Windows
+refuses a file that became dirty (`0x80070179`), pinned (`0x80070188`), or open
+in another application (`0x80070187`), the content stays local and counts as
+not freed. Automatic cleanup also keeps a planned file whose known revision,
+pin, sync state, or allocated size changed after planning. Freed totals count
+the bytes each dehydration actually released.
+
+**Last reviewed: 2026-10-17.** On a local Windows 11 sync root,
+`WindowsCloudFilesDehydrationTest` hydrates a synthetic file, observes nonzero
+allocated bytes, frees them through Free up space and through automatic cleanup,
+and reopens the content. It also keeps a dirty edit, a pinned file, and a file
+held open by another process local. This evidence does not cover Explorer's own
+Free up space command or a published MSI.
+
 ## WinGet delivery
 
 `tools/create-winget-manifests.ps1` generates a three-file WinGet manifest from

tools/kotlin-file-size-baseline.txt

@@ -51,8 +51,8 @@ ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/DesktopFileSyncRemoteT
 ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/DesktopNextcloudServices.kt|6161
 ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/DesktopVirtualRangeCache.kt|2759
 ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/LinuxVirtualFileSystem.kt|1697
-ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudFilesJna.kt|1030
-ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudFilesProvider.kt|2338
+ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudFilesJna.kt|1023
+ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudFilesProvider.kt|2319
 ui/src/desktopTest/kotlin/dev/obiente/nextcloudnative/app/DesktopVirtualRangeCacheTest.kt|2479
 ui/src/desktopTest/kotlin/dev/obiente/nextcloudnative/app/JvmSupportIntakeTest.kt|2888
 ui/src/desktopTest/kotlin/dev/obiente/nextcloudnative/app/LinuxVirtualFileSystemTest.kt|2149

ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudDehydration.kt

@@ -0,0 +1,85 @@
+package dev.obiente.nextcloudnative.app
+
+import com.sun.jna.Native
+import com.sun.jna.WString
+import com.sun.jna.platform.win32.WinNT
+import com.sun.jna.ptr.IntByReference
+import java.nio.file.Path
+
+private const val FILE_ATTRIBUTE_PINNED = 0x0008_0000
+
+/**
+ * Only an unpinned, in-sync file placeholder may release its local content. Dirty content is the
+ * only copy of a local edit and pinned content is kept on request, so neither is a candidate.
+ *
+ * The decision uses native find data: a connected file placeholder is a cloud reparse point that
+ * Java NIO reports as "other", never as a regular file.
+ */
+internal fun windowsCloudDehydratable(entry: WindowsCloudLocalEntry, fileAttributes: Int): Boolean =
+    entry.file &&
+        fileAttributes and FILE_ATTRIBUTE_PINNED == 0 &&
+        entry.inspection.state == WindowsCloudPlaceholderEntryState.InSync
+
+/**
+ * Windows keeps content that became dirty, pinned or opened by another application after
+ * [windowsCloudDehydratable] accepted it. The content stays local and nothing is freed.
+ */
+internal fun isWindowsCloudDehydrationRefusedResult(result: Int): Boolean =
+    result == 0x80070020.toInt() || // HRESULT_FROM_WIN32(ERROR_SHARING_VIOLATION): no exclusive oplock
+        result == 0x80070179.toInt() || // HRESULT_FROM_WIN32(ERROR_CLOUD_FILE_NOT_IN_SYNC)
+        result == 0x80070187.toInt() || // HRESULT_FROM_WIN32(ERROR_CLOUD_FILE_IN_USE)
+        result == 0x80070188.toInt() || // HRESULT_FROM_WIN32(ERROR_CLOUD_FILE_PINNED)
+        result == 0x80070323.toInt() // HRESULT_FROM_WIN32(ERROR_OPLOCK_HANDLE_CLOSED)
+
+/**
+ * CfDehydratePlaceholder requires an exclusive handle, and Microsoft recommends an exclusive
+ * oplock so another application's open breaks it instead of failing. Returns false when Windows
+ * keeps the content, including when the oplock cannot be taken or breaks before dehydration.
+ */
+internal fun CldApi.dehydrateExclusively(path: Path): Boolean {
+    val protectedHandle = WinNT.HANDLEByReference()
+    val flags = windowsCloudOpenFileFlags(write = false, exclusive = true)
+    val opened = CfOpenFileWithOplock(WString(path.toWindowsWin32Path()), flags, protectedHandle)
+    if (isWindowsCloudDehydrationRefusedResult(opened)) return false
+    checkWindowsCloudHResult(opened, "open a Windows Cloud Files placeholder for dehydration")
+    try {
+        // A reference fails once another application's open has broken the oplock.
+        if (CfReferenceProtectedHandle(protectedHandle.value).toInt() == 0) return false
+        try {
+            val handle = CfGetWin32HandleFromProtectedHandle(protectedHandle.value)
+            val result = CfDehydratePlaceholder(handle, 0L, -1L, 0, null)
+            if (isWindowsCloudDehydrationRefusedResult(result)) return false
+            checkWindowsCloudHResult(result, "dehydrate a Windows Cloud Files placeholder")
+            return true
+        } finally {
+            CfReleaseProtectedHandle(protectedHandle.value)
+        }
+    } finally {
+        CfCloseHandle(protectedHandle.value)
+    }
+}
+
+/**
+ * Dehydrates the files an eviction plan selected, rechecking each at execution time. A file that
+ * another revision replaced, or that was pinned, edited or resized since planning, is kept.
+ * Returns the bytes actually released.
+ */
+internal fun WindowsCloudFilesApi.dehydratePlannedEvictions(
+    evictions: List<VirtualFileEviction>,
+    currentFile: (relativePath: String) -> Pair<WindowsCloudFileIdentity, Path>?,
+): Long = evictions.sumOf { eviction ->
+    val (identity, path) = currentFile(eviction.key.relativePath) ?: return@sumOf 0L
+    val unchanged = identity.remoteRevision == eviction.expectedLocalRevision &&
+        !isPinned(path) &&
+        placeholderState(path) == WindowsCloudPlaceholderState.InSync &&
+        allocatedBytes(path) == eviction.sizeBytes
+    if (unchanged) dehydrate(path) else 0L
+}
+
+/** Bytes allocated on disk; a dehydrated placeholder keeps its logical size but allocates none. */
+internal fun KernelFileApi.allocatedFileBytes(path: Path): Long {
+    val high = IntByReference()
+    val low = GetCompressedFileSizeW(WString(path.toWindowsWin32Path()), high)
+    if (low == -1 && Native.getLastError() != 0) return 0L
+    return ((high.value.toLong() and 0xffff_ffffL) shl 32) or (low.toLong() and 0xffff_ffffL)
+}

ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudFilesJna.kt

@@ -406,13 +406,8 @@ internal class JnaWindowsCloudFilesApi(
 
     override fun localChildren(directory: Path): List<Path> = localEntries(directory).map { it.path }
 
-    override fun allocatedBytes(path: Path): Long {
-        if (!Files.isRegularFile(path)) return 0L
-        val high = IntByReference()
-        val low = kernelFiles.GetCompressedFileSizeW(WString(path.toWindowsWin32Path()), high)
-        if (low == -1 && Native.getLastError() != 0) return 0L
-        return ((high.value.toLong() and 0xffff_ffffL) shl 32) or (low.toLong() and 0xffff_ffffL)
-    }
+    override fun allocatedBytes(path: Path): Long =
+        if (withFindData(path, false) { localEntry(path, it).file }) kernelFiles.allocatedFileBytes(path) else 0L
 
     override fun lastAccessedAtEpochMillis(path: Path): Long =
         withFindData(path, 0L) { findData -> findData.ftLastAccessTime.toTime().coerceAtLeast(0L) }
@@ -528,17 +523,15 @@ internal class JnaWindowsCloudFilesApi(
     }
 
     override fun dehydrate(path: Path): Long {
-        if (!Files.isRegularFile(path)) return 0L
-        val size = Files.size(path)
-        withWindowsFileHandle(path, write = true) { handle ->
-            checkWindowsCloudHResult(
-                cldApi.CfDehydratePlaceholder(handle, 0L, -1L, 0, null),
-                "dehydrate a Windows Cloud Files placeholder",
-            )
-        }
-        return size
+        if (!withFindData(path, false) { windowsCloudDehydratable(localEntry(path, it), it.dwFileAttributes) }) return 0L
+        // An online-only placeholder needs no native call, so a pass does not scale with the remote tree.
+        val allocated = kernelFiles.allocatedFileBytes(path).takeIf { it > 0L } ?: return 0L
+        return if (cldApi.dehydrateExclusively(path)) allocated else 0L
     }
 
+    private fun localEntry(path: Path, findData: WinBase.WIN32_FIND_DATA): WindowsCloudLocalEntry =
+        findData.windowsCloudLocalEntry(path, cldApi::CfGetPlaceholderStateFromFindData)
+
     override fun close() {
         callbacksByConnection.keys.toList().forEach { key -> runCatching { disconnect(key) } }
     }

ui/src/desktopMain/kotlin/dev/obiente/nextcloudnative/app/WindowsCloudFilesProvider.kt

@@ -227,6 +227,7 @@ internal interface WindowsCloudFilesApi : AutoCloseable {
     fun localContentStamp(path: Path): WindowsCloudLocalContentStamp = nioWindowsCloudLocalContentStamp(path)
     fun markInSyncIfUnchanged(path: Path, expected: WindowsCloudLocalContentStamp): WindowsCloudInSyncCommit =
         markInSyncWhenStampMatches(path, expected)
+    /** Returns the bytes released, or 0 when the entry is not an unpinned in-sync file or Windows kept it. */
     fun dehydrate(path: Path): Long
 }
 
@@ -906,7 +907,8 @@ internal class WindowsCloudFilesProvider(
 
     override fun closed(info: WindowsCloudCallbackInfo, deleted: Boolean) {
         if (callbacksPaused.get() || deleted || info.fileIdentity == null) return
-        val identity = runCatching { requireIdentity(info, expectDirectory = null) }.getOrNull() ?: return
+        // After a blocked commit Windows still reports the last in-sync size; only hydration needs it.
+        val identity = runCatching { requireIdentity(info, null, recordedSize = false) }.getOrNull() ?: return
         if (identity.directory) return
         val localPath = root.resolve(identity.path.replace('/', File.separatorChar)).normalize()
         if (!localPath.startsWith(root) || !Files.exists(localPath)) return
@@ -915,12 +917,7 @@ internal class WindowsCloudFilesProvider(
             if (api.placeholderState(localPath) != WindowsCloudPlaceholderState.Dirty) return
             pendingWritebacks += identity.path
             submitPathOperation(identity.path) {
-                val current = api.placeholderIdentity(localPath)
-                    ?.let { encoded -> runCatching { WindowsCloudFileIdentityCodec.decode(encoded) }.getOrNull() }
-                    ?.takeIf { it.accountId == backend.accountId && it.path == identity.path && !it.directory }
-                    ?: knownIdentities[identity.path]
-                    ?: identity
-                writeback.writeBack(localPath, current)
+                writeback.writeBack(localPath, identity.path, fallback = knownIdentities[identity.path] ?: identity)
             }
         }
     }
@@ -1041,9 +1038,7 @@ internal class WindowsCloudFilesProvider(
                 if (requestedBytes > 0L && freed >= requestedBytes) return@forEach
                 val path = localPath(identity)
                 if (path.startsWith(root) && api.placeholderState(path) == WindowsCloudPlaceholderState.InSync) {
-                    val allocated = api.allocatedBytes(path)
-                    api.dehydrate(path)
-                    freed += allocated
+                    freed += api.dehydrate(path)
                 }
             }
         return freed
@@ -1082,20 +1077,9 @@ internal class WindowsCloudFilesProvider(
             availableFreeBytes = Files.getFileStore(root).usableSpace,
             nowEpochMillis = nowEpochMillis,
         )
-        var freed = 0L
-        plan.evictions.forEach { eviction ->
-            val identity = knownIdentities[eviction.key.relativePath] ?: return@forEach
-            val path = localPath(identity)
-            if (
-                !api.isPinned(path) &&
-                api.placeholderState(path) == WindowsCloudPlaceholderState.InSync &&
-                api.allocatedBytes(path) == eviction.sizeBytes
-            ) {
-                api.dehydrate(path)
-                freed += eviction.sizeBytes
-            }
+        return api.dehydratePlannedEvictions(plan.evictions) { relative ->
+            knownIdentities[relative]?.let { identity -> identity to localPath(identity) }
         }
-        return freed
     }
 
     fun summary(): WindowsCloudFilesSummary = synchronized(corruptRootStableAccessLock) {
@@ -1847,13 +1831,14 @@ internal class WindowsCloudFilesProvider(
     private fun requireIdentity(
         info: WindowsCloudCallbackInfo,
         expectDirectory: Boolean?,
+        recordedSize: Boolean = true,
     ): WindowsCloudFileIdentity {
         val bytes = requireNotNull(info.fileIdentity) { "The Cloud Files callback has no identity." }
         val identity = WindowsCloudFileIdentityCodec.decode(bytes)
         require(identity.accountId == backend.accountId) { "The Cloud Files callback belongs to another account." }
         requireWindowsCloudCallbackPath(root, info.normalizedPath, identity.path)
         if (expectDirectory != null) require(identity.directory == expectDirectory)
-        require(identity.size == info.fileSize || identity.directory) { "The Cloud Files callback size is stale." }
+        require(!recordedSize || identity.size == info.fileSize || identity.directory) { "The Cloud Files callback size is stale." }
         knownIdentities[identity.path] = identity
         return identity
     }
@@ -2106,11 +2091,7 @@ internal class WindowsCloudFilesProvider(
                         deferredWhenPaused = { if (!runtimeStopping.get()) deferredLocalChanges.add(local) },
                         allowWhilePaused = allowWhilePaused,
                     ) {
-                        val current = requireNotNull(api.placeholderIdentity(local)) {
-                            "The dirty Windows placeholder has no recoverable identity."
-                        }.let(WindowsCloudFileIdentityCodec::decode)
-                        require(current.path == original.path) { "The dirty Windows placeholder identity is not safe to recover." }
-                        writeback.writeBack(local, current)
+                        writeback.writeBack(local, original.path)
                     }
                 }
             }