Skip to content

Commit c1b4beb

Browse files
committed
Merge FEATURE-ci-v3-runtime-green: fix Windows timer exit and FreeBSD evidence transport
2 parents 8f30839 + 9300e15 commit c1b4beb

8 files changed

Lines changed: 235 additions & 1 deletion

File tree

‎.github/workflows/v3-package-validation.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ jobs:
155155
- name: freebsd_instance
156156
uses: vmactions/freebsd-vm@c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e
157157
with:
158+
copyback: 'false'
158159
release: '14.3'
159160
mem: 8192
160161
cpu: 2

‎.github/workflows/v3-release-acceptance.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,7 @@ jobs:
289289
cp -R "$RUNNER_TEMP/selected" build/ci-selected-input
290290
- uses: vmactions/freebsd-vm@c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e
291291
with:
292+
copyback: 'false'
292293
release: '14.3'
293294
mem: 8192
294295
cpu: 2

‎ChangeLog‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
Version 3.0.0 - 2026-10-10
2+
Avoid Windows DLL shutdown hangs by retaining the default timer service
3+
during process termination while preserving ordinary timer cleanup.
4+
Report the saved Winsock error code when host WSAPoll fails.
5+
Use explicit SCP evidence transport for FreeBSD CI, disabling redundant
6+
whole-workspace copyback without changing native test gates.
27
Connect native TLS-off/on builds, replay and complete installed matrices to CI.
38
Add bounded sanitizer fuzz smoke, persistent trusted corpora and scheduled
49
campaign code that activates after default-branch cutover.

‎examples/host_poll.hpp‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
#include <cstdint>
4242
#include <optional>
4343
#include <stdexcept>
44+
#include <string>
4445
#include <vector>
4546

4647
#include <httpserver/server/readiness.hpp>
@@ -99,7 +100,10 @@ inline std::vector<srv::readiness_event> wait(
99100
for (const auto& interest : interests) fds.push_back(registration(interest));
100101
#if defined(_WIN32)
101102
const auto ready = ::WSAPoll(fds.data(), static_cast<ULONG>(fds.size()), timeout(deadline));
102-
if (ready < 0 && ::WSAGetLastError() != WSAEINTR) throw std::runtime_error("WSAPoll failed");
103+
if (ready < 0) {
104+
const int error = ::WSAGetLastError();
105+
if (error != WSAEINTR) throw std::runtime_error("WSAPoll failed (WSA error " + std::to_string(error) + ")");
106+
}
103107
#else
104108
const auto ready = ::poll(fds.data(), static_cast<nfds_t>(fds.size()), timeout(deadline));
105109
if (ready < 0 && errno != EINTR) throw std::runtime_error("poll failed");
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# FEATURE-ci-v3-runtime-green
2+
3+
Status: Stages 1 and 3 implemented with focused proof; complete CI and remaining supported runtime repairs are pending. Groundwork validation is explicitly prohibited until FULL CI IS GREEN.
4+
Base: `v3` at `8f3083977f49706e66cefbb3ee8b6f6685469719`.
5+
Working branch: `task/FEATURE-ci-v3-runtime-green`. User requested integration into `v3` before full CI completion; retain incomplete verification explicitly.
6+
Plan: primary checkout `.groundwork-plans/FEATURE-ci-v3-runtime-green-plan.md`.
7+
Journal: primary checkout `.debug/v3-ci-green-VSvXxx/journal.md`.
8+
Stage evidence: `.debug/v3-ci-green-VSvXxx/runtime-green-stage1/`.
9+
10+
## Stage 1: default Windows timer shutdown
11+
12+
- [x] Compare cached shutdown detection with event substitution and unconditional singleton leakage.
13+
- [x] Preserve actual native62 RED and reliable LLDB unwind; do not relaunch original captured suites.
14+
- [x] Add and observe a failing bounded Windows DLL exit regression, with ordinary explicit timer retirement coverage.
15+
- [x] Cache the documented Windows shutdown query before constructing the default timer thread; abandon that service only during actual process termination. Preserve ordinary cleanup, POSIX behavior and public headers/API/ABI.
16+
- [x] Register `task_timer_shutdown` in Automake; verify DLL exit, explicit retirement, ordinary static-executable teardown, existing Linux resume/lifetime tests, changed-file lint and diff.
17+
- [ ] Full exact-source CI on the working branch, including actual MSYS2 package/archive lanes.
18+
- [ ] Groundwork validation, only after full CI is green; caller owns publication and eventual merge/cleanup.
19+
20+
The actual original CI/local native62 program completes all 734 checks but hangs during DLL process detach. The accepted 24-frame LLDB unwind reaches `timer_queue` condition-variable destruction from `ExitProcess`/`LdrShutdownProcess`. Evidence is retained at `.debug/ci-windows-dump-entry-dbl8orzx/offline-analysis/mac-approved-result.json` and `mac-lldb-session.stdout.txt`; the earlier DbgHelp `.bss` chain is rejected. The original 180-second gate remains FAILED/NOT_RECHECKED.
21+
22+
Microsoft documents that process exit kills the other threads before DLL detach, so even lock acquisition, joining and synchronization-object destruction can deadlock. The private Windows owner resolves `RtlDllShutdownInProgress` before creating its queue/thread, and releases its ownership only when that cached function reports process termination. There is no module lookup, allocation, locking, joining or synchronization destruction on that path. The operating system reclaims this single process-lifetime service. Ordinary destruction still deletes the queue; the POSIX factory is byte-identical. If the documented Windows 10+ query is unavailable, the ordinary cleanup fallback remains; legacy-platform shutdown is not claimed fixed.
23+
24+
Requirements preserved: PRD-V3N-REQ-024/025 and architecture v3 section 3.1 (timeout/cancellation, exactly-once delivery and executor affinity). Public representation, scheduling, cancellation, deadlines, explicit timer ownership and callback delivery are unchanged. No forced application exit, DLL pinning or public shutdown interface is introduced.
25+
26+
## Focused proof and context limits
27+
28+
A C-only consumer of the retained actual CI DLL reaches its default timer factory and prints `DEFAULT_TIMER_READY`, then fails to exit within 9 seconds; its retained handle is retired, with total 9.449 seconds. No C++ standard-library object crosses that ABI boundary. This is a new minimal regression, not another original native62 launch.
29+
30+
The new registered C++ regression runs a matching small three-owner DLL (`resume_signal.cpp`, `task.cpp`, `executor.cpp`) with explicit retirement and default-exit children. The final test against original timer source is RED: exit 1, child wait timeout 258 and destruction-during-shutdown marker, total 9.372 seconds. Against fixed source it is GREEN: exit 0, total 0.491 seconds, no shutdown destruction marker. Each child has a 9-second wait and at most 1 second for owned retirement. An inherited event observes pending-capture destruction during actual process termination, so the assertion also protects runtimes whose condition variable does not itself hang. Static-executable normal CRT destruction is permitted by this test and passes separately. Ordinary explicit retirement joins the worker and releases a capture with an hour-long pending deadline.
31+
32+
Linux focused tests pass: `task_timer_shutdown`; `task_resume_signal` 11 tests/134 checks; `task_resume_signal_lifetime` 3 tests/1816 checks. These use current source compiled with GNU13/C++20/pthreads and only the three owning concurrency translation units. Changed-file cpplint and `git diff --check` pass. `autoreconf -fi` registers the new program and inherited library linkage; existing AC_OUTPUT and .PHONY warnings are recorded without unrelated repair. No full baseline or broad suite was run.
33+
34+
Native Windows context is Windows 11 build26200, cross LLVM-MinGW23.1.4/MinGW15 with its UCRT startup/headers, MSYS2 GCC15.2 standard-library headers/import libraries in an owned cache, and the retained actual CI MSVCRT libstdc++/libgcc/winpthread DLLs. Actual CI uses Windows Server2022 and LLVM23.1.0 with MSYS2 mingw64 throughout. This focused mixed-CRT proof is qualified; it does not replace full actual CI. No provider, SDK or compiler was built or globally installed. A first downloaded UCRT GCC package was unused after original imports identified MSVCRT.
35+
36+
Preserved setup failures: sandboxed WSL native launch was unavailable before normal escalation; initial future-based fixture had unresolved libstdc++ TLS symbols, then a corrected TLS build exposed missing runtime once-call exports (native entry-point status C0000139). That setup failure is not RED. Replacing only test readiness synchronization with an already-supported Win32 event corrected the fixture boundary; the corrected native failure above is the behavioral RED. No production workaround was based on the setup error. Raw commands, logs, receipts, hashes and bounded ownership cleanup are retained in stage evidence.
37+
38+
Darwin PSK failures, Windows host WSAPoll consumer failure and incidental Linux/macOS/BSD assessments retain their previous UNKNOWN/closed-budget states. No unrelated source repair, original diagnostic retry, push, CI dispatch, review agent, Groundwork validation, merge or cleanup occurred in stage 1.
39+
40+
## Related WSAPoll error visibility
41+
42+
- [x] Immediately save `WSAGetLastError()` when the existing `WSAPoll` call fails and include its numeric value in the existing exception message.
43+
- [x] Preserve interrupted-poll handling and all polling, timing and shutdown semantics.
44+
45+
This caller-authorized message change follows proven timer stage commit `70d0fba7`. It supplies useful error evidence in the next already-required CI run; it does not diagnose or repair the prior consumer exit 6, whose cause remains UNKNOWN and captures remain closed. There are no per-poll traces, socket queries, clocks, locks or error suppression changes.
46+
47+
A bounded Windows test substitutes only the external WSAPoll boundary to return WSAEINVAL through the actual Winsock last-error API. Before the change it returns 2 because the exception omits numeric 10022 (0.310s); after it exits 0 (0.303s), also checking that WSAEINTR still produces no exception. This controlled message regression is not a reproduction of the original consumer failure. Actual Windows compilation, GNU syntax checking of `examples/external_event_loop.cpp`, changed-header cpplint and diff checks pass. Commands, sources and receipts are under stage evidence `host-poll-message/`.
48+
49+
Additional ordinary cleanup proof: a static-linked Windows default service releases its pending callback during normal executable CRT teardown before a later registered atexit observer checks the capture; exit 0 in 0.391s (`ordinary-default-native-test.json`). The default service is therefore not unconditionally immortal.
50+
51+
## Stage 3: FreeBSD evidence transport setup
52+
53+
- [x] Preserve actual current8f native package PASS, redundant automatic-copyback FAIL, and explicit fresh SCP recovery/upload PASS as separate receipts.
54+
- [x] Set `copyback: 'false'` on the pinned FreeBSD VM action in package validation and release acceptance.
55+
- [x] Pass the existing 13 focused workflow tests, actionlint and exact source-preservation proof.
56+
- [ ] Full working-branch CI with the corrected VM setup (not dispatched here).
57+
58+
Current8f package job `121358499066` passed installed native TLS=no at 03:49UTC, TLS=yes and identical installed headers at 04:03UTC. Its later automatic whole-workspace rsync copyback failed with protocol error 12 after receiving 4,049,536,821 bytes. Explicit fresh SCP recovery then succeeded and uploaded artifact `12626459963`, 776,035,888 bytes, producer-reported digest `597ded253263153a8ae4bb617df42defaaceb55c930978e013e4063253532894`. This is retained raw-log/metadata evidence; the artifact was not downloaded for stage3. It does not prove the separate cookie archive translation-unit gate, which remains pending.
59+
60+
The exact pinned action implementation SHA256 is `7f9e5e36ceae15937ec0aabbf7e4ce4ab2855b36aa3dfa0ce8e79e7c0fb9569e`; its post-run step 7 runs only when the lowercased copyback input is not `'false'`. Initial synchronization and guest execution are distinct earlier operations. The input is declared in the [pinned action metadata](https://raw.githubusercontent.com/vmactions/freebsd-vm/c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e/action.yml). Disabling only this redundant post-run copyback leaves the explicit required SCP transport responsible for retrieving evidence. The underlying rsync connection-loss mechanism is not claimed diagnosed.
61+
62+
Whole-file byte comparison proves that removing the single added scalar from each workflow restores the original file exactly. Guest prepare/run/trap/native commands, initial sync settings, pinned actions/toolchains, source checks, every gate, SCP error propagation, artifact paths and existing fallback handling are therefore unchanged. An existing fallback is preserved if present; automatic copyback is no longer a promised source for it. Failed SCP and missing evidence remain failures.
63+
64+
Focused checks: `python3 scripts/test-v3-ci-workflows.py FreeBSDNodePrerequisites -v` (9 tests), `python3 scripts/test-v3-package-workflow.py -v` (4 tests), cached actionlint on both workflows, and `git diff --check` all pass. No new mirror test, VM, timeout increase, retry, transport framework, baseline suite, CI dispatch, push, merge or Groundwork validation was performed. Stage3 commands, outputs, source hashes and preservation proof are retained at `.debug/v3-ci-green-VSvXxx/runtime-green-stage3/` in the primary checkout. PSK source remains untouched while its fresh native authorization is pending.
65+
66+
## User-directed integration before CI completion
67+
68+
The user requested “merge back to v3, then keep fixing the remaining CI issues.”
69+
Integrate the focused repairs and ChangeLog update now. Full CI remains incomplete;
70+
Groundwork validation has not run and stays deferred until full CI is green.
71+
This integration does not mark the overall CI repair complete or reopen closed
72+
diagnostic batches. Draft PR #382 carries the original working-branch runs.

‎src/concurrency/resume_signal.cpp‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,13 @@
44
#include <utility>
55
#include <vector>
66

7+
#ifdef _WIN32
8+
#ifndef NOMINMAX
9+
#define NOMINMAX
10+
#endif
11+
#include <windows.h>
12+
#endif
13+
714
namespace httpserver::detail {
815

916
auto resume_node::claim() noexcept -> bool {
@@ -121,8 +128,27 @@ auto timer_queue::run() -> void {
121128
}
122129

123130
timer_queue& default_timer_queue() {
131+
#ifdef _WIN32
132+
struct service {
133+
using shutdown_query = BOOLEAN (NTAPI*)();
134+
// Resolve before starting the timer thread. Late loader lookups can
135+
// themselves acquire locks orphaned by Windows process termination.
136+
shutdown_query terminating = reinterpret_cast<shutdown_query>(GetProcAddress(GetModuleHandleW(L"ntdll.dll"), "RtlDllShutdownInProgress"));
137+
std::unique_ptr<timer_queue> queue = std::make_unique<timer_queue>();
138+
139+
~service() {
140+
// ExitProcess has already killed other threads before DLL detach.
141+
// Leave this process-lifetime queue to the OS; even destroying its
142+
// mutex/condition_variable can deadlock on a killed waiter.
143+
if (terminating && terminating()) static_cast<void>(queue.release());
144+
}
145+
};
146+
static service owner;
147+
return *owner.queue;
148+
#else
124149
static timer_queue queue;
125150
return queue;
151+
#endif
126152
}
127153

128154
} // namespace httpserver::detail

‎test/Makefile.am‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -584,6 +584,11 @@ EXTRA_DIST += integ/native_websocket_e2e.sh
584584

585585
task_resume_signal_lifetime_SOURCES = unit/task_resume_signal_lifetime_test.cpp
586586
task_resume_signal_lifetime_LDADD = $(top_builddir)/src/libhttpserver.la
587+
588+
# Windows children bound DLL process-detach retirement; POSIX checks ordinary
589+
# explicit timer ownership without changing its process lifetime semantics.
590+
check_PROGRAMS += task_timer_shutdown
591+
task_timer_shutdown_SOURCES = unit/task_timer_shutdown_test.cpp
587592
websocket_drain_SOURCES = unit/websocket_drain_test.cpp
588593
websocket_drain_LDADD = $(top_builddir)/src/libhttpserver_v3core.la
589594
websocket_drain_race_SOURCES = unit/websocket_drain_race_test.cpp
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
/* Copyright (C) 2026 Sebastiano Merlino; SPDX-License-Identifier: LGPL-2.1-or-later */
2+
#ifdef _WIN32
3+
#ifndef NOMINMAX
4+
#define NOMINMAX
5+
#endif
6+
#include <windows.h>
7+
#endif
8+
9+
#include <chrono>
10+
#include <cstdint>
11+
#include <cstdio>
12+
#include <cstdlib>
13+
#include <cstring>
14+
#ifndef _WIN32
15+
#include <future>
16+
#endif
17+
#include <memory>
18+
#include <string>
19+
#include <utility>
20+
#include <httpserver/concurrency/resume_signal.hpp>
21+
22+
namespace {
23+
using httpserver::detail::timer_queue;
24+
25+
bool timer_runs(timer_queue& queue) {
26+
#ifdef _WIN32
27+
std::shared_ptr<void> ready(CreateEventW(nullptr, TRUE, FALSE, nullptr), [](void* event) { CloseHandle(event); });
28+
if (!ready) return false;
29+
queue.schedule(timer_queue::clock::now(), [ready] { SetEvent(ready.get()); });
30+
return WaitForSingleObject(ready.get(), 1000) == WAIT_OBJECT_0;
31+
#else
32+
auto ready = std::make_shared<std::promise<void>>();
33+
auto done = ready->get_future();
34+
queue.schedule(timer_queue::clock::now(), [ready] { ready->set_value(); });
35+
return done.wait_for(std::chrono::seconds(1)) == std::future_status::ready;
36+
#endif
37+
}
38+
39+
int retire_explicit_timer() {
40+
std::weak_ptr<int> pending_capture;
41+
{
42+
timer_queue queue;
43+
if (!timer_runs(queue)) return 1;
44+
auto capture = std::make_shared<int>(42);
45+
pending_capture = capture;
46+
queue.schedule(timer_queue::clock::now() + std::chrono::hours(1), [capture] {});
47+
capture.reset();
48+
if (pending_capture.expired()) return 2;
49+
}
50+
// Retirement joins the worker and releases pending callbacks promptly,
51+
// even though their deadlines are an hour away.
52+
return pending_capture.expired() ? 0 : 3;
53+
}
54+
55+
#ifdef _WIN32
56+
int run_child(const wchar_t* mode) {
57+
wchar_t image[32768];
58+
const DWORD length = GetModuleFileNameW(nullptr, image, 32768);
59+
if (length == 0 || length == 32768) return 4;
60+
SECURITY_ATTRIBUTES attributes{sizeof(attributes), nullptr, TRUE};
61+
HANDLE destroyed = CreateEventW(&attributes, TRUE, FALSE, nullptr);
62+
if (destroyed == nullptr) return 5;
63+
std::wstring command = L"\"" + std::wstring(image, length) + L"\" " + mode + L" " + std::to_wstring(reinterpret_cast<std::uintptr_t>(destroyed));
64+
STARTUPINFOW startup{};
65+
startup.cb = sizeof(startup);
66+
PROCESS_INFORMATION child{};
67+
if (!CreateProcessW(image, command.data(), nullptr, nullptr, TRUE, 0, nullptr, nullptr, &startup, &child)) {
68+
CloseHandle(destroyed);
69+
return 5;
70+
}
71+
CloseHandle(child.hThread);
72+
const DWORD wait = WaitForSingleObject(child.hProcess, 9000);
73+
DWORD code = 6;
74+
if (wait == WAIT_OBJECT_0) {
75+
if (!GetExitCodeProcess(child.hProcess, &code)) code = 7;
76+
} else {
77+
// Only this retained child handle is retired; no name-based cleanup.
78+
TerminateProcess(child.hProcess, 8);
79+
WaitForSingleObject(child.hProcess, 1000);
80+
}
81+
CloseHandle(child.hProcess);
82+
// Captures may retire normally in a static executable's CRT teardown,
83+
// but must survive DLL process detach after other threads are killed.
84+
if (WaitForSingleObject(destroyed, 0) != WAIT_TIMEOUT) code = 10;
85+
CloseHandle(destroyed);
86+
if (code != 0) std::fprintf(stderr, "timer child failed: wait=%lu exit=%lu\n", wait, code);
87+
return code == 0 ? 0 : 1;
88+
}
89+
#endif
90+
} // namespace
91+
92+
int main(int argc, char** argv) {
93+
#ifdef _WIN32
94+
if (argc == 3 && std::strcmp(argv[1], "--default-exit") == 0) {
95+
using shutdown_query = BOOLEAN (NTAPI*)();
96+
const auto terminating = reinterpret_cast<shutdown_query>(GetProcAddress(GetModuleHandleW(L"ntdll.dll"), "RtlDllShutdownInProgress"));
97+
if (!terminating) return 11; // documented Windows 10+ contract
98+
auto& queue = httpserver::detail::default_timer_queue();
99+
if (!timer_runs(queue)) return 9;
100+
HANDLE destroyed = reinterpret_cast<HANDLE>(static_cast<std::uintptr_t>(std::strtoull(argv[2], nullptr, 10)));
101+
auto marker = std::shared_ptr<void>(destroyed, [terminating](void* event) {
102+
if (terminating()) SetEvent(event);
103+
});
104+
queue.schedule(timer_queue::clock::now() + std::chrono::hours(1), [marker = std::move(marker)] {});
105+
// Leave the DLL timer worker waiting when ExitProcess retires other
106+
// threads, before invoking DLL static destructors under loader lock.
107+
Sleep(100);
108+
std::puts("default timer active; returning from main");
109+
std::fflush(stdout);
110+
return 0;
111+
}
112+
if (argc == 3 && std::strcmp(argv[1], "--retire") == 0) return retire_explicit_timer();
113+
if (run_child(L"--retire") != 0) return 1;
114+
return run_child(L"--default-exit");
115+
#else
116+
static_cast<void>(argc);
117+
static_cast<void>(argv);
118+
return retire_explicit_timer();
119+
#endif
120+
}

0 commit comments

Comments
 (0)