Repository navigation
feat(vhost-user): add client socket mode so the daemon can reconnect - #2
Conversation
| # wait for here: the daemon retries until it appears. Permissions are | ||
| # fixed from the `started` hook, after QEMU created the socket. | ||
| log(f"client mode: {unit} started, waiting for {domain_socket_from_config(domain)}") | ||
| return 0 |
There was a problem hiding this comment.
QEMU cannot bind client-mode socket
High Severity
In client mode the hook never grants QEMU write access to /run/vhost-user-wg before the VM starts. That directory is created 0755 and systemd then owns it as vhost-user-wg, so a libvirt-qemu frontend cannot bind() the socket the daemon is waiting to connect to. The started chown never runs, and the guest NIC never comes up.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 80180e9. Configure here.
| // session. Keeping them separate keeps the two shutdown paths apart. | ||
| let eventfd = EventFd::new(EFD_NONBLOCK).ok()?; | ||
| let consumer_clone = eventfd.try_clone().ok()?; | ||
| let notifier_clone = eventfd.try_clone().ok()?; |
There was a problem hiding this comment.
Exit eventfd leaked every session
Medium Severity
Each new VhostUserDaemon calls exit_event, which allocates a fresh eventfd. The framework takes the consumer via into_raw_fd for epoll_ctl and never closes it, so one fd is leaked per session. Reconnects (QEMU probe plus every later session) now accumulate fds until LimitNOFILE is hit.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 80180e9. Configure here.
QEMU's vhost-user chardev has no reconnect: once the daemon's socket
direction makes QEMU the client (`<source ... mode='client'/>`), a daemon
restart severs the virtio-net device for the rest of the VM's life. That is
what took a bank VM off the network when an unattended-upgrades run restarted
the backend service (QEMU logged `vhost VQ 0/1 ring restore failed: -22` and
never retried).
Add `[vhost_user] mode = "server" | "client"` (default `server`, so existing
configs are untouched; also `--vhost-user-mode`). In client mode the frontend
binds the socket (`<source ... mode='server'/>`), and the daemon connects with
a 250 ms retry that aborts on SIGTERM, then reconnects whenever a session
ends. The guest keeps its NIC across daemon restarts and frontend restarts.
Making reconnection actually work required fixing four latent bugs:
* Session-end errors were classified by substring ("Disconnected"), but
`Error::Disconnected` renders as "peer disconnected", so every clean
disconnect was treated as fatal. Match the error variants instead; server
mode's reconnect loop was broken by the same mistake.
* A reused `VhostUserDaemon` keeps the frontend's claim, so the next session's
SET_OWNER fails with InvalidOperation("already claimed") and the reconnect
is rejected forever. Build one daemon per session; the `WgNetBackend` Arc
(WireGuard state, DHCP leases, RX queue) is what survives.
* `exit_event()` handed the framework a dup of the backend's shutdown
eventfd, so `VhostUserHandler::drop`'s `send_exit_event()` tripped our
EXTRA_TOKEN_EXIT, killed the vring worker and left the datapath dead. Give
the framework its own eventfd.
* `run()` blocked in `signal_thread.join()` when the serve loop ended on its
own, so a fatal error looked like a hang until the next SIGTERM. Close the
signal-hook self-pipe before joining.
Also: register the backend's external fds once per daemon (epoll_ctl ADD on an
already-registered fd returns EEXIST, which killed the second session), exit
instead of opening another session when a session ends because of SIGTERM, and
teach the libvirt hook the client direction: prepare no longer waits for a
socket the frontend has not created yet, the configured socket path is read
from the TOML instead of assumed, and `started` chowns/chmods the frontend's
socket for the daemon — best effort, it must never abort a VM start.
Client mode prerequisites (documented in docs/operations.md and the unit file):
the socket must live where the frontend can create it, and the daemon must be
in the socket's group (`SupplementaryGroups=kvm`), because libvirt starts QEMU
with `server=on` and no `wait=off` — QEMU blocks until a backend connects, so
there is no window to fix permissions after the fact.
Tests: new tests/integration_client_mode.rs (connect + ARP round-trip, retry
until a late listener appears, reconnect after a frontend restart), client-mode
support in the mock frontend harness, config/CLI tests for `mode`, and a
rewritten exit-event test. Verified end-to-end against real libvirt + QEMU
8.2.2 with an Ubuntu 24.04 cloud-image guest: guest DHCPs 10.66.66.2 over the
vhost-user link, reaches the internet through the WireGuard tunnel, and keeps
working after both a daemon restart and a VM restart.
80180e9 to
f478362
Compare
| if !connect_to_frontend(&mut daemon, socket_path, exit_fd)? { | ||
| return Ok(()); | ||
| } | ||
| } |
There was a problem hiding this comment.
UDP events livelock before session
High Severity
External fds are now registered on the vring worker before start/connect_to_frontend, while handle_event still returns without draining EXTRA_TOKEN_UDP when guest memory is unset. The io_uring eventfd stays level-triggered and readable, so any WireGuard datagram during the connect-retry (or server accept) wait busy-loops that worker at 100% CPU until SET_MEM_TABLE. Client-mode reconnects and the original libc6/needrestart case are exactly that window.
Reviewed by Cursor Bugbot for commit f478362. Configure here.
| // session. Keeping them separate keeps the two shutdown paths apart. | ||
| let eventfd = EventFd::new(EFD_NONBLOCK).ok()?; | ||
| let consumer_clone = eventfd.try_clone().ok()?; | ||
| let notifier_clone = eventfd.try_clone().ok()?; |
There was a problem hiding this comment.
Exit eventfd leaked every session
Medium Severity
exit_event now allocates a fresh eventfd for each daemon, and a daemon is created per session. The framework takes the consumer via into_raw_fd into epoll_ctl and never closes that fd. Each QEMU probe, VM reboot, or client reconnect therefore leaks a file descriptor (and the eventfd object) for the life of the process.
Reviewed by Cursor Bugbot for commit f478362. Configure here.
…sion Addresses the Cursor Bugbot findings on this PR. **Exit eventfd leaked one fd per session (medium).** `VringEpollHandler::new` hands the consumer fd to `epoll_ctl` via `into_raw_fd()` and never closes it, so the fd stayed open for the life of the process. That was harmless with a single daemon per process, but there is now one daemon — hence one such fd — per vhost-user session (QEMU probe, reconnect, ...). Vendored patch: close the consumer fd right after registration. The registration keeps working because epoll keys on the file description and `notifier` is a dup of the same eventfd. Measured with 12 connect/disconnect sessions against one daemon: fd count 13 → 13, idle CPU 0.00s/5s. **`EXTRA_TOKEN_UDP` could livelock the vring worker (high).** I had moved the external-fd registration before `start`/`connect`, while `handle_event` still returned early for the WG socket token when guest memory was not mapped yet — the io_uring eventfd stayed readable and the level-triggered epoll spun the worker. Two fixes: register the fds only once a session exists (each session has its own daemon, so no `EEXIST`), and drain the WG socket even without guest memory (frames are only queued in the backend, no vring needed) so a packet arriving between connect and SET_MEM_TABLE cannot spin either. **Client mode could not bind the default socket directory (high).** In client mode QEMU creates the socket, but the default `/run/vhost-user-wg` belongs to the daemon and is 0755, so a `libvirt-qemu` QEMU could not bind there. The hook now chgrps the socket's directory to the QEMU group and adds group write, so the documented stock path works. E2E re-run on the same libvirt + QEMU 8.2.2 + Ubuntu 24.04 guest, now with the **default** socket directory: hook logs `chgrp libvirt-qemu + group-write on /run/vhost-user-wg`, QEMU binds the socket, guest gets 10.66.66.2 by DHCP and pings through the WireGuard tunnel 3/3 (~3.3 ms); `systemctl restart` of the daemon under the live VM leaves the guest answering 3/3; a VM restart brings it back answering 3/3.
|
Addressed the three Bugbot findings in
E2E re-run, this time with the default socket directory (real libvirt + QEMU 8.2.2, Ubuntu 24.04 cloud-image guest):
Test suite: |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 5 total unresolved issues (including 4 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 87b0efc. Configure here.
| parent.chmod(parent.stat().st_mode | stat.S_IWGRP | stat.S_IXGRP) | ||
| log(f"chgrp {group} + group-write on {parent}") | ||
| except (LookupError, OSError) as exc: | ||
| log(f"chgrp {group} on {parent} failed ({exc}); QEMU may not be able to bind") |
There was a problem hiding this comment.
QEMU dir grant races systemd RuntimeDirectory
Medium Severity
make_dir_writable_for_qemu chgrps and group-writes the socket parent so QEMU can bind, including the stock path /run/vhost-user-wg. That directory is the unit's shared RuntimeDirectory, so any vhost-user-wg@* start chmod/chowns it back to vhost-user-wg 0755 and drops group-write. If that happens after this hook returns and before QEMU binds — parallel VM starts or another instance coming up in that window — QEMU gets EACCES and the guest NIC never appears. The hook still returns success, so libvirt does not see the permission failure.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 87b0efc. Configure here.
There was a problem hiding this comment.
Fixed in c8c6111 — and you are right that the silent-success part was the worse half.
Client-mode sockets must not live in the unit's shared RuntimeDirectory, so the hook no longer pretends it can make that work:
make_dir_writable_for_qemunow returns a bool instead of only logging, and treatsrealpath(parent) == /run/vhost-user-wgas unserviceable.handle_preparestops the daemon and returns non-zero in that case, so libvirt fails the start with a visible error instead of launching a NIC-less guest. The journal says why and what to use instead:
vhost-user-wg-hook: client mode: socket /run/vhost-user-wg/vhost-wg.sock is inside the daemon's shared RuntimeDirectory /run/vhost-user-wg; any vhost-user-wg@* start resets that directory's group/mode, so QEMU may fail to bind. Point [vhost_user] socket at a dedicated directory instead, e.g. /var/lib/libvirt/vuwg/<domain>.sock
vhost-user-wg-hook: client mode: refusing to start vhost-wg with socket /run/vhost-user-wg/vhost-wg.sock
→ virsh start: error: Hook script execution failed: ... unexpected exit status 1
- docs/operations.md (client-mode prerequisites + the domain XML sample) and the hook header now require a dedicated directory, e.g.
/var/lib/libvirt/vuwg/<domain>.sock, which the hook creates with the QEMU group.
Verified end-to-end on QEMU 8.2.2 + Ubuntu 24.04:
- negative case above: VM refused, nothing half-started;
- dedicated dir: hook logs
chgrp libvirt-qemu + group-write on /var/lib/libvirt/vuwg, QEMU binds, guest DHCPs10.66.66.2and pings through the WireGuard tunnel 3/3;systemctl restart vhost-user-wg@vhost-wgunder the live VM → guest 3/3; VM restart → guest 3/3.
…ctory Bugbot follow-up on the previous commit: `make_dir_writable_for_qemu` granted the QEMU group access to the socket's directory even when that directory is the unit's shared `RuntimeDirectory` (`/run/vhost-user-wg`). Every `vhost-user-wg@*` start recreates that directory as `vhost-user-wg` 0755, so the grant can be revoked between the hook returning and QEMU's `bind()` — a parallel VM start is enough — and the guest then comes up without a NIC while the hook reports success and libvirt sees nothing. Client-mode sockets must live in a directory systemd does not manage: * the hook now detects the shared-runtime-dir case, explains it in the journal (`/etc/libvirt/hooks/qemu.d/vhost-user-wg` output via `vhost-user-wg-hook`), stops the daemon and returns non-zero, so libvirt fails the VM start with a visible `Hook script execution failed` instead of starting a NIC-less guest; * `make_dir_writable_for_qemu` reports failure instead of only logging, which is what enables the above; * docs and the operations sample use a dedicated directory (`/var/lib/libvirt/vuwg/<domain>.sock`) for client mode and explain why. E2E on real libvirt + QEMU 8.2.2 + Ubuntu 24.04 guest: * **negative**: client mode with `socket = "/run/vhost-user-wg/vhost-wg.sock"` → hook logs the reason, `virsh start` fails with `Hook script execution failed ... unexpected exit status 1`, no VM starts; * **positive** (dedicated dir): hook logs `chgrp libvirt-qemu + group-write on /var/lib/libvirt/vuwg`, QEMU binds the socket, guest DHCPs 10.66.66.2 and pings through the tunnel 3/3; daemon restart under the live VM → 3/3; VM restart → 3/3.


Why
A restarted backend takes the guest's NIC down permanently when the daemon
binds the socket. Production hit it on 2026-11-13: an
unattended-upgradesrun upgraded
libc6, Ubuntu'sneedrestartAPT hook ransystemctl restart … vhost-user-wg@<vm>, and the guest stayed "network cableunplugged" until the VM was power-cycled. QEMU's vhost-user chardev has no
reconnectoption, so once the daemon it connects to goes away the virtio-netdevice has no backend for the rest of the VM's life (QEMU log:
vhost VQ 0/1 ring restore failed: -22).Reconnecting is only possible if the frontend owns the socket, so add that
direction to the daemon.
What
[vhost_user] mode = "server" | "client"(defaultserver; also--vhost-user-mode):<source mode='…'/>server(default)clientclientserverClient mode connects with a 250 ms retry (aborted by SIGTERM via a poll on the
backend's exit eventfd) and reconnects whenever a session ends.
Latent bugs that had to be fixed for reconnection to work
s.contains("Disconnected")never matched —
Error::Disconnectedrenders as"peer disconnected"— soevery clean disconnect was fatal. Now matched on the error variants; this
also repairs server mode's own reconnect loop.
VhostUserDaemonrejects the next session. The framework'sVhostUserHandlerkeepsowned, so a secondSET_OWNERreturnsInvalidOperation("already claimed")and the reconnect fails forever. Onedaemon per session now; the
WgNetBackendArc(WireGuard state, DHCPleases, RX queue) is the state that survives.
exit_event()handed outdups of the backend's exit fd, so
VhostUserHandler::drop'ssend_exit_event()trippedEXTRA_TOKEN_EXIT, killed the vring worker andleft the datapath dead. The framework gets its own eventfd.
run()hung on teardown. When the serve loop ended on its own,signal_thread.join()blocked until the next SIGTERM, so a fatal errorlooked like a hang. The signal-hook self-pipe is closed before joining.
Plus: external fds are registered once per daemon (
epoll_ctl(ADD)twice →EEXIST, which killed the second session), and a session that ends because ofSIGTERM ends the process instead of opening another one.
Hook
The libvirt hook learns the client direction:
prepare/beginstarts the daemon and returns — there is no socket to waitfor yet (the frontend creates it), and the daemon retries on its own.
[vhost_user] socketin the TOML instead ofassumed to be
/run/vhost-user-wg/<domain>.sock(client deployments usuallyneed a directory the frontend may write to).
startedchowns/chmods the frontend's socket for the daemon, best effort —it can no longer abort a VM start (found by the end-to-end run: a
chownTypeError there made libvirt tear a VM back down).
End-to-end verification (real libvirt + QEMU 8.2.2, not the mock harness)
Ubuntu 24.04 cloud-image guest on
qemu:///system, TOMLmode = "client",<source mode='server'/>:(
vhost_user_client_connect_retry) until QEMU binds the socket, thenvhost_user_client_connected; QEMU logs the successful renegotiationwarning; VM runs.
enp8s0gets10.66.66.2/30withdefault via 10.66.66.1 … proto dhcp(served by the daemon over thevhost-user ring), the daemon persists the lease, and the guest reaches
106.12.199.25through the WireGuard tunnel (3/3 replies, ~3.3 ms).systemctl restart vhost-user-wg@<vm>with theVM running: QEMU logs the disconnect (
ring restore failed: -22), thenimmediately re-initialises on the new backend connection
(
vhost-user backend supports VHOST_USER_PROTOCOL_F_CONFIG), the new daemonnegotiates and re-handshakes the tunnel within milliseconds, and the guest
answers 3/3 pings with the same address. This is exactly what used to kill
the NIC.
virsh destroy+start): hook stops/starts the daemon,reads the correct socket path, chowns it, guest comes back and is reachable
again (3/3 replies).
Client mode prerequisites (documented in
docs/operations.mdand the unit)server=onand nowait=off, so QEMU blocks until a backend connects andthere is no window to chmod the socket. QEMU creates it as
libvirt-qemu:kvm 0775, so run the daemon withSupplementaryGroups=kvm(comment added to the packaged unit).
Tests
tests/integration_client_mode.rs: connect + ARP round-trip, retry untila listener that appears 1 s late, and reconnect after a frontend restart
(listener gone, then re-bound) with traffic working again afterwards.
FrontendMode(we bind/accept) andrestart_frontend_listener().modeparsing (default,client, unknown value rejected),the CLI override, and a rewritten exit-event test asserting the framework's
exit event is independent of the backend's shutdown fd.
cargo test --lib: 173 passed. Fullcargo test --no-fail-fastgreen exceptintegration_dhcp::test_lease_persistence_across_restart, which failsidentically on
master(pre-existing, reproduced in a clean worktree).cargo clippy --all-targets -- -D warnings -D clippy::unwrap_used -D clippy::expect_used -D clippy::panic -D clippy::as_conversionsclean,cargo fmt --checkclean.Deployment notes
Behaviour of existing configs is unchanged (default
server). Switching a VMto client mode needs all of: new binary, updated hook,
mode = "client"in theembedded TOML,
<source mode='server'/>in the domain XML, and the kvm groupfor the unit. Server mode still cannot survive a daemon restart — that is a
property of QEMU's chardev, not of this daemon.
Also: SIGTERM while a session is live still cannot unwind
daemon.wait()(the framework's request thread blocks inrecv); the unit'sTimeoutStopSec=5scovers it, as before.Note
Medium Risk
Changes core vhost-user session lifecycle, VM networking behavior, and libvirt hook permissions; default
servermode preserves existing configs but misconfigured client mode can block VM start or leave guests without a NIC.Overview
Adds
[vhost_user] mode(serverdefault, orclient) and--vhost-user-mode, so the daemon can either bind the Unix socket (today’s behavior) or connect to a frontend-owned socket with 250ms retries—letting the guest keep virtio-net across daemon restarts when QEMU uses<source mode='server'/>.The serve loop is reworked for multi-session survival: one
VhostUserDaemonper session (sharedWgNetBackendstate), variant-based session teardown (not string matching), separate framework vs backend shutdown eventfds, register external fds once per daemon, drain WG traffic before guest memory is set, and clean shutdown via polling the exit fd / closing the signal-hook pipe.Libvirt hook reads
socket+modefrom embedded TOML: client mode prepares a dedicated socket directory for QEMU, skips waiting atprepare, chowns the socket atstarted, and refuses client sockets under the sharedRuntimeDirectory. Docs, examples, and systemd noteSupplementaryGroups=kvmfor client mode.New
integration_client_modetests and harnessFrontendMode; small vendor patch closes leaked exit-eventfds per session.Reviewed by Cursor Bugbot for commit c8c6111. Bugbot is set up for automated code reviews on this repo. Configure here.