Skip to content

feat(vhost-user): add client socket mode so the daemon can reconnect - #2

Merged
feisuzhu merged 3 commits into
masterfrom
feisuzhu/vuwg-client-mode
Nov 17, 2026
Merged

feisuzhu merged 3 commits into
masterfrom
feisuzhu/vuwg-client-mode

Conversation

@feisuzhu

@feisuzhu feisuzhu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

A restarted backend takes the guest's NIC down permanently when the daemon
binds the socket. Production hit it on 2026-11-13: an unattended-upgrades
run upgraded libc6, Ubuntu's needrestart APT hook ran
systemctl restart … vhost-user-wg@<vm>, and the guest stayed "network cable
unplugged" until the VM was power-cycled. QEMU's vhost-user chardev has no
reconnect option, so once the daemon it connects to goes away the virtio-net
device has no backend for the rest of the VM's life (QEMU log:
vhost VQ 0/1 ring restore failed: -22).

Reconnecting is only possible if the frontend owns the socket, so add that
direction to the daemon.

What

[vhost_user] mode = "server" | "client" (default server; also
--vhost-user-mode):

config <source mode='…'/> who binds daemon restart
server (default) client daemon guest loses its NIC
client server frontend daemon reconnects, guest unaffected

Client mode connects with a 250 ms retry (aborted by SIGTERM via a poll on the
backend's exit eventfd) and reconnects whenever a session ends.

Latent bugs that had to be fixed for reconnection to work

  1. Session-end errors were misclassified. s.contains("Disconnected")
    never matched — Error::Disconnected renders as "peer disconnected" — so
    every clean disconnect was fatal. Now matched on the error variants; this
    also repairs server mode's own reconnect loop.
  2. A reused VhostUserDaemon rejects the next session. The framework's
    VhostUserHandler keeps owned, so a second SET_OWNER returns
    InvalidOperation("already claimed") and the reconnect fails forever. One
    daemon per session now; the WgNetBackend Arc (WireGuard state, DHCP
    leases, RX queue) is the state that survives.
  3. The framework shared our shutdown eventfd. exit_event() handed out
    dups of the backend's exit fd, so VhostUserHandler::drop's
    send_exit_event() tripped EXTRA_TOKEN_EXIT, killed the vring worker and
    left the datapath dead. The framework gets its own eventfd.
  4. run() hung on teardown. When the serve loop ended on its own,
    signal_thread.join() blocked until the next SIGTERM, so a fatal error
    looked like a hang. The signal-hook self-pipe is closed before joining.

Plus: external fds are registered once per daemon (epoll_ctl(ADD) twice →
EEXIST, which killed the second session), and a session that ends because of
SIGTERM ends the process instead of opening another one.

Hook

The libvirt hook learns the client direction:

  • prepare/begin starts the daemon and returns — there is no socket to wait
    for yet (the frontend creates it), and the daemon retries on its own.
  • the socket path is now read from [vhost_user] socket in the TOML instead of
    assumed to be /run/vhost-user-wg/<domain>.sock (client deployments usually
    need a directory the frontend may write to).
  • started chowns/chmods the frontend's socket for the daemon, best effort —
    it can no longer abort a VM start (found by the end-to-end run: a chown
    TypeError there made libvirt tear a VM back down).

End-to-end verification (real libvirt + QEMU 8.2.2, not the mock harness)

Ubuntu 24.04 cloud-image guest on qemu:///system, TOML mode = "client",
<source mode='server'/>:

  • VM start: hook starts the daemon first; daemon retries
    (vhost_user_client_connect_retry) until QEMU binds the socket, then
    vhost_user_client_connected; QEMU logs the successful renegotiation
    warning; VM runs.
  • Guest datapath: guest enp8s0 gets 10.66.66.2/30 with
    default via 10.66.66.1 … proto dhcp (served by the daemon over the
    vhost-user ring), the daemon persists the lease, and the guest reaches
    106.12.199.25 through the WireGuard tunnel (3/3 replies, ~3.3 ms).
  • The incident scenario — systemctl restart vhost-user-wg@<vm> with the
    VM running: QEMU logs the disconnect (ring restore failed: -22), then
    immediately re-initialises on the new backend connection
    (vhost-user backend supports VHOST_USER_PROTOCOL_F_CONFIG), the new daemon
    negotiates and re-handshakes the tunnel within milliseconds, and the guest
    answers 3/3 pings with the same address. This is exactly what used to kill
    the NIC.
  • VM restart (virsh destroy + start): hook stops/starts the daemon,
    reads the correct socket path, chowns it, guest comes back and is reachable
    again (3/3 replies).

Client mode prerequisites (documented in docs/operations.md and the unit)

  • The socket must live in a directory the frontend can create it in.
  • The daemon must be able to connect to it: libvirt starts QEMU with
    server=on and no wait=off, so QEMU blocks until a backend connects and
    there is no window to chmod the socket. QEMU creates it as
    libvirt-qemu:kvm 0775, so run the daemon with SupplementaryGroups=kvm
    (comment added to the packaged unit).

Tests

  • New tests/integration_client_mode.rs: connect + ARP round-trip, retry until
    a listener that appears 1 s late, and reconnect after a frontend restart
    (listener gone, then re-bound) with traffic working again afterwards.
  • Mock frontend harness gained FrontendMode (we bind/accept) and
    restart_frontend_listener().
  • Unit tests for mode parsing (default, client, unknown value rejected),
    the CLI override, and a rewritten exit-event test asserting the framework's
    exit event is independent of the backend's shutdown fd.
  • cargo test --lib: 173 passed. Full cargo test --no-fail-fast green except
    integration_dhcp::test_lease_persistence_across_restart, which fails
    identically on master (pre-existing, reproduced in a clean worktree).
  • cargo clippy --all-targets -- -D warnings -D clippy::unwrap_used -D clippy::expect_used -D clippy::panic -D clippy::as_conversions clean,
    cargo fmt --check clean.

Deployment notes

Behaviour of existing configs is unchanged (default server). Switching a VM
to client mode needs all of: new binary, updated hook, mode = "client" in the
embedded TOML, <source mode='server'/> in the domain XML, and the kvm group
for the unit. Server mode still cannot survive a daemon restart — that is a
property of QEMU's chardev, not of this daemon.

Also: SIGTERM while a session is live still cannot unwind
daemon.wait() (the framework's request thread blocks in recv); the unit's
TimeoutStopSec=5s covers it, as before.


Note

Medium Risk
Changes core vhost-user session lifecycle, VM networking behavior, and libvirt hook permissions; default server mode preserves existing configs but misconfigured client mode can block VM start or leave guests without a NIC.

Overview
Adds [vhost_user] mode (server default, or client) and --vhost-user-mode, so the daemon can either bind the Unix socket (today’s behavior) or connect to a frontend-owned socket with 250ms retries—letting the guest keep virtio-net across daemon restarts when QEMU uses <source mode='server'/>.

The serve loop is reworked for multi-session survival: one VhostUserDaemon per session (shared WgNetBackend state), variant-based session teardown (not string matching), separate framework vs backend shutdown eventfds, register external fds once per daemon, drain WG traffic before guest memory is set, and clean shutdown via polling the exit fd / closing the signal-hook pipe.

Libvirt hook reads socket + mode from embedded TOML: client mode prepares a dedicated socket directory for QEMU, skips waiting at prepare, chowns the socket at started, and refuses client sockets under the shared RuntimeDirectory. Docs, examples, and systemd note SupplementaryGroups=kvm for client mode.

New integration_client_mode tests and harness FrontendMode; small vendor patch closes leaked exit-eventfds per session.

Reviewed by Cursor Bugbot for commit c8c6111. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

# wait for here: the daemon retries until it appears. Permissions are
# fixed from the `started` hook, after QEMU created the socket.
log(f"client mode: {unit} started, waiting for {domain_socket_from_config(domain)}")
return 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QEMU cannot bind client-mode socket

High Severity

In client mode the hook never grants QEMU write access to /run/vhost-user-wg before the VM starts. That directory is created 0755 and systemd then owns it as vhost-user-wg, so a libvirt-qemu frontend cannot bind() the socket the daemon is waiting to connect to. The started chown never runs, and the guest NIC never comes up.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 80180e9. Configure here.

Comment thread src/datapath/mod.rs
// session. Keeping them separate keeps the two shutdown paths apart.
let eventfd = EventFd::new(EFD_NONBLOCK).ok()?;
let consumer_clone = eventfd.try_clone().ok()?;
let notifier_clone = eventfd.try_clone().ok()?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exit eventfd leaked every session

Medium Severity

Each new VhostUserDaemon calls exit_event, which allocates a fresh eventfd. The framework takes the consumer via into_raw_fd for epoll_ctl and never closes it, so one fd is leaked per session. Reconnects (QEMU probe plus every later session) now accumulate fds until LimitNOFILE is hit.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 80180e9. Configure here.

QEMU's vhost-user chardev has no reconnect: once the daemon's socket
direction makes QEMU the client (`<source ... mode='client'/>`), a daemon
restart severs the virtio-net device for the rest of the VM's life. That is
what took a bank VM off the network when an unattended-upgrades run restarted
the backend service (QEMU logged `vhost VQ 0/1 ring restore failed: -22` and
never retried).

Add `[vhost_user] mode = "server" | "client"` (default `server`, so existing
configs are untouched; also `--vhost-user-mode`). In client mode the frontend
binds the socket (`<source ... mode='server'/>`), and the daemon connects with
a 250 ms retry that aborts on SIGTERM, then reconnects whenever a session
ends. The guest keeps its NIC across daemon restarts and frontend restarts.

Making reconnection actually work required fixing four latent bugs:

* Session-end errors were classified by substring ("Disconnected"), but
  `Error::Disconnected` renders as "peer disconnected", so every clean
  disconnect was treated as fatal. Match the error variants instead; server
  mode's reconnect loop was broken by the same mistake.
* A reused `VhostUserDaemon` keeps the frontend's claim, so the next session's
  SET_OWNER fails with InvalidOperation("already claimed") and the reconnect
  is rejected forever. Build one daemon per session; the `WgNetBackend` Arc
  (WireGuard state, DHCP leases, RX queue) is what survives.
* `exit_event()` handed the framework a dup of the backend's shutdown
  eventfd, so `VhostUserHandler::drop`'s `send_exit_event()` tripped our
  EXTRA_TOKEN_EXIT, killed the vring worker and left the datapath dead. Give
  the framework its own eventfd.
* `run()` blocked in `signal_thread.join()` when the serve loop ended on its
  own, so a fatal error looked like a hang until the next SIGTERM. Close the
  signal-hook self-pipe before joining.

Also: register the backend's external fds once per daemon (epoll_ctl ADD on an
already-registered fd returns EEXIST, which killed the second session), exit
instead of opening another session when a session ends because of SIGTERM, and
teach the libvirt hook the client direction: prepare no longer waits for a
socket the frontend has not created yet, the configured socket path is read
from the TOML instead of assumed, and `started` chowns/chmods the frontend's
socket for the daemon — best effort, it must never abort a VM start.

Client mode prerequisites (documented in docs/operations.md and the unit file):
the socket must live where the frontend can create it, and the daemon must be
in the socket's group (`SupplementaryGroups=kvm`), because libvirt starts QEMU
with `server=on` and no `wait=off` — QEMU blocks until a backend connects, so
there is no window to fix permissions after the fact.

Tests: new tests/integration_client_mode.rs (connect + ARP round-trip, retry
until a late listener appears, reconnect after a frontend restart), client-mode
support in the mock frontend harness, config/CLI tests for `mode`, and a
rewritten exit-event test. Verified end-to-end against real libvirt + QEMU
8.2.2 with an Ubuntu 24.04 cloud-image guest: guest DHCPs 10.66.66.2 over the
vhost-user link, reaches the internet through the WireGuard tunnel, and keeps
working after both a daemon restart and a VM restart.
@feisuzhu
feisuzhu force-pushed the feisuzhu/vuwg-client-mode branch from 80180e9 to f478362 Compare November 13, 2026 09:56

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/datapath/mod.rs
if !connect_to_frontend(&mut daemon, socket_path, exit_fd)? {
return Ok(());
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UDP events livelock before session

High Severity

External fds are now registered on the vring worker before start/connect_to_frontend, while handle_event still returns without draining EXTRA_TOKEN_UDP when guest memory is unset. The io_uring eventfd stays level-triggered and readable, so any WireGuard datagram during the connect-retry (or server accept) wait busy-loops that worker at 100% CPU until SET_MEM_TABLE. Client-mode reconnects and the original libc6/needrestart case are exactly that window.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit f478362. Configure here.

Comment thread src/datapath/mod.rs
// session. Keeping them separate keeps the two shutdown paths apart.
let eventfd = EventFd::new(EFD_NONBLOCK).ok()?;
let consumer_clone = eventfd.try_clone().ok()?;
let notifier_clone = eventfd.try_clone().ok()?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exit eventfd leaked every session

Medium Severity

exit_event now allocates a fresh eventfd for each daemon, and a daemon is created per session. The framework takes the consumer via into_raw_fd into epoll_ctl and never closes that fd. Each QEMU probe, VM reboot, or client reconnect therefore leaks a file descriptor (and the eventfd object) for the life of the process.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit f478362. Configure here.

…sion

Addresses the Cursor Bugbot findings on this PR.

**Exit eventfd leaked one fd per session (medium).** `VringEpollHandler::new`
hands the consumer fd to `epoll_ctl` via `into_raw_fd()` and never closes it, so
the fd stayed open for the life of the process. That was harmless with a single
daemon per process, but there is now one daemon — hence one such fd — per
vhost-user session (QEMU probe, reconnect, ...). Vendored patch: close the
consumer fd right after registration. The registration keeps working because
epoll keys on the file description and `notifier` is a dup of the same eventfd.
Measured with 12 connect/disconnect sessions against one daemon: fd count 13 →
13, idle CPU 0.00s/5s.

**`EXTRA_TOKEN_UDP` could livelock the vring worker (high).** I had moved the
external-fd registration before `start`/`connect`, while `handle_event` still
returned early for the WG socket token when guest memory was not mapped yet —
the io_uring eventfd stayed readable and the level-triggered epoll spun the
worker. Two fixes: register the fds only once a session exists (each session has
its own daemon, so no `EEXIST`), and drain the WG socket even without guest
memory (frames are only queued in the backend, no vring needed) so a packet
arriving between connect and SET_MEM_TABLE cannot spin either.

**Client mode could not bind the default socket directory (high).** In client
mode QEMU creates the socket, but the default `/run/vhost-user-wg` belongs to
the daemon and is 0755, so a `libvirt-qemu` QEMU could not bind there. The hook
now chgrps the socket's directory to the QEMU group and adds group write, so
the documented stock path works.

E2E re-run on the same libvirt + QEMU 8.2.2 + Ubuntu 24.04 guest, now with the
**default** socket directory: hook logs `chgrp libvirt-qemu + group-write on
/run/vhost-user-wg`, QEMU binds the socket, guest gets 10.66.66.2 by DHCP and
pings through the WireGuard tunnel 3/3 (~3.3 ms); `systemctl restart` of the
daemon under the live VM leaves the guest answering 3/3; a VM restart brings it
back answering 3/3.
@feisuzhu

Copy link
Copy Markdown
Contributor Author

Addressed the three Bugbot findings in 87b0efc:

finding fix
Exit eventfd leaked every session (medium) VringEpollHandler::new passes the consumer fd to epoll_ctl and never closed it; with one daemon per session that is one fd per probe/reconnect. Vendored patch closes it after registration (epoll keys on the file description, and notifier is a dup — so events keep working). Measured: 12 connect/disconnect sessions against one daemon, fd count 13 → 13; idle CPU 0.00 s / 5 s.
UDP events livelock before session (high) External fds are registered only once a session exists (each session has its own daemon, so no EEXIST), and the WG socket is now drained even when guest memory is not mapped yet — frames only go into the backend queue, no vring needed — so a datagram arriving between connect and SET_MEM_TABLE cannot spin the level-triggered epoll either.
QEMU cannot bind client-mode socket (high) In client mode the hook now chgrps the socket's directory to the QEMU group and adds group write, so the documented default /run/vhost-user-wg works (it is created for the daemon, 0755). The daemon-side requirement is unchanged: SupplementaryGroups=kvm (or whatever group owns the socket).

E2E re-run, this time with the default socket directory (real libvirt + QEMU 8.2.2, Ubuntu 24.04 cloud-image guest):

hook: chgrp libvirt-qemu + group-write on /run/vhost-user-wg
hook: client mode: vhost-user-wg@vhost-wg.service started, waiting for /run/vhost-user-wg/vhost-wg.sock
hook: chown vhost-user-wg + chmod 0660 on /run/vhost-user-wg/vhost-wg.sock
# dir drwxrwxr-x vhost-user-wg libvirt-qemu, socket srw-rw---- vhost-user-wg vhost-user-wg
  • guest: enp8s0 UP 10.66.66.2/30, default via 10.66.66.1 … proto dhcp, ping 106.12.199.25 → 3/3 received, ~3.3 ms
  • systemctl restart vhost-user-wg@vhost-wg under the live VM: QEMU logs the disconnect then re-initialises on the new connection, new daemon re-handshakes within milliseconds, guest → 3/3 received
  • virsh destroy + start: hook re-prepared the dir/socket, guest back → 3/3 received

Test suite: cargo test --no-fail-fast run twice, integration_client_mode 5× — all green except integration_dhcp::test_lease_persistence_across_restart, which fails the same way on master (pre-existing, reproduced in a clean worktree). Note for anyone re-running locally: the harness allocates WG ports from 51820 upward, so a daemon of yours listening on 51820 makes every integration test fail with "daemon never created its socket".

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 87b0efc. Configure here.

parent.chmod(parent.stat().st_mode | stat.S_IWGRP | stat.S_IXGRP)
log(f"chgrp {group} + group-write on {parent}")
except (LookupError, OSError) as exc:
log(f"chgrp {group} on {parent} failed ({exc}); QEMU may not be able to bind")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QEMU dir grant races systemd RuntimeDirectory

Medium Severity

make_dir_writable_for_qemu chgrps and group-writes the socket parent so QEMU can bind, including the stock path /run/vhost-user-wg. That directory is the unit's shared RuntimeDirectory, so any vhost-user-wg@* start chmod/chowns it back to vhost-user-wg 0755 and drops group-write. If that happens after this hook returns and before QEMU binds — parallel VM starts or another instance coming up in that window — QEMU gets EACCES and the guest NIC never appears. The hook still returns success, so libvirt does not see the permission failure.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 87b0efc. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c8c6111 — and you are right that the silent-success part was the worse half.

Client-mode sockets must not live in the unit's shared RuntimeDirectory, so the hook no longer pretends it can make that work:

  • make_dir_writable_for_qemu now returns a bool instead of only logging, and treats realpath(parent) == /run/vhost-user-wg as unserviceable.
  • handle_prepare stops the daemon and returns non-zero in that case, so libvirt fails the start with a visible error instead of launching a NIC-less guest. The journal says why and what to use instead:
vhost-user-wg-hook: client mode: socket /run/vhost-user-wg/vhost-wg.sock is inside the daemon's shared RuntimeDirectory /run/vhost-user-wg; any vhost-user-wg@* start resets that directory's group/mode, so QEMU may fail to bind. Point [vhost_user] socket at a dedicated directory instead, e.g. /var/lib/libvirt/vuwg/<domain>.sock
vhost-user-wg-hook: client mode: refusing to start vhost-wg with socket /run/vhost-user-wg/vhost-wg.sock
→ virsh start: error: Hook script execution failed: ... unexpected exit status 1
  • docs/operations.md (client-mode prerequisites + the domain XML sample) and the hook header now require a dedicated directory, e.g. /var/lib/libvirt/vuwg/<domain>.sock, which the hook creates with the QEMU group.

Verified end-to-end on QEMU 8.2.2 + Ubuntu 24.04:

  • negative case above: VM refused, nothing half-started;
  • dedicated dir: hook logs chgrp libvirt-qemu + group-write on /var/lib/libvirt/vuwg, QEMU binds, guest DHCPs 10.66.66.2 and pings through the WireGuard tunnel 3/3; systemctl restart vhost-user-wg@vhost-wg under the live VM → guest 3/3; VM restart → guest 3/3.

…ctory

Bugbot follow-up on the previous commit: `make_dir_writable_for_qemu` granted
the QEMU group access to the socket's directory even when that directory is the
unit's shared `RuntimeDirectory` (`/run/vhost-user-wg`). Every
`vhost-user-wg@*` start recreates that directory as `vhost-user-wg` 0755, so the
grant can be revoked between the hook returning and QEMU's `bind()` — a parallel
VM start is enough — and the guest then comes up without a NIC while the hook
reports success and libvirt sees nothing.

Client-mode sockets must live in a directory systemd does not manage:

* the hook now detects the shared-runtime-dir case, explains it in the journal
  (`/etc/libvirt/hooks/qemu.d/vhost-user-wg` output via `vhost-user-wg-hook`),
  stops the daemon and returns non-zero, so libvirt fails the VM start with a
  visible `Hook script execution failed` instead of starting a NIC-less guest;
* `make_dir_writable_for_qemu` reports failure instead of only logging, which is
  what enables the above;
* docs and the operations sample use a dedicated directory
  (`/var/lib/libvirt/vuwg/<domain>.sock`) for client mode and explain why.

E2E on real libvirt + QEMU 8.2.2 + Ubuntu 24.04 guest:

* **negative**: client mode with `socket = "/run/vhost-user-wg/vhost-wg.sock"`
  → hook logs the reason, `virsh start` fails with
  `Hook script execution failed ... unexpected exit status 1`, no VM starts;
* **positive** (dedicated dir): hook logs `chgrp libvirt-qemu + group-write on
  /var/lib/libvirt/vuwg`, QEMU binds the socket, guest DHCPs 10.66.66.2 and
  pings through the tunnel 3/3; daemon restart under the live VM → 3/3; VM
  restart → 3/3.
@feisuzhu
feisuzhu merged commit 18804c1 into master Nov 17, 2026
11 checks passed
@feisuzhu
feisuzhu deleted the feisuzhu/vuwg-client-mode branch November 17, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant