Skip to content

fix: resolve containerd pid reliably before sending SIGHUP - #2127

Open
jpschmitt02 wants to merge 1 commit into
NVIDIA:mainfrom
jpschmitt02:fix/signal-containerd-passcred-race
Open

jpschmitt02 wants to merge 1 commit into
NVIDIA:mainfrom
jpschmitt02:fix/signal-containerd-passcred-race

Conversation

@jpschmitt02

@jpschmitt02 jpschmitt02 commented Oct 7, 2026 •

Copy link
Copy Markdown

Description

Fixes #2126.

SignalContainerd resolves containerd's PID from socket peer credentials, but enables SO_PASSCRED after connect(). containerd's gRPC server writes its first frames immediately on accept, and the kernel only records peer credentials on data queued while SO_PASSCRED is set — so the credentials can parse as pid=0 (with overflow uid/gid 65534). The PID was passed to kill(2) unvalidated, and kill(0, SIGHUP) signals the calling process's own process group: the toolkit killed itself, logged "Successfully signaled containerd", and crash-looped, with the real containerd kill landing on a later retry — well after node startup had completed. Details, production evidence, and a standalone reproducer are in #2126.

This change:

  • extracts PID resolution into a containerdPid helper and enables SO_PASSCRED before connect() via net.Dialer.Control, so every message from the server carries credentials;
  • rejects non-positive peer PIDs with an error, which feeds the existing retry loop (6 attempts, 5s backoff) instead of a signal.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Unit tests passing (make test)
  • Lint checks passing (make lint)
  • Test cases are added for new code paths
  • Commits are signed-off and cryptographically signed

Testing

  • New unit tests: TestContainerdPid resolves the PID of a mock unix server that writes immediately on accept (the containerd behavior that triggers the race), and TestContainerdPidNeverZero runs 200 iterations as a regression test — on the previous code this resolves pid=0 for a fraction of attempts (1–3/200 on an idle machine; 200/200 with a 50ms delay before setsockopt), with the fix it is deterministic.
  • go build, go vet, golangci-lint run (v2.6.1), and the package test suite pass on go 1.26 (linux/arm64 container).
  • The standalone reproducer in [Bug]: SignalContainerd can SIGHUP its own process group: SO_PASSCRED enabled after connect can resolve ucred.Pid=0 #2126 shows the before/after kernel behavior against both a live containerd 2.2.7 socket and the mock server.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@jpschmitt02
jpschmitt02 force-pushed the fix/signal-containerd-passcred-race branch 2 times, most recently from 7716d8c to c2617c5 Compare October 7, 2026 17:19

@cdesiniotis cdesiniotis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpschmitt02 thank you for the contribution! I left some minor review comments.

Comment thread cmd/nvidia-ctk-installer/container/runtime/containerd/containerd_linux.go Outdated
Comment thread cmd/nvidia-ctk-installer/container/runtime/containerd/containerd_linux.go Outdated
Comment thread cmd/nvidia-ctk-installer/container/runtime/containerd/containerd_linux.go Outdated
SignalContainerd enabled SO_PASSCRED after connect(). containerd's gRPC
server writes its first frames immediately on accept, and the kernel only
records peer credentials on data queued while SO_PASSCRED is set, so the
credentials could parse as pid 0. The pid was passed to kill(2)
unvalidated, and kill(0, SIGHUP) signals the calling process's own
process group: the toolkit killed itself, logged "Successfully signaled
containerd", and crash-looped until a retry resolved the real pid,
restarting containerd well after node startup had completed.

Enable SO_PASSCRED before connect() via net.Dialer.Control so every
message carries credentials, and reject non-positive peer pids so a
failed resolution feeds the existing retry loop instead of a signal.

Fixes NVIDIA#2126

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Justin Schmitt <justin.schmitt@baseten.co>
@jpschmitt02
jpschmitt02 force-pushed the fix/signal-containerd-passcred-race branch from c2617c5 to e9fd70f Compare October 8, 2026 21:29
@cdesiniotis

Copy link
Copy Markdown
Contributor

/ok to test e9fd70f

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 37847168109

Coverage increased (+0.2%) to 44.795%

Details

  • Coverage increased (+0.2%) from the base build.
  • Patch coverage: 19 uncovered changes across 1 file (24 of 43 lines covered, 55.81%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
cmd/nvidia-ctk-installer/container/runtime/containerd/containerd_linux.go 43 24 55.81%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 13564
Covered Lines: 6076
Line Coverage: 44.8%
Coverage Strength: 0.45 hits per line

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SignalContainerd can SIGHUP its own process group: SO_PASSCRED enabled after connect can resolve ucred.Pid=0

3 participants