Skip to content

fix: keep annotations added after CreatePod when virtual-kubelet pushes a Pod's status - #148

Merged
InftyAI-Agent merged 1 commit into
mainfrom
fix/vnode-status-patch
Nov 18, 2026
Merged

InftyAI-Agent merged 1 commit into
mainfrom
fix/vnode-status-patch

Conversation

@kerthcet

Copy link
Copy Markdown
Member

Problem

Nebula's virtual kubelet removes any annotation added to a Pod after CreatePod, within a second. For example, a controller's controller.kubernetes.io/pod-deletion-cost mark is lost, so a ReplicaSet scale-down can delete a busy Pod and keep an idle one. Pods on real kubelets keep their marks.

The cause:

  1. The handler keeps its own copy of each Pod, taken at CreatePod, and the poll loop hands that copy to virtual-kubelet on every tick.
  2. virtual-kubelet v1.11.0 writes the copy with UpdateStatus at resourceVersion "0", so the write never conflicts (node/pod.go:241-245).
  3. The pods/status endpoint resets only spec, deletionTimestamp and ownerReferences, so the copy's labels and annotations are written too (strategy.go:205-223).
  4. When an annotation is added, virtual-kubelet sees the Pod differs from its copy and pushes the copy again at once (node/podcontroller.go:349-358).

Change

  • The PodClient Nebula gives virtual-kubelet sends UpdateStatus as a JSON patch that replaces /status on the status subresource. Labels and annotations others set stay.
  • The patch first tests the Pod's UID, so a late push for a deleted Pod is refused by one recreated under the same name, as an update was.
  • Every other PodClient call goes to the client unchanged.
  • Comments that said the status write drops metadata are corrected.

A JSON patch, not a merge patch, so a status field the provider empties is still cleared. Nebula's endpoint and instance-ID annotations reach the API server only through patchMeta.

Testing

New unit tests on client-go's fake clientset: a stale copy's status push keeps the annotation the live Pod gained and still updates (and clears) status fields; a push with an old UID is refused and leaves the recreated Pod alone. go test ./pkg/... ./internal/... and make lint pass.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings November 18, 2026 14:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 18, 2026
virtual-kubelet pushes the provider's status with UpdateStatus on the
provider's copy of the whole Pod, at resourceVersion "0". The
pods/status endpoint resets only spec, deletionTimestamp and
ownerReferences, so the push also writes that copy's labels and
annotations. VK re-pushes its last copy as soon as it sees the Pod's
annotations differ from it, so an annotation added since the copy was
taken is removed within a second. A pod-deletion-cost,
which tells a ReplicaSet scale-down which Pods to spare, is one such
annotation.

The PodClient handed to VK now sends UpdateStatus as a JSON patch that
replaces /status on the status subresource. A JSON patch, not a merge
patch, so a field the new status leaves out is cleared, as the update
cleared it. The patch first tests the copy's UID, so a late push is
refused by a Pod recreated under the same name, as an update was.
Nebula's own annotations already reach the API server through
patchMeta.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@InftyAI-Agent InftyAI-Agent added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 18, 2026
@kerthcet
kerthcet force-pushed the fix/vnode-status-patch branch from f43b772 to 5f73b13 Compare November 18, 2026 14:37
Copilot AI balanced review requested due to automatic review settings November 18, 2026 14:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@kerthcet

Copy link
Copy Markdown
Member Author

/lgtm
/approve
/kind bug

@kerthcet kerthcet changed the title fix(vnode): keep annotations added after CreatePod when virtual-kubelet pushes a Pod's status fix: keep annotations added after CreatePod when virtual-kubelet pushes a Pod's status Nov 18, 2026
@InftyAI-Agent InftyAI-Agent added lgtm Looks good to me, indicates that a PR is ready to be merged. bug Categorizes issue or PR as related to a bug. and removed do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 18, 2026

@InftyAI-Agent InftyAI-Agent left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: PR has both lgtm and approved labels

@InftyAI-Agent
InftyAI-Agent merged commit f4b96e3 into main Nov 18, 2026
24 of 26 checks passed
InftyAI-Agent pushed a commit that referenced this pull request Nov 19, 2026
…us (#148)" (#149)

This reverts commit f4b96e3.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. bug Categorizes issue or PR as related to a bug. lgtm Looks good to me, indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants