[pull] master from kubernetes:master - #1924
Open
pull[bot] wants to merge 8709 commits into
Open
Conversation
This reverts commit 36a27dc.
Update the gossip docs for the removal that shipped in kOps 1.37, with the deprecation and removal timeline, the changes shipped in kOps 1.36, and a step-by-step guide for migrating existing gossip clusters to None-DNS using kOps 1.36 before upgrading. Note that nodes no longer install or run protokube, and that the gossipConfig and dnsControllerGossipConfig cluster spec fields are deprecated, have no effect, and are rejected by validation when set. Align the 1.36 release notes with the removal plan from #18240, use None-DNS wording in the docs that still described clusters as created without DNS, and remove the outdated AWS China guide, which documented kOps 1.10 procedures and is superseded by the None-DNS support in the getting started guide.
chore(networking): bump aws-cni to version 1.22.4
Remove gossip leftovers
pkg/dns/hosts (the /etc/hosts writer used by none-DNS clusters) and pkg/etcd (etcd cluster spec parsing from volume tags) are still used outside protokube, so relocate them ahead of the protokube removal. protokube temporarily imports the new locations.
protokube no longer had any responsibilities after the gossip DNS removal: nodes stopped installing and running it, and its only remaining function was serving gossip DNS. Delete the protokube tree, the forked memberlistmesh library, and the gcediscovery package whose only consumer was protokube's GCE volume provider. Remove the protokube build targets and stop publishing its release assets. The gossip well-known ports are kept (and the two dns-controller ones restored) as Deprecated constants so the port numbers are not reused. The release promotion and validation scripts learn an OPTIONAL_BINARIES mechanism, probing artifacts.k8s.io per version, so older releases that still ship protokube (and channels, dropped in kOps 1.36) keep full coverage while newer releases skip them. Only a confirmed 404 counts as absent; any other probe outcome fails the scripts.
github.com/hashicorp/memberlist and github.com/weaveworks/mesh are no longer used after the protokube removal.
Removed concept of CCM node as it is dynamic. Fixed KubController -> KubeController. Added setting kops role labels on nodes via kops-channel.
Fixed APIServer label test.
Fix IG HasAPIServer to RunsAPIServer
Remove the protokube binary
docs: gossip removal in kOps 1.37 and migration guide
The DIGITALOCEAN_ACCESS_TOKEN, HCLOUD_TOKEN and SCW_* exports in the bootstrap script existed only for protokube's gossip seed discovery. Workers stopped receiving them in kOps 1.36 and protokube itself was removed in kOps 1.37, so the control-plane exports and their persistence in /etc/sysconfig/kops-configuration have no consumer left. All remaining consumers (etcd-manager, kops-controller, dns-controller, CCM, CSI) receive credentials via manifests built at kops update time. Also write /etc/sysconfig/kops-configuration with mode 0600; it can still contain state store and OpenStack credentials and was previously world-readable.
…URST Signed-off-by: ronaldngounou <rngounou@amazon.com>
…odeSyncs Signed-off-by: ronaldngounou <rngounou@amazon.com>
Skip the dns-controller Route 53 grant on the control-plane instance role when useServiceAccountExternalPermissions is set; dns-controller and external-dns get dedicated IAM roles in that mode, matching how the other addon permissions are handled. Scope the kube-router and kindnet ec2:ModifyInstanceAttribute grants to cluster-tagged instances, matching the CCM and IRSA variants of the same permission. Add golden coverage for the kube-router node policy.
Narrow instance cloud permissions to actual consumers
The list of expected files in Go was a second copy of the test data directory, which now also has to describe conditional output: an instance role that needs no permissions has no inline IAM policy. Nothing is lost by dropping it. hack/update-expected.sh never deletes existing expected files, so a file that is wrongly no longer generated keeps failing the comparison until its expected file is deleted too.
Grant ec2:DescribeInstanceTypes only where something on the instance calls it: nodeup Nvidia GPU detection and the kubelet MaxPods calculation for AWS VPC CNI and Cilium ENI IPAM. The cluster autoscaler grant now honors awsUseStaticInstanceList, whose branch was a no-op because the action was also inserted unconditionally. Drop ec2:DescribeRegions from the etcd-manager permissions, as it reads the region from the instance metadata service. Move ec2:DescribeInstances to a kops-controller helper, which needs it for node verification and identification independently of etcd-manager. Render policies without statements as empty documents instead of erroring, so that roles which need no permissions, such as the bastion role, have no inline policy. Updating an existing cluster deletes the policy created for it before.
isHealthy() marked an object unhealthy on any status condition with a "False" status, regardless of the condition type. Controllers routinely publish conditions that carry no readiness meaning: KEDA sets "Paused" to "False" on every ScaledObject and ScaledJob it is actively scaling, and a healthy Node reports "MemoryPressure", "DiskPressure" and "PIDPressure" as "False". Such objects were permanently unhealthy, so an addon containing one could never be applied successfully. Since kops 1.36 the kops-channels pod reports readiness from the result of the last apply iteration, so this also leaves the pod permanently not ready and cluster validation failing. Only consider condition types whose "True" status means the object is healthy, and ignore the rest. Abnormal-true conditions are deliberately not interpreted, matching the previous behaviour for those. Signed-off-by: Joakim Olsson <joakim@unbound.se>
The KopsConfig reconciler and the node config server both failed when the cluster channel could not be read, e.g. a file:// channel set by the e2e harness that only exists on the machine that created the cluster. The channel only provides optional defaults here, so warn and continue like apply_cluster does. With a nil channel, instance group spec population then fails resolving a default machine image on GCE. The image is chosen by the CAPI infrastructure provider and is not used for nodeup config generation, so give the placeholder instance groups a placeholder image.
kube-proxy special-cased only control-plane nodes for the localhost address and used the API internal name on dedicated apiserver nodes. Those nodes have no /etc/hosts record for that name (and no DNS record in dns=none clusters, previously masked by gossip), so kube-proxy could never reach the API and never programmed cluster IPs. As a result the local kube-apiserver could not call any admission webhook by service, failing with a timeout regardless of where the webhook pods run. Use the same HasAPIServer condition as the kubeconfig builders.
Firewall source tags only match traffic sent from an instance's primary address, so pod-sourced traffic is matched by CIDR instead. That rule only targeted node instances, leaving pods unable to reach pods hosted on control plane or dedicated apiserver nodes, such as addons that are pinned to the control plane. Grant pod CIDRs the same limited port set that nodes get, so host ports such as etcd stay protected.
keypair-ids.yaml covers every keypair on the host, while the server keystore only loads the CAs it is configured to serve, so the etcd CAs always trigger this warning on the CAPI node bootstrap path.
The clusterapi, splitkcp and ai-conformance scenarios each installed cert-manager from their own pinned upstream release URL, with two different versions across the three scripts. Enable the bundled certmanager.io addon in the cluster spec instead, so the tree has a single cert-manager definition and version bumps happen only in the addon. The explicit readiness waits are no longer needed: the addon runs its pods with the system-cluster-critical priority class, and cluster validation during kops-up fails while any such pod is pending or not ready.
test(e2e): deploy cert-manager via the certmanager.io addon
No Prow job runs this script. All eight DigitalOcean jobs invoke "make test-e2e-install" and kubetest2 directly from their job config, and the JOB_NAME it hardcodes, pull-kops-e2e-kubernetes-do-kubetest2, no longer exists anywhere in test-infra. The e2e-do-kubetest2 testgrid tab it was named for now belongs to pull-kops-e2e-kubernetes-do-fqdn, which does not use the script. It could not work if it were run. It pins Kubernetes 1.27 via stable-1.27.txt, well below OldestSupportedKubernetesVersion, and it calls "go get sigs.k8s.io/kubetest2/...@latest" from the repo root, which on supported Go versions edits go.mod instead of installing anything. Last touched in 2023.
Now that the e2e scenarios enable the certmanager.io addon rather than applying an upstream manifest, this echo is the last place in the tree naming a cert-manager version, and it named an older one than the addon deploys. Point it at the addon's version.
Remove the orphaned digital-ocean e2e scenario
Remove unused hack scripts and a duplicate hash generator line
Every YAML example under docs/ was strict-decoded through the kOps API scheme, and every documented kops command was checked against the 1.37 flag set. Fix examples that current kOps would reject: * networking.kubeRouter and calico.IPIPMode do not match the JSON tags in the API types; they are kuberouter and ipipMode * clusterAutoscaler.scaleDownUtilizationThreshold and a kubeAPIServer env value were unquoted numbers, but both fields are strings * an InstanceGroup in the Spot Ocean docs had its spec nested under metadata * iam_roles used the long-removed zones field instead of subnets * the GPU taint was shown as a Kubernetes taint object, but the kOps taints field takes strings * a missing line continuation split the GCE create cluster command Rename the flags that moved to control-plane naming (--master-zones, --master-size, --master-count, --master-volume-size, --master-interval) and --vpc, which is now --network-id. kops toolbox instance-selector takes the instance group name as a positional argument. Drop the containerRuntime and spec.docker sections, as validation now rejects containerRuntime: docker outright. Refresh stale versions and images: Kubernetes 1.36.4, containerd 2.3.4, and Ubuntu 24.04 in place of focal, CoreOS, kope.io and debian-9 references. The exported manifest in manifests_and_customizing_via_api was regenerated with kops create cluster --dry-run -o yaml. Remove the Ubuntu 20.04 and Debian 10 sections from the images doc. Both distros were removed in kOps 1.36 and were the only removed distros still listed under Supported Distros. Examples stay on the v1alpha2 apiVersion, which is what kOps writes.
docs: refresh CLI examples and manifests for kOps 1.37
A v1alpha2 spec whose spec.kubeAPIServer.oidcRequiredClaim held an entry without an "=" crashed the conversion with an index out of range, because strings.SplitN was indexed unconditionally. Return a field.Invalid naming the offending index instead. Build the OIDC settings before publishing them, too. The generated conversion copies the OIDC pointer straight across, so writing into out.Authentication.OIDC could mutate the caller's object, and returning the new error partway through would leave it half rewritten.
TestRoundTripTypes excluded Cluster, ClusterList, InstanceGroup and InstanceGroupList for both versions, so it only covered Host, Keyset and SSHCredential: none of which have hand-written conversions. Every field that v1alpha2 renames, flattens or inverts went untested, and there is no codegen signal to fall back on because v1alpha2 has no AWSSpec or GCESpec types at all, so a field added to either gets no warning. Fuzz each version with its own functions, since the two lose different fields. What those functions clear is the catalogue of what is silently dropped on write. Discover kinds from the scheme so a new type is covered without being listed, and fail if a new API version arrives without fuzzer functions describing what it loses. Add unit tests pinning the manual v1alpha2 conversions the fuzzer has to exclude: the inverted booleans, the control-plane/master renaming, the rootVolume flattening, the OIDC promotion including the cases the flag encoding cannot represent, and the cloudProvider string. sigs.k8s.io/randfill becomes a direct dependency; vendor is unchanged.
Ports the applicable pieces of upstream's daemonset modernization work. Six changes, in rough order of how much they matter: * Gate liveness behind a startup probe. We were giving kube-router initialDelaySeconds: 10 plus the default failureThreshold: 3 at periodSeconds: 3, so roughly 19 seconds before the kubelet kills it. But --cache-sync-timeout defaults to a full minute, so on a cluster large enough that informer sync runs long, kube-router CrashLoopBackOffs on startup and never comes up. The startup probe allows 150 seconds, and liveness only starts once it passes. * Add type: File to the kubeconfig hostPath. Without it, a missing kubeconfig gets silently papered over by the kubelet creating a directory at that path. nodeup writes the file before kubelet.service on every node running kube-router, so this can't regress a healthy node, it just turns a confusing failure into a loud FailedMount. * Update cniVersion 0.3.0 -> 1.0.0. Our oldest CNI plugin asset is containernetworking/plugins v1.3.0 (k8s 1.29), well past the v1.0.0 that added spec 1.0.0 support, and portmap ships in the same release. Note that because the initContainer only writes the conflist when it's absent, this reaches newly-provisioned nodes only. Existing nodes keep 0.3.0 until they're replaced, and since both versions work, the mixed state is harmless. * Drop dead RBAC grants. Since the service proxy moved to EndpointSlices, the only surviving v1.Endpoints reference is isEndpointsForLeaderElection, and nothing calls it. The extensions API version was removed in k8s 1.16 and we support 1.29+. * Drop the deprecated serviceAccount field, which came back in when the v2.10.0 bump re-pulled from upstream's older kubeadm manifest. It's the long-deprecated alias for serviceAccountName, both were set to the same value, and upstream has since removed it. * Add dnsPolicy: Default and a linux nodeSelector. The dnsPolicy is effectively a no-op on a hostNetwork pod because the kubelet already coerces ClusterFirst to Default, but it records the reasoning. The nodeSelector matches what calico, cilium, coredns and metrics-server already do. Also refreshes the source URL, which pointed at the v2.1.0 kubeadm manifest even though what we ship much more closely resembles generic-kuberouter-all-features.yaml. We deliberately keep the kind: User / name: system:kube-router subject on the ClusterRoleBinding. That's ours, not upstream's, since control plane nodes get an issued client cert with that CN.
We already mount the containerd socket and pass --runtime-endpoint=unix:///run/containerd/containerd.sock, which kube-router only ever uses for DSR, but we never set hostIPC: true. Upstream's DSR docs list that as mandatory, and their kubeadm-kuberouter-all-features-dsr.yaml manifest sets all three of hostNetwork, hostIPC and hostPID, so what we ship today is half-wired: the plumbing is there but DSR can't actually work. This does widen the pod's privileges, which is worth being explicit about. In practice it's a small step from where we already are, because the container runs privileged: true with hostNetwork and hostPID already, and hostIPC is the last piece DSR needs in order to enter the pod's network namespace through the runtime.
Since v2.8.0 kube-router defaults --strict-external-ip-validation to true, and its filters default-deny: with no ranges configured, FilterExternalIPs and FilterLoadBalancerIPs both return nothing, so every externalIP and loadBalancerIP drops out of IPVS and stops being advertised over BGP. We pin the flag off today to avoid breaking existing clusters, but that leaves no way to turn the feature on at all. Add externalIPRanges and loadBalancerIPRanges to KuberouterNetworkingSpec and emit the matching --service-external-ip-range and --loadbalancer-ip-range flags. Strict validation now follows the config: setting either list turns it on, leaving both empty keeps it off, so upgrades are unaffected and opting in is a deliberate act. There's no options builder for this because empty is semantically meaningful here, it's precisely the signal for "keep strict validation off", so there's nothing to default. Validation moves out of the inline block in validateNetworking into validateNetworkingKubeRouter, which lets us reuse the already-parsed serviceClusterIPRange. Each CIDR has to parse, has to match the cluster's IP family, and must not overlap the service cluster IP range. That last one is worth failing early on because kube-router rejects those at runtime anyway, so an overlap would otherwise turn into a silently dead range rather than an error at kops update cluster time. The family check is a predicate over the cluster's stack rather than a blanket IPv6 rejection, so allowing IPv6-only clusters later widens it instead of rewriting it. Two side effects are documented but not otherwise handled: --service-external-ip-range also feeds the network policy controller's firewall rules, which was the flag's original purpose upstream, and --loadbalancer-ip-range does not activate kube-router's load balancer IPAM, which is gated separately on --run-loadbalancer.
Two teardown failures where a leaked resource keeps the network alive
until `kops delete cluster` gives up:
Routes were deleted once all the instances were gone, but not once the
managed instance groups were gone. A MIG whose deletion is still blocked
- typically the control-plane MIG, which GCE refuses to delete while the
api BackendService refers to its instance group - recreates a
control-plane instance, and its route controller recreates the routes we
just deleted. Block route deletion on the MIGs as well.
Firewall rules created for a Service of type LoadBalancer were only
matched for externalTrafficPolicy: Cluster. With Local, the health check
is named for the load balancer rather than k8s-<clusterid>-node, so we
abandoned the whole load balancer as "not a k8s rule", and its
k8s-<loadbalancer>-http-hc firewall matched neither that path nor the
-node-http-hc rule.
Also order the ForwardingRule -> BackendService -> {HealthCheck, MIG}
deletions that GCE enforces, so each one no longer costs a retry
interval.
…tion kube-router Daemonset Modernization
fix: persist binariesLocation for AWS and GCE in v1alpha2
Round trip Cluster and InstanceGroup across API versions
aws: Generate NodePool requirements for instanceRequirements
…d-lb-firewalls gce: don't leave routes and LB firewalls behind on cluster deletion
aws: validate Karpenter InstanceGroups early
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )