Summary
The post-install,post-upgrade hook Job *-keda-apply-* is consistently OOMKilled (exit 137) on upgrade at the resource limits hardcoded in templates/keda-scaledobject.yaml (lines 481-487):
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 200m
memory: 256MiBecause that Job sets backoffLimit: 0, a single OOM fails the whole Helm operation. With chart 0.7.2 this made every upgrade fail — 5 consecutive attempts (hook Jobs …-keda-apply-{12,14,16,18,20}, of which I confirmed -12 and -14 as OOMKilled/exit 137 at the pod level before Helm's hook-delete-policy cleaned them up), each failure rolling the release back. The deployed application itself was healthy and serving traffic throughout; only the upgrade path was blocked.
Notably the post-install run of the same hook (…-keda-apply-1) succeeded on the same cluster at the same limit — see the mechanism note below, which I think explains the asymmetry.
The chart exposes no value to raise those limits (no hook*.resources key exists in values.yaml), which is the core of this request.
Where it dies
The hook's own log shows CRD discovery succeeding and the failure happening in the apply step:
=== KEDA ScaledObject Apply ===
Waiting for KEDA CRD...
KEDA CRD available
Applying ScaledObjects...
Applying sie-keda-so-0000
<OOMKilled>
So it is past "Step 1: Wait for KEDA CRD" and inside "Step 2".
Why install passes and upgrade fails, at the same limit on the same cluster — the ownership check in Step 2 is data-dependent:
existing="$(kubectl get scaledobject "$name" -n "$NAMESPACE" --ignore-not-found -o json)"
if [ -n "$existing" ] && ! printf '%s' "$existing" | jq -e ' ... ' ; then
On a fresh install nothing matches, existing is empty, and the jq branch is skipped. On every subsequent upgrade each ScaledObject is fetched as full JSON and piped through jq, and Step 3 then runs another namespace-wide kubectl get scaledobjects -l …. That is several extra kubectl/jq processes inside the same 256Mi cgroup that already holds a kubectl discovery cache for ~500 API resources. I have not profiled the individual allocations, so treat the split as the observation and this as the likely explanation — but it does match install-passes / every-upgrade-fails exactly, and it means the headroom needed scales with how many lanes a deployment has.
Why the usual workarounds do not apply
--no-hooks / upgrade.disableHooks — the chart README explicitly says not to use --no-hooks, --atomic or --cleanup-on-fail. We are currently forced to run upgrade.disableHooks: true anyway, which means upgrades no longer reconcile the KEDA ScaledObjects or run the health gates. We compensate with a CI check that blocks chart-version and worker-lane changes while that override is in place, but it is a workaround for a chart limitation, not a fix.- Post-renderers — we deploy via Flux. Helm does not pass hook manifests through post-renderers, so a Kustomize patch raising the memory has no effect on hook Jobs (verified empirically: patch applied, hook pods still ran at 256Mi). Flux helm-controller v1.3.0 has no
postRenderStrategy option to change that.
- Admission mutation — no mutating webhook (Kyverno/Gatekeeper/VPA) in this cluster to raise it either.
autoscaling.enabled: false — not viable: templates/keda-lifecycle.yaml then switches its hooks to pre-upgrade,pre-delete, and that Job is the same kubectl-in-a-Job pattern at an even smaller 128Mi limit (line 120).
That leaves no supported way to run this chart's hooks on a cluster like ours.
Request
Expose the hook Jobs' resources through values, so operators can size them for their cluster. Something as simple as:
hooks:
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {cpu: 200m, memory: 1Gi}applied to the kubectl-based hook Jobs would be enough. The same limitation affects the other kubectl-based hooks — templates/hooks/keda-ready-test.yaml (256Mi, line 385) and templates/keda-lifecycle.yaml (128Mi, line 120) — so a shared key covering all of them seems right.
A complementary improvement, if you would rather not make it configurable: backoffLimit: 0 turns any transient hook failure into a failed release. Even backoffLimit: 1-2 would make these hooks considerably less brittle.
Happy to open a PR for the values-driven version if that shape works for you.
Environment
- Chart / app version:
sie-cluster 0.7.2 / v0.7.2 (latest published at time of writing)
- Kubernetes: v1.34 (EKS), Bottlerocket nodes
- Deployed via Flux (helm-controller v1.3.0); external KEDA 2.20.2 and an external Prometheus (
autoscaling.enabled: true, keda.install: false)
- Cluster API surface: 471 CRDs, 539 API resources
- Hook image:
alpine/k8s:1.29.10 (as pinned by sie-cluster.keda.hookImage)
Summary
The
post-install,post-upgradehook Job*-keda-apply-*is consistently OOMKilled (exit 137) on upgrade at the resource limits hardcoded intemplates/keda-scaledobject.yaml(lines 481-487):Because that Job sets
backoffLimit: 0, a single OOM fails the whole Helm operation. With chart 0.7.2 this made every upgrade fail — 5 consecutive attempts (hook Jobs…-keda-apply-{12,14,16,18,20}, of which I confirmed-12and-14asOOMKilled/exit 137 at the pod level before Helm'shook-delete-policycleaned them up), each failure rolling the release back. The deployed application itself was healthy and serving traffic throughout; only the upgrade path was blocked.Notably the
post-installrun of the same hook (…-keda-apply-1) succeeded on the same cluster at the same limit — see the mechanism note below, which I think explains the asymmetry.The chart exposes no value to raise those limits (no
hook*.resourceskey exists invalues.yaml), which is the core of this request.Where it dies
The hook's own log shows CRD discovery succeeding and the failure happening in the apply step:
So it is past "Step 1: Wait for KEDA CRD" and inside "Step 2".
Why install passes and upgrade fails, at the same limit on the same cluster — the ownership check in Step 2 is data-dependent:
On a fresh install nothing matches,
existingis empty, and thejqbranch is skipped. On every subsequent upgrade each ScaledObject is fetched as full JSON and piped throughjq, and Step 3 then runs another namespace-widekubectl get scaledobjects -l …. That is several extrakubectl/jqprocesses inside the same 256Mi cgroup that already holds a kubectl discovery cache for ~500 API resources. I have not profiled the individual allocations, so treat the split as the observation and this as the likely explanation — but it does match install-passes / every-upgrade-fails exactly, and it means the headroom needed scales with how many lanes a deployment has.Why the usual workarounds do not apply
--no-hooks/upgrade.disableHooks— the chart README explicitly says not to use--no-hooks,--atomicor--cleanup-on-fail. We are currently forced to runupgrade.disableHooks: trueanyway, which means upgrades no longer reconcile the KEDAScaledObjects or run the health gates. We compensate with a CI check that blocks chart-version and worker-lane changes while that override is in place, but it is a workaround for a chart limitation, not a fix.postRenderStrategyoption to change that.autoscaling.enabled: false— not viable:templates/keda-lifecycle.yamlthen switches its hooks topre-upgrade,pre-delete, and that Job is the same kubectl-in-a-Job pattern at an even smaller 128Mi limit (line 120).That leaves no supported way to run this chart's hooks on a cluster like ours.
Request
Expose the hook Jobs' resources through values, so operators can size them for their cluster. Something as simple as:
applied to the kubectl-based hook Jobs would be enough. The same limitation affects the other kubectl-based hooks —
templates/hooks/keda-ready-test.yaml(256Mi, line 385) andtemplates/keda-lifecycle.yaml(128Mi, line 120) — so a shared key covering all of them seems right.A complementary improvement, if you would rather not make it configurable:
backoffLimit: 0turns any transient hook failure into a failed release. EvenbackoffLimit: 1-2would make these hooks considerably less brittle.Happy to open a PR for the values-driven version if that shape works for you.
Environment
sie-cluster0.7.2 /v0.7.2(latest published at time of writing)autoscaling.enabled: true,keda.install: false)alpine/k8s:1.29.10(as pinned bysie-cluster.keda.hookImage)