functions: deploy exits 0 with endpoints left un-updated after 429 quota errors, and prints no error summary

[gustavfrid]

[REQUIRED] Environment info

firebase-tools: 15.22.2

Platform: Linux (ubuntu-latest GitHub Actions runner), Node 24.16.0

[REQUIRED] Test case

A 2nd-gen functions codebase with ~81 endpoints, all in one region (europe-west3), deployed with:

firebase deploy --only functions --project <project> --force

The codebase is a single bundle, so every endpoint's sourceHash changes on any source change and all ~81 are updated on every deploy. That exceeds cloudfunctions.googleapis.com/v2_write_requests_regional, which is 60 mutations/min/region and is documented as not increasable, so the deploy reliably takes HTTP 429s and leans on the executor's retries.

I have not reduced this to a minimal repro — it is a race that shows up under sustained 429 pressure. Roughly: deploy enough endpoints in one region that the mutation quota is exceeded for several minutes.

[REQUIRED] Steps to reproduce

1. Deploy ~80+ 2nd-gen functions to a single region in one firebase deploy --only functions.
2. Let the Cloud Functions API return 429 Quota exceeded for quota metric 'Per project mutation requests' and limit 'Per project mutation requests per minute per region' for some of them.
3. Observe the exit code and compare the deployed revisions against the source.

[REQUIRED] Expected behavior

Either every endpoint deploys, or the command fails. src/deploy/functions/release/index.ts looks like it intends the latter — after Fabricator::applyPlan it calls reporter.printErrors(summary) and then:

const allErrors = summary.results.filter((r) => r.error).map((r) => r.error) as Error[];
if (allErrors.length) {
  ...
  throw new FirebaseError("There was an error deploying functions", { ...opts, exit: 2 });
}

[REQUIRED] Actual behavior

The command exited 0 with two endpoints never updated, and printed no error summary.

From the run's log:

• 81 endpoints logged functions: updating Node.js 24 (2nd Gen) function <name>(europe-west3)...
• 79 logged functions[<name>(europe-west3)] Successful update operation.
• One logged the 429 and then ⚠ functions: failed to update function projects/<project>/locations/europe-west3/functions/<name>, and nothing further
• One logged only updating ... and then nothing at all — no success, no error

And critically, none of these appeared:

• Functions deploy had errors with the following functions: (reporter.printErrors)
• Deploy complete!
• any artifact-cleanup output — setupArtifactCleanupPolicies is the statement right after printTriggerUrls, so execution seems not to have reached it either

The last line of the step was the final endpoint's success message, ~30 ms before the CI step ended. Exit code 0.

Verified against the Cloud Functions v2 API afterwards: both endpoints were still on their previous revision, and stayed there for three days across two further "successful" deploys of the same shape (the same bundle hash means they were retried and dropped the same way each time). Every other endpoint showed an updateTime inside the deploy window.

So it looks like the process can leave applyPlan — or exit outright — without summary.results carrying the failed operations, so the allErrors check finds nothing to throw and the CLI reports success. I have not been able to pin down the mechanism from the outside; wrapOperation does record op.error for non-retryable codes, so my guess is something in the throttler's retry-exhaustion path, but that is speculation.

Impact

This is a silent partial deploy. CI reports green, and production keeps serving old code for the affected functions with nothing anywhere to indicate it. A non-zero exit — or even just the existing error summary — would have surfaced it immediately.

Happy to supply more of the raw log or test a patch.

[IzaakGough]

Hi @gustavfrid, thanks for raising this. We will review as soon as possible.

[IzaakGough]

Hi @gustavfrid, thanks for the detailed write-up, the log breakdown is really helpful.

Sustained 429s on a deploy this size are expected given the per-region mutation quota, but exiting 0 with endpoints left on their old revision and no error summary is not. That's the part we should fix: a deploy that doesn't fully apply needs to fail loudly. I'll dig into this further and post updates here.