Skip to content

[GSoC 2026] Active cleanup of orphaned subscriptions for the taxirides topic - #39728

Open
HansMarcus01 wants to merge 1 commit into
apache:masterfrom
HansMarcus01:feat-cleaning-orphaned-suscriptions
Open

[GSoC 2026] Active cleanup of orphaned subscriptions for the taxirides topic#39728
HansMarcus01 wants to merge 1 commit into
apache:masterfrom
HansMarcus01:feat-cleaning-orphaned-suscriptions

Conversation

@HansMarcus01

Copy link
Copy Markdown
Contributor

Summary of Changes

This pull request switches the execution mode of the Pub/Sub subscription cleaner for NYC taxi rides from simulation to active deletion by changing dry_run=True to dry_run=False inside:
stale_cleaner.pyclean_pubsub_subscriptions()

Why This Change is Safe (7-Day Empirical Validation)

Prior to enabling active deletion, we ran the subscription cleaner in dry_run=True (Simulation Mode) for 7 days (from August 4, 2026, to August 11, 2026).

We monitored the state of the tracked resources by periodically downloading and analyzing the JSON audit state stored in GCS:

gs://apache-beam-testing-pabloem/stale_cleaner/pubsub_subscription.json

Key Empirical Findings:

  • 100% Resource Abandonment: Multiple subscriptions under the pattern taxirides-realtime_beam_-... created on August 4 remained active, completely unconsumed, and were continuously updated every single day until August 11 without being acknowledged.
  • Daily Accumulation Pattern: Between 4 and 5 new orphaned subscriptions were added daily and remained in the backlog.

JSON State Evidence (August 11 Audit):

{
    "projects/apache-beam-testing/subscriptions/taxirides-realtime_beam_-6413074580631200911": {
        "resource_name": "projects/apache-beam-testing/subscriptions/taxirides-realtime_beam_-6413074580631200911",
        "creation_date": "2026-08-04T00:53:24.441535",
        "last_update_date": "2026-08-11T12:24:57.889076"
    },
    "projects/apache-beam-testing/subscriptions/taxirides-realtime_beam_3858484106805878348": {
        "resource_name": "projects/apache-beam-testing/subscriptions/taxirides-realtime_beam_3858484106805878348",
        "creation_date": "2026-08-04T00:53:24.441562",
        "last_update_date": "2026-08-11T12:24:57.889077"
    }
}

Verification & Testing

  • Unit test have been run successfully
  • No other prefixes are impacted; the active deletion only targets the explicit NYC taxi prefix taxirides-realtime_beam_.

@github-actions github-actions Bot added the infra label Aug 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Abacn added as fallback since no labels match configuration

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant