Skip to content

feat(cleanup): add per-type row limits to existing jobs - #7842

Open
TheodoreSpeaks wants to merge 5 commits into
stagingfrom
feat/bounded-cleanup
Open

TheodoreSpeaks wants to merge 5 commits into
stagingfrom
feat/bounded-cleanup

Conversation

@TheodoreSpeaks

@TheodoreSpeaks TheodoreSpeaks commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Accept per-type row limits on the existing log and soft-delete cleanup endpoints, returning HTTP 202 and one queued job ID.
  • Pass a shared remaining budget through the existing cleanup selectors so limits apply across every workspace and organization chunk. Omitted types are skipped.
  • Serialize retention jobs, disable automatic task retries, and use smaller owner-discovery pages for manual runs. Preserve existing retention rules, billing, storage, and child cleanup behavior.

Type of Change

  • New feature

Testing

  • 104 focused contract, route, dispatch, worker, and cleanup-helper tests passed.
  • Lint, all 46 audits, block registry, docs manifest, and app/auth type checks passed.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Sep 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 15, 2026 6:15pm UTC

Request Review

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@TheodoreSpeaks I have started the AI code review. It will take a few minutes to complete.

@greptile-apps

greptile-apps Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge because no accepted new finding or outstanding previous finding remains.

Summary

This PR adds bounded, manually dispatched retention cleanup with per-resource budgets, dry-run support, shared queueing, transactional cleanup coordination, progress reporting, and durable storage retries.

  • Adds validated bounded-cleanup parameters to the log and soft-delete maintenance endpoints.
  • Coordinates root and child cleanup through guarded transactions and shared global budgets.
  • Persists storage cleanup events with metadata-generation snapshots to avoid affecting changed bindings.
  • Replaces partial Vitest module mocks with explicit dependency mocks.
  • Documents operational invocation, failure handling, and gradual rollout.

Diagram

sequenceDiagram
    participant Operator
    participant API as Cleanup API
    participant Queue as Retention Queue
    participant Worker as Bounded Worker
    participant DB as PostgreSQL
    participant Outbox
    participant Storage

    Operator->>API: Authenticated bounded request
    API->>Queue: Dispatch limits, batch size, dryRun, requestId
    Queue->>Worker: Run serialized cleanup
    Worker->>DB: Select bounded eligible roots
    Worker->>DB: Lock/recheck and mutate guarded rows
    DB->>Outbox: Persist storage intent in transaction
    Worker->>Outbox: Attempt run-owned events
    Outbox->>DB: Lock and verify captured bindings
    Outbox->>Storage: Delete object with deadline
    Outbox->>DB: Tombstone verified metadata
    Worker-->>Queue: Checkpoint progress and stop reason
Loading

Reviews (4) · Last reviewed commit: "fix(cleanup): guard storage retries by f..."

Comment thread apps/sim/background/cleanup-logs-bounded.ts Outdated
Comment thread apps/sim/background/cleanup-logs-bounded.ts Outdated
Comment thread apps/sim/background/cleanup-soft-deletes-bounded.ts Outdated
Comment thread apps/sim/lib/cleanup/bounded-delete.ts Outdated
@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@TheodoreSpeaks I have started the AI code review. It will take a few minutes to complete.

Comment thread apps/sim/background/cleanup-logs-bounded.ts Outdated
@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@TheodoreSpeaks I have started the AI code review. It will take a few minutes to complete.

Comment thread apps/sim/lib/cleanup/storage-outbox.ts Outdated
Comment thread apps/sim/background/cleanup-bounded.test.ts Outdated
@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@TheodoreSpeaks I have started the AI code review. It will take a few minutes to complete.

@TheodoreSpeaks TheodoreSpeaks changed the title feat(cleanup): add bounded manual retention runs feat(cleanup): add per-type row limits to existing jobs Sep 15, 2026
@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@greptile

@TheodoreSpeaks

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@TheodoreSpeaks I have started the AI code review. It will take a few minutes to complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant