fix(knowledge): release kept connector documents in the background so removing a large source cannot time out by waleedlatif1 · Pull Request #8165 · simstudioai/sim

waleedlatif1 · 2026-09-22T22:07:11Z

Summary

Removing a connector while keeping its documents timed out on any sizeable source and showed "Connection is busy. Try removing it again in a moment." Every retry hit the same timeout
Cause: the request transaction deleted the connector row, ON DELETE SET NULL nulled connector_id on every document, and projection_source_acl_sync then rewrote every enabled chunk in embedding_search and embedding_keyword_tin. Those rows are indexed on connector_id, so none of the updates could be HOT; each one re-inserted into every index. All of it had to finish inside the 10 s statement_timeout
Keep-documents now works like delete-documents: one short transaction admits storage against the quota, sets the new detached_at, disables both syncs, and queues a knowledge.connector.detach outbox event. The request writes no document
The detach worker handles the next 100 documents per transaction. It nulls at most 250 of their projection rows per table first, and flips those documents only once none of their rows still names the connector, so the trigger has nothing left to rewrite. Each flipped page is billed in its own transaction, so a document is billable exactly when it no longer names a connector. When nothing references the connector, it drains history and member rows and deletes the row, using removeDrainedConnector, which is shared with the deletion worker
connectorIsLive() now also requires detached_at IS NULL. A detached connector can't be synced, listed, updated, or removed again, while document visibility still checks only archived_at/deleted_at, so kept documents stay readable and searchable throughout
The keep path's knowledge-base lock drops from FOR UPDATE to FOR SHARE. FOR UPDATE conflicted with the FOR KEY SHARE that every in-flight document and chunk write holds, so even small sources could fail the 5 s lock_timeout
Knowledge-base restore no longer sets a detached connector back to active
Migration 0376 adds the nullable detached_at column: additive, no default, metadata-only
The knowledge integration runner now installs the projection source/ACL trigger (script 0021), so integration tests exercise the same trigger production runs

Type of Change

Bug fix

Testing

New unit tests for the detach worker: paging, a large document released across several transactions, billing and rollback, outdated events, cancellation. Updated tests for the request path
Knowledge integration suite run against real Postgres with the production triggers installed. A new test releases a document with 600 chunks and checks that no search row still names the source. search-source-progress and storage-accounting pass. Two failures remain locally (confluence-enrollment, the filename-search paging test); both also fail on unmodified staging against the same database
bun run type-check, bun run lint, check:audits (47), check:migrations origin/staging, docs-manifest:check

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

… removing a large source cannot time out

vercel · 2026-09-22T22:07:17Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 22, 2026 10:33pm UTC

cubic-dev-ai

All reported issues were addressed across 26 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-09-22T22:12:27Z

Confidence Score: 5/5

The PR appears safe to merge; no outstanding correctness, security, migration, or repository-rule issue was identified.

Summary

This PR moves connector document retention out of the request transaction and into bounded, durable outbox processing while preserving storage accounting and access behavior.

Adds a detached connector lifecycle and migration-backed reservation state.
Releases projection rows and documents in bounded transactions before deleting the drained connector.
Carries detach reservations through payer transfers, workspace moves, and storage reconciliation.
Excludes detached connectors from synchronization, management, restoration, credential access, and connector listings.
Extends unit and PostgreSQL integration coverage for paging, billing, cancellation, projection cleanup, and large documents.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Remove connector and keep documents] --> B[Lock connector and billing state]
  B --> C[Charge retained bytes]
  C --> D[Set detached_at and reservation]
  D --> E[Queue detach outbox event]
  E --> F[Release bounded projection rows]
  F --> G{Projection rows remain?}
  G -- Yes --> F
  G -- No --> H[Detach bounded document page]
  H --> I[Consume reservation]
  I --> J{Documents remain?}
  J -- Yes --> F
  J -- No --> K[Drain history and member rows]
  K --> L[Settle remaining reservation]
  L --> M[Delete connector]

_{Reviews (4) · Last reviewed commit: "test(workspaces): add knowledge_connecto..."}

…it when the release finishes

waleedlatif1 · 2026-09-22T22:19:26Z

@greptile

waleedlatif1 · 2026-09-22T22:19:27Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T22:19:33Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 28 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…use credentials to removed connectors

waleedlatif1 · 2026-09-22T22:27:04Z

@greptile

waleedlatif1 · 2026-09-22T22:27:05Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T22:27:11Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 31 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…r fixture schema

waleedlatif1 · 2026-09-22T22:33:04Z

@greptile

waleedlatif1 · 2026-09-22T22:33:05Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T22:33:12Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 32 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

fix(knowledge): release kept connector documents in the background so…

8cb378f

… removing a large source cannot time out

vercel Bot deployed to Preview September 22, 2026 22:09 View deployment

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/orchestration/connectors.ts Outdated

Comment thread apps/sim/lib/knowledge/service.ts

Comment thread apps/sim/lib/knowledge/connectors/deletion.ts

Comment thread apps/sim/lib/knowledge/connectors/queue.ts Outdated

greptile-apps Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/orchestration/connectors.ts Outdated

Comment thread apps/sim/lib/knowledge/connectors/detachment.ts

fix(knowledge): reserve kept connector storage at removal and settle …

2ba26c6

…it when the release finishes

vercel Bot deployed to Preview September 22, 2026 22:20 View deployment

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/connectors/member-access.ts

greptile-apps Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/orchestration/connectors.ts

fix(knowledge): carry detach reservations through payer moves and ref…

cef636b

…use credentials to removed connectors

vercel Bot temporarily deployed to Preview September 22, 2026 22:26 Inactive

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

test(workspaces): add knowledge_connector to the payer-move lock-orde…

d6f3923

…r fixture schema

vercel Bot temporarily deployed to Preview September 22, 2026 22:33 Inactive

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

waleedlatif1 merged commit 7823118 into staging Sep 22, 2026
34 checks passed

waleedlatif1 deleted the fix/connector-remove-busy branch September 22, 2026 22:45

This was referenced Sep 22, 2026

fix(knowledge): bound the members-mode tombstone pass by what a run changed #8172

Merged

v0.8.57: knowledgebase connector improvements, distributed oauth refresh #8180

Merged

Conversation

waleedlatif1 commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Sep 22, 2026 •

edited

Loading

vercel Bot commented Sep 22, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 22, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading