fix(search): use compact candidates for filtered vector retrieval

[icecrasher321]

Summary

• Broad searches with live user permissions now retrieve a bounded shortlist from compact binary HNSW indexes, then rerank stored vectors by cosine similarity before authorization and content hydration. This avoids traversing full-precision vectors for every inaccessible neighbor.
• Keep exact ranking for small or explicit document/tag scopes. Add concurrent, replay-safe candidate indexes for every supported embedding width; existing cosine indexes remain available to other search paths.
• Return known retrieval deadlines as incomplete tool results even when no passages are available. Keep timeout diagnostics and evidence limitations without presenting recoverable searches as failed tools. Companion: https://github.com/simstudioai/mothership/pull/496.
• Record candidate count, ranking strategy, and separate candidate-scan/rerank timings. Extend the existing integration harness with dense vectors, exact-neighbor recall checks, and a real-model recovery case.

Type of Change

• Bug fix

Testing

• Knowledge/search/tool regression tests and a fresh TypeScript check.
• Real PostgreSQL application harness with 45,000 synthetic vectors: authorization, source filters, revocation, concurrent searches, SQL deadlines, Search-tab parity, and exact-neighbor recall.
• Local Sim + Go + real-model test: first search exceeded its SQL deadline; Assistant searched again, progressively read the document, and cited both requested facts without timeout narration.
• Migration replay and index selection at all five widths; migration safety, lint, API validation, and the full audit suite.
• Existing contract generators and synchronization checks passed in both repositories.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 12, 2026 12:25am UTC

[github-actions]

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

• ❌ simstudioai/mothership#496 — OPEN, not merged (targets staging) — fix(assistant): recover quietly from incomplete search coverage

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable correctness, security, migration, or repository-rule issue remains.

Summary

• Adds compact HNSW expression indexes for every supported embedding width through a replay-safe concurrent migration.
• Preserves exact ranking for explicit document and structured-tag scopes and falls back to exact ranking when approximate candidates underfill.
• Reports candidate strategy, candidate count, and separate scan/rerank timings.
• Treats explicitly allowed retrieval deadlines as successful but incomplete search outcomes and extends regression and PostgreSQL integration coverage.

Diagram

sequenceDiagram
  participant Caller
  participant Search as Knowledge search
  participant Binary as Binary HNSW index
  participant Store as Embedding store
  participant Auth as Live authorization
  Caller->>Search: Query, filters, and principal
  Search->>Search: Probe scope and choose ranking strategy
  alt Explicit document/tag scope or approximate underfill
    Search->>Store: Exact cosine-ranked page
  else Broad permission-filtered search
    Search->>Binary: Retrieve bounded compact candidates
    Binary-->>Search: Candidate embedding IDs
    Search->>Store: Rerank IDs by stored-vector cosine distance
  end
  Search->>Auth: Resolve current connector grants
  Auth-->>Search: Authorized sources
  Search->>Store: Hydrate authorized passages
  Store-->>Caller: Ranked results and retrieval status

Loading

_{Reviews (1) · Last reviewed commit: "fix(search): use compact candidates for ..."}