Skip to content

[task-service] 🤖 fix: correlate workspace-turn liveness - #3915

Open
coadler wants to merge 5 commits into
mainfrom
fix/workspace-turn-false-interruptions
Open

[task-service] 🤖 fix: correlate workspace-turn liveness#3915
coadler wants to merge 5 commits into
mainfrom
fix/workspace-turn-false-interruptions

Conversation

@coadler

@coadler coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make workspace-turn liveness depend on exact turn correlation.

Background

A stale workspace-turn handle could remain running when unrelated work started in the same target workspace. The stale handle could then consume a maxParallelAgentTasks slot.

A second race could interrupt a newly persisted queued handle while sendMessage completed its preflight work.

This branch previously contained a broad pre-refactor handoff patch. The new branch starts from current main and contains only the liveness and capacity fix.

Implementation

  • Match active streams by task handle, owner workspace, and turn ID.
  • Read direct and inherited workspace-turn correlation from compaction follow-ups.
  • Match queued or preparing continuations through exact workspace-turn metadata.
  • Reserve a queued handle until sendMessage exposes queue or stream correlation.
  • Associate auto-retry and bash-monitor continuations with the active registration.
  • Treat positively unrelated target activity as evidence that the old handle is stale.
  • Carry ownership evidence through stale settlement to protect adopted disposable workspaces.
  • Repair stale handles before parallel-slot accounting.

Risks

Moderate. This changes workspace-turn recovery and task admission. The tests cover streams, compaction, queued continuations, preflight races, retry paths, monitor wakes, ownership transfer, and unrelated activity.


Generated with xum • Model: openai:gpt-5.6-sol • Thinking: xhigh • Cost: $0.19

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the workspace-turn lifecycle changes.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the implementation. The prior response only contained the completed security review.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fbe142a01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed both Codex findings in f5674e42d.

  • PRRT_kwDOPxxmWM6bQLsI: An uncorrelated stream ending now stays nonterminal only while an exact correlated continuation is pending or streaming. The code no longer relies on terminal resettlement.
  • PRRT_kwDOPxxmWM6bQLsL: The terminal report continuation is accepted before queued progress is removed. An enqueue failure keeps the progress wake available.

Added regression tests for waiter preservation, handoff order, enqueue failure, and explicit cancellation.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Both P1 findings are addressed in f5674e42d. Please review the new handoff behavior.

@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The security review completed, but no code-review approval appeared. Please run the code review for f5674e42d.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5674e42de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The CI unit test now matches the queued terminal continuation behavior in 07eba9212. Please review the latest commit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07eba9212c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
@chatgpt-codex-connector

This comment has been minimized.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed PRRT_kwDOPxxmWM6bQkb8 in 3da714e8c.

The terminal attention record is now persisted before the background continuation starts. The onAccepted callback marks it delivered before stream startup continues. This prevents the fast stream-end from missing the record and causing a later duplicate drain.

Added a regression test that invokes onAccepted during sendMessage and verifies no pending attention remains.

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The final attention-ordering finding is addressed in 3da714e8c. Please review the latest commit.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3da714e8cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the current Codex findings in a5f36592d.

  • Clear queued progress for every finalized group member: cleanup now covers every finalization.taskIds entry.
  • Hide the reservation from concurrent attention drains: owner-scoped notification claims now prevent duplicate delivery.
  • Re-arm attention when accepted stream startup fails: all accepted failure paths restore pending attention and schedule retry.

The fix also preserves delivery callbacks after correlation removal.

Stable child generations now use generation-specific terminal queue keys.

Local validation passed:

  • bun test src/node/services/taskService.test.ts
  • bun test src/node/services/workspaceService.test.ts
  • bun test src/node/services/messageQueue.test.ts
  • make static-check

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review commit a5f36592d.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5f36592d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/messageQueue.ts Outdated
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed Return a removal result even without callbacks in a64261fa1.

MessageQueue.removeWorkspaceTurn() now returns an empty callback object after a successful callback-free removal.

AgentSession.removeQueuedWorkspaceTurn() therefore emits the queue change, updates the background queue flag, and returns true.

Added a regression for a workspace-turn entry with no callbacks.

Local validation passed:

  • bun test src/node/services/messageQueue.test.ts
  • bun run tsgo --noEmit -p tsconfig.main.json
  • make static-check

@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review commit a64261fa1.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: a64261fa1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

This comment has been minimized.

@coadler
coadler added this pull request to the merge queue Aug 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 21, 2026
@coadler
coadler added this pull request to the merge queue Aug 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 21, 2026
@coadler

coadler commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

All findings are resolved and local validation passes. Please run the code review for 0e8fde0ff.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e8fde0fff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The latest commits address all reported continuation and restart-recovery races.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: f4616ece24

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4616ece24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts Outdated
Comment thread src/node/services/taskService.ts Outdated
@coadler
coadler force-pushed the fix/workspace-turn-false-interruptions branch from f4616ec to 10023aa Compare September 1, 2026 20:33
@coadler coadler changed the title [task-service] 🤖 fix: prevent false workspace turn interruptions [task-service] 🤖 fix: correlate workspace-turn liveness Sep 1, 2026
@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

The branch was rebuilt from current main. The previous broad diff is replaced by a focused liveness and capacity fix.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T22:26:40.889431Z 3f190cc Manual request
🔒 Security Review Completed 2026-09-01T22:25:38.480189Z 3f190cc Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

The branch was rebuilt from current main. These three findings refer to removed taskService code and are no longer present in the diff:

  • PRRT_kwDOPxxmWM6bVDyl: fallback delivery callbacks were removed with the old implementation.
  • PRRT_kwDOPxxmWM6bVDyo: deferred monitor-wake settlement logic was removed with the old implementation.
  • PRRT_kwDOPxxmWM6bVDyp: inferred interruption markers were removed with the old implementation.

The replacement diff only changes correlation-aware liveness and stale-handle capacity accounting.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 10023aa1ac

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10023aa1ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/workspaceTurnManager.ts Outdated
Comment thread src/node/services/workspaceTurnManager.ts Outdated
Mux added 3 commits September 1, 2026 15:46
Use exact stream and queued-turn correlation before slot accounting.
Keep busy-workspace admission tests aligned with exact handle liveness.
@coadler
coadler force-pushed the fix/workspace-turn-false-interruptions branch from 10023aa to 3c1d37b Compare September 1, 2026 20:46
@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Rebased onto current main at bf97dfd. Local workspace-turn tests and make static-check pass.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 3c1d37bb14

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c1d37bb14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/workspaceTurnManager.ts Outdated
Comment thread src/node/services/workspaceTurnManager.ts
@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed all four findings in 3f22e12:

  • PRRT_kwDOPxxmWM6eRQF5: stale disposable ownership is relinquished during unrelated activity. Settlement no longer removes the active workspace.
  • PRRT_kwDOPxxmWM6eRQF_: full-queue hasQueuedWorkspaceTurn matching keeps later queued turns live.
  • PRRT_kwDOPxxmWM6eRmcX: mock streams without StreamInfo remain ambiguous and use the active-handle fallback.
  • PRRT_kwDOPxxmWM6eRmca: active compaction requests recover correlation from followUpContent.workspaceTurnMetadata.

@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

All current findings are addressed. The full workspace-turn manager suite and make static-check pass.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 3f22e1209f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f22e1209f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/workspaceTurnManager.ts Outdated
Comment thread src/node/services/workspaceTurnManager.ts Outdated
Comment thread src/node/services/workspaceTurnManager.ts Outdated
Comment thread src/node/services/workspaceTurnManager.ts Outdated
Correlate compaction, retry, wake, queue, and send-preflight activity.

Carry workspace ownership evidence through stale settlement.

---

_Generated with `xum` • Model: `openai:gpt-5.6-sol` • Thinking: `xhigh` • Cost: `$0.19`_

<!-- mux-attribution: model=openai:gpt-5.6-sol thinking=xhigh costs=0.19 -->
@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed all four findings in 3f190cc89.

  • PRRT_kwDOPxxmWM6eSHhI: The compaction parser now reads direct followUpContent.muxMetadata and inherited workspaceTurnMetadata.
  • PRRT_kwDOPxxmWM6eSHhO: The liveness result now carries unrelated activity into settlement. A fresh correlated continuation still cancels settlement.
  • PRRT_kwDOPxxmWM6eSHhV: A scoped creation reservation keeps a persisted queued handle live through sendMessage preflight.
  • PRRT_kwDOPxxmWM6eSHhY: Retry and monitor-wake flags now require the matching active handle registration. Unrelated flags settle stale handles.

@coadler

coadler commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review commit 3f190cc89.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 3f190cc897

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f190cc897

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +3888 to +3891
const hasUncorrelatedCreationReservation =
creationReservations != null &&
creationReservations.size > 0 &&
!hasCorrelatedCreationReservation;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep reservations from superseding prior turns

When handle A is live only through its active registration—for example, after a deferred stream-end while descendant work remains—and follow-up B enters send preflight, B's reservation makes hasUncorrelatedCreationReservation true for A. A concurrent task-list or capacity scan therefore terminally interrupts A before B reaches queue admission; if B's send subsequently fails, no replacement turn ever existed and A's waiter/result has already been lost. Creation reservations should keep their own handle live without becoming supersession evidence for other handles until the new send is admitted.

AGENTS.md reference: AGENTS.md:L150-L150

Useful? React with 👍 / 👎.

Comment on lines +3902 to +3903
const hasUncorrelatedRetryOrWake =
!hasActiveRegistration && (hasPendingAutoRetry || hasPendingBashMonitorWake);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep restart retries live before registration

During startup recovery of a non-task workspace, WorkspaceService.initialize() can schedule the correlated auto-retry before any process-local active registration is restored. While that retry is pending, this condition classifies it as uncorrelated solely because hasActiveRegistration is false, so a task-list or capacity scan settles the durable handle as interrupted and rejects its waiter even though the same turn is about to resume. Pending restart retries need correlation from their preserved message metadata, or an equivalent registration established before liveness checks.

AGENTS.md reference: AGENTS.md:L112-L112

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant