Perf: avoid repeated orchestration-history scans for tracing by berndverst · Pull Request #788 · microsoft/durabletask-dotnet

Bernd Verst (berndverst) · 2026-07-24T22:16:41Z

Summary

OnRunOrchestratorAsync in src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs scanned orchestration history to correlate tracing events unconditionally, and rescanned the full past-events list for every qualifying new event to find the originating TaskScheduled/SubOrchestrationInstanceCreated event — O(new events × past events), even when no tracing listener was registered.

This is a precise, internal-only performance fix:

Added TraceHelper.HasListeners, a cheap check backed by ActivitySource.HasListeners(), and gated all tracing-correlation work in OnRunOrchestratorAsync behind it, so nothing is scanned when no listener is registered.
Replaced the LINQ Concat-based scan for the ExecutionStarted event with a single-pass local function.
Added TraceHistoryEventLookup (src/Shared/Grpc/Tracing/TraceHistoryEventLookup.cs), which builds one dictionary index per work item (lazily, at most once per event type) for O(1) repeated lookups instead of rescanning PastEvents per new event, while preserving the exact original first-wins (SubOrchestrationInstanceCreated) and last-wins (TaskScheduled) semantics for duplicate event IDs.

No public API changes. Trace output, error handling, and orchestration replay determinism are unaffected — this only changes how existing history is looked up, not what is looked up or emitted.

Tests

test/Worker/Grpc.Tests/TraceHistoryEventLookupTests.cs (new): direct unit tests for TraceHistoryEventLookup covering duplicate-event-ID first/last-wins semantics, no-match, event-type filtering, and empty input.
test/Worker/Grpc.Tests/GrpcDurableTaskWorkerTests.cs (added):
- A regression test asserting the no-listener fast path completes work items normally without performing trace-event lookups.
- An end-to-end test with duplicate event IDs in history that registers a real ActivityListener and asserts the resulting activities reflect the original first/last-wins correlation semantics.
Full Worker.Grpc.Tests suite passes (145/145).
Worker.Grpc builds cleanly across all target frameworks (netstandard2.0;net6.0;net8.0;net10.0).

Fixes #769

OnRunOrchestratorAsync previously scanned orchestration history to correlate tracing events unconditionally, and for every qualifying new event it rescanned the full past-events list to find the originating TaskScheduled/SubOrchestrationInstanceCreated event - O(new events x past events) work, even when no tracing listener was registered. - Add TraceHelper.HasListeners, a cheap check backed by ActivitySource.HasListeners(), and gate all tracing-correlation work in OnRunOrchestratorAsync behind it so nothing is scanned when no listener is registered. - Replace the LINQ Concat-based scan for the ExecutionStarted event with a single-pass local function. - Add TraceHistoryEventLookup, which builds one dictionary index per work item (built lazily, once) for O(1) repeated lookups instead of rescanning PastEvents per new event, while preserving the exact original first-wins (SubOrchestrationInstanceCreated) and last-wins (TaskScheduled) semantics for duplicate event IDs. No public API changes; trace output, error handling, and replay determinism are unaffected. Adds regression tests for the no-listener fast path, first/last-wins correlation semantics end-to-end, and direct unit tests for TraceHistoryEventLookup. Fixes #769 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c538f3e-308d-48a3-af7f-b3baf90e97b2

Copilot

Pull request overview

Improves the gRPC worker’s tracing correlation performance by avoiding unnecessary orchestration-history scans when tracing is disabled and by indexing past history events for O(1) correlation lookups when tracing is enabled. This aligns with the SDK’s goal of keeping worker-side orchestration processing efficient, especially for long-running instances with large histories.

Changes:

Added a cheap listener check (TraceHelper.HasListeners) and gated trace-correlation work in OnRunOrchestratorAsync behind it.
Replaced repeated past-history scans with a per-work-item indexed lookup (TraceHistoryEventLookup) that preserves prior first/last-wins semantics for duplicate event IDs.
Added unit and worker-level tests covering listener/no-listener behavior and duplicate-ID correlation semantics.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/Worker/Grpc.Tests/TraceHistoryEventLookupTests.cs	Adds focused unit tests for the new indexed lookup, including duplicate-ID first/last-wins semantics and filtering by event type.
test/Worker/Grpc.Tests/GrpcDurableTaskWorkerTests.cs	Adds end-to-end worker tests for the no-listener fast path and for correlation semantics under a real `ActivityListener`.
src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs	Gates tracing correlation behind `HasListeners`, replaces LINQ concat scan with a single-pass helper, and uses the indexed lookup for repeated correlation.
src/Shared/Grpc/Tracing/TraceHistoryEventLookup.cs	Introduces a lazily-built, cached dictionary index for past events to avoid O(N×M) repeated scans.
src/Shared/Grpc/Tracing/TraceHelper.cs	Exposes `HasListeners` backed by `ActivitySource.HasListeners()` to allow callers to skip trace-correlation work when tracing is disabled.

…history-scan-perf

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Bernd Verst (berndverst) · 2026-08-01T05:33:14Z

Quantitative performance impact

I compared live head d223fc2 with its merge base in a focused Release/.NET 10 harness on Windows x64 (AMD EPYC 7763). It used the actual TraceHistoryEventLookup/TraceHelper.HasListeners code and the exact old lookup algorithm. Activity creation, the rest of worker execution, and gRPC were excluded, so these are correlation-stage measurements rather than fleet-wide orchestration averages.

Let H be past-event count and N_t/N_s qualifying task/sub-orchestration completion counts.

Path	Old past-event inspections	New work
No tracing listener	execution-start search + up to `N_t*H + sum(sub match positions)`	one `HasListeners()` check; zero history enumeration
Listener, task completions	`N_t*H`	one `H`-event task index + `N_t` dictionary lookups
Listener, sub-orchestration completions	`sum(sub match positions)`	one `H`-event sub index + `N_s` dictionary lookups

Representative, best, and adverse cases

“Representative” below means a selected ordinary shape, not production telemetry.

Case	Listener / shape	Correlation latency, old -> new	Allocation/work item	Interpretation
Average-style representative	Off; `H=100`, 5 task completions	4.40 us -> ~6 ns	1,056 -> 0 B	The history work is eliminated; the new value is near harness overhead, so the ratio is not meaningful
Listener-on ordinary task case	`H=100`, 5 task completions	4.38 -> 2.00 us (-54.3%)	1,056 -> 4,704 B (+3,648 B)	Faster CPU path, higher temporary allocation
Listener-on ordinary mixed case	`H=100`, 3 task + 3 early sub matches	2.55 -> 3.43 us (~+34.5%)	1,024 -> 4,280 B (+3,256 B)	Small/early-match histories can regress
Best measured large replay	Off; `H=10,000`, 1,000 task completions	79.8 ms -> near harness overhead	160,256 -> 0 B	Removes the entire quadratic correlation scan
Listener-on large fan-out	`H=10,000`, 1,000 task completions	80.46 ms -> 333.5 us (-99.585%, ~241x)	160,256 -> 451,592 B (+291,336 B)	Large CPU/latency gain traded for a temporary index
Listener-on mixed fan-out	`H=10,000`, 500 task + 500 sub completions	36.98 ms -> 490.7 us (-98.673%, ~75x)	128,256 -> 431,438 B (+303,182 B)	Same tradeoff with two lazy indexes
Constructed adverse case	Listener on; one sub match at position 2 of `H=10,000`	166 ns -> 95.7 us (~576x as long)	352 -> 296 B	Old `FirstOrDefault` stops at event 2; new code builds the full lazy index

The no-listener ~6 ns figures are close to benchmark-harness cost and should be read as “the scan was removed,” not as a credible millions-of-operations throughput claim.

Memory and GC

The lazy indexes reference existing events; they do not copy protobuf messages. On this x64 runtime, estimated incremental dictionary storage while live is approximately 28-56 B per unique indexed ID, including capacity slack/array headers but excluding the existing event objects.

50 task entries: about 2.7 KiB retained, 4.7 KiB cumulatively allocated
5,000 task entries: about 236 KiB retained, 452 KiB allocated
Two 3,333-entry indexes: about 227 KiB retained, 431 KiB allocated

Thus:

No listener: both scan CPU and correlation allocations fall to zero.
Listener + repeated lookups: CPU falls from O(N*H) to O(H+N), but temporary memory becomes O(unique history IDs).
Listener + one early sub lookup: CPU can regress from O(1) expected early exit to O(H) index construction.
A listener that samples every activity out still makes HasListeners() true and pays indexing costs.

Whole-worker throughput interpretation

Correlation-stage speedup does not translate directly to orchestration throughput. Using Amdahl's law:

If the old correlation work is 10% of turn CPU and the new path removes it, modeled turn throughput improves by about 11.1%.
If a large fan-out replay spends 50% of turn CPU in old correlation scans and that work becomes negligible, the modeled ceiling is about 2x turn throughput.

Those are sensitivity examples. Actual gains require telemetry for listener prevalence and the distributions of H, qualifying new events, event mix, and match positions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01dc5b7c-c742-45b7-ae8c-5cda9bcfbd2c

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

sophiatev · 2026-08-11T23:40:57Z

+/// events (e.g. "TaskCompleted") back to the history event that scheduled them (e.g. "TaskScheduled").
+/// </summary>
+/// <remarks>
+/// The indexes are built lazily, at most once per instance, and cached for the lifetime of the instance. This


Can we modify this comment? It's misleading because the index is not cached for the lifetime of the instance, but rather just for the lifetime of the workitem for the instance. The instance could have multiple work items, in which case we recreate this map, right?

So maybe just changing to "cached for the lifetime of the instance work item" would be more accurate

sophiatev · 2026-08-11T23:41:31Z

+    /// <param name="eventId">The event ID to look up.</param>
+    /// <returns>The matching event, or <see langword="null"/> if none is found.</returns>
+    /// <remarks>
+    /// If more than one "TaskScheduled" event shares the given event ID, the last one encountered (in history


We should actually throw an exception in this case, this situation should be impossible. It should be 1:1

sophiatev · 2026-08-11T23:41:49Z

+    /// <param name="eventId">The event ID to look up.</param>
+    /// <returns>The matching event, or <see langword="null"/> if none is found.</returns>
+    /// <remarks>
+    /// If more than one "SubOrchestrationInstanceCreated" event shares the given event ID, the first one


sophiatev · 2026-08-11T23:44:15Z

+    }
+
+    [Fact]
+    public async Task DispatchWorkItem_OrchestratorRequest_WithActivityListener_UsesFirstAndLastWinsSemantics()


As per the comment in TraceHistoryEventLookup, we actually want to throw an exceptoin in this case, so let's update the test appropriately

sophiatev · 2026-08-11T23:44:27Z

+    // TraceHelper, so they are kept in this class (whose test methods xunit runs sequentially by default) to
+    // avoid flaky interference between them.
+    [Fact]
+    public async Task DispatchWorkItem_OrchestratorRequest_NoActivityListeners_SkipsTracingWorkAndCompletes()


Can we confirm that the TraceHistoryEventLookup methods are not invoked in this case?

sophiatev · 2026-08-11T23:44:36Z

Can we add any integration tests for this?

sophiatev · 2026-08-11T23:45:31Z

+public class TraceHistoryEventLookupTests
+{
+    [Fact]
+    public void GetTaskScheduledEvent_DuplicateEventIds_ReturnsLastMatch()


Same comment about new semantics

sophiatev · 2026-08-11T23:45:37Z

+    }
+
+    [Fact]
+    public void GetSubOrchestrationInstanceCreatedEvent_DuplicateEventIds_ReturnsFirstMatch()


Copilot AI lite review requested due to automatic review settings July 24, 2026 22:16

Copilot started reviewing on behalf of Bernd Verst (berndverst) July 24, 2026 22:18 View session

github-code-quality Bot found potential problems Jul 24, 2026

View reviewed changes

Comment thread src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into berndverst-fix-trace-…

d223fc2

…history-scan-perf

Copilot AI review requested due to automatic review settings July 27, 2026 18:37

Copilot started reviewing on behalf of Bernd Verst (berndverst) July 27, 2026 18:37 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Merge origin/main into PR #788

d20bd2c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01dc5b7c-c742-45b7-ae8c-5cda9bcfbd2c

Copilot AI review requested due to automatic review settings August 4, 2026 19:50

Copilot started reviewing on behalf of Bernd Verst (berndverst) August 4, 2026 20:00 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

sophiatev reviewed Aug 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

Perf: avoid repeated orchestration-history scans for tracing - #788

Perf: avoid repeated orchestration-history scans for tracing#788
Bernd Verst (berndverst) wants to merge 3 commits into
mainfrom
berndverst-fix-trace-history-scan-perf

Bernd Verst (berndverst) commented Jul 24, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Bernd Verst (berndverst) commented Aug 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

sophiatev Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Bernd Verst (berndverst) commented Jul 24, 2026

Summary

Tests

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Bernd Verst (berndverst) commented Aug 1, 2026

Quantitative performance impact

Representative, best, and adverse cases

Memory and GC

Whole-worker throughput interpretation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants