Skip to content

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation - #7050

Open
pabloDeputter wants to merge 19 commits into
masterfrom
pablo/fix-botocore-sigv4-trace-propagation
Open

fix(boto3): Fix botocore SigV4 failures caused by post-sign trace propagation#7050
pabloDeputter wants to merge 19 commits into
masterfrom
pablo/fix-botocore-sigv4-trace-propagation

Conversation

@pabloDeputter

@pabloDeputter pabloDeputter commented Aug 5, 2026

Copy link
Copy Markdown
Member

Description

Summary of issue

  • pre-4.12.0 ddtrace basically behaved like Sentry before: propagation headers were added by the HTTP client after botocore already had calculated the SigV4 signature. So baggage was not included in SignedHeaders. Any later modifications to the value did not invalidate the request.
  • 4.12.0 moves the header injection to botocore's before-sign event. It adds baggage, ... and x-datadog-* before signing. Any later modifications to the value DO invalidate the request, thus later HTTP-client injection is suppressed to avoid duplicate headers.
  • How failure happens:
    1. incoming FastAPI request contains Sentry or other W3C baggage
    2. datadog extracts that baggage into its active context
    3. boto3 request is created
    4. datadog’s before-sign handler writes the baggage to the AWS request
    5. botocore includes baggage in the SigV4 signature
    6. sentry’s httplib runs afterwards and updates or adds another baggage value
    7. AWS receives a different baggage value from the one that was signed. Because request was "tampered" with after signing, AWS rejects the request with 403 Forbidden or SignatureDoesNotMatch.

Changes

  • inject sentry propagation through before-sign handler, so final baggage and sentry-trace values are created before SigV4 signing.
  • existing third-party baggage is merged into one header.
  • http.client propagation is delayed until endheaders(), when the complete request headers and SigV4 SignedHeaders are available. Existing baggage header is never mutated after it already was signed.
  • propagation is skipped for presigned requests, since adding headers could require a future caller to reproduce these exact headers.

Issues

Resolves: #7031 & PY-2667
Related issues in dd-trace-py: #19477 & #19358

Reminders

- merge Sentry baggage with existing vendor (e.g. Datadog) baggage in botocore's`before-sign` hook; avoiding post-sign header tampering that invalidates the SigV4 signature.
- Skip propagation for presigned requests

Fixes: #7031 & PY-2667
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codecov Results 📊

104653 passed | ⏭️ 6692 skipped | Total: 111345 | Pass Rate: 93.99% | Execution Time: 364m 10s

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +266
Passed Tests 📈 +251
Failed Tests
Skipped Tests 📈 +15

All tests are passing successfully.

✅ Patch coverage is 95.92%. Project has 2486 uncovered lines.
✅ Project coverage is 90.16%. Comparing base (base) to head (head).

Files with missing lines (3)
File Patch % Lines
sentry_sdk/utils.py 93.94% ⚠️ 2 Missing and 4 partials
sentry_sdk/integrations/boto3.py 96.15% ⚠️ 1 Missing and 1 partials
sentry_sdk/integrations/stdlib.py 97.22% ⚠️ 1 Missing and 1 partials
Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
+ Coverage    90.14%    90.16%    +0.02%
==========================================
  Files          193       193         —
  Lines        25183     25274       +91
  Branches      9176      9224       +48
==========================================
+ Hits         22700     22788       +88
- Misses        2483      2486        +3
- Partials      1429      1434        +5

Generated by Codecov Action

- trace-header injection is delayed until `endheaders()` after all request headers are available.
- avoid duplicate propagation headers.
- no tampering/modifying of already signed baggage

Fixes: #7031 & PY-2667
@pabloDeputter
pabloDeputter marked this pull request as ready for review August 6, 2026 11:12
@pabloDeputter
pabloDeputter requested a review from a team as a code owner August 6, 2026 11:12
Comment thread sentry_sdk/integrations/stdlib.py Outdated
Comment thread sentry_sdk/integrations/stdlib.py Outdated
return rv

def endheaders(self: "HTTPConnection", *args: "Any", **kwargs: "Any") -> "Any":
trace_headers = getattr(self, "_sentrysdk_trace_headers", ())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does calling sentry_sdk.get_current_scope().iter_trace_propagation_headers() here instead of in putrequest() work?
It would be best to avoid stashing stuff on the HTTPConnection instance if we can help it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion :) header generation is moved into endheaders() instead of being stored on the connection.

Comment thread sentry_sdk/integrations/stdlib.py Outdated
Comment thread sentry_sdk/integrations/stdlib.py
@alexander-alderman-webb

Copy link
Copy Markdown
Contributor

I haven't forgot about this, it's just complex so I'll likely only re-review fully at the start of next week.

…` + support for SigV4 query/presigned authentication

Refs: #7031 & PY-2667
Comment thread sentry_sdk/integrations/stdlib.py Outdated
Comment on lines +72 to +73
client.meta.events.register("before-sign", _inject_third_party_baggage)
client.meta.events.register_last(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boto3 event handlers leak across tests via shared default session

Event handlers registered on client.meta.events leak across tests because boto3.client() uses the global default session. Create a dedicated boto3.Session() per test, or unregister handlers afterwards.

Evidence
  • boto3.client() internally uses a global default Session, whose event_emitter is shared across all callers.
  • tests/integrations/boto3/test_s3.py avoids this exact pitfall by creating a dedicated boto3.Session() at module scope.
  • tests/integrations/boto3/aws_mock.py explicitly calls meta.events.unregister('before-send', self) on teardown.
  • The new test file never unregisters its before-sign handlers, so handlers from earlier parametrized runs or test functions persist in the shared emitter.

Identified by Warden · code-review · SV9-PJR

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boto3.client() reuses the global session, event emitters are copied. See below for example:

import boto3
c1 = boto3.client("dynamodb")
c2 = boto3.client("dynamodb")
calls = []

def handler(**kwargs):
    calls.append(kwargs)
    
c1.meta.events.register("before-sign", handler)
# handler registered on c1 is not called for c2.
c2.meta.events.emit("before-sign", request=None, signature_version="v4")

assert calls == []
assert c1.meta.events._emitter is not c2.meta.events._emitter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 HeadObject returns 403 when sentry_sdk.start_span and Datadog botocore instrumentation are both active

2 participants