Skip to content

fix(parser): limit the size of decompressed gzip data - #5986

Merged
dreamorosi merged 2 commits into
mainfrom
fix/gzip-decompression-size-limit
Nov 16, 2026
Merged

dreamorosi merged 2 commits into
mainfrom
fix/gzip-decompression-size-limit

Conversation

@dreamorosi

Copy link
Copy Markdown
Contributor

Summary

Changes

Please provide a summary of what's being changed

Parser and JMESPath decompressed gzip data from events without an output size limit. Gzip can expand data roughly 1,000×, so a payload well within Lambda's limits could use up the function's memory and kill the runtime before parser or the error handler could report anything. This PR limits decompressed output to 16 MiB by default, and users can change the limit with POWERTOOLS_MAX_DECOMPRESSED_SIZE.

  • commons: added getMaxDecompressedSize() to @aws-lambda-powertools/commons/utils/env. It reads POWERTOOLS_MAX_DECOMPRESSED_SIZE at call time and falls back to 16 MiB when the value isn't a positive integer, like the other commons env helpers.
  • parser: Base64Encoded, used by the built-in Kinesis schemas and envelopes and the batch base64 transformer, and CloudWatchLogsSchema now pass maxOutputLength to gunzipSync. Data over the limit fails as a normal parse error: Decompressed data exceeds the maximum size of <limit> bytes, set POWERTOOLS_MAX_DECOMPRESSED_SIZE to change the limit. Other decompression errors behave as before. The issue didn't list CloudWatchLogsSchema, which decompresses gzip with its own code.
  • jmespath: powertools_base64_gzip passes maxOutputLength too. Data over the limit is handled like corrupt gzip input already is, and the expression evaluates to undefined.
  • Tests use a small limit through vi.stubEnv. Coverage stays at 100% in commons, parser and jmespath. With the issue's repro (320 MiB decompressed, --max-old-space-size=256), safeParse now returns success: false, where before the process ran out of memory.
  • Docs: one row in docs/environment-variables.md, and one sentence each in the Parser and JMESPath docs.

Behaviour change: gzip data that expands past 16 MiB is now rejected. Users who expect bigger payloads can raise the limit with POWERTOOLS_MAX_DECOMPRESSED_SIZE.

Issue number: closes #5977


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Gzip can expand data by about 1,000x, so a payload well under Lambda's limits
could decompress to more memory than the function has and kill the runtime
before parser or JMESPath could report anything.

Parser and JMESPath now decompress at most 16 MiB, configurable in bytes with
POWERTOOLS_MAX_DECOMPRESSED_SIZE. Data that expands past the limit fails like
any other invalid input.

Closes #5977
@dreamorosi dreamorosi self-assigned this Nov 16, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/L PRs between 100-499 LOC label Nov 16, 2026
@dreamorosi
dreamorosi enabled auto-merge (squash) November 16, 2026 15:53
@dreamorosi
dreamorosi merged commit 7ac0769 into main Nov 16, 2026
43 checks passed
@dreamorosi
dreamorosi deleted the fix/gzip-decompression-size-limit branch November 16, 2026 15:55
dreamorosi added a commit that referenced this pull request Nov 16, 2026
…limit

After the size limit from #5986, the over-limit case throws a FunctionEvaluationError instead of evaluating to undefined.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L PRs between 100-499 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: gzip decompression in parser and JMESPath has no output size limit

2 participants