Repository navigation
fix(parser): limit the size of decompressed gzip data - #5986
Merged
Merged
Conversation
Gzip can expand data by about 1,000x, so a payload well under Lambda's limits could decompress to more memory than the function has and kill the runtime before parser or JMESPath could report anything. Parser and JMESPath now decompress at most 16 MiB, configurable in bytes with POWERTOOLS_MAX_DECOMPRESSED_SIZE. Data that expands past the limit fails like any other invalid input. Closes #5977
sdangol
approved these changes
Nov 16, 2026
dreamorosi
enabled auto-merge (squash)
November 16, 2026 15:53
dreamorosi
added a commit
that referenced
this pull request
Nov 16, 2026
…limit After the size limit from #5986, the over-limit case throws a FunctionEvaluationError instead of evaluating to undefined.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Parser and JMESPath decompressed gzip data from events without an output size limit. Gzip can expand data roughly 1,000×, so a payload well within Lambda's limits could use up the function's memory and kill the runtime before parser or the error handler could report anything. This PR limits decompressed output to 16 MiB by default, and users can change the limit with
POWERTOOLS_MAX_DECOMPRESSED_SIZE.getMaxDecompressedSize()to@aws-lambda-powertools/commons/utils/env. It readsPOWERTOOLS_MAX_DECOMPRESSED_SIZEat call time and falls back to 16 MiB when the value isn't a positive integer, like the other commons env helpers.Base64Encoded, used by the built-in Kinesis schemas and envelopes and the batchbase64transformer, andCloudWatchLogsSchemanow passmaxOutputLengthtogunzipSync. Data over the limit fails as a normal parse error:Decompressed data exceeds the maximum size of <limit> bytes, set POWERTOOLS_MAX_DECOMPRESSED_SIZE to change the limit. Other decompression errors behave as before. The issue didn't listCloudWatchLogsSchema, which decompresses gzip with its own code.powertools_base64_gzippassesmaxOutputLengthtoo. Data over the limit is handled like corrupt gzip input already is, and the expression evaluates toundefined.vi.stubEnv. Coverage stays at 100% in commons, parser and jmespath. With the issue's repro (320 MiB decompressed,--max-old-space-size=256),safeParsenow returnssuccess: false, where before the process ran out of memory.docs/environment-variables.md, and one sentence each in the Parser and JMESPath docs.Behaviour change: gzip data that expands past 16 MiB is now rejected. Users who expect bigger payloads can raise the limit with
POWERTOOLS_MAX_DECOMPRESSED_SIZE.Issue number: closes #5977
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.