Skip to content

Duplicate top-level keys blow up format detection: 32 KB file emits a 1.2 GB diagnostic #443

Description

@Wahbeh-Mohammad

What happens

A YAML spec with many duplicate top-level keys makes morphic validate either emit a gigantic
diagnostic or never finish.

Fixture Result on main (3d40d98)
32 KB, ~6.5k duplicate x: y lines 8.1 s, 1.2 GB on stderr
128 KB, ~26k duplicates no result after 150 s

Found 2026-09-08 while verifying #435. Not caused by that PR; it exists on main. #435 widens the
exposure from the 64 KiB prefix to the whole file, since detection there parses the entire
document when the prefix does not answer.

Why

  • Detect (compilers/openapi/detect.go) fully decodes the mapping with yaml.Unmarshal just to
    read the openapi / swagger key.
  • yaml.v3's duplicate-key error is quadratic in size. Measured in isolation:
    204 duplicates → 1.1 MB error string; 819 → 18.7 MB; 1,638 → 76 MB (each doubling ×4).
  • diag.OneLine (compilers/openapi/internal/diag/diag.go) copies the entire parser error into the
    openapi/undecodable-source diagnostic message, with no length cap. The CLI then prints it.

Fix ideas

  1. Cap the length of a parser error quoted into a diagnostic (first line, or first N bytes, plus "…").
  2. Do not decode the whole mapping in detection at all; scan for the top-level key and read its value.
    This is the same change the fix(compilers/openapi): detect a version key past the sniff cap #435 review asks for.
  3. Optionally, an explicit duplicate-top-level-key check before parsing, reported once.

Reproduce

python3 -c "print('openapi: 3.0.3\ninfo: {title: t, version: v}\npaths: {}\n' + 'x: y\n'*6553)" > dup-32k.yaml
morphic validate dup-32k.yaml 2> err.txt; ls -l err.txt   # ~1.2 GB

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugDefect in shipped behavior

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions