Skip to content

fix(compress): pin utf-8 and newline on file I/O - #763

Open
francojeferson wants to merge 1 commit into
JuliusBrussee:mainfrom
francojeferson:fix/compress-windows-text-mode-io
Open

fix(compress): pin utf-8 and newline on file I/O#763
francojeferson wants to merge 1 commit into
JuliusBrussee:mainfrom
francojeferson:fix/compress-windows-text-mode-io

Conversation

@francojeferson

Copy link
Copy Markdown

Fixes #762. Also covers #686 (and #652, #655, #533).

What

Path.read_text / Path.write_text default to the locale encoding and to CRLF translation on Windows. Both defaults mutate the user's file.

Encoding. Non-ASCII output goes out in the locale codec. An em dash becomes a bare 0x97 under cp1252, so the file stops being valid UTF-8. validate.py then read it back through the same default, saw a correct em dash, and printed Validation passed — the round-trip is self-consistent and wrong, so the corruption ships silently.

Line endings. Every processed file comes back CRLF. Measured on a real run: a 67-line LF file restored from backup at +67 bytes, one CR per line. git diff then reports every line changed.

Overlap with #683

#683 already fixes the encoding half, and this PR contains that same change — the two touch identical call sites, so splitting them would just conflict. What this adds on top:

Happy to rebase down to newline-only if #683 lands first. No claim on that work intended.

Notes

  • Reads go through open() rather than read_text(newline=...). That parameter is 3.13+ and CLAUDE.md states the skill supports Python 3.10+.
  • validate.py deliberately does not get newline="". The validator needs universal-newline normalization so a CRLF original and an LF candidate still compare equal — pinning it there would make validate_code_blocks fail on every CRLF file.
  • The no-op comparison needed normalizing too. The body now carries the source's CRLF while Claude always answers LF, so compressed_body.strip() == body.strip() would never match and the "output identical to input" guard would stop firing on CRLF files entirely. The existing test_identical_compressed_output_does_not_touch_disk caught this — it is green again.

Tests

New tests/test_compress_encoding.py, 13 cases: match_line_endings unit tests, read_exact round-trips, and end-to-end through compress_file with call_claude stubbed, so no API key is needed.

python3 -m unittest tests.test_compress_encoding   # Ran 13 tests, OK
python3 -m unittest tests.test_compress_safety     # Ran 5 tests, OK

tests.test_hooks reports 3 errors and npm test 2 failures on my Windows box. Both are identical on a clean checkout of main and unrelated to this change.

CI will resync plugins/caveman/skills/caveman-compress/scripts/ on merge.

Path.read_text / Path.write_text default to the locale encoding and to
CRLF translation on Windows. Two user-visible consequences:

- Non-ASCII output is written in the locale codec. An em dash lands as a
  bare 0x97 under cp1252, so the file stops being valid UTF-8. validate.py
  read it back through the same default and reported success, because the
  round-trip is self-consistent and wrong.
- Every processed file gets its line endings rewritten to CRLF. A 67-line
  LF file came back 67 bytes larger.

Pin encoding="utf-8" and newline="" on every read and write, then restore
the source file's line endings on the way out since Claude always answers
in LF. Reads go through open() rather than read_text(newline=...) because
that parameter is 3.13+ and the skill supports 3.10+.

Also normalize both sides of the no-op comparison. The body now carries
the source's CRLF while Claude answers LF, so a raw compare would never
match and the "output identical to input" guard would stop firing on CRLF
files entirely.

detect.py gets the same encoding fix. It only classifies, so it cannot
corrupt a file, but on a non-UTF-8 locale it can misread prose as code and
silently skip it.

Refs JuliusBrussee#762, JuliusBrussee#686
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

caveman-compress rewrites every line ending to CRLF on Windows - survives the #683 encoding fix

1 participant