caveman-compress rewrites every line ending to CRLF on Windows — survives the #683 encoding fix

Scope

This is not the encoding bug. That one is #686 (and #652, #655, #533), with #683 and #678 open against it. This is a second, independent defect at the same call sites that none of those touch: after #683 merges, the compressor still rewrites the line endings of every file it processes.

Filing separately so it doesn't get closed along with the encoding work.

The bug

Path.write_text() opens in text mode. On Windows that means newline=None, which translates every \n to \r\n on the way out. An LF-ending source file comes back CRLF, so the next git diff reports every line changed.

Measured on a 67-line LF file:

before compress   7015 bytes
after  restore    7082 bytes    # +67, one CR per line

Adding encoding="utf-8" as #683 does has no effect on this — encoding and newline translation are independent parameters.

Why newline="\n" on the writes is not the fix

The obvious patch is wrong in the other direction. Reads go through read_text(), which applies universal-newline translation, so original_text is always LF in memory regardless of what is on disk. Forcing newline="\n" on the writes would then flatten genuine CRLF files to LF.

Both sides have to be pinned, and separately, the LLM's output (always LF) has to be matched back to whatever the source actually used.

Reproduction

No API key needed — stub call_claude and run the real compress_file:

import sys
from pathlib import Path
sys.path.insert(0, "<plugin>/skills/caveman-compress")
from scripts import compress as C

C.call_claude = lambda prompt: "Plain prose, no heading, no code.\nSecond line.\n"

for label, eol in (("lf", "\n"), ("crlf", "\r\n")):
    src = Path(f"probe_{label}.md")
    raw = f"Plain prose file, no heading, no code.{eol}Second line here.{eol}".encode()
    src.write_bytes(raw)
    C.backup_dir_for(src).joinpath(src.stem + ".original.md").unlink(missing_ok=True)
    C.compress_file(src)
    out = src.read_bytes()
    print(label, "output keeps source EOL:", eol.encode() in out and
          (b"\r\n" in out) == (eol == "\r\n"))
    print(label, "backup byte-identical:  ",
          C.backup_dir_for(src).joinpath(src.stem + ".original.md").read_bytes() == raw)

On Windows before the patch: the LF case comes back CRLF, and the backup is not byte-identical to the source. On Linux/macOS both pass, which is why this has gone unnoticed alongside the encoding bug.

Patch

Applies on top of #683. Tested against both LF and CRLF sources.

     m = FRONTMATTER_REGEX.match(text)
     if m:
         return m.group(1), m.group(2)
     return "", text
 
+
+def match_line_endings(text: str, reference: str) -> str:
+    """Rewrite text's line endings to match the reference document's.
+
+    File I/O below pins ``newline=""`` so Python performs no translation of
+    its own. The LLM always returns LF, so a CRLF source still needs its
+    endings restored explicitly.
+    """
+    crlf = reference.count("\r\n")
+    lf = reference.count("\n") - crlf
+    normalized = text.replace("\r\n", "\n")
+    if crlf > lf:
+        return normalized.replace("\n", "\r\n")
+    return normalized
+
+
+def read_exact(path: Path) -> str:
+    """Read text as UTF-8 with no newline translation.
+
+    ``Path.read_text`` only grew a ``newline`` parameter in 3.13, so go through
+    ``open`` to keep this working on older interpreters.
+    """
+    with path.open(encoding="utf-8", errors="ignore", newline="") as fh:
+        return fh.read()
+
+
 # Filenames and paths that almost certainly hold secrets or PII.

-    original_text = filepath.read_text(encoding="utf-8", errors="ignore")
+    original_text = read_exact(filepath)

-    compressed = frontmatter + compressed_body
+    compressed = match_line_endings(frontmatter + compressed_body, original_text)

     backup_path.write_text(original_text, encoding="utf-8", newline="")
-    backup_readback = backup_path.read_text(encoding="utf-8", errors="ignore")
+    backup_readback = read_exact(backup_path)

-    filepath.write_text(compressed, encoding="utf-8")
+    filepath.write_text(compressed, encoding="utf-8", newline="")

             # Restore original on failure
-            filepath.write_text(original_text, encoding="utf-8")
+            filepath.write_text(original_text, encoding="utf-8", newline="")

-        compressed = call_claude(
-            build_fix_prompt(original_text, compressed, result.errors)
-        )
-        filepath.write_text(compressed, encoding="utf-8")
+        compressed = match_line_endings(
+            call_claude(build_fix_prompt(original_text, compressed, result.errors)),
+            original_text,
+        )
+        filepath.write_text(compressed, encoding="utf-8", newline="")

validate.py should not get newline="". The validator needs universal-newline normalization so a CRLF original and an LF candidate still compare equal — pinning it there would make validate_code_blocks fail on every CRLF file.

Results after the patch

[lf]   output keeps '\n', zero CR bytes, backup byte-identical (118 == 118)
[crlf] output keeps '\r\n',              backup byte-identical (120 == 120)

Two notes on #683 while you're in here

It misses detect.py. #686's body lists the extensionless-file content sniff as a call site needing the fix, but #683 only patches compress.py and validate.py. detect.py:93 still has a bare read_text(errors="ignore"):

         try:
-            text = filepath.read_text(errors="ignore")
+            text = filepath.read_text(encoding="utf-8", errors="ignore")

It classifies rather than writes, so it cannot corrupt a file — but on a non-UTF-8 locale it can misclassify a prose file as code and silently skip it.

Watch the Python floor if anyone adds newline= to a read. Path.read_text() only accepts newline= on 3.13+; Path.write_text() has had it since 3.10. The repo declares no minimum Python version and the skill docs just say python3 -m scripts, so a read_text(..., newline="") would TypeError for anyone on 3.10–3.12. That is why the patch above routes the reads through open() instead.

Environment

• Windows 11 Pro 26200
• Python 3.14.6
• caveman plugin 0d95a81d35a9
• No ANTHROPIC_API_KEY, so the claude --print CLI path is used