Skip to content

fix: use explicit utf-8 encoding in compress/validate file I/O - #683

Open
Abhishaex wants to merge 1 commit into
JuliusBrussee:mainfrom
Abhishaex:fix/windows-compress-encoding
Open

fix: use explicit utf-8 encoding in compress/validate file I/O#683
Abhishaex wants to merge 1 commit into
JuliusBrussee:mainfrom
Abhishaex:fix/windows-compress-encoding

Conversation

@Abhishaex

Copy link
Copy Markdown

On Windows, Python's read_text/write_text default to the system codepage (cp1252), which crashes or silently truncates when content contains non-ASCII characters (emojis, CJK, wenyan output).

Fixes #652, fixes #655.

On Windows, Python's read_text/write_text default to the system codepage
(cp1252), which crashes or silently truncates when content contains
non-ASCII characters (emojis, CJK, wenyan output).

Fixes JuliusBrussee#652, fixes JuliusBrussee#655.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@francojeferson

Copy link
Copy Markdown

Heads up — this misses detect.py. #686's body lists the extensionless-file content sniff as a call site needing the fix, but the diff here only covers compress.py and validate.py.

         try:
-            text = filepath.read_text(errors="ignore")
+            text = filepath.read_text(encoding="utf-8", errors="ignore")

It only classifies, so it can't corrupt a file. But on a non-UTF-8 locale it can misdecode prose, trip a CODE_PATTERNS match, and silently skip a file the user explicitly asked to compress.

Separately, encoding= on its own only fixes half of the Windows text-mode default. write_text still translates \n to \r\n, so every processed file comes back with its line endings rewritten — measured a 67-line LF file returning 67 bytes larger. Filed that as #762 with a patch in #763.

#763 contains this PR's change verbatim, because both touch identical call sites and splitting them would just conflict. Not trying to step on this one — happy to rebase #763 down to newline-only if this merges first.

JuliusBrussee added a commit that referenced this pull request Aug 3, 2026
…655, #686)

Every read/write resolved to the locale codec (cp1252/cp949 on Windows):
non-ASCII files were silently mojibake'd, and because Path.write_text
truncates before encoding, a UnicodeEncodeError left the target at 0 bytes.
The backup readback check couldn't catch it — it read back with the same
wrong codec.

- encoding=utf-8 pinned on every I/O call site (compress, validate,
  detect, benchmark); validate now decodes strict — it's the fidelity gate
- write_text_atomic: encode first, temp file in same dir, fsync, preserve
  permissions, os.replace; temp unlinked on any failure
- fix-retry pass gains the same empty-output guard as the first pass
- fix-retry preamble leak (#588): output must start at the original's
  structural anchor (frontmatter/heading) or the attempt is rejected
- primary-write failure now prints the backup path — users hitting the
  crash had no idea a backup existed
- extract_inline_codes: strip fences via the CommonMark-aware extractor;
  the old column-0 regex leaked indented fences into inline-code pairing,
  causing false validation failures (extracted from PR #619's diagnosis)
- SKILL.md/README/SECURITY corrected: backups live in the out-of-tree data
  dir (#420), not beside the source file

Supersedes PRs #683 #678 #626 #534 and the fence fix from #619 with a
local implementation. 58 python tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf
kvag0 added a commit to Wibx-LABS/caveman that referenced this pull request Aug 3, 2026
* fix(skill): negation safety, tool-call silence, language-drift hardening

Verified against filed reports and applied minimal rewordings:
- never-drop-negations rule — dropping not/never inverts instructions (JuliusBrussee#670)
- tool-call silence defined: no preamble/progress notes around calls (JuliusBrussee#713, JuliusBrussee#714)
- language rule de-named: removes Portuguese/Spanish attractor tokens that
  drove drift under compaction; every emitted line in session language
  (JuliusBrussee#665, JuliusBrussee#654, JuliusBrussee#701, JuliusBrussee#539)
- particles/postpositions exempt from article-dropping — Korean/Japanese
  case markers are grammar, not filler (JuliusBrussee#680)
- classical chars scoped to wenyan modes only; Auto-Clarity example marked
  format-only so warnings render in session language (JuliusBrussee#679, JuliusBrussee#680)
- switch line covers all six levels + off (JuliusBrussee#549, JuliusBrussee#670)
- persisted-text boundary: docs/issues/PR text/memory files write normal
  (JuliusBrussee#483, JuliusBrussee#562, JuliusBrussee#582, JuliusBrussee#670)
- wenyan-full row: character reduction labeled chars-not-tokens

Supersedes PRs JuliusBrussee#715 JuliusBrussee#702 JuliusBrussee#684 JuliusBrussee#685 JuliusBrussee#667 JuliusBrussee#654 JuliusBrussee#658 JuliusBrussee#549 JuliusBrussee#670 JuliusBrussee#483 JuliusBrussee#562 JuliusBrussee#582
with local implementations. Eval snapshots need regeneration (API-gated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf

* fix(hooks): shared parser, envelope unwrap, resume-safe SessionStart

- extract mode parsing into src/hooks/caveman-parse.js, consumed by both
  the Claude tracker and the opencode plugin — fixes the three JuliusBrussee#602 drifts
  (brevity triggers missing, bogus level overwrote flag with default,
  independent modes unreachable via expanded templates)
- unwrap Claude Code's <command-name>/<command-args> slash envelope: real
  slash-UI /caveman <level> and /caveman off were silent no-ops (JuliusBrussee#537);
  foreign envelopes skip natural-language detection entirely
- SessionStart branches on payload source: startup resets to configured
  default, resume/clear/compact preserve a valid existing flag (JuliusBrussee#691)
- scheduled-task prompts (<scheduled-task marker) skip flag mutation and
  reinforcement so unattended runs aren't hijacked
- per-turn reinforcement honors repo-local defaultMode off via
  getDefaultMode(cwd) gate — read-only, never deletes the shared flag
  (JuliusBrussee#634; rejects JuliusBrussee#532's cross-session flag deletion)
- reinforcement anchor shrunk ~57%, opencode line kept identical (JuliusBrussee#660)
- statusline setup nudge shown once, gated by .caveman-nudge-shown (JuliusBrussee#661)
- /caveman-stats delivered via hookSpecificOutput.additionalContext so the
  macOS desktop app renders it (JuliusBrussee#618)
- safeWriteFlag: retry rename on Windows sharing violations, always unlink
  temp in finally — no more .caveman-active.<pid>.<ts> litter (JuliusBrussee#511 JuliusBrussee#578)
- statusline.sh exits 0 on empty suffix file — non-zero exit was hiding
  the whole status bar (JuliusBrussee#711)
- cavecrew-model-overrides resolves plugin root across layouts; env model
  overrides were a silent no-op (JuliusBrussee#645)
- opencode dev-tree loader: base require on the loaded file so
  caveman-parse's relative require resolves in both layouts

Supersedes PRs JuliusBrussee#623 JuliusBrussee#674 JuliusBrussee#700 JuliusBrussee#691 JuliusBrussee#634 JuliusBrussee#660 JuliusBrussee#661 JuliusBrussee#692 JuliusBrussee#632 JuliusBrussee#622 JuliusBrussee#657
JuliusBrussee#578 JuliusBrussee#511 JuliusBrussee#645 JuliusBrussee#590 JuliusBrussee#498 JuliusBrussee#501 with local implementations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf

* feat(stats): surface net savings, not just gross (JuliusBrussee#145, JuliusBrussee#677)

/caveman-stats only reported gross output savings, hiding the regime where
the ~1,250-token/turn rule injection costs more than compression saves —
the exact case docs/HONEST-NUMBERS.md warns about.

- Est. rule overhead: per-turn injected-rule input cost x turns
  (CAVEMAN_RULE_OVERHEAD_TOKENS overrides the default 1250)
- Est. net: saved minus overhead; when negative, says plainly that caveman
  cost more than it saved for this workload
- shown only for attributed uniform spans (follows JuliusBrussee#601 attribution);
  history rows without turn counts are excluded from net, never guessed
- statusline suffix stays gross savings, unchanged semantics

Reimplements the idea from PR JuliusBrussee#718 locally. 46 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf

* fix(compress): stop Windows data loss — UTF-8 + atomic writes (JuliusBrussee#652, JuliusBrussee#655, JuliusBrussee#686)

Every read/write resolved to the locale codec (cp1252/cp949 on Windows):
non-ASCII files were silently mojibake'd, and because Path.write_text
truncates before encoding, a UnicodeEncodeError left the target at 0 bytes.
The backup readback check couldn't catch it — it read back with the same
wrong codec.

- encoding=utf-8 pinned on every I/O call site (compress, validate,
  detect, benchmark); validate now decodes strict — it's the fidelity gate
- write_text_atomic: encode first, temp file in same dir, fsync, preserve
  permissions, os.replace; temp unlinked on any failure
- fix-retry pass gains the same empty-output guard as the first pass
- fix-retry preamble leak (JuliusBrussee#588): output must start at the original's
  structural anchor (frontmatter/heading) or the attempt is rejected
- primary-write failure now prints the backup path — users hitting the
  crash had no idea a backup existed
- extract_inline_codes: strip fences via the CommonMark-aware extractor;
  the old column-0 regex leaked indented fences into inline-code pairing,
  causing false validation failures (extracted from PR JuliusBrussee#619's diagnosis)
- SKILL.md/README/SECURITY corrected: backups live in the out-of-tree data
  dir (JuliusBrussee#420), not beside the source file

Supersedes PRs JuliusBrussee#683 JuliusBrussee#678 JuliusBrussee#626 JuliusBrussee#534 and the fence fix from JuliusBrussee#619 with a
local implementation. 58 python tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf

* fix(install): rename bin/ to cli/, harden Windows quoting, clean uninstall

Marketplace fix (JuliusBrussee#712, JuliusBrussee#705): Claude Desktop rejects plugins containing a
top-level bin/ directory, and .claude-plugin/marketplace.json packages the
repo root, so the installer directory is now cli/. Every reference updated
(package.json bin entry + files, shims, docs, tests, caveman-init require
path). Supersedes PR JuliusBrussee#726.

Security (PR JuliusBrussee#717 verified): quoteWinArg only quoted on whitespace/quotes,
leaving cmd.exe metacharacters (& | ^ < > % parens) unescaped on the
shell:true Windows spawn path. Attacker-influenced arguments (--with-init
cwd, --with-mcp-shrink value) could chain commands. Trigger regex now
covers the metacharacter set; quoting logic split into a platform-
independent, unit-tested helper.

Also:
- uninstall removes .caveman-active.prev, .caveman-mode-log.jsonl,
  .caveman-statusline-suffix, .caveman-nudge-shown; keeps
  .caveman-history.jsonl with a printed note; dry-run now says
  'would remove' instead of lying (JuliusBrussee#635, supersedes PRs JuliusBrussee#693 JuliusBrussee#636)
- Array.isArray guard in rewriteLegacyManagedHookCommands — malformed
  hook event no longer crashes the installer mid-run (supersedes PR JuliusBrussee#646)
- gemini extensions install --consent: the security prompt hung every
  piped/non-interactive install forever (JuliusBrussee#676, part of PR JuliusBrussee#664)
- OpenClaw skill stamps the real PINNED_REF version instead of hardcoded
  1.0.0; new --no-always flag for load-on-demand installs (supersedes
  PR JuliusBrussee#720)
- shims scope NPM_CONFIG_ALLOW_GIT=all to the npx call — npm >=12
  defaults allow-git to none and EALLOWGITs github: installs (JuliusBrussee#698)
- .codex/config.toml ships hooks + codex_hooks keys so auto-activation
  works on both sides of the codex-cli rename (JuliusBrussee#617)
- caveman-help card shows the Windows config path (%APPDATA%) (JuliusBrussee#723)
- caveman-parse.js added to HOOK_FILES, opencode payload (.cjs), and the
  regenerated checksums.sha256; manifest now matches shipped hook
  contents — release must bump PINNED_REF to a tag containing these files

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf

* ci: run the full test suite on every push and PR

Tests existed but nothing ran them in CI. Node 18/20/22 matrix runs the
installer suite plus every standalone tests/test_*.js runner; a python job
runs unittest discover (compress tests mock the Claude call, no network).
Node is set up in the python job too — several python tests shell out to
node for hook checks. Inspired by PR JuliusBrussee#695, minus the scanner extras.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf

* docs: clarify token savings by workload

Update README to separate the 65% chat-style prose benchmark from full agentic coding runs, adding JetBrains’ independent 86-task result (8.5% output-token savings). Adds context on why both numbers are valid, explains workload-dependent ceilings, and aligns Caveman 2 positioning around measuring real savings on each team’s own traffic.

* release: pin install ref to v1.10.0

The sweep adds src/hooks/caveman-parse.js and regenerates
checksums.sha256 to include it. The detached install path fetches
hooks from RAW_BASE = raw.githubusercontent.com/<repo>/<PINNED_REF>,
and the integrity gate refuses any hook that doesn't match the pinned
release. Left at v1.9.1 that ref has no caveman-parse.js, so
curl | bash 404s and fails integrity for new users.

Bump must land inside the tagged commit so v1.10.0 contains both the
new hooks and a PINNED_REF pointing at itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012SSMrNKRooMGmJ1yNFpLMy

* chore: sync SKILL.md copies [skip ci]

---------

Co-authored-by: Julius Brussee <104168679+JuliusBrussee@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants