fix: use explicit utf-8 encoding in compress/validate file I/O - #683
fix: use explicit utf-8 encoding in compress/validate file I/O#683Abhishaex wants to merge 1 commit into
Conversation
On Windows, Python's read_text/write_text default to the system codepage (cp1252), which crashes or silently truncates when content contains non-ASCII characters (emojis, CJK, wenyan output). Fixes JuliusBrussee#652, fixes JuliusBrussee#655. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Heads up — this misses try:
- text = filepath.read_text(errors="ignore")
+ text = filepath.read_text(encoding="utf-8", errors="ignore")It only classifies, so it can't corrupt a file. But on a non-UTF-8 locale it can misdecode prose, trip a Separately, #763 contains this PR's change verbatim, because both touch identical call sites and splitting them would just conflict. Not trying to step on this one — happy to rebase #763 down to newline-only if this merges first. |
…655, #686) Every read/write resolved to the locale codec (cp1252/cp949 on Windows): non-ASCII files were silently mojibake'd, and because Path.write_text truncates before encoding, a UnicodeEncodeError left the target at 0 bytes. The backup readback check couldn't catch it — it read back with the same wrong codec. - encoding=utf-8 pinned on every I/O call site (compress, validate, detect, benchmark); validate now decodes strict — it's the fidelity gate - write_text_atomic: encode first, temp file in same dir, fsync, preserve permissions, os.replace; temp unlinked on any failure - fix-retry pass gains the same empty-output guard as the first pass - fix-retry preamble leak (#588): output must start at the original's structural anchor (frontmatter/heading) or the attempt is rejected - primary-write failure now prints the backup path — users hitting the crash had no idea a backup existed - extract_inline_codes: strip fences via the CommonMark-aware extractor; the old column-0 regex leaked indented fences into inline-code pairing, causing false validation failures (extracted from PR #619's diagnosis) - SKILL.md/README/SECURITY corrected: backups live in the out-of-tree data dir (#420), not beside the source file Supersedes PRs #683 #678 #626 #534 and the fence fix from #619 with a local implementation. 58 python tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf
* fix(skill): negation safety, tool-call silence, language-drift hardening Verified against filed reports and applied minimal rewordings: - never-drop-negations rule — dropping not/never inverts instructions (JuliusBrussee#670) - tool-call silence defined: no preamble/progress notes around calls (JuliusBrussee#713, JuliusBrussee#714) - language rule de-named: removes Portuguese/Spanish attractor tokens that drove drift under compaction; every emitted line in session language (JuliusBrussee#665, JuliusBrussee#654, JuliusBrussee#701, JuliusBrussee#539) - particles/postpositions exempt from article-dropping — Korean/Japanese case markers are grammar, not filler (JuliusBrussee#680) - classical chars scoped to wenyan modes only; Auto-Clarity example marked format-only so warnings render in session language (JuliusBrussee#679, JuliusBrussee#680) - switch line covers all six levels + off (JuliusBrussee#549, JuliusBrussee#670) - persisted-text boundary: docs/issues/PR text/memory files write normal (JuliusBrussee#483, JuliusBrussee#562, JuliusBrussee#582, JuliusBrussee#670) - wenyan-full row: character reduction labeled chars-not-tokens Supersedes PRs JuliusBrussee#715 JuliusBrussee#702 JuliusBrussee#684 JuliusBrussee#685 JuliusBrussee#667 JuliusBrussee#654 JuliusBrussee#658 JuliusBrussee#549 JuliusBrussee#670 JuliusBrussee#483 JuliusBrussee#562 JuliusBrussee#582 with local implementations. Eval snapshots need regeneration (API-gated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf * fix(hooks): shared parser, envelope unwrap, resume-safe SessionStart - extract mode parsing into src/hooks/caveman-parse.js, consumed by both the Claude tracker and the opencode plugin — fixes the three JuliusBrussee#602 drifts (brevity triggers missing, bogus level overwrote flag with default, independent modes unreachable via expanded templates) - unwrap Claude Code's <command-name>/<command-args> slash envelope: real slash-UI /caveman <level> and /caveman off were silent no-ops (JuliusBrussee#537); foreign envelopes skip natural-language detection entirely - SessionStart branches on payload source: startup resets to configured default, resume/clear/compact preserve a valid existing flag (JuliusBrussee#691) - scheduled-task prompts (<scheduled-task marker) skip flag mutation and reinforcement so unattended runs aren't hijacked - per-turn reinforcement honors repo-local defaultMode off via getDefaultMode(cwd) gate — read-only, never deletes the shared flag (JuliusBrussee#634; rejects JuliusBrussee#532's cross-session flag deletion) - reinforcement anchor shrunk ~57%, opencode line kept identical (JuliusBrussee#660) - statusline setup nudge shown once, gated by .caveman-nudge-shown (JuliusBrussee#661) - /caveman-stats delivered via hookSpecificOutput.additionalContext so the macOS desktop app renders it (JuliusBrussee#618) - safeWriteFlag: retry rename on Windows sharing violations, always unlink temp in finally — no more .caveman-active.<pid>.<ts> litter (JuliusBrussee#511 JuliusBrussee#578) - statusline.sh exits 0 on empty suffix file — non-zero exit was hiding the whole status bar (JuliusBrussee#711) - cavecrew-model-overrides resolves plugin root across layouts; env model overrides were a silent no-op (JuliusBrussee#645) - opencode dev-tree loader: base require on the loaded file so caveman-parse's relative require resolves in both layouts Supersedes PRs JuliusBrussee#623 JuliusBrussee#674 JuliusBrussee#700 JuliusBrussee#691 JuliusBrussee#634 JuliusBrussee#660 JuliusBrussee#661 JuliusBrussee#692 JuliusBrussee#632 JuliusBrussee#622 JuliusBrussee#657 JuliusBrussee#578 JuliusBrussee#511 JuliusBrussee#645 JuliusBrussee#590 JuliusBrussee#498 JuliusBrussee#501 with local implementations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf * feat(stats): surface net savings, not just gross (JuliusBrussee#145, JuliusBrussee#677) /caveman-stats only reported gross output savings, hiding the regime where the ~1,250-token/turn rule injection costs more than compression saves — the exact case docs/HONEST-NUMBERS.md warns about. - Est. rule overhead: per-turn injected-rule input cost x turns (CAVEMAN_RULE_OVERHEAD_TOKENS overrides the default 1250) - Est. net: saved minus overhead; when negative, says plainly that caveman cost more than it saved for this workload - shown only for attributed uniform spans (follows JuliusBrussee#601 attribution); history rows without turn counts are excluded from net, never guessed - statusline suffix stays gross savings, unchanged semantics Reimplements the idea from PR JuliusBrussee#718 locally. 46 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf * fix(compress): stop Windows data loss — UTF-8 + atomic writes (JuliusBrussee#652, JuliusBrussee#655, JuliusBrussee#686) Every read/write resolved to the locale codec (cp1252/cp949 on Windows): non-ASCII files were silently mojibake'd, and because Path.write_text truncates before encoding, a UnicodeEncodeError left the target at 0 bytes. The backup readback check couldn't catch it — it read back with the same wrong codec. - encoding=utf-8 pinned on every I/O call site (compress, validate, detect, benchmark); validate now decodes strict — it's the fidelity gate - write_text_atomic: encode first, temp file in same dir, fsync, preserve permissions, os.replace; temp unlinked on any failure - fix-retry pass gains the same empty-output guard as the first pass - fix-retry preamble leak (JuliusBrussee#588): output must start at the original's structural anchor (frontmatter/heading) or the attempt is rejected - primary-write failure now prints the backup path — users hitting the crash had no idea a backup existed - extract_inline_codes: strip fences via the CommonMark-aware extractor; the old column-0 regex leaked indented fences into inline-code pairing, causing false validation failures (extracted from PR JuliusBrussee#619's diagnosis) - SKILL.md/README/SECURITY corrected: backups live in the out-of-tree data dir (JuliusBrussee#420), not beside the source file Supersedes PRs JuliusBrussee#683 JuliusBrussee#678 JuliusBrussee#626 JuliusBrussee#534 and the fence fix from JuliusBrussee#619 with a local implementation. 58 python tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf * fix(install): rename bin/ to cli/, harden Windows quoting, clean uninstall Marketplace fix (JuliusBrussee#712, JuliusBrussee#705): Claude Desktop rejects plugins containing a top-level bin/ directory, and .claude-plugin/marketplace.json packages the repo root, so the installer directory is now cli/. Every reference updated (package.json bin entry + files, shims, docs, tests, caveman-init require path). Supersedes PR JuliusBrussee#726. Security (PR JuliusBrussee#717 verified): quoteWinArg only quoted on whitespace/quotes, leaving cmd.exe metacharacters (& | ^ < > % parens) unescaped on the shell:true Windows spawn path. Attacker-influenced arguments (--with-init cwd, --with-mcp-shrink value) could chain commands. Trigger regex now covers the metacharacter set; quoting logic split into a platform- independent, unit-tested helper. Also: - uninstall removes .caveman-active.prev, .caveman-mode-log.jsonl, .caveman-statusline-suffix, .caveman-nudge-shown; keeps .caveman-history.jsonl with a printed note; dry-run now says 'would remove' instead of lying (JuliusBrussee#635, supersedes PRs JuliusBrussee#693 JuliusBrussee#636) - Array.isArray guard in rewriteLegacyManagedHookCommands — malformed hook event no longer crashes the installer mid-run (supersedes PR JuliusBrussee#646) - gemini extensions install --consent: the security prompt hung every piped/non-interactive install forever (JuliusBrussee#676, part of PR JuliusBrussee#664) - OpenClaw skill stamps the real PINNED_REF version instead of hardcoded 1.0.0; new --no-always flag for load-on-demand installs (supersedes PR JuliusBrussee#720) - shims scope NPM_CONFIG_ALLOW_GIT=all to the npx call — npm >=12 defaults allow-git to none and EALLOWGITs github: installs (JuliusBrussee#698) - .codex/config.toml ships hooks + codex_hooks keys so auto-activation works on both sides of the codex-cli rename (JuliusBrussee#617) - caveman-help card shows the Windows config path (%APPDATA%) (JuliusBrussee#723) - caveman-parse.js added to HOOK_FILES, opencode payload (.cjs), and the regenerated checksums.sha256; manifest now matches shipped hook contents — release must bump PINNED_REF to a tag containing these files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf * ci: run the full test suite on every push and PR Tests existed but nothing ran them in CI. Node 18/20/22 matrix runs the installer suite plus every standalone tests/test_*.js runner; a python job runs unittest discover (compress tests mock the Claude call, no network). Node is set up in the python job too — several python tests shell out to node for hook checks. Inspired by PR JuliusBrussee#695, minus the scanner extras. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016ySX6TBWZuvFze4ajf7Hpf * docs: clarify token savings by workload Update README to separate the 65% chat-style prose benchmark from full agentic coding runs, adding JetBrains’ independent 86-task result (8.5% output-token savings). Adds context on why both numbers are valid, explains workload-dependent ceilings, and aligns Caveman 2 positioning around measuring real savings on each team’s own traffic. * release: pin install ref to v1.10.0 The sweep adds src/hooks/caveman-parse.js and regenerates checksums.sha256 to include it. The detached install path fetches hooks from RAW_BASE = raw.githubusercontent.com/<repo>/<PINNED_REF>, and the integrity gate refuses any hook that doesn't match the pinned release. Left at v1.9.1 that ref has no caveman-parse.js, so curl | bash 404s and fails integrity for new users. Bump must land inside the tagged commit so v1.10.0 contains both the new hooks and a PINNED_REF pointing at itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012SSMrNKRooMGmJ1yNFpLMy * chore: sync SKILL.md copies [skip ci] --------- Co-authored-by: Julius Brussee <104168679+JuliusBrussee@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
On Windows, Python's read_text/write_text default to the system codepage (cp1252), which crashes or silently truncates when content contains non-ASCII characters (emojis, CJK, wenyan output).
Fixes #652, fixes #655.