[chrhoffmann]

Summary

Fixes two ways the bytecode fabricator can hang a pkg build forever, and
makes every other fabricator failure either fall back cleanly or abort
loudly, never stall or ship wrong bytecode.

Problems fixed

1. Shared header buffer aliasing (reachable, Windows CI hangs). The
parent wrote both request headers from one reused 4-byte buffer. When
child.stdin was still draining the previous request, the first header
was queued by reference and then overwritten with the body length before
it was sent. The child then read the wrong snap size and waited forever.
Confirmed in production on GitLab Windows runners
(comment):
19 of 28 instrumented builds hung, every hung request started with a
non-empty writableLength, and using separate header buffers took hangs
from 2/4 to 0/5. Each request now builds its own header buffers
(buildFabricatorRequestChunks).

2. V8 trace flags corrupting the response stream. --options trace-gc
(also print-bytecode, trace-ic, ...) is passed to the fabricator child
and prints to its stdout, which was also the response channel. The parent
read trace text as a frame size and waited forever. Responses now come back
on a dedicated pipe at fd 3, and the child's stdout is ignored. Checked
with a 3.8MB file and --options trace-gc: the old code fails with
Invalid blob size header: 842150491, the new code compiles it.

Behaviour

Fabricator outcome	Result
V8 refuses a file (exit 2), or cachedData over 256MB	Retried once, then --fallback-to-source ships source, otherwise skip + warn
Request part over 256MB	Not retried; fallback or skip
No response within 60s	Child SIGKILLed; not retried; fallback or skip
Corrupt frame / channel desync / child exit 65	Build aborts, even with --fallback-to-source, with remedy hints

User-visible changes:

• Builds using V8 trace flags in --options no longer hang.
• A file that takes over 60s to compile now falls back or is skipped
  instead of hanging the build.
• A desynced channel aborts the build instead of degrading every remaining
  file to source. The message points at --no-bytecode and the issue
  tracker.
• A failed build no longer leaves a partial binary at --output.
• Failure messages carry the last 2000 bytes of the child's stderr from
  that request, so Pkg: Cached data not produced. is visible without
  --debug.
• Request frame parts are capped at 256MB (headers are signed i32). Source
  files between 256MB and 512MB that used to compile are now per-file
  failures.

Changes

lib/fabricator.ts

• Requests on stdin, responses on fd 3, child stdout ignored.
• Each child keeps one set of listeners from spawn and one pending request.
  Output with no request pending, or bytes left after a response, is a
  protocol error. Children are evicted when they exit and respawned on the
  next call.
• Size headers are validated on both sides against
  FABRICATOR_MAX_FRAME_PART_SIZE. Frames are assembled in linear time:
  chunks are only concatenated once enough bytes for the next parse step
  have arrived.
• Child framing violations exit 65 (FABRICATOR_PROTOCOL_EXIT_CODE),
  outside Node's reserved exit codes. Exit 2 still means "V8 refused this
  file".
• Failures carry a code (FABRICATOR_FRAME_TOO_LARGE, FABRICATOR_TIMEOUT,
  FABRICATOR_PROTOCOL, read with fabricatorErrorCode()); coded failures
  are never retried by fabricateTwice.

lib/producer.ts

FABRICATOR_PROTOCOL aborts the build through wasReported() with hint
lines; everything else keeps the existing fallback/skip path. When
producer() fails for any reason it closes the output stream and removes
the partial file before rejecting.

Docs

• docs/ARCHITECTURE.md: new "Bytecode Fabricator IPC" section with the
  frame layout, size cap and the outcome table above.
• docs-site/guide/bytecode.md: notes that --fallback-to-source covers
  compile failures and timeouts, but not a corrupted fabricator exchange.

Tests (test/unit/fabricator.test.ts)

• Child script: multi-frame decode across chunk splits, invalid snap/body
  headers exit 65, zero-length parts.
• fabricate() against the real child, with misbehaviour injected through
  --require preload bakes: stdout noise is ignored, corrupt response,
  trailing bytes, unsolicited output (child restarts), exit 65, compile
  failure, timeout then respawn, two requests on one child, a response
  split across chunks, and exit 65 while a 1MB request is still being
  written.
• fabricateTwice: plain failures retried once, coded failures never.
• producer() with a minimal fake base binary: a protocol error aborts
  even with --fallback-to-source and removes the partial output; a
  compile failure falls back to source.

Validation

• yarn build, yarn lint clean
• yarn test:unit: 311 pass, fabricator suite repeated 30x without flakes
• node test/test.js node22 no-npm "test-50-*" passes locally (except
  test-50-ignore-files, which fails only inside a .claude/ worktree
  path and passes in a normal checkout)

[robertsLando]

Hi @chrhoffmann and thanks for your PR! I will back from vacation on Monday and i will review this ASAP!

[codecov]

Codecov Report

❌ Patch coverage is 96.85230% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.26%. Comparing base (8ddf2aa) to head (9384f2a).

Files with missing lines	Patch %	Lines
lib/fabricator.ts	96.85%	12 Missing ⚠️
lib/producer.ts	96.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #294      +/-   ##
==========================================
+ Coverage   87.62%   88.26%   +0.64%     
==========================================
  Files          23       23              
  Lines        7926     8217     +291     
  Branches     1214     1270      +56     
==========================================
+ Hits         6945     7253     +308     
+ Misses        974      957      -17     
  Partials        7        7              

Files with missing lines	Coverage Δ
lib/producer.ts	89.72% <96.87%> (+3.03%)	⬆️
lib/fabricator.ts	97.29% <96.85%> (+4.22%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[robertsLando]

Summary

Verdict: Ship with minor changes.

The child-side rewrite is genuinely correct — I fuzzed it against every one of the 119 split offsets plus byte-at-a-time delivery of a 3-frame stream and a 3MB body: zero failures. The three new tests are real regressions, not tautologies: against the base script the multi-frame case yields frames=1, expected 2, and the bad-header case exits 0 silently, which is the actual silent-corruption path. Dropping the raw console.log(stdout.toString()) binary dump is right. buildFabricatorRequestChunks is genuinely wired into fabricate(), not test-only.

Two things to settle before merge, and one correction to the PR description.

Top 3 risks

1. The one reachable fabricator hang lives in the parent decoder this PR didn't touch.
2. Non-debug users now see strictly less on failure than before this PR.
3. A protocol corruption silently degrades to --fallback-to-source, so a real bug ships as a green build.

Themes

• One protocol, two half-implementations. The child got while + bounds guards + trailing-byte retention; the parent decoder got none of it. Four independent review passes landed on this.
• The correct parser exists only in the test file. parseBlobFrames handles multi-frame and negative sizes; production's onData does neither.
• Diagnosability moved backwards, in a PR whose stated purpose is explaining failures.

---

Findings that fall outside the diff (can't be line-anchored)

Major — lib/fabricator.ts:160-171, the parent's onData

This reads sizeOfBlob with none of the guards just added to the child, and it is reachable, not theoretical:

bakes come from the user's --options (lib/index.ts:171), and fabricate's filter strips only --prof/--v8-options/--trace-opt/--trace-deopt. So --options trace-gc writes GC traces to stdout, straight into the framed response stream:

• Buffer.from('[123:0x5').readInt32LE(0) = 858927451 → stdout.length >= 4 + sizeOfBlob never becomes true, cb never fires, and pkg hangs forever (there is no timeout anywhere in this path).
• A byte with the high bit set gives a negative size → stdout.length >= 4 + (-1) is true → Buffer.alloc(-1) throws ERR_OUT_OF_RANGE inside a 'data' handler, uncaught, crashing pkg instead of routing through onError / --fallback-to-source.

This is pre-existing, so I'm not treating it as a merge blocker — but it's the only actual deadlock in this file, and the PR title is "prevent stdio deadlocks". Worth either fixing here or being explicit that it's out of scope.

Separately: onData consumes one frame and drops everything past 4 + sizeOfBlob, and removeListener('data', ...) does not pause a flowing stream. Today that's masked only because producer.ts:478 issues one request per Multistream callback — and note the child's old stdin = Buffer.alloc(0) used to enforce that lock-step. This PR removes that enforcement, so the child can now pipeline while the parent still can't read it.

Major — no timeout on the child's response

If the child accepts a valid frame and then hangs, fabricate never calls back and pkg stalls with no log line. That's the failure mode immediately adjacent to the one this PR targets.

Major — tests don't reach the parent half

The tests exercise the extracted script and helper; fabricate() itself, the parent decode, and the stderr routing change are all untested — and that untested half is where every remaining gap is. Driving the tests through fabricate() with a Target whose binaryPath is process.execPath would cover both halves, and would remove the need for the two new exports.

Minor — lib/fabricator.ts:143

onClose(code: number) but 'close' emits number | null. Runtime behaviour is fine; the type isn't.

---

On the PR description

I tried to reproduce all three claimed root causes. Results:

• Claim A (shared header buffer aliasing) — real Node hazard, not reachable here. A repro confirms Writable.write() retains Buffers by reference: with 1MB pre-filled to force backpressure, the base sends H1=3145728 instead of H1=36. But at the mutation point only h (4B) + snap (~40B) are in flight — far under the 64KB pipe buffer — so uv_try_write completes them synchronously. Without pre-filled backpressure (4 configurations tried, including a child delaying reads by 400ms) the base is uncorrupted. Worth the 2-line fix as latent-hazard cleanup; it is not a hang anyone is hitting.
• Claim B (trailing-byte truncation) — theoretical. fabricate has exactly one caller (lib/producer.ts:478), inside a Multistream factory, and multistream calls _next() only from the current stream's onEnd. Targets are serialized too (lib/index.ts:313-326). One request → full response → next. Trailing bytes can never exist on the child's stdin.
• Claim C (debug stderr causes backpressure deadlock) — not a bug. The base passes process.stdout as stdio[2], so Node inherits the fd rather than creating an unread pipe. Repro: child writes 5MB to stderr → exit 0 in 587ms. The PR's 'pipe' does correctly attach a reader, so it doesn't introduce one either. Net: a routing change, not a deadlock fix.

The changes are still worth having — but the description reads as three observed deadlocks, and I could not reproduce any of them as such. Could you share the actual reproducer, or retitle to drop the deadlock claim? Also, the validation section cites npm run test:unit / npm run lint / npm run build; this repo is yarn-only at the root (npm would create a stray package-lock.json).

Two things verified clean, for the record: non-ASCII snap paths work correctly (toString('utf8', 4, 4 + sizeOfSnap) takes byte offsets, matching Buffer.from(snap) — checked end-to-end with /snapshot/ünïcodé-èà.js), and the child.stderr listener is attached once per spawn inside if (!child) with kill() deleting the cache key, so there's no leak. The 256MB ceiling also can't regress real payloads — bodies are per-file source buffers and module.wrap already caps at Node's 512MB string limit.

---

Coverage: correctness, DRY, performance, design/API, tests, operability, readability. Security not run — no files in its lane (build-time IPC, no auth/crypto/network surface). No prior unresolved review threads.

[robertsLando]

Deep review — 4 files, +777 / −65 LOC

Verdict: Needs work. Two blockers, both in the wiring around a parser that is itself solid.

Top 3 risks

1. The pipelined fast path answers a request that was never sent, delivering another file's bytecode — the exact silent corruption this PR exists to stop (lib/fabricator.ts:398).
2. The 60s timeout is tagged as a protocol error, so a slow-but-healthy compile now hard-fails with --fallback-to-source ignored (lib/fabricator.ts:277).
3. The stated root cause — --options trace-gc writing to the child's stdout — is documented but not removed. The frame channel still shares stdout, so that flag turns from a hang into an unbuildable configuration.

Strengths. The corrected, honestly-labelled problem statement in the PR description is genuinely rare and made this review much faster. Tests assert against the production parser rather than a copy. 11 of the 13 threads from the previous round are verifiably addressed. The frame layout and exit-code contract are now written down in docs/ARCHITECTURE.md.

Themes

• Guards on a channel that's still polluted. The PR identifies stdout-sharing as the root cause, then hardens the reader instead of moving the frames off stdout. Everything in the trace-gc family below follows from that one decision.
• New fatal class, old plumbing. FABRICATOR_PROTOCOL is now the only build-aborting error in fabricator.ts, but it doesn't go through wasReported(), doesn't appear in the user-facing docs, and doesn't clean up the partially-written output file.
• The parser is tested; the state machine around it isn't. Both blockers live in code the suite never drives, which is why they survived a green run.

---

Two findings that fall outside the diff

[Major · Design] lib/fabricator.ts — the bakes filter is the actual root cause, and it's untouched.
fabricate() strips only --prof / --v8-options / --trace-opt / --trace-deopt. But --options trace-gc writes to the child's stdout (measured: ~15KB stdout, 0 stderr), as do trace-gc-verbose, print-bytecode, trace-ic and trace-turbo. This PR converts that from an infinite hang into a hard, unescapable abort — a supported user flag becomes an unbuildable configuration with no override. The durable fix is to stop sharing stdout between V8 traces and frames: move the frame channel to a dedicated fd (stdio[3]). Widening the filter is the cheaper stopgap.

[Minor · Docs] docs-site/ still promises the old contract.
guide/bytecode.md:59, guide/targets.md:99 and guide/getting-started.md:127 all state that --fallback-to-source ships source "when bytecode generation fails". The new always-fatal exception is documented only in the contributor-facing docs/ARCHITECTURE.md. Users hitting the new abort will read the guide and conclude pkg is broken.

---

Coverage

Specialists run: Correctness, DRY (repo-wide), Performance, Design/API/BackCompat, Tests, Operability, Readability. Every lane returned findings; none came back clean.

Not run: Security — no changed file matched its lane. The child's spawn arguments are build-time flags rather than untrusted input, and the -e script is a module constant with only numeric constants interpolated.

Reviewed against a dedicated worktree at 64a8921, so repo-wide reuse checks and sibling-file context were complete.

[manelpb]

Thanks for this PR. We have production evidence that the shared header buffer aliasing is reachable. On Windows it is the cause of pkg hanging forever in the bytecode phase, so the PR fixes a real bug, not just hardens against a latent one.

Environment: GitLab.com hosted Windows runners (saas-windows-medium-amd64: Windows Server 2022, 2 vCPU). pkg 6.23.0 targets node22-win-x64 with bytecode enabled. The host node is 22.16.0 and the fabricator is fetched-v22.23.2-win-x64.

Symptom: the build hangs forever after the walker finishes. pkg and the fabricator child both sit at 0 CPU with every thread waiting, and there is no output or error. 19 of 28 instrumented builds hung.

Evidence the aliasing is the cause:

We patched fabricate() to log child.stdin.writableLength at the start of each request.

• In all 3 hung runs, the request the build stalled on started with writableLength = 498. That is exactly the previous request's snap (85 B), body header (4 B) and body (409 B), still in flight.
• So the new request's first write(h) was queued by reference. Then h.writeInt32LE(b.length, 0) overwrote it with the body length before it was sent, and the child read the body length as sizeOfSnap.
• A successful run made 6,552 requests, and none of them started with a non-zero writableLength.

We then ran arms side by side in the same window, so all ran under the same runner conditions:

Variant	Completed	Hung
unpatched 6.23.0	2/4	2/4
child.stdin.write(Buffer.from(h)) for both header writes	5/5	0/5
per-request timeout + respawn (no buffer fix)	4/4	0/4, one stall caught and recovered

One note on how #294 handles this case. With the aliasing, sizeOfSnap becomes a small valid number, 423 in the stall above, so it passes the new frame-size check. The separate header buffers in buildFabricatorRequestChunks are what actually prevent the hang. The validation and the 60s timeout would only turn it into a loud failure instead. So I'd treat the buffer change as the core fix rather than hygiene.

It would be great to see this merged and released. We're happy to test a prerelease on the same runners.

Until a release ships, this workaround fixes it in CI after npm install -g @yao-pkg/pkg@6.23.0: patch both child.stdin.write(h); calls in lib-es5/fabricator.js to child.stdin.write(Buffer.from(h));.

[robertsLando]

finalize-pr: 5 review rounds, exit with no Blocker/Major open

Decided by the maintainer

• Response channel moved from stdout to a dedicated fd 3 pipe, so V8 trace flags (--options trace-gc, …) can't corrupt frames (round 1).
• A 60s response timeout gets its own FABRICATOR_TIMEOUT code. It is not retried and falls back to source or skips the file, with no tuning knob (round 1). Raised again in rounds 2 and 4 (a slow emulated build skips the file when --fallback-to-source is off) and kept as decided.
• Partial --output file on a failed build: fixed in this PR (producer() removes it), even though it predates this PR.
• Simplicity review: core cuts applied (parser folded in, single stderr buffer, no killed flag, one error message format). The child stays a multi-frame parser.

Dropped, with reasons

• Retry-then-success logged at debug only: a warning would print once per file on Node versions that hit the compile retry.
• Double cap on the stderr tail: resolved by the simplification (one 2000-byte buffer).
• Extra tests on code added during review (late events after kill, oversize request through the callback, clean exit while pending): more tests on the loop's own hardening. The paths are covered by neighbouring tests.
• Fixed 50ms write gap in the child-script split test is not verified to split chunks: assertion would need instrumentation inside the child.
• shutdown() not called when a library caller's build fails: pre-existing, and the CLI exits the process anyway.