nixd terminates during textDocument/diagnostic, failing niche (ubuntu-latest) on main

[ibakard]

main is currently red: the niche (ubuntu-latest) batch fails because nixd terminates during a textDocument/diagnostic request.

Latest main run: https://github.com/oraios/serena/actions/runs/30703366827 (commit c7dbe3bd) → job niche (ubuntu-latest)
→ 1 failed, 53 passed, 1 skipped, 1974 deselected, 1 xfailed

It also fails on any PR branched from that commit — on #1800 it is the only red check out of 20.

Symptom

test/solidlsp/nix/test_nix_basic.py::TestNixLanguageServer::test_file_diagnostics[nix]

test/solidlsp/nix/test_nix_basic.py:256: in test_file_diagnostics
    assert_file_diagnostics(
test/solidlsp/util/diagnostics.py:12: in assert_file_diagnostics
    diagnostics = language_server.request_text_document_diagnostics(relative_file_path, min_severity=1)
src/solidlsp/ls.py:961: in request_text_document_diagnostics
    response = self.server.send.text_document_diagnostic(
...
E   solidlsp.ls_exceptions.SolidLSPException: Error processing request textDocument/diagnostic with params:
E   {'textDocument': {'uri': '.../test/resources/repos/nix/test_repo/diagnostics_sample.nix'}}
E   (caused by LanguageServerTerminatedException: Language server stdout read process terminated unexpectedly)

----------------------------- Captured stdout call -----------------------------
ERROR solidlsp.ls_process:_read_ls_process_stderr:646 - Language server stderr reader thread terminated unexpectedly
ERROR solidlsp.ls_process:_read_ls_process_stdout:625 - LanguageServerTerminatedException: Language server stdout read process terminated unexpectedly
ERROR solidlsp.ls_process:_send_payload:663 - Failed to write to stdin: [Errno 32] Broken pipe

The process is gone, not merely erroring: both reader threads die and stdin is a broken pipe. All ten preceding nix tests in the module pass, so nixd survives symbol/reference/definition work and dies on the diagnostics pull specifically.

When it started

niche (ubuntu-latest) was last green on main at c74373e2 (run 30694340061). Only three commits have landed since: c958aa02, f4ecb1a5, c7dbe3bd. Of those, f4ecb1a5 ("fix(ls): re-raise LS termination from pull-diagnostics fallback", for #1770) is the only one touching this code path — it makes request_text_document_diagnostics re-raise LanguageServerTerminatedException instead of falling back to published diagnostics.

There are two readings, and I cannot separate them from CI logs alone:

1. nixd has been dying here for some time and the fallback silently masked it; the Pull-diagnostics fallback in ls.py can swallow LanguageServerTerminatedException, bypassing language-server restart #1770 fix correctly unmasked a real crash. The last green run's log shows no LS-termination errors — but that is not evidence against this, because pytest only prints captured logs for failing tests and CI runs without --log-cli-level, so a swallowed crash in a passing test would leave no trace.
2. nixd itself started crashing recently and f4ecb1a5 is merely coincident. Worth noting that nixd is installed unpinned from nixos-unstable (nix-env -f '<nixpkgs>' -iA nixd), so the binary under test can change between runs independently of anything in this repo.

Either way it looks deterministic rather than flaky: two independent runs, identical failure.

One further note on the test path

assert_file_diagnostics calls language_server.request_text_document_diagnostics(...) directly, so there is no tools_base layer above it to perform the restart-and-retry that the new raise is there to enable ("Termination must propagate so tools_base can restart the LS and retry"). However the nixd crash itself is resolved, a propagated termination in a direct-call test is an unconditional failure.

Possible leads

• The stderr reader thread died too, so nixd's own output was not captured. Capturing its stderr and exit status would show why it dies on diagnostics_sample.nix — that seems like the thing to know first.
• Pinning nixd in the workflow would make the niche batch reproducible over time and would settle reading 1 vs 2 for good.

I ran into this while triaging a red check on #1800 (an Erlang change that cannot touch the nix path) and could not find an existing issue for it. Happy to help if any of this is useful.