Pin the version against pyproject; point the bug template at --version - #45
Merged
Conversation
…rsion `tooltrace --version` already works. Nothing tested that it kept working, and nothing compared `FRAMEWORK_VERSION` to the version `pyproject.toml` declares — the sibling project api-verity-lab carried `__version__ = "0.1.0"` against a declared 0.2.0 for exactly that reason. `tests/test_version_is_reportable.py` closes both: the flag exits 0 and prints the version, and the two places the number is written down must agree. The bug-report template's version field carried the placeholder `0.1.0 (pip show tooltrace-bench)`. The version in it was two releases stale (0.3.0 today) and would go stale again on the next release, so the placeholder now names the command rather than a number: reporters run `tooltrace --version` and paste the answer.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
webdevsamran
added a commit
that referenced
this pull request
Sep 12, 2026
`docs/feature-status.md` grades 122 capabilities and calls itself this
project's verification artifact. Its path check inspected only backtick-quoted
tokens containing `/` or ending `.py` -- and 97 of the 122 rows cited bare prose
("clustering module", "calibration sets module", "conformance fixtures module").
So roughly three quarters of the table was never verified by anything, which is
how four rows survived as `I` with no implementation behind them:
- #45 multi-judge adapters and #46 judge calibration datasets -> `D` (declared
only). No file matching `judge*.py` or `calibrat*.py` has ever existed; the
only mentions of the word are a reserved `judge_config` field that is never
assigned, a `ScoringContract.judge_required` declaration, and a lint warning.
- #42 abstention/calibration tasks and #121 backup/restore tooling -> `N` (not
implemented). Nothing in the package mentions abstention or clarification, and
neither backup nor restore appears in the server or in the
`docs/self-hosting.md` that #121 cited as its evidence.
Seventy further rows now cite a path and the symbols inside it, replacing prose.
Every citation was verified against the file before being written -- a
fabricated citation would be the same defect in a new coat.
Three mechanisms stop this recurring:
- an `I`/`E`/`P` row must cite something inspectable (`N` and `D` are exempt by
definition -- their content is that nothing ships);
- a claim probe rejects any row claiming a capability whose implementation is
absent from disk, with a non-vacuity test asserting the probe still matches
rows 45 and 46 and still does *not* match row 44, which legitimately claims
independence *from* a judge;
- the path resolver moved to `scripts/check_doc_code_refs.py` and runs in CI
across all 25 documents. The test imports it rather than keeping a private
copy, which is how the two drifted in the first place.
That checker immediately found `docs/differentiators.md` citing four modules
that do not exist, one of them `scoring/judges.py`. Its "judge-independent"
section claimed multi-judge disagreement reporting and calibration-drift
datasets; it now states what is true, which is the stronger claim -- scoring is
judge-*free*, so a third party can recompute a score from the bundle alone.
The resolver also had a real bug: `lstrip("./")` strips *characters*, so a
dotfile citation silently lost its leading dot. Now `removeprefix`.
The checker is deliberately narrow -- a path needs a separator and a known
extension -- and skips fenced blocks and blockquotes, because a dated correction
note exists precisely to say "this row used to cite X". A checker that cries
wolf is one people learn to ignore.
Adds one step to the existing `python` job. No job names changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
tooltrace --versionalready works — this is not a bug fix, it is a guard.During a final verification pass I installed all four sibling projects' wheels into clean venvs and ran
--versionon each. Two had no such flag; this repo was fine. But nothing here tested that, and nothing comparedFRAMEWORK_VERSIONto the versionpyproject.tomldeclares.That comparison is exactly what was missing in
api-verity-lab, where__version__sat at0.1.0against a declared0.2.0— invisible because no test looked.What this adds
tests/test_version_is_reportable.py:--versionexits 0 and prints the versionFRAMEWORK_VERSION == pyproject's versionThe number is written down in more than one place. This makes disagreement fail the build rather than ship.
Bug template
The version field's placeholder was:
Two problems:
0.1.0is two releases stale (0.3.0 today), and it would go stale again on the next release. It now names the command instead of a number —— so reporters run the tool and paste the answer, and there is nothing left to drift.
Checks
ruff check,ruff format --check,mypyall clean