test(company-monitoring): freeze viability and evaluation contract - #6186
Merged
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…blic docs site
docs/research/** publishes to the Mintlify site: neither docs/.mintignore nor
docs/.mintlifyignore excludes research/, and a sibling page with no docs.json
navigation entry is live in production today. Both new pages would therefore
have gone public on merge.
That contradicts this PR's own checklist ("no public methodology or API/MCP
contract is published") and would have published a present-tense statement that
the current shared WorldMonitor OpenRouter route does not enforce provider.zdr,
the unit economics for an unannounced product, and the pending X commercial-use
approval status.
Move both records under docs/internal/, which is already excluded by both ignore
files and already holds this genre of audit and protocol document. internal/
fails closed, where a research/ ignore entry would be an opt-out a later docs
refactor could silently drop.
No content changes; the contract test and fixture are untouched.
Found by code review of #6186.
Claude-Session: https://claude.ai/code/session_01HWw5pNo1KJrzgRPxEypkyz
…ifiable A mutation sweep of the committed contract killed only 12 of 36 guard mutants: most gates could be deleted outright and the suite stayed green. Two attacks also landed green -- weakening the base-rate floors and recomputing the digest, and emptying the entire privacy deny-list. Coverage (24 -> 0 surviving mutants across the guards touched here): - point floors, recompute-mismatch guards (incl. the 1e-12 tolerance), every Stage 0 composition guard, the usefulness protocol and identity guards, the provider-policy declarations, and every pinned result container now have a negative test asserting the exact reason code and a stop decision. - the calibration-ceiling branch of evaluateAdmissionMetric was unreachable by any test; it now has one, including the fail-closed missing-measurement case. - the bootstrap percentile order statistic is extracted so its off-by-one is observable; the committed vector's 8999th and 9000th estimates are equal, so it could not be killed through the bootstrap alone. Integrity: - the frozen floors are pinned as independent test literals, as the STOP record already was. The digest is derivable from the file it constrains, so alone it catches accidental drift but not a deliberate coordinated edit. - every top-level protocol key must be declared digest-covered or digest-exempt, closing the hand-written projection against a new section escaping the digest. - evidence digests reject the degenerate single-character shape the suite itself used to model a pass; the synthetic fixtures now use distinct real digests. - chronology is anchored on the digest-covered frozenAt, not only on approvedAt. Privacy (both guards were provably vacuous -- deleting either kept 13/13 green): - fixture string values are now an allow-list (enum token, 64-hex digest, opaque cm_* id, RFC3339 stamp) instead of a URL/email deny-list that admitted bare domains, protocol-relative hosts, s3:// URIs, handles, phones and free text. - root, approval and stage0 are exact-key pinned; the deny-list is normalized and broadened as defence in depth, and is now exercised where no pin applies. - direction and status values are validated against their declared enums. Also derives the per-customer useful bar from minimumUsefulRatePerCustomer instead of a hardcoded 7, and range-checks the admission numerator. The fixture and the approved threshold digest are unchanged: this hardens the verification, not the contract. 13 -> 32 tests. Verified with a two-sided mutation sweep (no-op control survives, all 18 mutants killed) and biome clean. Found by code review of #6186. Claude-Session: https://claude.ai/code/session_01HWw5pNo1KJrzgRPxEypkyz
…e follow-ups Clears the four items deferred from the review of #6186. Engine extraction (one implementation, reachable by production): - shared/company-monitoring-evaluation.ts now holds the statistics, canonical JSON, schema pinning and gate logic. Nothing was reachable outside the test before, so any future dark contract would have re-derived the rules from prose. - evaluateStage0 takes the approved digest as an argument. The digest literal, the recorded STOP reasons and the frozen floors stay in the test source, so weakening a gate still cannot be done by editing the engine alone -- and viability.md's claim that the digest is compared against a literal held outside the frozen protocol stays true. Operator affordance: - scripts/recompute-company-monitoring-digest.mjs prints the digest and names the two approval sites to update; --check verifies fixture, test literal and recomputation agree. Previously the only way to obtain a new digest was to provoke an AssertionError and scrape its diff. It deliberately writes neither site: that is a product-owner decision, not a keystroke. Typechecking (this is what npm run typecheck never covered): - tsconfig.contract-tests.json + typecheck:contract-tests, wired into the EXISTING required typecheck job so no branch-protection context changes. - It immediately found real defects, not just unused imports: evaluateAdmissionMetric computed its lower bound from the unvalidated result.numerator rather than the range-checked local, percentileOrderStatistic could return undefined into an exact-equality assertion, and several index accesses were unguarded. Admission quality: - evaluateAdmissionQuality is a real caller for evaluateAdmissionMetric, which had none, leaving the frozen metric table inert. It is deliberately NOT part of evaluateStage0: every admission metric is defined over published decisions or a Stage 3/4 blind corpus, so folding them in would make Stage 0 unsatisfiable rather than stricter. Unmeasured metrics and unrecognised measurements are both stop reasons, and a test pins the boundary that a Stage 0 `continue` says nothing about admission quality. The fixture and the approved threshold digest are unchanged. 32 -> 33 tests, typecheck and biome clean, and a re-run two-sided mutation sweep across both files kills all 21 mutants with the no-op control surviving. Found by code review of #6186. Claude-Session: https://claude.ai/code/session_01HWw5pNo1KJrzgRPxEypkyz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Company Monitoring now has a preregistered, machine-enforced Stage 0 contract that prevents paid-provider or customer-visible runtime until the viability, usefulness, provider-policy, privacy, and cost gates all pass. The committed baseline is intentionally STOP: only fixtures and dark contracts are permitted, and future empirical results are restricted to opaque aggregate evidence.
The frozen decision package requires minimum denominators plus exact one-sided confidence bounds, the same ten historical impacts to pass with two external target customers, separately evidenced Exa/X/model runtime policy, and an exact one-account 500-company cost package. The modeled $110.8659375 monthly cost passes its $125 arithmetic ceiling, but cannot override missing empirical or provider evidence.
Thresholds, methods, bootstrap seed, provider requirements, and workload shape are integrity-bound to a named approval digest. Tests independently recompute that digest, the Stage 0 decision, statistical bounds, calibration fixture, privacy schemas, usefulness rules, and cost arithmetic so a status-only or point-estimate-only edit cannot promote the product.
Related: #6003
Related: #6002
Type of change
Affected areas
/api/*)Checklist
npm run typecheck)Documentation Alignment Checklist
Validation
npm run lintexited 0; the repository's existing 35 warnings and 9 infos remain outside this four-file diff.npm run test:dataexposed only the managed sandbox blocking macOSmktempinside the pre-push-hook fixture. The exact affected test passed 15/15 outside the sandbox, and the real pre-push hook then completed green outside the sandbox.Post-Deploy Monitoring & Validation
There is no runtime or deploy behavior to monitor from this PR. After merge, the operational state must remain STOP with paid-provider calls, publication, REST writes, workspace behavior, and alerts disabled. The product owner may consider a later
continuerecord only after the frozen base-rate, rediscovery, two-customer usefulness, provider-approval/runtime-enforcement, and measured-cost evidence all pass the machine contract.New concepts
Exact one-sided promotion bounds
A point estimate says what happened in the observed sample; a one-sided lower bound asks how weak the true rate could plausibly be at the frozen confidence level. Requiring both keeps a small or lucky sample from promoting a product on its headline percentage alone.
This contract uses Clopper-Pearson for binomial rates and Garwood for a Poisson event rate, recomputing both independently from aggregate counts. Exact bounds are conservative and appropriate for a hard promotion gate; they are not a substitute for representative sampling or a reason to ignore selection bias.
Screenshots
Not applicable; this change has no visual or customer-visible runtime surface.