root-cause(scrapers): a repair can persist only by freezing a field, because refusal is not a capability

[quntao-z]

Summary

A repair can make a correction stick in exactly one way today: write manuallyLockedFields, which freezes the field against every future improvement (#2612). 107 of the 122 remaining lock instances exist for that reason alone. #2542 built the capability that was supposed to replace it, and #3153 gave a second source the ability to use it, but retraction answers a different question from the one these locks are asking.

Absence and refusal are different capabilities, and only absence exists. Field retraction handles "the page stopped saying it". These locks hold values that are wrong, not gone: a journal article page, a personal site, another institution's profile. The page still carries the value, so a correct source will keep asserting it and retraction will correctly never fire. The repo has now recorded this shape from three directions: a probe verdict alone is not a durable refusal (#2567), omission is not absence (#2647), and this.

Measured on Development

33 rows lock websiteUrl while storing no value. 28 live rival assertions block them. Running each through researchHomeWebsiteUrlDecision, the per-rule refusal vocabulary #3068 landed:

verdict	count
ADMITTED by every rule	22
yale-path-vocabulary	2
person-profile-or-directory-path	2
listing-or-index	2

Two separable findings.

The refusal vocabulary is audit-only. Its only caller in the whole tree is researchHomeWebsiteUrlRefusalAuditCore.ts. Nothing in the materialize or serve path consults it, so it names defects and prevents none. That is why 6 of the 28 blocking values are ones the repo can already classify as inadmissible and still derives anyway. Those 6 need no new capability, only a caller, and they are the cheap half of this issue.

The other 22 are genuinely per-row. No rule refuses them and none should: tedcohenlab.org is a real lab site, just not this row's. Admissibility here is a judgement about a value on a row, not about a URL shape, so no vocabulary can reach it.

What to build

A durable refusal record: a positive, dated, evidence-bearing statement that one specific stored value is not admissible at one field on one row, attributed to the rule or operation that refused it. Shape follows fieldLockProvenance, and the attribution vocabulary follows #3068.

Requirements, and the traps each one is designed against.

• Readable by materialization. A repair persists by recording a refusal instead of writing a lock. That is the entire point of the issue.
• It removes a candidate, it does not pin a field. A refusal keys on a normalized value, so a later, better value at the same field is untouched. This is the answer to "have you reinvented the lock": a lock makes the field unreachable, a refusal makes one value unreachable and lets the corpus decide among the rest.
• Withdrawable. An explicit withdrawnAt plus reason, retired by a reviewed operation, so a refusal recorded under a rule that later changes does not become permanent.
• Not expressible as superseded, and this is provable rather than asserted. superseded retires one observation row; a re-scrape mints a fresh live row carrying the same value, which is precisely the fix(scrapers): the observation engine cannot retract a field a source stops asserting, so a stale websiteUrl is permanent #2542 mechanism and why re-scraping does not retract a dead sourceUrl. A refusal is a standing judgement about a (row, field, value) triple, so it survives re-observation. The same argument rules out the four rollback reasons already in use: they are retirement records attached to observations, not admissibility rules about values. 181,047 rows carry superseded and not one of them can stop a value coming back.

Definition of done

Stored-data, so merging delivers nothing. Done is: a refusal recorded against one of the 22 on Development, materialize run, the wrong value shown not to return, that row's lock released, and the served surface re-read. A lock released without the refusal actually holding is the August 2026 regression in a new place. Mutation-checked both ways: a refusal that suppresses a good value must turn a test red, and a missing refusal must let the wrong value back.

Refs #2612, #2542, #3068, #3135.

[quntao-z]

The capability landed in #3178 and is proven end to end, but reopening: 20 of the 21 eligible rows still carry the defect, and each is a per-row judgement rather than a sweep.

The two properties you asked me to state explicitly

How a refusal is superseded, so it never becomes a permanent veto. Two independent ways, and the first is structural rather than procedural.

1. A refusal keys on a normalized value, not on a field. refusedResolverObservations drops only the matching observations before resolveAllFields runs, so a later, better value at the same field wins normally and the row keeps improving. A field whose every candidate is refused resolves to nothing, which is the retraction the repair was reaching for. This is the whole difference from manuallyLockedFields, which removes the field itself.
2. withdrawnAt plus a required reason, via research-entity:refuse-field-value --withdraw, for when a rule changes or a judgement was wrong.

Both are mutation-checked: making a refusal suppress the whole field goes red, and making a withdrawn refusal still apply goes red.

Why it cannot be superseded or a rollback reason, proven rather than asserted. The integration probe supersedes the observation, re-runs the pass, and the wrong value comes back - that is the #2542 mechanism itself, and why 181,047 superseded rows cannot stop a value returning. It then records a refusal instead and the value stays out. The rollback reasons lose for the same reason: they are retirement records about observation rows, not admissibility rules about values.

The finding that made it actually work

The resolver screen alone did not hold. deriveResearchEntityWebsiteUrl promotes a cited sourceUrl into an empty websiteUrl slot and was gated only by manuallyLockedFields. That is why clearing a wrong websiteUrl never stuck, and why a lock was the only thing that could ever hold one: the resolver dropped the value and this lane put it straight back from the citation. A refusal that reaches one derivation path and not the other reaches neither.

End-to-end proof, Development

A FACULTY_RESEARCH_AREA row whose single live rival websiteUrl is a journals.sagepub.com DOI article page cited from a School of Management faculty directory.

step	result
RED control: release lane with the lock ignored	engine derives the DOI, plannedReleases: 0
record the refusal	1 live refusal, row re-gated
release lane again	engine derives (empty), plannedReleases: 1, applied
materialize with manuallyLockedFields: []	screen fires, the DOI does not return
served surface	detail route still serves the row, student_ready, no websiteUrl, unchanged for students

The lock came off because the correction became durable, not because it was forced off. Corpus 76 rows / 122 instances to 75 / 121; keptEngineDisagrees 33 to 32.

A false refusal I nearly recorded

My first pick was a student_ready LAB row whose blocked value was medicine.yale.edu/lab/<name>/ with that row's own profile as the citing source. That URL looks correct for the row, so refusing it would have written a false judgement and released a lock on a lie. I re-picked on a demonstrable predicate. Recording it because the failure mode of this capability is a wrong refusal, and no test catches that - only a person reading the row does. That is the argument against ever sweeping these.

What remains

• 20 eligible rows: locked websiteUrl, storing nothing, one live rival that every rule admits. Each needs a read of the row before a refusal is recorded. The hosts split roughly into other-institution profiles, personal sites, publisher platforms, one unparseable value, and several medicine.yale.edu paths that may well be correct and should be re-examined rather than refused.
• The 6 values the existing vocabulary already refuses need no refusal record at all. They need researchHomeWebsiteUrlDecision to have a caller on the write path, since today its only caller is an audit. That is the cheap half and it is still open.

[quntao-z]

Read all 21 eligible rows one at a time. No sweep, per the near-miss recorded above. Reached 7; the other 14 are a different finding.

Delivered

2 released with no judgement at all, purely because #3183 wired the refusal vocabulary onto the write path: their blocked values are a department directory page and a team-member page on another lab's site, which the vocabulary already classified and now declines at adoption. Those needed no refusal record.

5 refused and released, each on a demonstrable defect rather than unfamiliarity:

the blocked value	rule recorded
a local filesystem path pasted out of a document, ending in a news article URL	operator_judgement
a defunct expertise-database CGI endpoint	not_this_rows_research
a journal's editor's-picks page	not_this_rows_research
another researcher's lab site, on a row that is a different person	wrong_owner
another institution's scholar profile, on a Yale row	wrong_owner

All 7 verified the same way: re-materialize with manuallyLockedFields: [], confirm the value does not return, re-read getResearchGroupDetail including the tier. 7 of 7 clean, every tier unchanged, no served websiteUrl gained or lost. The two operator_review rows still correctly return null from the detail route, as they did before.

Corpus: 75 rows / 121 lock instances to 68 rows / 114, and keptEngineDisagrees 32 to 25.

The finding in the other 14, which is more useful than the releases

Most of these locks are holding a value that looks correct. Reading them one at a time, 13 or 14 of the 21 have a blocked value that is a plausible or clearly correct research home for that row:

• a medicine.yale.edu/lab/<name>/ path whose citing source is that same person's profile, twice
• a campuspress.yale.edu/<name>group/ site cited from that person's chemistry profile
• five personal academic domains and Yale personal-homepage paths, cited from that person's own department roster entry
• a department research-group subdomain, a genome-analysis centre the row's person directs, a fertility centre, and a cancer-centre resource page

Refusing any of these would have written a false judgement and released a lock on a lie, which is precisely the near-miss recorded above and why none of them was touched.

That reframes #2612's backlog. The lock population is not mostly wrong values frozen in place; it is mostly rows where a repair cleared a correct URL and then had to freeze the field to keep it cleared. The likely author is the promotion-regression repair, which cleared URLs that Production had regressed. So the open question for those rows is not "what refusal do they need" but "why was a correct value cleared, and should it be restored" - a restore question, not a refusal one, and it should not be answered by this issue's mechanism.

Two of the 21 are declined only by yale-path-vocabulary, which #3183 deliberately does not wire to the write path, and both of those values look correct on reading, which is exactly the argument for that exclusion.

Still open

• The 14 correct-looking locks need a restore decision rather than a refusal. Worth its own issue once someone confirms the promotion-regression attribution, which fieldLockProvenance cannot answer because all of these record unknown.
• 87 lock instances remain keptNotRevisitable, all unknown reason, unchanged by this work.
• root-cause(scrapers): field retraction has never retracted anything, and the recorded reason is wrong #3135 still owes the two real roster runs over roughly 118 live pages, low yield expected.

Noting for whoever reads the working tree: the primary checkout currently carries another session's uncommitted client changes and a scratch script. I did not touch them; all of my work here was data operations plus the merged #3183.

[quntao-z]

Probed the 16 locks the engine wants to overwrite: none of its proposed URLs is alive

After #3135's roster runs, 16 websiteUrl locked instances read keep_engine_disagrees, all of them storing no value while the engine proposes a URL. I described that as "the page still carries a link, so releasing the lock would restore it". That framing was incomplete in the way that matters: "the page carries a link" is a statement about the source page, not about whether the link's target answers.

Probed serially with a browser user agent, 1.5s apart, classified by the repo's own classifySourceLinkHealth so the verdicts are read the way the corpus reads them:

verdict	targets	what it means
GONE (404)	7	the resource is gone
does not resolve (ENOTFOUND, no DNS record)	5	dead by this repo's own rule
private address (10.x)	1	not publicly reachable rather than dead
inconclusive, a server answered	3	1 connection reset, 1 expired certificate, 1 certificate altname mismatch
HEALTHY	0

So 12 of 16 are dead by the repo's rule, 1 is unreachable rather than gone, 3 answered inconclusively, and not one is a live page.

This is not a restore cohort, and nothing should be restored

The locks are holding an empty value correctly. What is wrong is the engine's proposal: it keeps offering a URL whose target is gone, so the disagreement is the engine being wrong rather than the lock being stale. Releasing any of these would publish a dead link.

That is precisely this issue's shape: the engine has no way to record that a value was refused, so it re-proposes the same dead URL on every pass and the only thing that stops it reaching a student is a frozen field. #3208 landing "a dead page is not an admissible research website" is the capability these 16 need; retraction is not, and #3135 is closed on that basis.

Two cautions for whoever picks these up

The 3 inconclusive ones must not be treated as dead. A TLS error and a connection reset both mean a server answered, and the standing rule is that 403/429/5xx/timeout never retire a link. The private-address one is the same class: it is a host that exists and is not publicly routable.

And a caution about my own instrument, since it matters for anyone re-running it: my throwaway probe labelled 5 unresolvable hosts as "private address" because it keyed on an error message rather than on a resolved address. Only 1 of the 6 genuinely resolves to a private address; a dig check separated them. The repo's classifier was right in both cases, my label was not, and the correct split is the table above.

[quntao-z]

Addendum: #3225 moved the denominator under this probe, and the verdicts still stand

The 16 probed above were selected from a research-entity:release-field-locks report taken before #3225, which narrowed candidate selection to ask the engine for each locked field's engineValue instead of scanning observations. That report read 105 locked instances across 59 rows; locks now read 98 across 52. So the cohort boundary has moved and should be re-derived from a post-#3225 run before anyone treats 16 as the current count.

What does not move is the per-URL evidence, because a probe verdict is a fact about a host rather than about how the row was selected: of the 16 targets, 7 answered 404, 5 have no DNS record, 1 resolves to a private address, 3 answered inconclusively, and 0 were healthy. Any of those URLs still proposed by the engine after the narrowing is still a dead or unreachable target.

The practical consequence for this issue is unchanged and if anything sharper: #3225 established that the engine is the right thing to ask what a locked field wants, and this probe establishes that on these rows what it wants is a dead page. That is the refusal capability, not a retraction and not a restore.