Skip to content

Research-home descriptions: 511 student-ready-blocked entities have no stored prose to synthesise from (scraper gap) #1878

Description

@quntao-z

Context

Re-scoped 2026-09-22. The split of this cohort by cause, with current counts and the host ranking, is in this comment. The original numbers below are kept as the record of what was believed at the time.

This issue now tracks only the rows with no stored prose to synthesise from: 511 rows, 110 of which carry no hard blocker outside the description family. The card-derivation half of the original cohort was a distinct root cause and is tracked in #2959.


While repairing the thin/missing-description student-visibility blocker class (442 non-archived research entities blocked only by description-family hard reasons: missing_card_description + thin_description/missing_description), the grounded LLM-synthesis lane could only promote the subset that carries real source material.

Applied result (Development): 42 promoted to student_ready; 400 still blocked.

Root cause

The remaining entities have no usable source prose to synthesize from, even after the NIH/NSF grant-abstract fallback. They were never captured with a research description at scrape time. Source-availability breakdown over the class:

  • ~209 had stored text or a grant abstract (synthesizable); of the candidates actually attempted, 42 passed grounding + quality + genuine-prose gates.
  • ~233 were source-empty (no full description, no corroborating profile-synthesis text, no NIH/NSF grant abstract).

Dominant source hosts among the source-empty entities:

  • medicine.yale.edu - 71
  • orcid.org - 27
  • campuspress.yale.edu - 13
  • som.yale.edu - 7
  • wti.yale.edu - 6
  • long tail of departmental hosts

Proposed follow-up

Investigate the medicine.yale.edu faculty/lab scraper first (largest single gap): confirm whether the source pages carry a research-focus blurb that the scraper is not capturing into a fullDescription observation, and if so, capture it. ORCID-only entities likely need a different source (they carry no narrative research description on ORCID itself).

No entity should be promoted without genuine source-backed prose; these stay correctly blocked until re-scraped.

Related: #1876 / #1877 (record-scoped, durable synthesis tooling used for the 42 promotions).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions