Add review workflow to approve predicted phrases before injection - #5267
Draft
Kaushik-Kumar-CEG wants to merge 7 commits into
Draft
Add review workflow to approve predicted phrases before injection#5267Kaushik-Kumar-CEG wants to merge 7 commits into
Kaushik-Kumar-CEG wants to merge 7 commits into
Conversation
Kaushik-Kumar-CEG
marked this pull request as draft
August 16, 2026 17:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of #5244, builds on #5243. Adds a review pass in front of the injection tool
from #5262 so a maintainer sees a predicted phrase before it is written
a wrong required phrase drops a real license detection, so nothing reaches a rule
on the model's word alone
this PR:
review_ml_phrases.py- three commands.predictscores every predictedphrase and writes them to a jsonl file,
reviewwalks the uncertain ones pasta maintainer with a diff and y/n/e/q,
applyinjects what came back approved.--dry-run,--limit,--license-expression,--auto-threshold,--review-thresholdtest_review_ml_phrases.py- 67 tests, the tagger and tokenizer are faked andthe crf is built in the test, so no weights and no network
injection goes through
add_ml_phrases.inject, sois_good,find_phrase_spans_in_textandadd_required_phrase_to_rulestill do the work.extract_spansandfirst_subword_positionscome fromtrain_model. the diffpreview is
add_required_phrase_to_ruleunderdry_runrather than my own spanpicking, so what is shown is what gets written. nothing in
src/changesneeds #5222 and #5262 to import, same as #5262 itself
notes
predict_wordsdecodes with viterbi and returns no score, so there was nothingto build the tiers on.
predictscores the decoded path twice, once free andonce with every label but the decoded one pinned out of reach at the span's
words. the difference is the probability mass on paths that tag that span the way
it was decoded. it says what the model thinks, not that the phrase is right,
is_goodand the review step are still the gate--auto-threshold0.95 and--review-threshold0.60 are placeholders, thecheckpoint is not published yet so there was nothing to calibrate against.
predictprints a confidence histogram so the numbers can be picked from realdata later
phrases a human approves still get
source: ml_model, that is whatinjectsets. the review file keeps the decision and the original predicted text if you
want to tell them apart
Rule.dump()opens withoutnewline=''so on windows it rewrites every lineending and the diff covers the whole file. not from this script, it hits anything
calling
rule.dump(), but worth knowing before a batch run on windowschecks
67 tests pass with no model.
pytest tests/licensedcode/ -k required_phrasestill passes, 44 tests
fed the gold labels back in as a perfect prediction over 400 marked rules with
the markers stripped: 382 phrases survive
is_good, 379 inject, a secondapplyinjects nothing and leaves every file byte identical. one real writechecked with git diff and restored
the confidence maths is checked against brute force enumeration on a small crf,
agrees to 1e-12
disclosure : used Claude to help review and clean up bugs in script