Skip to content

feat(dedupe): injectable candidate scope and winner order, plus dedupe pools docs - #15819

Open
blakeaowens wants to merge 17 commits into
bugfixfrom
feature/dedupe-injectable-candidate-scope
Open

feat(dedupe): injectable candidate scope and winner order, plus dedupe pools docs#15819
blakeaowens wants to merge 17 commits into
bugfixfrom
feature/dedupe-injectable-candidate-scope

Conversation

@blakeaowens

@blakeaowens blakeaowens commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Description

Two related changes: a small seam in the deduplication engine, and the documentation for the Pro
feature built on top of it.

The seam. build_candidate_scope_queryset() decides which existing findings a new finding is
compared against, and the winner order decides which one becomes the original. Both were computed
internally with no way for a caller to influence them. This adds two optional keyword arguments,
candidate_qs and ordering_key, threaded through the finders, the candidate generators, the
batch matchers and both dispatchers, plus a scope_filter on the false-positive-history path.

Nothing changes when they are not supplied. Every call site forwards them only when set:

scope_kwargs = {}
if candidate_qs is not None:
    scope_kwargs["candidate_qs"] = candidate_qs
if ordering_key is not None:
    scope_kwargs["ordering_key"] = ordering_key

so the default path builds exactly the queryset and ordering it always did. The seam is
deliberately queryset-shaped rather than a callback or a settings flag: a caller narrows or widens
the candidate set by handing in a queryset, and the engine keeps ownership of how matching itself
works.

The documentation covers DefectDojo Pro's dedupe pools, which are the first consumer of the
seam: named groups of Assets whose findings deduplicate against each other, per matching kind.
The new page explains what a pool does and does not do (it changes which findings are eligible to
be compared, never how two are compared), why the reimport kind cannot widen scope even though it
appears alongside the two kinds that can, and the preview-then-acknowledge contract on the
retroactive actions.

Three existing statements became incomplete rather than wrong once pools exist, so they are
updated in the same change: the scope paragraph and the Pro algorithm summary in About
Deduplication, and that page's troubleshooting table, which previously offered only instance-wide
answers to a per-Asset scope problem. The Enabling Deduplication intro gains a pointer.

Test results

unittests/test_dedupe_injectable_scope.py is new and covers the seam directly: that an injected
queryset narrows and widens the candidate set, that an injected ordering key selects the original,
that omitting both reproduces the default scope and order exactly, that the false-positive
history path honours the same filter, and (in its second class) that the history scope hook is
consulted only when the caller passed no scope.

Run locally against a Pro-flavoured checkout:

python manage.py test unittests.test_dedupe_injectable_scope unittests.test_dedupe_flush_missing_original

21 tests (17 in the scope module, 4 in the flush module), all passing, and the new file is clean
under the OSS ruff pin. The wider deduplication suites were also run green on the consuming side.

False-positive history scope hook

do_false_positive_history_batch now asks the FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD
custom method for its search scope when the caller passes none, the same mechanism as the
existing candidate-filter hook. Without it, the post-import task (which passes no scope) always
searched the finding's own product, so a plugin that widens deduplication to a group of
products could not widen the history search the same way. A provider returns filter keyword
arguments or None; an explicit scope_filter from the caller is never second-guessed, and
with no provider configured the default is unchanged. Four tests pin the default, the provider,
a None answer, and the caller's precedence.

Release ordering

This has to land first, or in the same release train as the DefectDojo Pro change that consumes
it. That change calls the new signature in nine places (eight forwarding call sites on the finders
plus one direct call to build_candidate_scope_queryset), so against a released engine without
candidate_qs every one of them raises a TypeError. Both are milestoned for the same release,
which satisfies the constraint; the note is here so a later retarget of either one does not
quietly break the pairing.

The same ordering matters for the history hook, in a quieter way. An engine without
FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD support ignores the setting: nothing raises, but a
false positive marked in one pooled Asset no longer reaches a sibling on import, and the only
symptom is a Finding that stays active. So the pairing is a behaviour dependency as well as a
signature one.

The reverse direction is safe: both parameters default to None and reproduce the previous
queryset and ordering exactly, so this engine works unchanged with no consumer at all.

Documentation

Included in this PR, under docs/content/triage_findings/finding_deduplication/, plus a 3.3.0
section in docs/content/releases/pro/changelog.md carrying the two behaviour changes the Pro PR
introduces (false-positive history follows deduplication scope; the three global algorithms,
Global Component, Global Vulnerability ID and Global Locations, bounded to the pool for a pooled
Asset) and the two feature lines. The
dedupe-pools page states the Apply Now ceiling (10,000 Findings) and the tuning page describes
what disabling Cross Tool Deduplication does.

The fourth to seventh reviews' documentation items are in the docs commits (the seventh's is
the last commit: the Global Locations intro names the hub, and the em dashes on the Location Drift
Matching and About Deduplication pages are gone; every page this PR touches is now free of them): every deduplication page gives
the Finding Workflow path first with the previous menu's path in parentheses; the sidebar page
no longer lists the three retired tuner pages; the Global Component and Global Locations pages
describe the Matching Configuration procedure, say a pooled Asset is bounded to its pool, and no
longer claim an algorithm change recomputes hashes; the tuning page drops the Support-only and
Assets-only statements and explains that every tool's cross-tool cell starts as Disabled; the
pools page names the Dedupe Pool permissions the way the roles editor does and states the
designated-member rule and the age check; the changelog and overview count three global
algorithms and scope false-positive replication to same-tool matching.

@blakeaowens
blakeaowens requested a review from Maffooch as a code owner August 28, 2026 11:47
@blakeaowens blakeaowens added this to the 3.2.400 milestone Aug 28, 2026
@blakeaowens
blakeaowens marked this pull request as draft August 28, 2026 11:54
@Maffooch Maffooch modified the milestones: 3.2.400, 3.3.0 Aug 31, 2026
@blakeaowens
blakeaowens marked this pull request as ready for review September 5, 2026 01:55
@blakeaowens
blakeaowens force-pushed the feature/dedupe-injectable-candidate-scope branch from 7e52490 to fb6cb0b Compare September 6, 2026 18:16
blakeaowens and others added 13 commits September 7, 2026 04:14
…order

Deduplication derives its own candidate scope: a finding matches within its own
product, narrowed to a single engagement by the engagement checkbox, and that is
the only scope an installation can get. This makes both the scope and the
preference order injectable, so a plugin can express a different one, while
leaving every existing call site on exactly the behaviour it had.

build_candidate_scope_queryset gains candidate_qs. When supplied it replaces the
scope derivation; the loading strategy (defer, select_related, prefetch_related)
is still applied here, so a caller decides which findings are candidates while
the engine keeps deciding how to load them. Candidate confirmation walks
locations, vulnerability ids and CWEs per candidate, so a scope handed in without
those prefetches would silently turn one query into thousands.

The four match generators gain ordering_key: a plain sort key over candidates
that only changes which of several valid candidates is preferred.
_is_candidate_older still runs afterwards, so an ordering key cannot make a newer
finding win, and the global antisymmetry concurrent batches depend on is
unaffected. uid_or_hash applies it to the merged candidate set, since merging two
buckets loses their query order.

Both kwargs thread through find_candidates_for_deduplication_*, match_batch_*,
_dedupe_batch_* and their dispatchers. dedupe_batch_of_findings forwards them to
a custom deduplication method only when set, so a plugin that predates them sees
exactly the arguments it saw before.

false_positive_history gains the same seam as scope_filter, since it builds its
own queryset per algorithm from filter kwargs rather than filtering a supplied
one.

Every kwarg defaults to None and every default path is the previous code, so an
open-source install is unaffected. The pairwise engagement guard is deliberately
untouched: it stays correct for installs that rely on it, and a caller supplying
a scope owns expressing its own isolation.
…guard to bite

The candidate was created first, so it held the lower id and was a legal original
after all. Creating the target first makes the candidate genuinely newer, which is
the case the assertion is about.
Deduplication has always been scoped to one Asset, narrowable to an Engagement.
Pools are the other direction: a named group of Assets whose Findings
deduplicate against each other, per matching kind.

The new page covers what a pool is and is not (it changes which Findings are
eligible to be compared, never how two are compared), the per-kind membership
rule, why reimport appears alongside the two kinds that scope and yet cannot
widen scope, the preview-then-acknowledge contract on both retroactive actions,
where originals collect and why there is no newest-wins, and the parent-edges-
only subtree toggle.

Three statements elsewhere became incomplete rather than wrong, so they are
updated in the same batch: the scope paragraph and the Pro algorithm summary in
About Deduplication, its troubleshooting table (which offered only instance-wide
answers to a per-Asset scope problem), and the Enabling Deduplication intro.
The pools page said a reimport "does not widen what is compared" and left it
there. True of a reimport's own matching, and it reads as "reimports never
deduplicate across a pool", which is false. That reading is exactly how it was
caught.

Both halves are now stated. A reimport's own matching stays inside its Test,
because that matching decides whether a Finding is updated, created or closed
and is scoped to what the scan is authoritative over. The Findings it creates
are then deduplicated under same tool and cross tool, which are pool-scoped.

About Deduplication already said the second half in general terms ("Findings
that remain after Reimport Deduplication are still subject to Same-Tool
Deduplication"); it now names pools as a case of it, so the two pages agree.

The apply-now paragraph carried the same framing and is reworded: pooling for
reimport picks a formula rather than a scope, so there is no widened scope to
re-run, and Findings a reimport created are covered by the other two kinds.
The three tuner deduplication pages were replaced by a single Matching Configuration page, so
this page was describing a UI that no longer exists: a menu path that is gone, a tool dropdown
on one of three pages, and four screenshots of retired screens.

Two claims were not merely stale but wrong in a way that matters.

It promised that changing a tool's settings "automatically triggers a background re-hash of all
existing Findings". Changing an algorithm does not re-hash anything, and cannot: the algorithm
selects which already-stored value is compared, so there is nothing to recompute. A reader
following the old text would wait for a backlog re-hash that is never coming. That section is
replaced with what actually happens, and points at a pool's Apply Now for the case it was
reaching for.

It also described selecting hash fields, which is not possible in this release. Changing hash
fields changes how every stored hash was computed, so it needs a new generation written behind
it before matching moves across, and that is not shipped. The page says so and points at
support rather than describing a control that is not there.

The reference material that is still accurate is kept as is: the algorithms, Content
Fingerprint, the set-based vulnerability-id and CWE matchers, and location drift tracking.
Follows the Pro change restoring hash-field editing to Matching Configuration. The previous
revision said they were not editable and pointed at support, which was true for one commit and
is not now.

Splits the retroactive-re-hash guidance by axis rather than making one claim about both, since
they behave oppositely and conflating them is what made the original page wrong: changing hash
fields recomputes the tool's whole backlog in the background, and changing the algorithm
recomputes nothing at all.

Also records that hash fields are set on the instance default rather than per pool, and why: a
finding stores one hash and every other view of that finding reads it.
…d the nav paths

Four review findings on the dedupe docs batch.

**PRO__location_drift_matching.md still walked users to a deleted page.** It said Settings >
Finding Workflow > Reimport Deduplication and "Enable Track findings as locations change" — a
page this batch removes — while the tuning page claimed the toggle was Support-set. One of those
had to be wrong, and the drift page is the one a user follows step by step. The toggle is
self-serve again on Matching Configuration (the Pro side of this batch adds it), so both pages
now describe that, including the impact review, because turning it on or off changes which
fields the reimport hash is built from and recomputes the tool's backlog.

**Pooling narrows a global algorithm rather than leaving it alone.** The "Pools vs. the global
algorithms" table read as two independent choices at different blast radii, but pro/dedupe/scope.py
bounds Global Component and Global Locations to the pool once an Asset joins one for that kind.
A Global Component user who creates a pool would silently narrow matching they believed was
instance-wide. Called out under the table and on both PRO__global_* pages, which are the ones
that promise "across all Assets".

**The pages disagreed about where the feature lives.** Dedupe Pools said Settings > Deduplication
Settings > Dedupe Pools; the tuning page said Settings > Matching Configuration. Both entries are
in the same nav group, so every page now names the full path and says the two sit beside each
other.

Translations are left for the usual separate pass; only the English pages are updated here.
…dedupe pages

Apply Now runs inside the request and refuses above 50,000 findings, and reports a
reason rather than a silent zero when deduplication is off instance-wide. Neither was
written down, so a user with a large pool met the refusal with no way to anticipate it.

Also replaces the em dashes these pages introduced with colons, parentheses and
periods, per the house copy style.
…ing on every finder

build_candidate_scope_queryset took candidate_qs positionally while all five finders
made it keyword-only. Every caller already passes it by keyword, so closing the
asymmetry costs nothing and stops a positional service argument from ever landing in
the wrong slot.

The tests reached candidate_qs only through the hash path. Each of the five finders
builds its own base queryset, so any of the others could have dropped the argument and
stayed green. All five now assert the same property directly and in both directions:
the derived scope does not reach the other product, and a supplied one does. The
finders returning two maps (uid-or-hash, legacy) are checked on both, since forwarding
could reach one and not the other.
…mport membership as they ship, upgrade and custom-role notes

False-positive history now uses the same scope as deduplication, which narrows replication
for instances with engagement-scoped deduplication: an isolated engagement is excluded from
every other engagement's history and its own imports read only its own. About Deduplication
says so; before, the search always covered the whole asset.

The pools page and Deduplication Tuning both claimed a pool can give its members a different
algorithm. No shipping path creates a pool-level configuration row, so both now say per-pool
overrides are not yet available and every member uses the instance default.

The pools page offered a reimport membership kind and described it as selecting the formula a
reimport uses. The resolver never consults a pool for that kind, so the membership did nothing;
the kind is withdrawn from the page and the text explains why, and that findings a reimport
creates still deduplicate across the pool.

Deduplication Tuning gains an Upgrading section: the cutover from the tuner pages is one-way
and a database backup should precede the upgrade. The pools page gains a custom-roles note:
the upgrade carries Edit Tuner to all four pool permissions and View Tuner to View Dedupe Pool,
and only roles created afterwards need the grants made by hand.
…ng cross tool

Adds the 3.3.0 section to the Pro changelog with the two behaviour changes
the matching-configuration work introduces (false-positive history follows
deduplication scope; Global Component and Global Locations matching bounded
to the pool for a pooled Asset) and the two feature lines. The dedupe-pools
page states the lowered Apply Now ceiling (10,000 Findings), and the tuning
page says what setting a tool's cross-tool algorithm back to Disabled does.
…cedures for the new pages

Every deduplication page now gives the Finding Workflow path first, with the
previous menu's path in parentheses, the way the enabling page does. The
sidebar page no longer describes the three retired tuner pages. The global
algorithm pages describe the hub procedure instead of the tuner form, say that
a pooled Asset bounds them to its pool rather than that matching is always
global, and no longer claim an algorithm change recomputes hashes. The tuning
page drops the Support-only and Assets-only statements that pools and the
hub made false, and explains that every tool's cross-tool cell starts as
Disabled. The pools page names the Dedupe Pool permissions the way the roles
editor does, notes that the designated Asset must be a member and that the
engine's age check still applies, and says the subtree toggle pools only what
the caller can read. The changelog and the overview count three global
algorithms and scope false-positive replication to same-tool matching.
do_false_positive_history_batch accepts a scope_filter, but the post-import task
and the bulk edit call it without one, so a plugin that widens deduplication to a
group of products (FINDING_DEDUPE_BATCH_METHOD plus the candidate scope hook) could
not widen the history search the same way: a false positive marked in a sibling
product never reached a new import.

When the caller supplied no scope, the batch now asks the optional
FINDING_FALSE_POSITIVE_HISTORY_SCOPE_METHOD for one. The default stays the product,
a provider returning None keeps the default, and an explicit scope_filter always
wins. Four tests pin those rules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@blakeaowens
blakeaowens force-pushed the feature/dedupe-injectable-candidate-scope branch from fb6cb0b to a588d98 Compare September 7, 2026 10:38
blakeaowens and others added 4 commits September 7, 2026 15:39
…pages, table row, wording

The Global Component and Global Locations pages describe the per-cell Matching
Configuration procedure for cross tool and for reverting, and their step lists
are numbered without gaps. The component page's matching rule says the
instance-wide reach stops at a pool. The pools page's comparison table gains the
Global Vulnerability ID row the callout beneath it already named, and quotes the
subtree action the way the panel renders it. The sidebar page counts the two
deduplication pages. The tuning page says selecting the cell edits the fields.
The em dashes on these pages are replaced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Global Component and Global Locations pages said the algorithm becomes
available "in the Tuner"; they now name Settings > Finding Workflow > Matching
Configuration like the rest of the page. The remaining em dashes on the tuning
and Global Locations pages are replaced with colons, commas and parentheses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s on two pages

The Global Locations page said the algorithm "does not appear in the Tuner"
when Locations is off; it now names Settings > Finding Workflow > Matching
Configuration. The em dashes on the Location Drift Matching page and the one on
About Deduplication are replaced with colons, commas and parentheses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two older changelog entries and the enabling-deduplication page kept an em dash
each; with these replaced, every documentation page this PR touches is free of
them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants