[chinapandaman]

Resolves #2099 (action plan item E3-01, review R01).

Baseline

At db35aa0a (pypdf 6.21.4), the appendix A reproduction still turned a 4,099-byte source into a 4,296-byte one-page output that contained EXCLUDED_SECRET_SECOND_PAGE. The widget's /P was object (8, 0), not the output page (3, 0), and that page's /Parent /Count was 2.

A three-page test form leaked every excluded marker into pages[0]: page 2's text and image, a field value only on page 2, a sibling's value under a shared parent, and other pages' appearances of a shared text field and a radio group. There were three leak paths:

• each widget's /P copied the source page tree
• a shared /Parent copied its other /Kids and their pages
• push-button GoTo actions in /A and /AA copied their target pages

pypdf's PdfWriter.append(pages=...) leaks through those actions too. The other clone paths (merge, draw(), coordinate grid, bulk field creation) also copied each widget's source page and page tree as stray objects.

Changes

• One clone plan: a new ClonePlan class in PyPDFForm/lib/clone.py builds the destination graph for every clone path. It is used through clone_page_annotations and get_page_streams in utils.py. Before cloning, it walks everything reachable from the copied annotations in a throwaway reader and edits it. No pypdf private state is used.

• Ownership: references to a copied page, including each annotation's own /P, point at its output page. References to the source catalog point at the output catalog.

• Shared fields: a field gets a new /Kids array without annotations that aren't copied and fields whose widgets are all on excluded pages, so a shared or parent field keeps only its copied part. The original array stays intact, so an action that refers to it is still checked. When a button field's kids are widgets and its /Opt, inherited or not, has one entry per kid, the field gets its own trimmed /Opt. Positional on-states (/0, /1, ...) are renumbered in /AS, /V and /DV (inherited or not), and the normal, rollover, and down appearances. Each remaining button gets its own /AP with renamed copies of its state dictionaries, so dictionaries shared with other fields are unchanged; appearance streams stay shared. A /V or /DV naming a removed button is deleted from the field, or set to /Off when a parent would still supply a value. Choice fields keep their /Opt.

• Cross-page policy (confirmed):

  • Links to excluded pages are still dropped (E2-07).
  • GoTo actions whose target page is excluded are removed; their button or field stays.
  • Any other reference to excluded content raises ValueError naming the page and entry, including fields that ResetForm/SubmitForm /Fields or Hide /T name by fully qualified name (direct or indirect strings), e.g. Cannot copy page 1: /Annots[7]/A/Fields[0] refers to content that is not being copied, ....

  Because .pages extracts every page at once, one rejected reference fails the whole extraction, and create extract exits 1 without writing output.

• Extraction: each page is copied from its own reader in one pass, without /Annots and article beads, then its checked annotations are cloned onto it. This replaces the remove-widgets, split, and re-clone pipeline, along with the single-page branch and page_num argument of copy_watermark_widgets.

• Docs: the User Guide warning on page extraction, the pages and create extract docstrings, and the E3-01 record in the action plan (separate commit).

• Expected PDFs: 54 regenerated (separate commit) and visually compared by the maintainer.

Validation

• New tests in tests/test_functional.py: nine tests, 19 cases. 18 fail against master. The exception is test_pages_keep_actions_naming_copied_fields, which guards against rejecting too much. Each review fix added cases that fail against the revision it fixed.
  • test_pages_exclude_other_page_content: the appendix A reproduction
  • test_pages_keep_only_own_part_of_cross_page_fields: a three-page pikepdf form with a shared field, parent fields, a cross-page radio group, a page 2 image and value, a popup, navigation buttons, a page action, and a link. Each page, pages[0:2], and a merge of pages 1 and 3 are checked across every serialized object for each page's markers, page ownership, /Kids, values, and actions.
  • test_pages_reject_other_cross_page_references: ResetForm, Hide, and /IRT references to page 2 and 3 content, and the same targets by name in ResetForm, SubmitForm, and Hide, as strings, arrays, and indirect strings
  • test_pages_check_targets_sharing_pruned_kids: a ResetForm /Fields that is the field's own /Kids array or a copy of it, both rejected
  • test_pages_keep_actions_naming_copied_fields: names of copied, shared, and unknown fields are kept
  • test_pages_renumber_positional_radio_states: positional radio states and /Opt after extracting either page, a single-stream rollover, a choice field whose option count matches its widgets, and no change after draw()
  • test_pages_renumber_shared_and_inherited_radio_states: a shared /N and /D state dictionary, rollover appearances, and /Opt and /V inherited from a parent whose other kids are radio groups
  • test_pages_renumber_without_changing_other_fields: a state dictionary shared by a group split across pages and a group wholly on the extracted page, which keeps its selection
  • test_clone_paths_keep_page_ownership: page ownership and remapped GoTo targets after draw(), the coordinate grid, merging, and bulk field creation
• Expected PDFs: compared with the previous versions, page counts, annotations (subtype, full name, value, appearance state, rect), /Kids counts, decoded page contents, AcroForm field order, and reopened data/schema are identical. Stray page objects fall from 124 to 0, annotations with a foreign /P from 3,786 to 0, and total size from 29.6 MB to 27.9 MB.
• Manual checks:
  • create extract on a rejected PDF exits 1 without output.
  • A /Data reference to the catalog points at the output catalog after extraction, draw(), and merging.
  • On the committed samples, extracting all pages is faster than on master: 1.70 s vs 5.50 s for test_bulk_create_fields_stress_max.pdf, 0.12 s vs 0.31 s for sample_template_with_full_key.pdf.
• Checks: make test-all (487 passed, 100% coverage), make linting, ruff format --check, a strict MkDocs build, and git diff --check outside the expected PDFs (whose xref lines always end in a space) all passed, using local ./venv with standard zlib.

Not addressed

• Extraction is not redaction. Resources a page shares with other pages are kept, as are widgetless fields referenced by a copied action.
• Numeric on-states that don't match their own position (radios in unison) are not renumbered. An inherited /DV naming a removed button still becomes a local /Off, which PyPDFForm reads incorrectly until PPF-2112: Radio buttons are read as checkboxes when the group's default value is Off #2112 (radios with a default of Off read as checkboxes) is fixed. JavaScript that names fields is not checked. An /AA dictionary whose actions were all removed is left empty.
• When merging, a page-level action that goes to another page still copies that page outside the page tree. pypdf's page copy makes that copy, not annotation cloning, and it is unchanged from master.
• As before, extracted pages lack the source AcroForm's /DA, /DR, and widgetless fields. A field shared by pages extracted together becomes two fields when they are merged.
• Not confirmed separately: a shared field keeps its own /V when it names a button on an excluded page by a non-positional state.

🤖 Generated with Claude Code

[chinapandaman]

Reviewed at 02bcd3afc3f6c52eecdb5fda6ae3923d1e89c752. I found two P2 issues:

1. Keep radio export options and numeric appearance states consistent when pruning /Kids. (clone.py, lines 325–346)

   Using build_cross_page_form(), give the radio group /Opt [(FIRST_PAGE_EXPORT) (THIRD_PAGE_EXPORT)] and corresponding appearance states /0 and /1. Extracting page 3 leaves one /Kids entry, both /Opt entries, and the retained widget's on-state and parent /V at /1. The base revision retains both children, so its positional mapping remains consistent. The PR lists the unpruned /Opt array as a limitation, but pruning /Kids introduces an inconsistent radio field: the PDF specification requires one /Opt entry per child and numeric on-states corresponding to the child's position. Pruning should update /Opt and the associated /AP, /AS, and /V mappings together. See PDF 32000-1:2008, Table 227.

2. Apply cross-page reference rejection to field names as well as indirect references. (clone.py, lines 271–282)

   In the same fixture, set a page 1 button's action to << /S /ResetForm /Fields [(person.address)] >>, or << /S /Hide /T (person.address) >>. Extracting page 1 removes the page 2 person.address field but retains either action without raising ValueError. This leaves a dangling target and bypasses the documented rejection policy solely because the target uses its fully qualified name. Both string targets and indirect targets are permitted by the PDF specification. Resolve named field targets against the source field hierarchy and reject references to excluded fields under the same policy. See PDF 32000-1:2008, Tables 210 and 238.

Validation: make check-zlib reported standard zlib; make test-all passed all 474 tests with 100% statement coverage; make linting passed. Both cases above were reproduced separately. No tracked files were changed, and the maintainer-only visual PDF comparison was not repeated.

[chinapandaman]

Thanks. Both findings reproduced at 02bcd3af and are fixed in b5bae25c; the E3-01 record is updated in 5529465c.

1. Radio /Opt and positional states: when a button field's /Kids are pruned and its /Opt has one entry per kid, /Opt keeps only the remaining kids' entries. Positional on-states are renumbered to the new positions in each remaining kid's /AS and /AP /N and /D keys, and in the field's /V and /DV. A /V or /DV that names a removed position becomes /Off, since keeping it would select a different button. In your scenario, extracting page 3 gives /Opt [(THIRD_PAGE_EXPORT)] with the button and /V at /0; extracting page 1 gives /Opt [(FIRST_PAGE_EXPORT)] with /V /Off. Choice fields keep their /Opt, which lists options rather than kids; a test covers a choice field whose option count matches its widgets.
2. Named field targets: ResetForm/SubmitForm /Fields and Hide /T strings are resolved by fully qualified name against the source field hierarchy and rejected under the same rule as references. Both of your examples now raise Cannot copy page 1: /Annots[7]/A/Fields[0] ... and ... /A/T .... The index includes kids that were already pruned, which my first attempt missed: page 1's walk had already pruned person.address before the action was reached. Names of copied or unknown fields are kept.

Tests: four named-target cases (strings and arrays, in all three actions), a test that names of copied and unknown fields are kept, and a positional-radio test. The new reject cases and the radio test fail against 02bcd3af. No expected PDF changed. make test-all passed 480 tests with 100% coverage, and make linting, ruff format --check, a strict MkDocs build, and git diff --check passed (local ./venv, standard zlib).

Still not handled, as noted in the description: numeric states that don't match their own position (radios in unison), and field names inside JavaScript.

🤖 Generated with Claude Code

[chinapandaman]

Follow-up review at 5529465c29846a2b9905039fe381e1aeb8ba1e7e: both original examples now pass. I reproduced four remaining P2 issues, including two regressions in the new appearance-state renumbering.

1. Avoid renaming a shared appearance dictionary more than once. (clone.py, lines 441–455)

   Give a positional radio group three widgets, with index 0 on an excluded page and indices 1 and 2 on the retained page. For widget 2, let /AP /N and /D reference the same indirect appearance dictionary containing /2, and select that widget with /V /2 and /AS /2. After extraction, the state mapping is /1 -> /0 and /2 -> /1. Processing /N changes the shared dictionary's /2 to /1; processing /D changes that same entry again to /0. The field's /V and widget's /AS are /1, but neither appearance dictionary has that state. Reopening the output changes data["radio"] from 1 at the previous PR revision to None. Each distinct appearance dictionary needs a single transformation, or an independent copy where necessary.

2. Resolve inherited /Opt before pruning button widgets. (clone.py, lines 406–411)

   In build_cross_page_form(), give the radio group /Opt [(FIRST_EXPORT) (THIRD_EXPORT)] and positional states /0 and /1, then move /Opt onto a new parent field whose /Kids contains that group. Extracting page 3 leaves one widget, two inherited export options, and unchanged /V /1, /AS /1, and /AP /N /1. field.get(Opt, ...) misses the inherited options, so the original inconsistency remains for this valid hierarchy. Resolve the inherited value and write a pruned local override on the button field. /Opt is explicitly inheritable in PDF 32000-1:2008, Table 227.

3. Renumber rollover appearances along with normal and down appearances. (clone.py, line 445)

   Give the second widget in the two-widget positional-radio example a rollover appearance dictionary /AP /R << /1 ... >>. Extracting its page changes /V, /AS, and /AP /N to /0, while /AP /R still contains only /1. The selected state has no matching rollover appearance. Include /R in the renumbering pass, with the sharing protection described above. Rollover entries may be state dictionaries under PDF 32000-1:2008, Table 168.

4. Dereference array elements before checking named field targets. (clone.py, lines 386–388)

   Make (person.address) an indirect string object and use its reference as an element of ResetForm /Fields or Hide /T. Extracting page 1 removes person.address but retains either action without raising. The array element is an IndirectObject, so isinstance(target, str) skips it even though target.get_object() is the known excluded field name. Resolve each target before the string check. Indirect strings are permitted by PDF 32000-1:2008, section 7.3.10.

Validation: all four cases were reproduced separately and compared with the previous PR revision. make check-zlib reported standard zlib; make test-all passed all 480 tests with 100% statement coverage; make linting passed. All GitHub checks were green at this revision. No tracked files were changed, and the maintainer-only visual PDF comparison was not repeated.

[chinapandaman]

Thanks. All four reproduced at 5529465c and are fixed in 98dd8936; the E3-01 record is updated in aca46966.

1. Shared appearance dictionaries: each distinct appearance dictionary is renamed once per field, even when /N and /D (or several buttons) share it. In your three-button example, the selected button keeps /AS, /V, and its shared /N//D dictionary at /1, and the dictionary stays shared. data["radio"] is 1 again.
2. Inherited /Opt: /Opt, /V, and /DV are resolved through inheritance, and the trimmed or renumbered values are written on the trimmed radio field itself. The parent keeps its values. Renumbering only applies to fields whose kids are widgets, so a parent whose kids are radio groups doesn't get its own /Opt trimmed when the counts happen to match.
3. Rollover: /R is renumbered with /N and /D.
4. Indirect strings: each named target is resolved before the string check. Both of your examples now raise at /Annots[7]/A/Fields[0] and /Annots[7]/A/T[0].

One behavior change came out of this round. Writing /DV /Off for a default that pointed at a removed button exposed an existing discovery bug, filed as #2112. PyPDFForm reads a radio button as a checkbox when its group's /DV is /Off and the button's own /AS is /Off, and a group with a selected button then fails with AttributeError, which would have broken .pages. Per your decision, a /V or /DV naming a removed button is now deleted from the field. /Off is written only when a parent would still supply a value, which is the one remaining case #2112 affects.

Tests: indirect-string cases in the reject test, and a new test for a shared /N//D dictionary, rollover, inherited /Opt and /V with an unchanged parent, and an own /DV. The first radio test now covers a group without /DV. The new cases fail against 5529465c. No expected PDF changed. make test-all passed 483 tests with 100% coverage, and make linting, ruff format --check, a strict MkDocs build, and git diff --check passed (local ./venv, standard zlib).

🤖 Generated with Claude Code

[chinapandaman]

Reviewed at aca4696631e1fd745cb49cb464af4a57791177cf. All four reproductions from the previous review now pass. I found one remaining P2 issue:

Isolate appearance state dictionaries shared between different fields before renumbering. (clone.py, lines 460–480)

The renamed set prevents repeated transformations within one field, but renumbering still mutates dictionaries used by other fields. A concrete two-page reproduction:

• Radio group split has widget 0 on page 1 and widget 1 on page 2.
• Radio group retained has both widgets on page 2.
• Both groups have /Opt [(a) (b)], /V /1, and widget states /Off and /1.
• The second widget of each group uses the same indirect /AP /N state dictionary, containing /1 and /Off. Their /AP containers are separate; only the normal state dictionary is shared.

Before extraction, PdfWrapper(source).data is {"split": 1, "retained": 1}. After PdfWrapper(source).pages[1].read(), reopening gives {"split": 0, "retained": None}. Pruning split renames the shared dictionary's /1 to /0. The retained group keeps both widgets and its /V /1 and selected widget's /AS /1, but that widget no longer has a matching appearance state. Against the pre-PR base (db35aa0a), the same extraction preserves data["retained"] == 1.

The mutable state dictionaries need to be isolated for each field before renumbering, while the underlying appearance streams can remain shared. A global visited set alone would still allow a pruned field to change an unpruned field's dictionary. Please add a regression test that shares the dictionary between two groups with different retained widget sets and checks that the wholly retained group's selection survives.

Validation: replayed all four previous reproductions and independently reproduced this case. make check-zlib reported standard zlib; make test-all passed all 483 tests with 100% statement coverage; make linting passed. No tracked files were changed, and the maintainer-only visual PDF comparison was not repeated.

[chinapandaman]

Thanks. Reproduced at aca46966 ({"split": 0, "retained": None}) and fixed in 755baf94; the E3-01 record is updated in 5b1bf113.

Renumbering no longer edits state dictionaries in place. Each remaining button of a renumbered field gets its own /AP dictionary, with renamed copies of its normal, rollover, and down state dictionaries. Appearance streams stay shared, and buttons or entries of the same field that shared a state dictionary share its copy. A dictionary used by another field is therefore unchanged. Your example now gives {"split": 0, "retained": 1}. While making this change I also made sure that appearance entries given as a single stream stay as they are, and that no entry a button didn't have gets added.

Test: test_pages_renumber_without_changing_other_fields shares one normal state dictionary between a group split across pages and a group wholly on page 2. It checks that the retained group keeps its selection, /Opt, and states while the split group is renumbered, and that both still share the appearance stream. It fails against aca46966 with kept: None. The first radio test also checks a single-stream rollover appearance. No expected PDF changed. make test-all passed 484 tests with 100% coverage, and make linting, ruff format --check, and git diff --check passed (local ./venv, standard zlib).

🤖 Generated with Claude Code

[chinapandaman]

Reviewed at 5b1bf113f6965564bb6fec88da69d587a215cd51. All previously reported reproductions now pass. I found one remaining P2 issue:

Preserve action targets when pruning a shared /Kids array. (clone.py, lines 426–427)

kids[:] = ... mutates the source array itself. If another entry references that array, pruning the field also changes that entry before the guard can validate its original contents.

Using build_cross_page_form(), make the person field's /Kids an indirect array containing its name and address children. Give a page 1 button a ResetForm action whose /Fields points to that same array:

person = pdf.Root.AcroForm.Fields[1]
person.Kids = pdf.make_indirect(Array(person.Kids))
pdf.pages[0].Annots[7].A = Dictionary(
    S=Name.ResetForm, Fields=person.Kids
)

Extracting page 1 succeeds and silently changes the action's targets from [person.name, person.address] to [person.name]. With an independent /Fields array containing the identical references (Fields=Array(person.Kids)), extraction correctly raises Cannot copy page 1: /Annots[7]/A/Fields[1] .... The shared-array representation therefore bypasses the confirmed policy of rejecting action references to excluded fields rather than silently pruning them.

Assign a new array to the field's /Kids instead of mutating the original array. This leaves the action's original targets available for validation. Please add a regression test that compares shared and independent arrays and requires both to reject the excluded target.

Validation: replayed the prior reported reproductions and independently reproduced both forms of this case. make check-zlib reported standard zlib; make test-all passed all 484 tests with 100% statement coverage; make linting passed. No tracked files were changed, and the maintainer-only visual PDF comparison was not repeated.

[chinapandaman]

Thanks. Reproduced at 5b1bf113: the shared form extracted page 1 and left the action targeting only person.name, and the copied form raised. Fixed in c9d1a5b9; the E3-01 record is updated in 4e0fdbcf.

A pruned field now gets a new /Kids array, and the original array is left intact. The guard reaches the original array through the action and rejects the excluded child, so both forms raise Cannot copy page 1: /Annots[7]/A/Fields[1] ....

I also checked the other edits the guard makes in place: remapped page and catalog references, removed GoTo actions, /P, /AS, and a field's own /V and /DV. None of them depends on how the object was reached, so checking each object only once stays safe. Pruning /Kids was the only edit that did.

Test: test_pages_check_targets_sharing_pruned_kids makes person's /Kids an indirect array. It requires the ResetForm action to raise both when /Fields is that array and when it is a copy, and checks that the field is still pruned when no action refers to it. The shared case fails against 5b1bf113. No expected PDF changed. make test-all passed 487 tests with 100% coverage, and make linting, ruff format --check, and git diff --check passed (local ./venv, standard zlib).

🤖 Generated with Claude Code