Skip to content

fix(utils): make external plugins importable in spawned workers - #10126

Open
Excelius-Wang wants to merge 2 commits into
modelscope:mainfrom
Excelius-Wang:fix/external-plugin-unpickling
Open

Excelius-Wang wants to merge 2 commits into
modelscope:mainfrom
Excelius-Wang:fix/external-plugin-unpickling

Conversation

@Excelius-Wang

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Fix a regression from #9953: custom templates loaded through --external_plugins fail with ModuleNotFoundError: _swift_external_... when a spawn/forkserver DataLoader unpickles them. The worker-init replay added in #10033 runs after this failure.

Import plugins through a private namespace whose module names encode their absolute paths. A small standard-library import finder resolves those names in fresh workers; importlib handles caching and failed-import cleanup. This preserves same-filename isolation and retains the old hashed names as aliases for existing pickles after explicit plugin import. Legacy package descendants resolve to the same canonical modules, including nested and delayed imports.

Experiment results

  • Regression tests cover fresh-interpreter class/function/instance unpickling, same-named plugins, Unicode paths, caching, import retries, package-relative imports, legacy package pickles and module identity, failed-import retries, and real spawn/forkserver DataLoaders with plugin-defined callbacks.
  • Public STS-B, SQuAD, CMRC2018 and HumanEval data: 12 actual two-step CPU SwiftSft runs comparing pre-regression spawn with patched spawn/forkserver. Per-source losses and trained model-weight hashes match exactly.
  • The shipped STS-B preprocessor consumes all 1,379 test rows under patched spawn/forkserver with the same output hash as the pre-regression baseline.
  • All 43 relevant utility/packing tests pass together; changed-file pre-commit hooks and git diff --check also pass.

Tested on Linux/Python 3.12 with explicit start methods; no native Windows/Python 3.14 or GPU validation claimed. The unrelated existing streaming routing-tag lambda serialization issue is outside this fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant