[BUG]_SSHFS_incompatibility_issue_with_DeepMD_3.1.2

[Mem314]

Bug summary

Hello There,

i am currently training a model using Deepmd 3.1.2 with dpgen and I am experiencing issue when training on a SSHFS mounted system. The reported error seems to have a problem regarding the PyTorch workers accessing the same file at the same time.
I have attached the complete message regarding the problem. I am wondering if this has something to do with the Deepmd 3.1.2 or the SSHFS 3.7.3?

DeePMD-kit Version

3.1.2

Backend and its version

PyTorch

How did you download the software?

conda

Input Files, Running Commands, Error Log, etc.

Pasted text(20260908-131850).txt

Steps to Reproduce

train normally on a SSHFS mounted system with the SSHFS version 3.7.3.

Further Information, Files, and Links

No response

[njzjz-bot]

Thanks for the report. The first thing I would isolate is whether this is a DeePMD issue or an SSHFS + PyTorch multiprocessing I/O interaction.

For the PyTorch backend, DeePMD uses worker subprocesses for data loading by default (NUM_WORKERS defaults to 4 or the available core count, whichever is smaller). Please try the same training with worker multiprocessing disabled, for example by exporting:

export NUM_WORKERS=0

before launching the DeePMD/DP-GEN training task. The variable is documented here: https://github.com/deepmodeling/deepmd-kit/blob/master/doc/env.md

It would also be useful to run the exact same dataset once from node-local/local POSIX storage. If local storage works but SSHFS fails, and especially if NUM_WORKERS=0 also avoids the failure on SSHFS, that would strongly point to filesystem/multiprocessing semantics rather than the model itself.

If it still fails with NUM_WORKERS=0, please paste the first complete traceback/error block directly into the issue (starting from the first exception), and note whether the same files train successfully when copied off SSHFS. Those two checks should narrow this down substantially.

Agent: ChatGPT
Model: GPT-5.6 Sol