Bug: settings.py module docstring states config precedence that contradicts its own implementation

[gtouloumes]

TYPE: Bug report (template: .github/ISSUE_TEMPLATE/bug_report.md — auto-labels bug, triage; suggest adding documentation)

TITLE: settings.py module docstring states config precedence that contradicts its own implementation

Bug Report

Description

The datajoint.settings module docstring lists .secrets/ as outranking datajoint.json.
The implementation is the reverse: the config file wins, and .secrets/ only supplies
values nothing else set.

src/datajoint/settings.py lines 7–11 on master (3f8a79c):

Configuration sources (in priority order):

1. Environment variables (``DJ_*``)
2. Secrets directories (``.secrets/`` in project, ``/run/secrets/datajoint/``)
3. Project config file (``datajoint.json``, searched recursively up to ``.git/.hg``)

Two mechanisms in the same file contradict items 2 and 3:

• _load_secrets (line 716) assigns only when the target is still unset —
  if db_user is not None and self.database.user is None: (line 722), same for the
  password (line 727). For stores it skips attributes already present, with the comment
  "Only set if not already present (config / env vars win)" (line 748).
• _update_from_flat_dict (line 658) skips a file value when its env var is set, logging
  "Skipping {key} from file (env var {var} takes precedence)" (lines 682, 694, 705).

So the effective order is: programmatic → environment variables → datajoint.json →
.secrets/ (gap-fill only) → defaults.

Reproducibility

• OS: macOS 26 (also reproducible by inspection on any platform — no OS-specific paths involved)
• Python: 3.13
• DataJoint: 2.3.2; docstring unchanged on master (3f8a79c)
• MySQL version / deployment: not applicable — this is config resolution only, no
  connection is opened

mkdir -p /tmp/dj-precedence/.secrets && cd /tmp/dj-precedence
echo '{"database": {"host": "h.example.com", "user": "from_file"}}' > datajoint.json
echo "from_secrets" > .secrets/database.user
python -c 'import datajoint as dj; print("user =", dj.config.database.user)'

Result:

user = from_file

Adding DJ_USER=from_env yields from_env; removing user from datajoint.json
yields from_secrets. That is, each source is overridden by the one above it in the
effective order, not the documented one.

No error or warning is emitted — a value in datajoint.json silently takes precedence
over the same key in .secrets/.

Two related observations from the same reading, both undocumented:

• .secrets/ recognizes only database.user, database.password, and
  stores.<name>.<attr>. A file named for any other setting is ignored silently — e.g.
  .secrets/database.host has no effect (verified: host stays at the default).
• A .secrets/datajoint.json is not read at all; the directory is consumed file by file.

Expected Behavior

Either the docstring or the implementation should change:

1. If the current behavior is intended, the docstring is simply stale — swap items 2
   and 3 and note that .secrets/ fills values the file and environment did not set. I
   have this patch ready and will open a PR against this issue.
2. If the documented order is the intended design — local secrets overriding committed
   project config, which is arguably the more natural reading — then this is an
   implementation bug in _load_secrets, and the docstring is correct as written.

The _load_secrets comment at line 748 and the deliberate env-var check in
_update_from_flat_dict both suggest (1), and in practice the two rarely collide because
credentials are not supposed to live in a committed datajoint.json. But that is an
inference about intent, so flagging rather than patching blind.

Additional Research and Context

• Source: src/datajoint/settings.py lines 7–11 (docstring), 658/682/694/705
  (_update_from_flat_dict), 716/722/727/748 (_load_secrets).
• This docstring appears to be the origin of the same error in the documentation: four
  pages on docs.datajoint.com stated the inverted order, corrected in
  docs: correct config precedence and the .secrets/ file format datajoint-docs#258 against the behavior verified above. If resolution (2) is
  chosen instead, that PR needs reverting — happy to coordinate.
• dj.config.save_template() (line ~841) writes .secrets/database.user and
  .secrets/database.password as separate files, consistent with the per-key format and
  with resolution (1).