PoC: cache policy and cache scopes (#1089) — AI-assisted, for hole-poking only

[RonnyPfannschmidt]

Warning

This is a deliberately AI-assisted proof of concept, and it is not ready for deeper review.

I drove this with Claude Code over a long session — steering the design, rejecting several of its
proposals, and correcting it where it was wrong — but I have not yet given the diff the line-by-line
read I would want before asking anyone to spend real review time on it. Every commit carries a
Co-authored-by trailer.

What I actually want from this PR right now is hole-poking, for fun. Push on the design, the
naming, the on-disk layout, the edge cases. Please do not treat this as a review request and please
do not feel obliged to do a careful pass — if the shape is wrong I would rather find out now and throw
it away than have someone audit code that should not exist.

Closes #1089.

---

The idea

#1089 has been open since 2015. It never stalled on where to put the cache — pointing cache_dir at
~/.cache has always worked. It stalled on lifetime, per my own comment on the issue:

if the cache folder is outside of the working directory, its lifetime is unrelated to the working
directory […] imagine a unaware ci system that makes one folder per build could easily DOS the ci
server due to filling up ~/.cache of the ci user

So this PR is mostly not about the location. It is about two named concepts, and the location falls
out of them.

Cache scopes — how far a cached value travels

The thing that actually blocks a shared cache is that some cached values are not portable. --lf,
--nf and --sw state depends on what the interpreter in use actually collects. Today the only way to
express that is to move the whole cache directory — which is exactly what the TOX_ENV_DIR special case
in cacheprovider.py does:

cache_dir_default = ".pytest_cache"
if "TOX_ENV_DIR" in os.environ:
    cache_dir_default = os.path.join(os.environ["TOX_ENV_DIR"], cache_dir_default)

That multiplies entire cache trees, keyed off an environment variable belonging to one specific tool,
and it only ever helped tox — not nox, not a plain second virtualenv.

CacheScope names the property directly instead:

Scope	Valid for
SHARED (default)	the project, whatever runs it
PYTHON	one implementation + major.minor
ENV	one sys.prefix

config.cache.set("myplugin/collected", ids, scope=pytest.CacheScope.ENV)

Interpreter matching now happens inside one cache folder rather than by having many folders. A tox
run with six envs adds six small scopes instead of six full trees. SHARED stays at the existing v/
and d/ paths, so existing caches and third-party plugins need no migration; only pytest's own three
keys move, costing one benign invalidation.

Cache policy — where the directory lives

cache_policy = local | user. local is unchanged. user puts it in the platform user cache dir,
keyed by project, so nothing is written into the project at all. PYTEST_CACHE_POLICY sets it
machine-wide. cache_dir still wins over both and remains the escape hatch for anything the policies do
not name.

Because interpreter identity moved into scopes, the project key is the rootdir alone — so one project
gets one cache directory however many environments run it, and ephemeral environments (uv run --with …)
stop minting a tree per invocation.

The lifetime machinery, which is the actual point

Every cache directory gets a cache-info.json recording where it came from and which scopes it holds.
That makes the accumulation visible and collectable:

$ pytest --cache-list
user cache directory: /home/ronny/.cache/pytest

  DIRECTORY               SIZE      LAST USED  STATUS    ORIGIN
  myproject-1a2b3c4d5e6f  12.4 MiB  2 days     ok        /home/ronny/src/myproject
    env-venv-9f8e7d6c      1.1 MiB  2 days     ok        /home/ronny/src/myproject/.venv
    env-py312-0a1b2c3d   840.1 KiB  94 days    stale     /home/ronny/src/myproject/.tox/py312
  oldthing-9f8e7d6c5b4a  840.0 KiB  31 days    orphaned  /home/ronny/src/oldthing

2 directories, 3 scopes, 13.2 MiB total

$ pytest --cache-prune=stale      # drops the dead venv's state, keeps the project cache

--cache-prune requires a selector (all, orphaned, stale, or a glob), so a bare invocation can
never delete anything. Nothing is ever pruned automatically — a project on an unmounted volume looks
orphaned, which is exactly why.

TOX_ENV_DIR deprecation

Now redundant, so it is deprecated with a PytestRemovedIn10Warning. Nothing needs to replace it;
anyone who wants that location can write cache_dir = $TOX_ENV_DIR/.pytest_cache, which is the same
expression the default was built from and so resolves to the same directory by construction.

The commits

Twelve, each self-contained and independently green. The first four are enablers with no user-visible
behaviour change.

terminalwriter: add OSC 8 hyperlink support	no callers yet
pytester: isolate the user-level cache directory	latent bug: LOCALAPPDATA is not derived from USERPROFILE, so inner runs leaked to the real user cache on Windows always, and on Linux whenever XDG_CACHE_HOME was set
pathlib: resolve the user-level cache root behind an xdg extra	platformdirs as an optional dep (pytest[xdg]), imported lazily
cacheprovider: funnel cache directory resolution through one function	pure refactor
cacheprovider: introduce cache scopes	no behaviour change; nothing passes a non-default scope yet
cacheprovider: pin lastfailed, nodeids and stepwise to the env scope	the behaviour change the design is for
cacheprovider: record cache metadata
cacheprovider: add the cache_policy option
cacheprovider: deprecate the TOX_ENV_DIR cache_dir default
cacheprovider: add --cache-list
cacheprovider: add --cache-prune
docs: document cache policy, scopes and pruning

Nice side effect: test_stepwise.py's cache_dir = .cache workaround, added because "tox's cache
directory makes tests in this module flaky", is no longer needed and is removed.

Where I would start poking

Ordered by how much I would like to be argued with:

1. Default scope is SHARED. Back-compat-safe, but arguably the wrong semantic default — a stale
   cross-interpreter entry is subtly wrong, an over-pinned one merely costs a recompute. Changing it
   later relocates third-party plugin data, so it wants settling before this could ever land.
2. cache_dir + cache_policy is a two-option surface. The rule is one sentence — cache_dir
   always wins — but two knobs for one outcome is how we got TOX_ENV_DIR in the first place.
3. cache_dir's ini default is now "" rather than ".pytest_cache", which is what makes "unset"
   detectable. In-tree only pytest_report_header read it expecting the old value; third-party plugins
   might too.
4. The on-disk layout — s/<scope-id>/{v,d}/, scope ids like env-venv-9f8e7d6c, project dirs like
   myproject-1a2b3c4d5e6f7a8b. All of it is a guess at what reads well in ls.
5. Symlink resolution in the project key. Stops one project reached via two paths getting two caches;
   churns the key for rotating-symlink deploy layouts.
6. --cache-prune deletes without confirmation, consistent with --cache-clear, with --cache-list
   as the preview. And nothing locks the cache, so pruning concurrently with a run using it can race.
7. Epoch floats rather than ISO-8601 in the metadata, so a hand-edited timestamp cannot raise
   mid-listing.
8. OSC 8 hyperlinks are allow-by-default with a small deny-list. tmux usually reports
   TERM=screen-* and so loses links; PYTEST_HYPERLINKS=1 forces them.

Notes

• Needs Validate enum-valued configuration options with Literal types #14791 (Raise UsageError from getini for invalid configuration values) — without it an invalid
  cache_policy read at configure time surfaces as an INTERNALERROR. Found that the hard way; it is
  already on main.
• New published surface: the xdg extra. Extras are effectively permanent, and xdg is only accurate on
  Linux even though the feature covers macOS and Windows.
• user-policy tests skip rather than fail when platformdirs is absent, so a minimal-deps job stays
  green.