Unity built-in package resolving information collection

[ComradeVanti]

I am currently in the process of investigating how built-in package resolution works in Unity. I will share my findings here.

[ComradeVanti]

Right now I am investigating what kind of web requests Unity sends to determine built-in packages.

[ComradeVanti]

When pressing the "Refresh" button in the package manager, Unity first sends a GET request to https://packages.unity.com/com.unity.package-manager.metadata-2022.3. As you can see this includes the editors version. I was hopeful that we would get some interesting information in the response to this request, but it seems to just respond with an empty 304.

[ComradeVanti]

Next it sends a request to the registry for each built-in package to check if it exists on the registry.

Most return a 404 as you can see. But some are also 304.

Maybe someone has an idea of what the pattern here could be?

In any case it seems that at this point it already has a list of what packages to check, so the initial list comes from somewhere else.

[ComradeVanti]

We know that Unity stores built-in packages in a folder that comes with the editor. For example, on Linux this is ~/Unity/Hub/Editor/[Editor Version]/Editor/Data/Resources/PackageManager//BuiltInPackages.

As an experiment I removed the com.unity.sprite, which could be found there to see if it would then still send a request to check for it in the registry. The answer is yes.

I checked, and it does not seem that this will regenerate the package in the built-in packages folder. The only effect this has, is that now, packages which depend onsprites show a warning in the editor.

Edit: For clarity, the reason that only Animation has this error, is that my test project depends on it. Other packages which depend on Sprite but are not in my project, did not have the error symbol.

Edit 2: I just removed every single package from the built-in packages folder and it still send requests for all packages as before. The search for where these are stored goes on.

[ComradeVanti]

Found the origin I think. There seems to be a "root manifest" which can be found on Linux at ~/Unity/Hub/Editor/[Editor Version]/Editor/Data/Resources/PackageManager/Editor/manifest.json. It contains a list of packages like this:

{
  "schemaVersion": 4,
  "packages": {
    "com.unity.collections": {
      "isDiscoverable": true,
      "mustBeBundled": true,
      "version": "1.2.4",
      "minimumVersion": "1.2.4"
    },
    "com.unity.entities": {
      "isDiscoverable": true
    },
    "com.unity.netcode": {
      "isDiscoverable": true
    },
    // ...
  }
}

After removing com.unity.2d.animation from this list it no longer shows up in the editor and no request is sent for it. This implies that this is the list of packages that are shown in the package-manager for each editor and which are then checked for whether they are built-in or not.

[favoyang]

https://packages.unity.com/com.unity.package-manager.metadata-2022.3 is a standard NPM package endpoint. It's good that Unity manages internal packages overridden using a standard endpoint.

openupm view com.unity.package-manager.metadata-2022.3

com.unity.package-manager.metadata-2022.3@0.0.51 | proprietary or unlicensed | versions: 35

Contains metadata used by the 2022.3 Unity Editor.

dist
.tarball: https://download.packages.unity.com/com.unity.package-manager.metadata-2022.3/-/com.unity.package-manager.metadata-2022.3-0.0.51.tgz
.shasum: 47538211cff36c2e6c99d3d9ed73c51d019de3df

latest: 0.0.51

published at 2024-05-25T10:15:35Z

versions:
  0.0.51
  0.0.50
  0.0.49
  0.0.48
  0.0.47
  ...

Each version contains only one file: manifest-overrides.json. But it only contains a part of the internal packages that have been updated. That means Unity must have stored the internal package list somewhere else.

[ComradeVanti]

That means Unity must have stored the internal package list somewhere else.

Yes, I think this is it.

[favoyang]

Any idea why the endpoint returns an empty 304 for me?

Unity could send a If-Modified-Since HTTP header, and the server returns 304 (not modified). It means the cache is valid. You can reach that URL via browser to view the full content: https://packages.unity.com/com.unity.package-manager.metadata-2022.3

[ComradeVanti]

Here are the headers it sends

Accept: application/json, text/plain, */*
If-None-Match: "05a1f18d78f8632d6f76e464820ac4f2"
Authorization: Bearer 8pC54Bcag7YPrm75SN5O14KR5oHWPkr1QRGsqHMR6yU008f
Accept-encoding: gzip, deflate
User-Agent: axios/0.27.2
Host: packages.unity.com
Connection: close

Probably the If-None-Match

[favoyang]

It makes sense then. If-None-Match sends an e-tag and the server responds either 200 if it does not match or 304 if it matches.

[favoyang]

That means Unity must have stored the internal package list somewhere else.

Yes, I think this is it.

Good catch, then that file contains all internal packages shipped with the Editor binary.

[ComradeVanti]

I tried removing the manifest.json file to check if it would redownload if from somewhere. But no, this just caused Unity to print an error and do nothing.

[favoyang]

So Unity uses that manifest.json to lookup internal packages shipped with the Editor. And use the com.unity.package-manager.metadata-[2022.3] endpoint to manage internal package override.

[ComradeVanti]

Just downloaded a fresh .tar.xz of a Unity editor to check what is inside on installation. Sure enough, the manifest.json is there, as are all the packages in BuiltInPackages.