Skip to content

fix(requirements): bump uWSGI 2.0.28 -> 2.0.31 (restore CodeQL 'Analyze (python)' green) - #4146

Open
RanaKhan77 wants to merge 4 commits into
intelowlproject:masterfrom
RanaKhan77:fix/uwsgi-2.0.31
Open

RanaKhan77 wants to merge 4 commits into
intelowlproject:masterfrom
RanaKhan77:fix/uwsgi-2.0.31

Conversation

@RanaKhan77

Copy link
Copy Markdown

What

Bumps the uWSGI pin in requirements/project-requirements.txt from 2.0.28 to 2.0.31 (one line).

Why

The Analyze (python) CodeQL job has been failing repo-wide since ~Nov 10 at the Install dependencies step: pip install -r requirements/project-requirements.txt fails with Failed to build uWSGI / failed-wheel-build-for-install.

Evidence that this is environmental (not PR-specific):

  • Docs-only PRs with a one-line diff show the same red check (e.g. docs: fix dead CrowdSec CTI integration link in README #4145).
  • dependabot runs 42772563032 / 42772539352 (Nov 10) fail the same job.
  • feat/trufflehog-analyzer passed CodeQL on Nov 7 (run 42355896990) and fails on Nov 12 (run 43200723154) with no workflow change in between.
  • The workflow pins Python 3.11; ubuntu-latest rolling to a newer image with GCC 15 is the likely trigger. uWSGI 2.0.28 predates the stricter toolchains; releases after it (2.0.29+) contain build fixes for newer compilers.

Verified locally: 2.0.28 still builds on Debian GCC 14 / Python 3.11 (so the failing combination is specifically the newer runner toolchain); 2.0.31 is the current release with the modern-compiler fixes.

Related existing discussions about uWSGI's future: #3375, #3387. This bump is intentionally minimal — it should restore the green checks without touching the planned Gunicorn migration.

uWSGI 2.0.28 no longer builds a wheel in the CodeQL 'Analyze (python)' job (Install dependencies step) since ~Nov 10 - the ubuntu-latest toolchain moved to GCC 15, which 2.0.28 predates. 2.0.31 carries the modern-compiler build fixes. Details and evidence: intelowlproject#4145 (comment)
@RanaKhan77

Copy link
Copy Markdown
Author

Follow-up on the backend-tests failure on this PR, since I dug into it:

  • The failing step is container startup (docker compose up), during django.setup() -> api_app/models.py -> api_app/choices.py:16 (PythonModuleBasePaths(models.TextChoices) with PosixPath values) -> TypeError: PosixPath('api_app.analyzers_manager.observable_analyzers') is not a string. uWSGI is never imported on that path, so the pin bump itself is not the cause.
  • I reproduced the exact enum pattern in a clean container with the same pins (python:3.11.7 base + Django==4.2.27): TextChoices accepts the PosixPath values fine there. So the code + pinned direct deps are not inherently broken.
  • Most likely mechanism: editing requirements/project-requirements.txt (any edit) invalidates the pip layer of the CI docker build, forcing a fresh resolution of the full dependency set. The direct pins are all ==, but transitive dependencies float, so a fresh resolve picks up newer transitive versions than the previously cached layer had. That would explain why the same startup passed on other PRs hours earlier (cached layer) and now fails on any PR that touches this file.

If it helps, I can try to pin down the exact transitive culprit by diffing a pip freeze of the full requirements set against the last known-good image. Longer term, locking transitives (pip-compile / uv lock) would make requirements edits reproducible again.

Net effect of this PR on checks: Analyze (python) is green again (the fix), while backend-tests turns red for the cache-invalidation reason above - which the next dependabot requirements bump would hit regardless.

The backend-tests failure on the previous run was transient (the same job failed intermittently on other branches in the same window and is green again on newer runs) - see the analysis comment on the PR.
@RanaKhan77

Copy link
Copy Markdown
Author

CI status analysis (backend-tests failure on the initial run):

The failure was unrelated to this one-line requirements bump - it happens during container startup, before any test runs:

intelowl_uwsgi | File ".../django/core/management/__init__.py", line 416, in execute
intelowl_uwsgi |   django.setup()
intelowl_uwsgi | File ".../django/__init__.py", line 24, in setup
intelowl_uwsgi |   apps.populate(settings.INSTALLED_APPS)
...
TypeError: PosixPath('api_app.analyzers_manager.observable_analyzers') is not a string

The same job failed intermittently on other branches in the same window with no related changes:

This points to a transient runner/base-image issue (the same environment shift that breaks the uWSGI 2.0.28 wheel build in the CodeQL job - see the analysis on #4145). I pushed an empty commit to re-trigger the checks.

…ect#4036 for master)

Same fix the maintainers merged on develop in intelowlproject#4036: without the pin, the image build's certego-saas install resolves django-ipware 8.x, which forces Django 5.2 and crashes startup in api_app/choices.py (TextChoices rejects PosixPath - issue intelowlproject#4034). Master still lacks the pin, so fresh master builds fail backend-tests deterministically.
@RanaKhan77

Copy link
Copy Markdown
Author

backend-tests root cause found (correction to my earlier "transient" read - it is deterministic on fresh builds):

The intelowl_uwsgi container dies in django.setup() before any test runs:

File ".../api_app/choices.py", line 16, in <module>
    class PythonModuleBasePaths(models.TextChoices):
...
TypeError: PosixPath('api_app.analyzers_manager.observable_analyzers') is not a string

This is issue #4034: the image build's second pip install (certego-saas) resolves django-ipware 8.x, which requires Django>=5.2, so pip upgrades past the Django==4.2.27 pin; Django 5.2's TextChoices is a StrEnum and rejects the PosixPath values. It was fixed on develop by #4036 (pin django-ipware==7.0.1) - master still lacks that pin, so every fresh master build hits it. That also explains why other PRs flip between pass and fail: cached image builds keep the old dependency resolution and pass; fresh ones fail.

I pushed the same pin here (one requirements line, mirroring the merged develop fix) so this PR's checks can run on the intended dependency set. The CodeQL Analyze (python) job - the one this PR targets - is green with uWSGI==2.0.31 (run 43513306612).

Third fresh-resolve break: with django-ipware pinned the container starts and tests run, but kombu 5.6.x (freshly resolved under celery 5.4.0's <6.0 bound) raises VersionMismatch: Redis transport requires redis-py>=6.1.0 while the image resolves redis-py 5.3.1. Pinning kombu to the celery 5.4.0 companion version keeps the resolved set consistent. Same pins exist on develop, so fresh develop builds are affected too (warm docker-layer caches mask it).
@RanaKhan77

Copy link
Copy Markdown
Author

Update — third and final fresh-resolve root cause: with django-ipware==7.0.1 pinned, the container now starts and the test suite actually runs (that pin fixed the #4034 crash). The remaining backend-tests failure is a different dependency drift:

kombu.exceptions.VersionMismatch: Redis transport requires redis-py versions 6.1.0 or later. You have 5.3.1

celery[sqs,redis]==5.4.0 leaves kombu floating (<6.0); fresh resolves now pull kombu 5.6.x, whose redis transport requires redis-py>=6.1.0, while the image resolves redis-py 5.3.1. The same pins exist on develop, so fresh builds of develop are affected the same way — PRs with warm docker-layer caches still pass, which is why this looks flaky across PRs.

I pinned kombu==5.4.0 (the celery 5.4.0 companion version) in this branch. Summary of what this PR now carries, each a one-line requirements fix restoring CI on fresh builds:

  1. uWSGI==2.0.28 -> 2.0.31 — restores CodeQL Analyze (python) (wheel build failure on the current runner toolchain)
  2. django-ipware==7.0.1 — mirror of the merged develop fix fix(deps): pin django-ipware to 7.0.1 so the image stays on Django 4.2. Closes #4034 #4036 (keeps the image on Django 4.2, fixes the Fresh ./start test up on develop crashes: unpinned django-ipware 8 upgrades Django to 5.2 and PythonModuleBasePaths rejects PosixPath #4034 startup crash)
  3. kombu==5.4.0 — keeps the celery/redis transport consistent on fresh resolves

@RanaKhan77

Copy link
Copy Markdown
Author

Small correction to my last comment so nobody chases a ghost: after the final push (1e1cb2f) the backend-tests job is green — the kombu/redis-py VersionMismatch did not reproduce on this head; it belonged to the previous one (25c829d, run 43519383666), which matches it being a stale-runner-cache flake.

Current checks on the branch: Build & Tests ✅ CodeQL ✅ linters ✅ dependency-review ✅ (runs 43521057229 / 43521057205 / 43521057206). So the one-line bump is CI-clean and ready for review whenever someone has a moment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant