Skip to content

fix: repair master — lint, a dropped method, vendored drift, scorecard - #132

Open
Kartikey1306 wants to merge 3 commits into
embeddedos-org:masterfrom
Kartikey1306:fix/repair-master-lint-drift-scorecard
Open

fix: repair master — lint, a dropped method, vendored drift, scorecard#132
Kartikey1306 wants to merge 3 commits into
embeddedos-org:masterfrom
Kartikey1306:fix/repair-master-lint-drift-scorecard

Conversation

@Kartikey1306

@Kartikey1306 Kartikey1306 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Problem

CI — ebuild has been red on master since the 09-08 batch merge, and because the first failing step is ruff, everything behind it (mypy, pytest) has not run on master since. Vendored core drift and OSSF Scorecard are red on the same commit for unrelated reasons.

Verified against runs 34419838440 (CI — ebuild on 8b623d5, all nine Test legs at Lint (ruff)), 34419838427 (drift) and 34419838445 (Scorecard), and reproduced locally on master before every fix below.

What broke

Lint (ruff, 4 findings). A duplicate import shutil (F811), a lost trailing newline (W292), and two imports two hundred lines down test_ci_gate.py (E402) — that last one is my own #103, replayed onto a file that had moved. These three hunks are byte-identical to #122's (blobs e08444f, 3c25922, b1d5e0f), so whichever merges first leaves the other clean.

Type check + tests (never reached on master since 09-08). ebuild/packages/index_sync.py:354 calls PackageRecipe.to_dict(). #111 defined it; #112, merged five minutes later from a base without it, deleted it in its replay. mypy names it once; pytest fails nine test_index_sync cases with AttributeError. The method is restored verbatim from #111 (cc90078): it emits the package: / build: keys that parse_recipe() reads back, which an asdict() replacement would not — the YAML it writes has to round-trip through the same parser. #119 and #124 both propose re-implementations; this is a restoration of what master already had, not a third design.

Two more steps that had never passed, found on this PR's first run (run 34591499237), once ruff let the job past its first step:

  • mypy on 3.10 / 3.11 only. ebuild/plugins/__init__.py:46 — those interpreters' stubs type entry_points() as the deprecated mapping whose .get() wants an EntryPoints default (arg-type). The line carried # type: ignore[attr-defined], the wrong error code, so it suppressed nothing. Spelled out with a cast, byte-identical to ci: fix the lint findings that stop CI before any test runs #122's hunk (blob 54605f0).
  • yamllint on Windows only. The Windows runners check out with core.autocrlf=true, so every YAML file arrives CRLF and the default new-lines: unix rule rejected every line of templates.yaml and layers/eni/build.yaml. The step was added 09-03 and has never passed on that leg. A first attempt (new-lines: platform) failed the other way: auto-assign.yml carried one stray CR, so autocrlf left it alone and Windows then expected CRLF on its first line. Fixed properly: .gitattributes pins *.yml/*.yaml to eol=lf so every OS lints the same bytes, yamllint keeps its default unix rule, and the stray CR is gone.

Vendored core drift. #109 (dba3d83) edited core/eos/docs/three-way-alignment.md, a vendored copy pinned to eos 5544c98, so drift went 44 → 45 and the guard failed exactly as core/UPSTREAM.yaml says it should. Reverted to the pinned content — verified the result is blob 7f9c8c1, the same bytes as eos:docs/three-way-alignment.md at the pin. The alignment note itself is worth keeping; it belongs in ebuild's own docs or upstream in eos, not in the snapshot.

OSSF Scorecard. ossf/scorecard-action@v2.4.0 pulls gcr.io/openssf/scorecard-action, and gcr.io now refuses with "This API method requires billing to be enabled". v2.4.3 pulls from ghcr.io; eos already pins it and its Scorecard job is green.

Test plan

Verified, run the way ci.yml runs them:

ruff check .                                                    → All checks passed
yamllint .                                                      → clean
mypy . --ignore-missing-imports --no-strict-optional --exclude '^(layers|core|promo)/'
                                                                → no issues in 107 source files
python -m pytest tests/                                         → 680 passed, 1 skipped
python scripts/check_vendor_drift.py                            → core/eos 44/44 OK, core/eboot 46/46 OK

Not in this PR

EoSim Sanity's Windows and macOS legs install a wheel that has never been published (run 34561966328); #121 already replaces that with the clone the other legs use.

CI -- ebuild has been red on master since the 09-08 batch merge, and the
first failing step (ruff) has hidden the ones behind it.

Lint (ruff, all nine Test legs):
- test_build_dir_resolution.py imported shutil twice (F811).
- test_package_recipe.py lost its trailing newline (W292).
- test_ci_gate.py had `import itertools` / `import re` two hundred lines
  down (E402) -- my own embeddedos-org#103, replayed onto a file that had moved.
  These three hunks are byte-identical to embeddedos-org#122's, so either PR merging
  first leaves the other clean.

Type check and tests (never reached on master since 09-08):
- ebuild/packages/index_sync.py calls PackageRecipe.to_dict(), which embeddedos-org#111
  defined and embeddedos-org#112 -- merged five minutes later from a base without it --
  deleted in its replay. mypy names it once; pytest fails nine
  test_index_sync cases with AttributeError. The method is restored verbatim
  from embeddedos-org#111 (cc90078): it emits the `package:`/`build:` keys parse_recipe()
  reads back, which an asdict() replacement would not.

Vendored core drift:
- embeddedos-org#109 (dba3d83) edited core/eos/docs/three-way-alignment.md, a vendored copy
  pinned to eos 5544c98, so drift went 44 -> 45 and the guard failed as
  designed. Reverted to the pinned content (blob 7f9c8c1, the same bytes as
  eos:docs/three-way-alignment.md at the pin). The alignment note belongs in
  ebuild's own docs or upstream in eos, not in the snapshot.

OSSF Scorecard:
- ossf/scorecard-action@v2.4.0 pulls gcr.io/openssf/scorecard-action, and
  gcr.io now refuses the pull ("requires billing to be enabled"). v2.4.3
  pulls from ghcr.io; eos already pins it and its Scorecard job is green.

Not in this PR: EoSim Sanity's Windows/macOS legs install a wheel that has
never been published; embeddedos-org#121 (srpatcha) already replaces that with the clone
the other legs use.

Verified locally: ruff clean, yamllint clean, mypy clean over 107 files,
680 passed / 1 skipped, scripts/check_vendor_drift.py 44/44 and 46/46.
@codecov-commenter

codecov-commenter commented Sep 11, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 66.66667% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
ebuild/packages/recipe.py 68.42% 3 Missing and 3 partials ⚠️
ebuild/plugins/__init__.py 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…/3.11, yamllint on Windows

Both surfaced on this branch's first CI run, once ruff let the job get past
its first step.

- ebuild/plugins/__init__.py: on Python 3.10 and 3.11 the stubs type
  entry_points() as the deprecated mapping, and its .get() wants an
  EntryPoints default, so mypy fails with arg-type. The line carried a
  '# type: ignore[attr-defined]' -- the wrong error code, so it suppressed
  nothing. Spelled out with a cast, byte-identical to embeddedos-org#122's hunk (54605f0).

- .yamllint.yml: the Windows runners check out with core.autocrlf=true, so
  every YAML file arrives as CRLF and the default new-lines: unix rule
  rejected every line. The step was added on 09-03 and had never passed on
  that leg. new-lines: platform accepts the checkout's own convention.
…e checkout's line ending

new-lines: platform was the wrong fix. The Windows runners' autocrlf turns
LF files into CRLF -- except a file that already carries a stray CR, which
git leaves alone, and auto-assign.yml had one on its last line. So under
'platform' Windows expected CRLF and got LF on that file's first line, and
the leg was red again for the opposite reason.

Pin *.yml and *.yaml to eol=lf so every OS lints the same bytes, keep
yamllint's default unix rule, and drop the stray CR.

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the forensic write-up — it made this easy to verify. I re-ran each check on the branch: ruff clean, mypy loses exactly the two code errors (plugins/__init__.py:46, index_sync.py:354), pytest 674 passed with only the environment-only e2e build test failing, and check_vendor_drift.py is back to 44/44 where master fails at 45. The three-way-alignment.md blob hash matches the pinned 7f9c8c1, and I confirmed no other YAML file carried a CR before the .gitattributes pin.

Two optional follow-ups on the restored method, neither blocking:

  • ebuild/packages/recipe.py:92-114to_dict() predates install_args, so that field is dropped on round-trip through index_sync. Adding if self.install_args: data["install_args"] = list(self.install_args) would make it complete, and wrapping the other lists in list(...) avoids handing out live references.
    • Since this method was already lost once in a replay, a tiny test asserting parse_recipe(r.to_dict()) == r would make the next disappearance a red test rather than nine AttributeErrors.
      Approving — this should land first in ebuild. For the maintainers: this supersedes #122 (identical hunks) and #119/#124 (to_dict), and it will conflict with #127's unrelated recipe.py/plugins hunks.

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — ebuild#132 "fix: repair master — lint, a dropped method, vendored drift, scorecard"

head: f8cc357 author: Kartikey1306 ci: pass (30 checks green, Create GitHub Release skipped)

Verdict: The most complete diagnosis of master's red state in the current queue, and the only PR in this batch with a full green matrix behind it — nine Test legs, CodeQL, vendor drift, eleven EoSim targets. Every claim in the body that I could check independently held. One defect: the restored PackageRecipe.to_dict() does not emit install_args, so it is not the round-tripping serializer the body says it is. That is latent rather than live today, and the reasoning for why is below.

Findings

# Severity File:line Finding Recommended fix
1 Medium ebuild/packages/recipe.py, to_dict() install_args is dropped. parse_recipe() reads ten optional fields; to_dict() emits nine. Verified: a recipe with install_args=["DESTDIR=/stage","--strip"] serializes to ['build','dependencies','package','patches','url','version'] and reconstructs with install_args == []parse_recipe(yaml.safe_load(yaml.safe_dump(r.to_dict()))) != r. This matters because the write and the read are two halves of one loop inside the product: index_sync.py:354 writes these files into recipes_dir, get_recipe_dirs() at :122 hands that directory to RecipeRegistry, and registry.py:114 loads them back through load_recipe(). No data is lost today, because the only producer — the entry→dict mapping at index_sync.py:335-347 — never populates install_args either, so the field is always empty by the time to_dict() sees it. The defect is that both halves silently agree to drop a field the schema supports, and the first recipe that carries one loses it with no error. The body's claim that this "emits the package:/build: keys that parse_recipe() reads back" is right about the key names and wrong about completeness. Add if self.install_args: data["install_args"] = list(self.install_args) alongside the other conditional fields. #124's implementation emits all twelve and round-trips exactly — I verified that one too — so if the restoration-vs-reimplementation argument is what is keeping this hunk here, #124 already has the correct behaviour and could simply be taken instead. Add the missing key to index_sync.py:335-347 in the same pass, or the field can never arrive.
2 Medium core/eos/docs/three-way-alignment.md Reverting the vendored snapshot to the pin is the correct call and I am not disputing it — core/UPSTREAM.yaml says a vendored copy must match its pin, and the guard did its job. But the content being reverted away from is a correction: #109 replaced "25 YAML files / 25 board ports / ✅ Aligned" with "84 vs 83 vs 14/138 — three inventories describe the same set and nothing cross-checks them". After this merges, the repository again ships a table asserting ✅ Aligned for a set of inventories that #109 found do not agree. The body acknowledges this ("worth keeping; it belongs in ebuild's own docs or upstream in eos") but nothing in the diff or the PR carries it anywhere. A correction that is reverted with only a sentence in a PR body is a correction that is lost. Open the upstream eos issue or PR before merging this, and reference it from the diff — a line in ebuild's own docs, or the issue number in the CHANGELOG. The revert then reads as "moved", not "dropped".
3 Low .github/workflows/scorecard.yml:27 ossf/scorecard-action@v2.4.3 is pinned by tag. A mutable tag is exactly what Scorecard's own Pinned-Dependencies check penalises, and this repo already knows the pattern — .github/workflows/linked-issue.yml on master pins its reusable workflow to @92cb596c773496ec4df76717e8acf0e6b7700f73. Raising the Scorecard job from the floor while leaving the action itself unpinned leaves points on the table in the same file. Pin to the commit SHA for the v2.4.3 tag, with # v2.4.3 trailing so the version stays readable. Same for actions/checkout@v4 and github/codeql-action/upload-sarif@v3 at :24 and :32 if you want the check fully satisfied. Out of scope if you would rather keep this PR to the four named repairs — say so and it stands.
4 Low .gitattributes (new) *.yml/*.yaml text eol=lf is the right fix and the reasoning in the header comment is exactly right. One operational gap: .gitattributes governs future checkouts and commits, not files already in a contributor's working tree. Anyone with an existing Windows clone keeps CRLF locally until git add --renormalize . or a fresh checkout, so "it still fails for me" reports are likely. Add one line to the CHANGELOG or CONTRIBUTING telling Windows contributors to run git add --renormalize . once after pulling this.
5 Low ebuild/packages/recipe.py, to_dict() The conditional-emission branches have no test in this PR — which is how finding 1 survives review, and is corroborated by the coverage bot's report on this PR. A single round-trip assertion would have caught it. python\ndef test_to_dict_round_trips():\n r = PackageRecipe(name="d", version="1", url="https://e/x.tgz",\n build_system="cmake", install_args=["DESTDIR=/s"])\n assert parse_recipe(yaml.safe_load(yaml.safe_dump(r.to_dict()))) == r\n Fails today, passes with finding 1 fixed.

Architecture conformance

Conforms. §21 places ebuild in Tier 1 — Foundation. Every code change is inside that repo, and the one file under core/ is a revert toward the pinned upstream, which strengthens rather than crosses a boundary. §5.1 is not engaged by any hunk: no import, link line or manifest entry changes direction, and eBuild understands the complete graph but is not a runtime dependency still holds.

The to_dict() restoration is the §23.2 "Package format — versioned .epkg/.eapp metadata schema" contract, and finding 1 is a conformance gap against it: the serializer does not emit the whole schema the parser accepts. The .gitattributes and Scorecard hunks are infrastructure, which §21 assigns to the Infrastructure tier and .github/STANDARDS.md §"Security" names OpenSSF Scorecard and CodeQL explicitly — moving a Scorecard job from permanently-red to green is direct compliance work under that section, and the body's evidence (gcr.io now requiring billing, v2.4.3 pulling from ghcr.io, eos already pinned there) is the kind of concrete justification STANDARDS.md §"Standards-compliance assertions" asks for.

No API break (brief item 8): to_dict() is a new method, purely additive; cast(Mapping[str, Any], ...) is a runtime no-op; the .gitattributes and workflow changes touch no interface. No performance concern (brief item 9). Documentation (brief item 11): the _PIC_FLAGS-style stale comment in plugins/__init__.py is corrected in place, and .gitattributes documents its own reasoning — but there is no CHANGELOG entry for a PR that changes five subsystems, which is inconsistent with #125, #126 and #131 in the same queue, all of which added one.

On duplication (brief item 10): the body is straight about it — three test hunks and the plugins/__init__.py hunk are stated as byte-identical to #122's, and I confirmed the blob hashes match (e08444f, 3c25922, b1d5e0f, 54605f0). That is the honest way to declare an overlap. It does still mean #122 and this PR cannot both merge meaningfully, and that to_dict() now has four competing implementations in flight — #119 (asdict, emits internal key names), #124 (all twelve keys, round-trips), #127 (conditional, includes install_args) and this one. Only #124 and #127 preserve install_args.

Proposed changes

  1. Fix finding 1 — one if self.install_args: branch, plus the matching key at index_sync.py:335-347.
  2. Add the round-trip test from finding 5. It is the assertion that makes "restored verbatim from #111" a checkable claim rather than a provenance argument.
  3. Resolve finding 2 before merge: file the upstream correction somewhere durable and reference it.
  4. Decide the overlap with #122, #119, #124 and #127 as one decision rather than four. This PR plus #124's to_dict() is the smallest combination that leaves master green and loses nothing.
  5. Findings 3 and 4 are optional polish.

Not checked

  • pytest — NOT RUN by me. pytest is not importable on this host, and the local ebuild clone has a dirty working tree, left untouched per the rules of engagement. Unlike every other PR in this batch, this one does not need me to: nine Test (Python 3.x, os) legs are green on this head in CI, which is stronger evidence than a single local run. The body's 680 passed, 1 skipped is consistent with that but I did not reproduce the count.
  • mypy — NOT RUN by me. Not installed. The body's "no issues in 107 source files" is unverified locally; the green Test legs imply the Type check (mypy) step passed, since ci.yml runs it before the test step with no continue-on-error.
  • yamllint — NOT RUN by me. Not installed. The Windows CRLF diagnosis is the one part of this PR I could not check at all: I have no Windows runner, did not reproduce core.autocrlf=true behaviour, and did not confirm that the stray CR in auto-assign.yml was the specific file that broke the new-lines: platform attempt. I am taking the body's account on that, and the green Test (Python 3.x, windows-2022) legs are consistent with it.
  • Vendor drift — NOT RUN by me. python scripts/check_vendor_drift.py was not executed here; Compare core/ against pinned upstreams is green in CI on this head. I did not independently verify that the reverted file matches blob 7f9c8c1 at eos pin 5544c98.
  • Scorecard — NOT VERIFIED. I did not confirm that v2.4.3 pulls from ghcr.io, that gcr.io returns the billing error, or that eos's Scorecard job is green. All three are the body's claims. The CodeQL and Analyze (Python) checks green on this head do not cover the Scorecard workflow, which runs on a schedule.
  • The existing-comments.txt for this PR contains only the coverage bot; no human review points existed to avoid repeating.

Verified locally, against a git archive export of head f8cc3574:

  • ruff 0.16.5 check .All checks passed! The four findings present on origin/master are cleared.
  • to_dict() emits ['build','dependencies','package','patches','url','version'] for a recipe carrying install_args; parse_recipe(yaml.safe_load(yaml.safe_dump(...))) does not compare equal to the original, and install_args comes back []. Finding 1.
  • The four blobs the body cites as byte-identical to #122 match.
  • origin/master still carries all four ruff findings, so this PR's premise is current, not stale.

Automated architecture review of f8cc357421ac — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants