Skip to content

feat(parser): parse plain HTML — inline <script> blocks + DocSection anchors (#521) - #818

Open
joseedson18jc wants to merge 1 commit into
tirth8205:mainfrom
joseedson18jc:feat/html-embedded-scripts
Open

feat(parser): parse plain HTML — inline <script> blocks + DocSection anchors (#521)#818
joseedson18jc wants to merge 1 commit into
tirth8205:mainfrom
joseedson18jc:feat/html-embedded-scripts

Conversation

@joseedson18jc

Copy link
Copy Markdown

Linked issue

Closes #521

What & why

Implements plain-HTML parsing per the maintainer's guidance on #521 (credit to @dannymn for the proposal and reference design — this is a fresh implementation rebased on current main, since the reference branch was noted as 341 commits behind).

Inline <script> blocks — same delegation pattern as _parse_vue/_parse_svelte:

  • Iterative AST walk (script elements sit at arbitrary depth in real documents, and deeply nested pages must not hit the recursion limit)
  • <script src=…> references and non-JS payloads (application/json, importmap, text/x-template, …) are skipped; an absent/empty type and module/text/javascript-family types parse as JavaScript
  • Unquoted (type=module) and case-varied attributes handled
  • Line numbers offset back to the containing .html file
  • Relative imports inside type="module" blocks resolve from the HTML file's directory (source-side gate only; .html is deliberately not added as an import target, which would not be valid ES semantics)

DocSection anchors — per your integration requirements on the issue:

  • div/section/article with an idDocSection node (extra.html_id, full element line range)
  • CONTAINS edge from the File node to each DocSection, so children_of and structural traversal reach them
  • Anchor lookup only — no prose-content search claims; only the id is indexed
  • Duplicate ids: first occurrence wins (getElementById semantics)
  • Style blocks and heading anchors stay out of scope, as requested

Field validation on a real codebase: a 246 KB single-file dashboard (index.html, previously a bare File node) now yields 169 functions with accurate line ranges; semantic queries like "save the api key to browser storage" resolve to the correct function at rank 1.

How it was tested

uv run pytest tests/ --tb=short -q
# 2417 passed, 1 skipped, 2 xpassed in 32.40s

uv run ruff check code_review_graph/
# All checks passed!

uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional
# Success: no issues found in 70 source files

15 new tests in tests/test_multilang.py::TestHtmlEmbeddedScripts cover the regression list from the issue discussion: multiple and nested scripts, external (src=) and empty scripts, malformed HTML (unclosed element), duplicate ids, unquoted and case-varied attributes, .htm detection, line-number offsets, import/call edges across blocks, and CONTAINS reachability of every DocSection.

Checklist

  • Tests added for new functionality
  • All tests pass: uv run pytest tests/ --tb=short -q
  • Linting passes: uv run ruff check code_review_graph/
  • Type checking passes: uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional
  • Lines are at most 100 characters
  • Docs updated where behavior changed (README language coverage ×2, CHANGELOG)

🤖 Generated with Claude Code

…anchors (tirth8205#521)

Reuses the Vue/Svelte SFC delegation machinery for .html/.htm, per the
maintainer's guidance on tirth8205#521:

- Inline <script> blocks: walk the tree-sitter html AST iteratively
  (script elements sit at arbitrary depth, and deep documents must not
  hit the recursion limit), skip <script src=…> references and non-JS
  payloads (application/json, importmap, text/x-template, …), parse each
  body with the JavaScript grammar, and offset line numbers back to the
  containing .html file. Unquoted (type=module) and case-varied
  attributes are handled. Relative imports inside type="module" blocks
  resolve from the HTML file's directory.
- DocSection anchors: id-anchored div/section/article elements become
  DocSection nodes (extra.html_id, full element line range) with a
  CONTAINS edge from the File node, so children_of and structural
  traversal reach them. Anchor lookup only — no prose-content claims.
  On duplicate ids the first occurrence wins (getElementById
  semantics). Style blocks and heading anchors stay out of scope.

Closes the blind spot where single-file HTML apps produced only a bare
File node: a 246 KB dashboard index.html now yields 169 functions with
accurate line ranges.

15 new tests in tests/test_multilang.py::TestHtmlEmbeddedScripts cover
multiple and nested scripts, external/empty scripts, malformed HTML,
duplicate ids, unquoted and case-varied attributes, .htm detection,
line offsets, import/call edges, and CONTAINS reachability.

Closes tirth8205#521

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dannymn

dannymn commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Original reporter of #521 here. Thanks for picking this up, and for rebasing rather than trying to salvage my branch — that was the right call.

Some field validation, since I've been running an out-of-tree variant of this design against a real repo since June (currently CRG 2.3.7, 2,065 files, 64,747 nodes):

  • 16 HTML files produce 165 DocSection nodes + 313 Function nodes from inline <script> blocks.
  • The largest single document goes from one opaque File node to 45 DocSections + 51 Functions. Before the parser existed, semantic search over generated HLD documents returned nothing at all; now it resolves to the right section.

On your requirement #1 (CONTAINS edges from the file to each DocSection) — worth stating that this is not a hypothetical concern. My variant omits those edges, and the consequence is measurable: 0 of 25,957 DocSection nodes in my graph have a single incoming edge. They are reachable by direct search only; children_of, structural context, and architecture traversal all walk straight past them. Emitting CONTAINS is the difference between "searchable" and "part of the graph." Good catch, and I'm glad it's in the PR rather than deferred.

Your requirement #2 is also right to insist on. Anchor-id indexing is what this delivers; prose-content retrieval is a different feature and shouldn't be implied by it.

One gap, purely as a follow-up and explicitly not a request to widen this PR: this covers .html only. My variant also emits one DocSection per ATX heading for .md (fenced-code aware), and in practice that is where most of the value has been — 447 markdown files produce 25,792 DocSections, versus 165 from HTML. Design docs, specs, and plans are usually markdown, and they're currently invisible to the graph for the same reason HTML was. Happy to open a separate issue for that if it's of interest, so this PR can stay scoped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HTML parser: extract inline <script> functions + id-anchored DocSection nodes

2 participants