Milestones 1–2: PEG parser engine port and the accept/reject conformance gate

.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+      - run: go build ./...
+      - run: go vet ./...
+      - run: gofmt -l . && test -z "$(gofmt -l .)"
+      - run: go test -race ./...

.gitignore

@@ -0,0 +1,4 @@
+duckdb
+*.db
+*.db.wal
+*.zip

CLAUDE.md

@@ -0,0 +1,86 @@
+# darkwing development guide
+
+darkwing is a pure Go port of DuckDB v2.0's PEG parser. Read `PLAN.md` first —
+it is the authoritative design document. The engine is a faithful port of the
+C++ under `src/parser/peg/` in the DuckDB repo, pinned at the commit recorded
+in `internal/grammar/README.md`.
+
+## Layout
+
+- `token/` — token kinds, `Token{Kind, Text, Span}`, the five keyword
+  categories (built from the vendored `.list` files).
+- `lexer/` — port of `base_tokenizer.cpp` + `parser_tokenizer.cpp`.
+- `internal/grammar/` — vendored `.gram`/`.list` files (never edit them by
+  hand; they are re-vendored by the regeneration workflow) and the grammar
+  loader, a port of `peg_parser.cpp`/`parsed_grammar.cpp`.
+- `internal/matcher/` — matcher tree, packrat memoization, rule overrides and
+  `ParseResult`, a port of `matcher_factory.cpp`/`matcher.cpp`/
+  `parser_packrat.cpp` and `matcher/*.hpp`.
+- `cmd/debug-parse/` — dump tokens or the raw `ParseResult` tree for SQL
+  passed on the command line.
+- `internal/sqltest/` — sqllogictest `.test` reader (statement extraction
+  only; template substitutions are skipped, never expanded).
+- `internal/testfile/` — corpus storage format (cases separated by `==`,
+  SQL/expectation separated by `--`) plus `*.metadata.json` todo/skip
+  sidecars keyed by case content hash.
+- `internal/duckdbsrc/` — the pinned DuckDB CLI (oracle): locate, verify
+  version against the pin, run statements against `:memory:` and classify
+  Parser Error vs post-parse.
+- `cmd/regenerate/` — rebuild `parser/testdata/` from a DuckDB source tree
+  + the pinned binary. The only way corpus expectations change.
+- `cmd/next-test/` — print the next todo case from the corpus metadata.
+- `parser/` — corpus conformance harness (`parser_test.go`); the public
+  Parse API and AST arrive in milestone 3.
+
+## Rules of the port
+
+- **Effective behavior over textbook behavior.** darkwing replicates what the
+  pinned DuckDB build actually does, including acknowledged quirks: negative
+  lookahead (`!`) is parsed but ignored by the matcher; `/` binds only the
+  immediately preceding element (upstream grammars parenthesize sequences in
+  alternatives); `*`/`+`/`?` wrap only the last element of the current list.
+  Do not "fix" these — they are pinned by tests.
+- **Vendored files are verbatim.** `internal/grammar/duckdb/**` is copied
+  unmodified from upstream. Changing grammar behavior means advancing the pin,
+  never editing the files.
+- **Zero dependencies.** `go.mod` stays module line + Go version only.
+
+## Dev loop
+
+```
+go build ./... && go vet ./... && gofmt -l . && go test -race ./...
+```
+
+Debugging a parse:
+
+```
+go run ./cmd/debug-parse 'SELECT 1'            # ParseResult tree
+go run ./cmd/debug-parse -tokens 'SELECT 1'    # token dump
+```
+
+## Conformance loop
+
+The milestone-2 gate: darkwing accepts a statement iff the pinned DuckDB
+binary parses it, over the whole corpus (`go test ./parser`).
+
+```
+go run ./cmd/next-test                                  # pick the next todo case
+go test ./parser -run TestCorpus -check-parse 'FRAGMENT' # dump detail for it
+# ...fix the engine...
+go test ./parser                                        # gate
+```
+
+Some oracle rejects come from upstream's *transformer* (Parser Errors whose
+message is not "syntax error at or near ..."): the matcher alone cannot
+reject those, so they stay in todo metadata until the matching transformer
+lands (milestones 3-5). Todo entries carry a note saying why.
+
+Regenerating the corpus (needs a DuckDB checkout at the pinned commit and
+the matching nightly CLI; see `internal/grammar/README.md` for the pin):
+
+```
+DARKWING_DUCKDB=/path/to/duckdb-cli \
+  go run ./cmd/regenerate -duckdb-src /path/to/duckdb
+```
+
+Never commit binaries (the CLI, *.db files) — only source and corpus text.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 The sqlc Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

LICENSE.DUCKDB

@@ -0,0 +1,27 @@
+darkwing vendors grammar and keyword files from DuckDB and ports portions of
+DuckDB's PEG parser (tokenizer, grammar loader, and matcher) to Go. DuckDB is
+distributed under the MIT license, reproduced below.
+
+Source: https://github.com/duckdb/duckdb
+
+---
+
+Copyright 2018-2026 Stichting DuckDB Foundation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1 +1,50 @@
-# darkwing
+# darkwing
+
+*The terror that flaps in the night, for DuckDB.*
+
+darkwing is a hand-written, zero-dependency Go port of DuckDB v2.0's
+PEG-based SQL parser, built to power a first-class DuckDB engine in
+[sqlc](https://github.com/sqlc-dev/sqlc). See [PLAN.md](PLAN.md) for the full
+design.
+
+Unlike its siblings ([oliphant](https://github.com/sqlc-dev/oliphant),
+[marino](https://github.com/sqlc-dev/marino),
+[meyer](https://github.com/sqlc-dev/meyer),
+[teesql](https://github.com/sqlc-dev/teesql),
+[zetajones](https://github.com/sqlc-dev/zetajones),
+[doubleclick](https://github.com/sqlc-dev/doubleclick)), darkwing does not
+re-express the grammar as recursive descent: DuckDB's own production parser
+is an interpreter over machine-readable grammar text, so darkwing vendors the
+`.gram`/`.list` files verbatim and ports the engine — tokenizer, grammar
+loader, and matcher (with packrat memoization) — to Go.
+
+## Status
+
+Milestone 2 (corpus + accept/reject conformance) is in place: the corpus
+under `parser/testdata/` is extracted from DuckDB's own test suite and
+classified by the pinned DuckDB CLI (`cmd/regenerate`); `go test ./parser`
+enforces *darkwing accepts iff pinned DuckDB parses*, with remaining
+disagreements tracked in todo metadata (`cmd/next-test`).
+
+Milestone 1 (engine) is complete:
+
+- `token/` — token kinds and DuckDB's keyword categories
+- `lexer/` — port of the parsing tokenizer
+- `internal/grammar/` — vendored grammar (pinned upstream commit recorded in
+  `internal/grammar/README.md`) and the grammar loader
+- `internal/matcher/` — matcher tree, packrat memoization, rule overrides,
+  furthest-failure error reporting
+- `cmd/debug-parse` — dump tokens or the raw parse tree for SQL on the
+  command line
+
+Next: the typed AST and transformer core (milestone 3).
+
+```
+$ go run ./cmd/debug-parse 'SELECT 1'
+$ go run ./cmd/debug-parse -tokens 'SELECT * FROM t'
+```
+
+## License
+
+MIT (see `LICENSE`). Vendored DuckDB grammar files and the ported engine
+derive from MIT-licensed DuckDB source; see `LICENSE.DUCKDB`.

cmd/debug-parse/main.go

@@ -0,0 +1,69 @@
+// Command debug-parse tokenizes and matches SQL from the command line,
+// dumping either the token stream or the raw ParseResult tree — the
+// milestone-1 window into the engine, before transformers and a public
+// Parse API exist.
+//
+// Usage:
+//
+//	debug-parse [-tokens] 'SELECT 1'
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/sqlc-dev/darkwing/internal/matcher"
+	"github.com/sqlc-dev/darkwing/lexer"
+)
+
+func main() {
+	tokensOnly := flag.Bool("tokens", false, "dump the token stream instead of the parse tree")
+	flag.Parse()
+	if flag.NArg() == 0 {
+		fmt.Fprintln(os.Stderr, "usage: debug-parse [-tokens] <sql>")
+		os.Exit(2)
+	}
+	sql := strings.Join(flag.Args(), " ")
+	if err := run(sql, *tokensOnly); err != nil {
+		fmt.Fprintf(os.Stderr, "error: %v\n", err)
+		os.Exit(1)
+	}
+}
+
+func run(sql string, tokensOnly bool) error {
+	tokens, err := lexer.Tokenize(sql)
+	if err != nil {
+		return err
+	}
+	if tokensOnly {
+		for _, t := range tokens {
+			fmt.Printf("%-16s %4d..%-4d %q\n", t.Kind, t.Span.Start, t.Span.End, t.Text)
+		}
+		return nil
+	}
+	engine, err := matcher.Default()
+	if err != nil {
+		return err
+	}
+	results, err := engine.MatchAll(tokens)
+	if err != nil {
+		return err
+	}
+	statement := 0
+	for _, result := range results {
+		// TopLevelStatement <- Statement? (';'+ / EndOfInput): a
+		// separator-only match yields no statement, as upstream's
+		// TransformTopLevelStatement returns nullptr for it
+		if lr, ok := result.(*matcher.ListResult); ok && len(lr.Children) > 0 {
+			if opt, ok := lr.Children[0].(*matcher.OptionalResult); ok && !opt.HasResult() {
+				continue
+			}
+		}
+		statement++
+		fmt.Printf("-- statement %d\n", statement)
+		fmt.Print(matcher.Dump(result))
+	}
+	return nil
+}

cmd/next-test/main.go

@@ -0,0 +1,86 @@
+// Command next-test picks the next todo case from the conformance corpus:
+// the dev loop is next-test -> implement -> `go test ./parser
+// -check-parse '<fragment>'` (see CLAUDE.md).
+//
+// Usage:
+//
+//	next-test [-testdata parser/testdata] [-all]
+package main
+
+import (
+	"flag"
+	"fmt"
+	"io/fs"
+	"log"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+
+	"github.com/sqlc-dev/darkwing/internal/testfile"
+)
+
+func main() {
+	log.SetFlags(0)
+	testdata := flag.String("testdata", "parser/testdata", "testdata directory")
+	all := flag.Bool("all", false, "list every todo case instead of just the first")
+	flag.Parse()
+
+	root := filepath.Join(*testdata, "corpus")
+	var paths []string
+	err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		if !d.IsDir() && strings.HasSuffix(path, ".test") {
+			paths = append(paths, path)
+		}
+		return nil
+	})
+	if err != nil {
+		log.Fatalf("error: %v", err)
+	}
+	sort.Strings(paths)
+
+	total := 0
+	for _, path := range paths {
+		meta, err := testfile.ReadMetadata(path)
+		if err != nil {
+			log.Fatalf("error: %v", err)
+		}
+		if len(meta.Todo) == 0 {
+			continue
+		}
+		file, err := testfile.Read(path)
+		if err != nil {
+			log.Fatalf("error: %v", err)
+		}
+		byKey := make(map[string]*testfile.Case)
+		for i := range file.Cases {
+			byKey[file.Cases[i].Key()] = &file.Cases[i]
+		}
+		for _, key := range testfile.SortKeys(meta.Todo) {
+			c, ok := byKey[key]
+			if !ok {
+				continue
+			}
+			total++
+			want := "accept"
+			if c.Reject {
+				want = "reject: " + c.Error
+			}
+			fmt.Printf("%s\ncase: %s (%s)\nwant: %s\nsql:\n%s\n\n", path, key, meta.Todo[key], want, c.SQL)
+			if !*all {
+				return
+			}
+		}
+	}
+	if total == 0 {
+		fmt.Println("no todo cases - the corpus gate is green")
+		return
+	}
+	if *all {
+		fmt.Printf("%d todo cases\n", total)
+	}
+	os.Exit(0)
+}