Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
7e4a62e
Add plan for hand-written recursive descent parser rewrite
claude Aug 6, 2026
874a644
Add recursive-descent parser core with goyacc differential harness
claude Aug 6, 2026
5448200
Implement expression parsing in the RD parser
claude Aug 6, 2026
e776acc
Implement SELECT family in the RD parser
claude Aug 6, 2026
d620560
Implement DML statements in the RD parser
claude Aug 6, 2026
02e3de4
Add per-file statement dispatch registration for the RD parser
claude Aug 6, 2026
60b4e5c
Document the differential-harness dev loop
claude Aug 6, 2026
bc00ffa
Implement GRANT/REVOKE statements in the RD parser
claude Aug 6, 2026
2b3d8af
Implement the optimizer-hint parser in recursive descent
claude Aug 6, 2026
e9793b3
Checkpoint in-progress ports: types, SET/SHOW/txn/misc families
claude Aug 6, 2026
f0787f5
Implement SET, SHOW, transaction, and misc statements in the RD parser
claude Aug 6, 2026
724fc82
Checkpoint in-progress port: column definitions and constraints
claude Aug 6, 2026
dc8f8b0
Implement CREATE TABLE, column types, and constraints in the RD parser
claude Aug 6, 2026
5767909
Implement remaining CREATE kinds, DROP, RENAME, and FLASHBACK/RECOVER
claude Aug 7, 2026
06fdfbb
Implement the ALTER statement family in the RD parser
claude Aug 7, 2026
55a0bec
Implement stored procedures and SQL plan bindings in the RD parser
claude Aug 7, 2026
9550624
Implement admin, analyze, BRIE, and TiDB statements in the RD parser
claude Aug 7, 2026
5ee3e41
Add error-fidelity mode and corpus to the RD parser
claude Aug 7, 2026
54268a6
Remove goyacc: the recursive-descent parser is now the only parser
claude Aug 7, 2026
3e57a2a
Rename yy_parser.go to api.go and drop yacc-era dead declarations
claude Aug 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 34 additions & 4 deletions BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,44 @@ exports_files([
go_library(
name = "parser",
srcs = [
"api.go",
"digester.go",
"generate.go",
"hintparser.go",
"hint_token_kinds.go",
"hintparserimpl.go",
"keyword_classes.go",
"keywords.go",
"lexer.go",
"misc.go",
"parser.go",
"yy_parser.go",
"parse_admin.go",
"parse_alter.go",
"parse_analyze.go",
"parse_binding.go",
"parse_brie.go",
"parse_column.go",
"parse_create_index.go",
"parse_create_misc.go",
"parse_create_table.go",
"parse_create_view.go",
"parse_dml.go",
"parse_drop.go",
"parse_expr.go",
"parse_flashback.go",
"parse_func.go",
"parse_grant.go",
"parse_hint.go",
"parse_misc.go",
"parse_procedure.go",
"parse_rename.go",
"parse_select.go",
"parse_set.go",
"parse_show.go",
"parse_tidb.go",
"parse_txn.go",
"parse_types.go",
"rd_errors.go",
"rd_parser.go",
"redact.go",
"token_kinds.go",
],
importpath = "github.com/sqlc-dev/marino/parser",
visibility = ["//visibility:public"],
Expand Down Expand Up @@ -48,6 +77,7 @@ go_test(
"lexer_test.go",
"main_test.go",
"parser_test.go",
"rd_errors_test.go",
],
data = glob(["**"]),
embed = [":parser"],
Expand Down
44 changes: 44 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# marino development guide

marino is a hand-written recursive-descent parser for the MySQL dialect
(TiDB-flavored). It was ported from a goyacc grammar; PLAN.md records the
architecture of that rewrite and the decisions that survive it.

## Layout

- `parser/rd_parser.go` — parser core: token window over the streaming
lexer, dispatch (`rdRegister`, one owner per leading token), statement
loop with the historical statement-text bookkeeping.
- `parser/parse_*.go` — one file per statement family. Every nontrivial
parse function carries a comment naming the grammar production(s) it
implements (the grammar reference is the parser.y of the last goyacc
commit; see PLAN.md).
- `parser/parse_expr.go` / `parse_func.go` — the expression precedence
ladder and atoms.
- `parser/token_kinds.go`, `hint_token_kinds.go` — token constants and
lexer value types (snapshotted from the generated parser at removal;
hand-maintained now). NOTE: `any`, `recover`, `dump`, `format` shadow
Go builtins/imports inside package parser.
- `parser/keyword_classes.go` + `keywords.go` + `misc.go` tokenMap — the
keyword tables, kept in exact agreement by `TestKeywordConsistent`.
- `parser/parse_hint.go` — the optimizer-hint sub-parser.
- `parser/rd_errors.go` — error construction; messages reproduce the
goyacc format byte-for-byte, validated by `TestRDErrorFidelity`
against `parser/testdata/errors.json` (goldens recorded from the
goyacc parser before its removal).

## Rules

- The `ast` package is frozen, and the public `parser` API is stable.
- Error messages are part of the contract: `line N column M near "..."`
built from the offending token's recorded position; action errors use
`r.actionErrorf` (reduce-time lookahead position). Failures during
speculation report the farthest position reached (`farthestFail`).
- One deliberate deviation from goyacc, documented in PLAN.md:
`OriginTextPosition` is the deterministic production-start offset.
- Always run parser tests with a timeout: they are fast, and a hang
means an infinite loop in the parser:

```sh
go test ./... -count=1 -timeout 120s
```
34 changes: 4 additions & 30 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,40 +1,14 @@
.PHONY: all parser clean
.PHONY: all test fmt clean

all: fmt parser generate
all: fmt

test: fmt parser
test: fmt
sh test.sh

parser: parser/parser.go parser/hintparser.go

genkeyword: generate_keyword/genkeyword.go
go build -C generate_keyword -o ../parser/genkeyword

generate: genkeyword parser/parser.y
go generate ./parser/...

parser/parser.go: parser/parser.y bin/goyacc
@echo "bin/goyacc -o $@ -p yy -t Parser $<"
@bin/goyacc -o $@ -p yy -t Parser $< || ( rm -f $@ && echo 'Please check y.output for more information' && exit 1 )
@rm -f y.output

parser/hintparser.go: parser/hintparser.y bin/goyacc
@echo "bin/goyacc -o $@ -p yyhint -t hintParser $<"
@bin/goyacc -o $@ -p yyhint -t hintParser $< || ( rm -f $@ && echo 'Please check y.output for more information' && exit 1 )
@rm -f y.output

parser/%arser_golden.y: parser/%arser.y
@bin/goyacc -fmt -fmtout $@ $<
@(git diff --no-index --exit-code $< $@ && rm $@) || (mv $@ $< && >&2 echo "formatted $<" && exit 1)

bin/goyacc: goyacc/main.go goyacc/format_yacc.go
GO111MODULE=on go build -C goyacc -o ../bin/goyacc .

fmt: bin/goyacc parser/parser_golden.y parser/hintparser_golden.y
fmt:
@echo "gofmt (simplify)"
@gofmt -s -l -w . 2>&1 | awk '{print} END{if(NR>0) {exit 1}}'

clean:
go clean -i ./...
rm -rf *.out
rm -f parser/parser.go parser/hintparser.go parser/genkeyword
Loading
Loading