Skip to content

TestValidateConfigExample silently swallows example parsing errors #210

@G26karthik

Description

@G26karthik

Bug report:

In schema/example_test.go, the validate(t, examples) function iterates over examples extracted from docs/config.md. For each example it checks example.Err (the error from parsing that specific example block), but when reporting the failure it calls t.Error(err) instead of t.Error(example.Err).

The err variable in that scope is the result of extractExamples(...) from earlier in the function. By the time the loop runs, extractExamples has already succeeded, so err is nil. Calling t.Error(nil) is a no-op, which means example parsing failures pass the test instead of failing it.

Expected behavior:

If any documented example in docs/config.md cannot be parsed, TestValidateConfigExample should fail with that example's parsing error so the broken documentation can be caught in CI.

How to reproduce it:

  1. Clone the repo at current main.
  2. Temporarily corrupt one of the JSON examples inside json ... blocks in docs/config.md (e.g. remove a closing brace).
  3. Run go test ./schema -run TestValidateConfigExample -v.
  4. Observe that the test still passes despite the malformed example, because the error path uses the wrong variable.

Environment:

  • Model Spec version: v0.0.7 (also present in main at commit c971699)
  • OS: any (logic bug, platform-independent)
  • Kernel (e.g. uname -a): n/a
  • Others: Go 1.23.1 toolchain (per go.mod)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions