Make dealership_adv8 Test For SQL Files Check Deterministic #508
Make dealership_adv8 Test For SQL Files Check Deterministic #508
Conversation
| uses: astral-sh/setup-uv@v3 | ||
| with: | ||
| version: "0.4.23" | ||
| version: "0.6.0" |
There was a problem hiding this comment.
needed to update version to use new pyproject Syntax and get rid of the warning message.
| # Pin today so the generated VALUES clause (6 month-start | ||
| # timestamps) stays deterministic. Reference files reflect | ||
| # the Oct 2025 – Mar 2026 window (today = 2026-04-01). | ||
| fixed_today=pd.Timestamp("2026-04-01"), |
There was a problem hiding this comment.
This this would make the same 6 month range each time, wouldn't this be a problem next month for the expected result? Because defog_sql_text_dealership_adv8 changes every month (build dynamically) same with the defog data that uses current_date
There was a problem hiding this comment.
That's the point of the PR. We are fixing the date to be the same for generating the exact same SQL files. It's not impacting the execution and the expected results of the query itself.
The key is that both sides of the comparison are now frozen to the same fixed point for SQL file tests only.
fixed_today patches pd.Timestamp.today() to always return 2026-04-01, so the generated SQL will always produce the Oct 2025–Mar 2026 window. The reference .sql files are hardcoded that same window. Since both sides are pinned to the same value, the test will pass every month.
The execution test (test_defog_e2e) is separate and unaffected. It uses SQL-native DATE('now', ...) expressions that compute relative to the actual current date, so it always queries the real current 6-month window from the live database.
john-sanchez31
left a comment
There was a problem hiding this comment.
If it doesn't affect the e2e tests we're good
|
LGTM. |
Fix flaky
dealership_adv8SQL files test:impl_defog_dealership_adv8asks “what are the sales metrics for the last 6 months?” It computes those 6 month boundaries in Python using today’s date, then adds them as hardcoded timestamps into the generated SQL (e.g.VALUES ('2025-10-01'), ('2025-11-01'), ...). The SQL comparison test works by saving that generated SQL to a reference file and comparing future runs against it. Since the 6-month window shifts every month, the reference file goes stale on the 1st of every month and the SQL file test fails.Solution: Add a
fixed_today: pd.Timestamp | Nonefield toPyDoughSQLComparisonTest. When set,pd.Timestamp.todayis patched during SQL generation so the output is deterministic. The reference files are pinned to theOct 2025–Mar 2026 window (today = 2026-04-01). Execution tests are unaffected, they use SQL-native date expressions(DATE('now', ...)).AND
Migrate [tool.uv] dev-dependenciesto[dependency-groups] devinpyproject.tomlto silence a uv deprecation warning.