clickhouse: fix date overflow for initial load#4534
Draft
dtunikov wants to merge 5 commits into
Draft
Conversation
dtunikov
commented
Jul 3, 2026
| // Timestamps are micros since epoch; dates are days since epoch. Both encoded as Avro varint. | ||
| avroVal := t.UnixMicro() | ||
| if isDate { | ||
| avroVal = t.Unix() / 86400 |
Collaborator
Author
There was a problem hiding this comment.
had to move it here out of process[type] functions because we clamp t value for clickhouse
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
❌ 8 Tests Failed:
View the top 3 failed test(s) by shortest run time
View the full list of 2 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Contributor
❌ Test FailureAnalysis: A deterministic assertion failure in Test_MySQL_DateTime_ClickHouse_Range (expected nil but got 1900-01-01) reproduces identically across all MySQL/MariaDB matrix variants, indicating the PR's own date-overflow fix is incomplete rather than a flaky failure. |
Contributor
❌ Test FailureAnalysis: Real bug: the PR's own test Test_MySQL_DateTime_ClickHouse_Range deterministically fails across all 4 CH suites and both matrix jobs with a clear value mismatch (out-of-range nullable DATE replicated as 1900-01-01 instead of NULL), with no timeout/race/network signals. |
parseDateTime64BestEffortOrNull (used by the CDC normalize path) saturates out-of-range values to ClickHouse's [1900, 2299] boundary, clamping the date while preserving the time-of-day, and never nulls. Make the snapshot/Avro path do the exact same thing so snapshot and CDC stay consistent for the same source value, regardless of column nullability. Verified the Go clamp matches parseDateTime64BestEffortOrNull output exactly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
❌ Test FailureAnalysis: The PR's own new test Test_MySQL_DateTime_ClickHouse_Range fails deterministically with a logic assertion ("expected out-of-range nullable DATE to be NULL") across all 4 ClickHouse suite variants and all 3 matrix jobs, indicating the date-overflow fix is genuinely broken rather than flaky. |
Contributor
❌ Test FailureAnalysis: Real bug: Test_MySQL_DateTime_ClickHouse_Range fails deterministically across every matrix leg with an identical out-of-range datetime overflow (expected 2299-12-31 but got wrapped-around 1715-06-12), indicating a MySQL→ClickHouse datetime range-boundary conversion defect, not a flaky/timeout/race failure. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.