validator: fix flaky routing tests racing the BOOTSTRAPPING state - #555
Merged
Conversation
Several tests waited for the whole index listing of a node to become empty (or to shrink to a fixed length) after dropping an index. That assertion is not scoped to the test that makes it: a group runs against one shared cluster, and a test that panics before its `DROP KEYSPACE` leaves its indexes behind. Every later test in the group that waits for an empty listing then times out as well, so a single failure is reported as several and the one that actually broke is buried among the follow-up failures. Use the existing `wait_for_no_index()` helper, which waits for one specific index to disappear, in all of those places. Failures now stay attributed to the test that caused them.
The routing tests need an index to sit in the BOOTSTRAPPING state long enough to be observed and queried, and they were losing that race in CI. The wait for the state timed out after 60s even though the index had been built long before. Two things have to hold for the state to be observable, and neither did. The status a node reports for an index over the HTTP API is a snapshot that the engine refreshes on a timer, once per second by default, so a state shorter than that interval can pass between two refreshes and never be reported at all. Give the nodes of proxy-based groups a 100ms refresh interval, so the snapshot tracks the internal state closely enough to be sampled. Groups that do not manipulate the timing keep the production default, so it stays covered. The tables in these tests hold a single row, so the initial full scan costs only a handful of round trips and finishes in well under a second. The tests already delayed the traffic between the Vector Store and the DB to stretch that phase, but 100ms per frame stretched it to ~0.65s, below the refresh interval. Raise it to 500ms, which puts the phase an order of magnitude above the refresh interval.
The scylla-proxy request rules belong to the group's proxy actor, and the group fixture is set up once for the whole group, so nothing resets them between tests. A test that panics between enabling the rules and turning them off therefore leaves them in place for every test that runs after it, including tests that never touch the proxy at all. The follow-up tests then time out waiting for an index to be created or to reach SERVING, which reads as an unrelated failure. Turn the rules off in the no-TLS connection helper, which every proxy-based test calls before it installs any rule of its own, so each test starts from a known state. The call is a no-op for groups running without a proxy.
QuerthDP
changed the title
QuerthDP
marked this pull request as ready for review
August 13, 2026 15:11
There was a problem hiding this comment.
Pull request overview
Stabilizes validator routing tests by making index deletion checks specific and proxy timing deterministic.
Changes:
- Wait for targeted indexes to disappear.
- Increase routing delays and shorten proxy index-status refreshes.
- Reset stale proxy rules when opening test connections.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
crates/validator/src/routing.rs |
Stabilizes BOOTSTRAPPING-state tests. |
crates/validator/src/full_scan.rs |
Uses targeted index-removal waits. |
crates/validator/src/crud.rs |
Uses targeted index-removal waits. |
crates/validator/src/common.rs |
Configures refresh timing and resets proxy rules. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
swasik
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The validator routing tests has been failing on master as well as unrelated branches.
The index status reported over the HTTP API is a snapshot the engine refreshes once per second. The routing tests index a single-row table, so the whole BOOTSTRAPPING phase lasted ~0.65s and could fall between two refreshes, leaving the index to go from INITIALIZING straight to SERVING. The tests then timed out waiting for a state that was never reported. A panicking test also left its keyspace behind, so later tests waiting for an empty index listing failed too, which is why one flake showed up as several.
Changes:
The recurring
no SELECT permissionCDC warnings are unrelated. Dropping an index revokes the SELECT that VECTOR_SEARCH_INDEXING grants implicitly, before the readers shut down. They are emitted by scylla-cdc, one per stream reader, and appear in passing runs too.Fixes: VECTOR-852
Fixes: VECTOR-679
Refs: VECTOR-680