validator: create the default role before starting the Vector Store - #556
validator: create the default role before starting the Vector Store#556QuerthDP wants to merge 1 commit into
Conversation
The default-auth init paths started the Vector Store nodes first and only then created the 'vector_store' role, so every node spent the gap authenticating with credentials the cluster did not know yet. A node asked to authenticate an unknown role keeps rejecting it long after the role appears. Create the role before the Vector Store starts, and wait until every DB node accepts the credentials the Vector Store will use, since CREATE ROLE only guarantees that the coordinator knows the role. Fixes: VECTOR-861
There was a problem hiding this comment.
Pull request overview
Creates and verifies the default Vector Store database role before starting Vector Store nodes.
Changes:
- Waits for every ScyllaDB node to accept the default credentials.
- Reorders initialization so role setup precedes Vector Store startup.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
The merge-queue run for this PR failed on All three waits passed on the first attempt (no Why the probe did not prevent it
So probing Proposed patchConfine the probe to the node under test with an allow-list host filter. --- a/crates/validator/src/common.rs
+++ b/crates/validator/src/common.rs
@@ -23,11 +23,13 @@ use itertools::Itertools;
use scylla::client::session_builder::SessionBuilder;
+use scylla::policies::host_filter::AllowListHostFilter;
use scylla::response::query_result::QueryRowsResult;
use scylla::statement::Statement;
use std::collections::HashMap;
use std::iter;
use std::net::Ipv4Addr;
+use std::net::SocketAddr;
use std::sync::Arc;
@@ -138,12 +140,22 @@ async fn wait_for_role_login(actors: &TestActors, ip: Ipv4Addr, tls: bool) {
None
};
+ // Confine the probe to the node under test. Without a host filter the driver
+ // discovers the whole cluster and opens `DEFAULT_DB_USER` connections to every
+ // node, so probing one node authenticates against nodes that have not been
+ // verified yet - the very race this function exists to close.
+ let addr = SocketAddr::from((ip, DB_PORT));
+ let host_filter = Arc::new(
+ AllowListHostFilter::new([addr]).expect("failed to build the allow list host filter"),
+ );
+
wait_for(
|| async {
SessionBuilder::new()
- .known_node(ip.to_string())
+ .known_node_addr(addr)
.user(DEFAULT_DB_USER, DEFAULT_DB_PASSWORD)
.tls_context(tls_context.clone())
+ .host_filter(host_filter.clone())
.build()
.await
.is_ok()
},
Two caveats worth considering separately
Given 1 and 2, it may be worth downgrading |
swasik
left a comment
swasik
left a comment
There was a problem hiding this comment.
The fix still does not work. See my comment from Claude - in my opinion it makes sense - we should take into account what driver is doing on the lower layer.
I do not think any of them is the real issue. Shard propagation time should be minimal and I do not believe in bug in the authentication infrastructure. |
The default-auth init paths started the Vector Store nodes first and only then created the 'vector_store' role, so every node spent the gap authenticating with credentials the cluster did not know yet. A node asked to authenticate an unknown role keeps rejecting it long after the role appears.
Create the role before the Vector Store starts, and wait until every DB node accepts the credentials the Vector Store will use, since CREATE ROLE only guarantees that the coordinator knows the role.
Fixes: VECTOR-861