MDEV-37009 bulk insert into vector index by shabbann · Pull Request #5150 · MariaDB/server

shabbann · 2026-05-31T04:08:20Z

Add support for bulk insertion into MHNSW vector indexes during ALTER TABLE

gemini-code-assist

Code Review

This pull request introduces bulk insert support for hierarchical navigable small world (mhnsw) indexes during table copy operations. The reviewer identified several critical issues, including compilation errors from accessing bulk_insert_active through the wrong pointer, an uninitialized member variable, and an ignored return value during bulk insert initialization. Suggestions were also made to reset the active flag on failure and correct formatting to match the repository's style.

gemini-code-assist · 2026-05-31T04:10:12Z

+    DBUG_ASSERT(s->hlindexes() == (hlindex != NULL));
+    if (hlindex && hlindex->in_use)
+    {
+        if (hlindex->bulk_insert_active)
+            return mhnsw_bulk_insert_row(this, key_info + s->keys);
+        else
+            return mhnsw_insert(this, key_info + s->keys);
+    }
+    return 0;


The member bulk_insert_active was added to the TABLE struct in sql/table.h, but here it is accessed via hlindex->bulk_insert_active. Since hlindex is not of type TABLE*, this will cause a compilation error. It should be accessed directly as bulk_insert_active (or this->bulk_insert_active).

Additionally, please update the indentation to 2 spaces to match the repository's implicit style guide.

DBUG_ASSERT(s->hlindexes() == (hlindex != NULL)); if (hlindex && hlindex->in_use) { if (bulk_insert_active) return mhnsw_bulk_insert_row(this, key_info + s->keys); else return mhnsw_insert(this, key_info + s->keys); } return 0;

gemini-code-assist · 2026-05-31T04:10:12Z

+int TABLE::hlindexes_bulk_insert_begin(ha_rows rows)
+{
+    if (hlindex && hlindex->in_use)
+    {
+        hlindex->bulk_insert_active= true;
+        return mhnsw_bulk_insert_begin(this, key_info + s->keys, rows);
+    }
+    return 0;
+}
+
+int TABLE::hlindexes_bulk_insert_end()
+{
+    if (hlindex && hlindex->in_use)
+    {
+        hlindex->bulk_insert_active= false;
+        return mhnsw_bulk_insert_end(this, key_info + s->keys);
+    }
+    return 0;
+}


There are three issues here:

bulk_insert_active is a member of TABLE, not hlindex, so accessing it via hlindex->bulk_insert_active will cause a compilation error.

If mhnsw_bulk_insert_begin fails (returns a non-zero error code), bulk_insert_active should be reset to false so that subsequent inserts do not incorrectly attempt bulk insertion.

The indentation should be updated to 2 spaces to match the repository's implicit style guide.

int TABLE::hlindexes_bulk_insert_begin(ha_rows rows) { if (hlindex && hlindex->in_use) { bulk_insert_active= true; int err= mhnsw_bulk_insert_begin(this, key_info + s->keys, rows); if (err) bulk_insert_active= false; return err; } return 0; } int TABLE::hlindexes_bulk_insert_end() { if (hlindex && hlindex->in_use) { bulk_insert_active= false; return mhnsw_bulk_insert_end(this, key_info + s->keys); } return 0; }

gemini-code-assist · 2026-05-31T04:10:13Z

+  if (!to->s->long_unique_table)
  {
-    to->file->ha_start_bulk_insert(from->file->stats.records,
-                                   ignore ? 0 : HA_CREATE_UNIQUE_INDEX_BY_SORT);
-    bulk_insert_started= 1;
+      to->file->ha_start_bulk_insert(from->file->stats.records,
+              ignore ? 0 : HA_CREATE_UNIQUE_INDEX_BY_SORT);
+      bulk_insert_started= 1;
+
+      if (to->s->hlindexes())
+      {
+          to->hlindexes_bulk_insert_begin(from->file->stats.records);
+          hlindex_bulk_started= 1;
+      }
  }


The return value of to->hlindexes_bulk_insert_begin is currently ignored. If the bulk insert initialization fails, we should handle the error, print it, and avoid setting hlindex_bulk_started to 1 so that we do not proceed with the bulk insert or attempt to end it later.

Additionally, please update the indentation to 2 spaces to match the repository's implicit style guide.

if (!to->s->long_unique_table) { to->file->ha_start_bulk_insert(from->file->stats.records, ignore ? 0 : HA_CREATE_UNIQUE_INDEX_BY_SORT); bulk_insert_started= 1; if (to->s->hlindexes()) { if ((error= to->hlindexes_bulk_insert_begin(from->file->stats.records))) { to->file->print_error(error, MYF(0)); } else { hlindex_bulk_started= 1; } } }

gemini-code-assist · 2026-05-31T04:10:13Z

+  if (hlindex_bulk_started && to->hlindexes_bulk_insert_end() && error <= 0)
+  {
+      if (!thd->is_error())
+          to->file->print_error(my_errno, MYF(0));
+      error= 1;
+  }
+  hlindex_bulk_started=0;


Please update the indentation to 2 spaces and add proper spacing around operators (e.g., hlindex_bulk_started = 0;) to match the repository's implicit style guide.

if (hlindex_bulk_started && to->hlindexes_bulk_insert_end() && error <= 0) { if (!thd->is_error()) to->file->print_error(my_errno, MYF(0)); error= 1; } hlindex_bulk_started= 0;

gemini-code-assist · 2026-05-31T04:10:13Z

  */
  bool alias_name_used;              /* true if table_name is alias */
  bool get_fields_in_item_tree;      /* Signal to fix_field */
+  bool bulk_insert_active;           /* mhnsw bulk_insert_started flag */


The new member variable bulk_insert_active is not initialized. To prevent potential uninitialized memory bugs, it is highly recommended to initialize it to false directly in the struct definition using C++ member initialization.

bool bulk_insert_active= false; /* mhnsw bulk_insert_started flag */

…orrectly set the grefs already faster by ~15%

iMineLink

Thank you for sharing the first draft @shabbann, I did a preliminary review and found minor things and a missing initialization of MHNSW_Share::bulk_active worth checking.
Possibly out of scope, but it may be interesting to enable falling back to normal insert if some memory budget is exceeded for the bulk insert, to avoid unexpected OOM issues.

iMineLink · 2026-06-05T10:16:03Z

@@ -510,7 +510,7 @@ class MHNSW_Share : public Sql_alloc
  const uint M;
  metric_type metric;
  bool use_subdist;
-
+  bool bulk_active;


Can you explicitely initialize this to false?

iMineLink · 2026-06-05T10:16:38Z

+
+      if (to->s->hlindexes())
+      {
+          to->hlindexes_bulk_insert_begin(from->file->stats.records);


I think error handling is needed here.

iMineLink · 2026-06-05T10:18:31Z

+      return err;
+  }
+
+  if (int err= graph->file->ha_end_bulk_insert())


I think this needs to be placed inside a SCOPE_EXIT or similar, to avoid premature exit on error if node->save() fails in the loop above.

iMineLink · 2026-06-05T10:21:34Z

+  if (my_init_dynamic_array(PSI_INSTRUMENT_MEM, &bulk->nodes, sizeof(FVectorNode*),
+                            rows + rows * 0.1, rows, MYF(0)))
+  {
+    delete bulk;


I think this delete bulk statements and the one few lines below can be removed.

iMineLink · 2026-06-05T10:24:20Z

@@ -1012,6 +1012,8 @@ int FVectorNode::load_from_record(TABLE *graph)
  FVector *vec_ptr= FVector::align_ptr(tref() + tref_len());
  memcpy(vec_ptr->data(), v->ptr(), v->length());
  vec_ptr->postprocess(ctx->use_subdist, ctx->vec_len);
+  if (ctx->metric == COSINE)
+    vec_ptr->abs2= 0.5f;


I see you already prepared #5184 to fix this separately, good.

iMineLink · 2026-06-05T11:08:53Z

+    delete bulk;
+    return err;
+  }
+


Here you may add DBUG_ASSERT(!bulk->ctx->start); to document that the bulk build assumes an empty target graph

shabbann marked this pull request as draft May 31, 2026 04:09

gemini-code-assist Bot reviewed May 31, 2026

View reviewed changes

vuvova requested review from iMineLink and vuvova May 31, 2026 08:35

vuvova added the GSoC label May 31, 2026

gkodinov added the External Contribution All PRs from entities outside of MariaDB Foundation, Corporation, Codership agreements. label Jun 1, 2026

shabbann added 2 commits June 4, 2026 09:38

init

a0e7cf3

with one thread we fully build the graph in memory and then save to c…

ffaa6a1

…orrectly set the grefs already faster by ~15%

shabbann force-pushed the MDEV-37009-bulk-insert-into-vector-index branch from 2c44b07 to ffaa6a1 Compare June 4, 2026 09:16

iMineLink reviewed Jun 5, 2026

View reviewed changes

Uh oh!

Conversation

shabbann commented May 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 31, 2026

Choose a reason for hiding this comment

Uh oh!

iMineLink left a comment

Choose a reason for hiding this comment

Uh oh!

iMineLink Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

iMineLink Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

iMineLink Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

iMineLink Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

iMineLink Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

iMineLink Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants