SOLR-17725: CoreAdmin API to upgrade an index in-place to facilitate Solr upgrade across major versions #3903

rahulgoswami · 2025-11-30T06:20:37Z

https://issues.apache.org/jira/browse/SOLR-17725

Description

Provide a /admin/cores api (action=UPGRADECOREINDEX) to upgrade older lucene segments to latest version (by targeted reindexing). Calling this REST endpoint on an index created in version X-1 (assuming you are on Solr version X) would reindex documents in any segments of the older version (X-1) so that they are re-written in version X. This would help prepare the index for when Solr is upgraded to X+1 without having to recreate the index from source (as is the requirement today).

Solution

This API helps to upgrade an index where all fields are either stored or docValues enabled or copy field target.

Here's the algorithm:

Set the merge policy as LatestVersionMergePolicy (which is a filter merge policy that delegates to whichever policy is actual defined in solrconfig or the TieredMergePolicy by default). This policy filters out any segments with minVersion=X-1 and only allows the latest version (X) segments to participate in merges. This ensures that the older version segments don't merge and "pollute" the index over the course of the reindexing.
Iterate over the segments and for any segment which is a pure X-1 version segment (result of flush; minVersion=X-1, version=X-1) or a result of merge between version X-1 and X segments (minVersion=X-1, version=X), reindex all live documents in the segment.
commit and validate that all segments in the index have been converted to version X.
Reset the merge policy back to the original merge policy.

This works because after the change in apache/lucene#14607 (and the backport to 10x) opening an index no longer cares about which version the index was originally created in. Lucene just needs all of the segments to be of version X or later in order to be able to open the index in version X+1 (even though the index could have been originally created in version X-1)

[AI Coding Tools Disclosure: Only use of AI coding tools was in the tests (UpgradeCoreIndexActionTest.java). Claude Code and OpenAI Codex were used to generate, review, iterate and fix the tests mentioned in the "Tests" section. The same were also manually reviewed thereafter for correctness. ]

Tests

For an index with below segments:
seg1 - pure 9x
seg2 - pure 10x
seg3 - minVersion 9x, version 10x (result of merge between 9x and 10x segements)

seg1 and seg3 get reindexed into new 10x segment, and get deleted. seg2 remains untouched (as expected).

DocValues-only fields are preserved during reindexing (this is tested specifically since we are reading docs from segments and decorating the SolrInputDocument with docValues thereafter)
older segments get deleted by the end of the process
async mode works fine. Tested by calling /admin/cores?action=UPGRADECOREINDEX&async=request_id. And later calling action=REQUESTSTATUS&requestid=request_id
For an index where all segments belong to the latest major version, calling the UPGRADECOREINDEX API is a no-op, and returns "upgradeStatus":"NO_UPGRADE_NEEDED" in the response.

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide
I have added a changelog entry for my change

rahulgoswami · 2025-11-30T06:27:17Z

First (half-baked) draft. WIP.

…ion for custom update.chain

rahulgoswami · 2025-12-23T06:50:10Z

Looking good. Basic (manual) tests with /admin/cores?action=UPGRADECOREINDEX working fine. Will publish response with segment status before and after index upgrade. Also need to add tests.

… has been renamed. It's being used in this feature..

rahulgoswami · 2025-12-31T07:11:51Z

Sample response in synchronous mode (/admin/cores?action=UPRGADECOREINDEX&core=techproducts)

{
  "responseHeader":{
    "status":0,
    "QTime":10193
  },
  "core":"techproducts",
  "numSegmentsEligibleForUpgrade":2,
  "numSegmentsUpgraded":2,
  "upgradeStatus":"UPGRADE_SUCCESSFUL"
}

Sample response in async mode ("/admin/cores?action=UPRGADECOREINDEX&core=techproducts&async=request_id" and checking status with action=REQUESTSTATUS)

{
  "responseHeader":{
    "status":0,
    "QTime":0
  },
  "STATUS":"completed",
  "response":{
    "core":"techproducts",
    "numSegmentsEligibleForUpgrade":2,
    "numSegmentsUpgraded":2,
    "upgradeStatus":"UPGRADE_SUCCESSFUL"
  }
}

rahulgoswami · 2026-01-02T02:42:19Z

@dsmiley @broustant @janhoy This is ready for review. Would appreciate your inputs. Thanks!

…gments reliably

…keep forbiddenApisTest happy

dsmiley

Partial review.

Thanks for working on this!

@broustant you may find some of this rather familiar from index encryption

dsmiley · 2026-01-07T05:27:16Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class UpgradeCoreIndex extends CoreAdminAPIBase {


Please always add at least a line of javadoc to new public classes.

dsmiley · 2026-01-07T05:37:20Z

solr/api/src/java/org/apache/solr/client/api/model/UpgradeCoreIndexRequestBody.java

+
+  @Schema(description = "Request ID to track this action which will be processed asynchronously.")
+  @JsonProperty
+  public String async;


Maybe @gerlowskija might review the V2 API spect.

But what I notice here is this async --- I'm suspicious as I figure it'd be handled in a more general way without each command needing to redundantly specify it, which is basically all commands.

dsmiley · 2026-01-07T05:39:04Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+        coreName,
+        requestBody.async,
+        "upgrade-index",
+        () -> {


a stylistic point but this lambda is way too big; please use another technique like define a method, and call it with a one-liner lambda.

dsmiley · 2026-01-07T05:40:33Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+        () -> {
+          try (SolrCore core = coreContainer.getCore(coreName)) {
+
+            log.info("Received UPGRADECOREINDEX request for core: {}", coreName);


Logging at request level seems like a framework infrastructure level concern (probably already logged), not one unique to this particular command

dsmiley · 2026-01-07T05:42:14Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+            int numSegmentsEligibleForUpgrade = 0, numSegmentsUpgraded = 0;
+            try {
+              iwRef = core.getSolrCoreState().getIndexWriter(core);
+              if (iwRef != null) {


Surely this can't be null?!

dsmiley · 2026-01-07T05:57:41Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+              if (iwRef != null) {
+                IndexWriter iw = iwRef.get();
+                if (iw != null && originalMergePolicy != null) {


again; null checks needless

dsmiley · 2026-01-07T06:02:28Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+      if (resolvedChain != null) {
+        return resolvedChain;
+      } else {
+        log.error(


I think this should throw an error (and not log)

dsmiley · 2026-01-07T06:05:47Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+    // Try to find chain configured in /update handler
+    String updateChainName = null;
+    SolrRequestHandler reqHandler = core.getRequestHandler("/update");
+
+    NamedList initArgs = ((RequestHandlerBase) reqHandler).getInitArgs();
+
+    if (initArgs != null) {
+      // Check invariants first
+      Object invariants = initArgs.get("invariants");
+      if (invariants instanceof NamedList) {
+        updateChainName = (String) ((NamedList) invariants).get(UpdateParams.UPDATE_CHAIN);
+      }
+
+      // Check defaults if not found in invariants
+      if (updateChainName == null) {
+        Object defaults = initArgs.get("defaults");
+        if (defaults instanceof NamedList) {
+          updateChainName = (String) ((NamedList) defaults).get(UpdateParams.UPDATE_CHAIN);
+        }
+      }
+    }


I think we're trying too hard to find the default chain. If it's actually this hard, then this code should live elsewhere and we just call it. But I'm still suspicious. If it wasn't so late I'd go look.

dsmiley · 2026-01-07T06:06:50Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+
+  private boolean isIndexUpgraded(SolrCore core) throws IOException {
+
+    try (FSDirectory dir = FSDirectory.open(Path.of(core.getIndexDir()));


woah woah... lets instead get the Directory from the factory from the core. See SolrIndexSplitter.

dsmiley · 2026-01-07T06:08:05Z

solr/core/src/java/org/apache/solr/handler/admin/api/UpgradeCoreIndex.java

+
+  private void doCommit(SolrCore core) throws IOException {
+    try (LocalSolrQueryRequest req = new LocalSolrQueryRequest(core, new ModifiableSolrParams())) {
+      CommitUpdateCommand cmd = new CommitUpdateCommand(req, false);


please add a little EOL comment on what "false" means

rahulgoswami · 2026-01-08T04:36:16Z

@dsmiley Thanks for an initial review. I am working on addressing the review comments.

rahulgoswami and others added 3 commits November 20, 2025 03:11

Add merge policy to block older segments from participating in merges

1ea5cce

extend FilterMergePolicy for delegation and modularize logic

cdc9978

First draft: Add UPGRADECOREINDEX as a CoreAdmin API

4be5ca1

github-actions bot added client:solrj cat:index cat:api labels Nov 30, 2025

rahulgoswami changed the title ~~CoreAdmin API to upgrade an index in-place to facilitate Solr upgrade across major versions~~ SOLR-17725: CoreAdmin API to upgrade an index in-place to facilitate Solr upgrade across major versions Nov 30, 2025

rahulgoswami marked this pull request as draft November 30, 2025 06:24

rahulgoswami added 2 commits November 30, 2025 01:55

fix incorrect early-exit while iterating over segments

052cda6

merge with main

faff01f

github-actions bot removed the cat:index label Dec 9, 2025

rahulgoswami added 2 commits December 12, 2025 02:13

restore original merge policy; injectable factory for test stubs; opt…

c9e3f3e

…ion for custom update.chain

cleanup and optimizations

daaaecb

rahulgoswami force-pushed the index-upgrade branch from 3085bea to daaaecb Compare December 12, 2025 07:14

rahulgoswami added 10 commits December 12, 2025 02:22

more cleanup

1fa4f2a

get default update chain if update.chain is not defined for /update

de455df

fix getUpdateProcessorChain()

c98418b

gradlew tidy

57b5842

fix getUpdateProcessorChain()

abd584e

1) incorrect LeafReader closure.2) Fix validation

bdfa291

cleanup

feb70ac

interrupt handling and validation cleanup

393c081

async mode may cause merge policy to be reset immediately

b335e8e

Fix null response

6a354fd

rahulgoswami marked this pull request as ready for review December 23, 2025 06:50

changelog

088c599

rahulgoswami mentioned this pull request Dec 29, 2025

Rename LatestVersionMergePolicy for a more intuitive name compatibility with the factory #3986

Closed

8 tasks

Merge branch 'main' into index-upgrade since LatestVersionMergePolicy…

dfd7786

… has been renamed. It's being used in this feature..

rahulgoswami added 3 commits December 30, 2025 11:52

LatestVersionMergePolicy renamed

3655ef4

Fix issue with running in async mode

a50fc8d

Decorate response with bookkeeping info and upgrade state

3607110

rahulgoswami added 3 commits January 1, 2026 18:41

Wrap exceptions in CoreAdminAPIBaseException

0f10339

Wrap exceptions in CoreAdminAPIBaseException

b8b5deb

Handle exceptions in processSegment to preserve the main failure

983f2d6

rahulgoswami added 2 commits January 2, 2026 03:15

keeping validateLogCalls and forbidden apis check happy

d005712

Add UPGRADECOREINDEX to ref guide for CoreAdmin API

c8f2df2

github-actions bot added the documentation Improvements or additions to documentation label Jan 4, 2026

rahulgoswami added 3 commits January 5, 2026 00:41

do UpdateHandler.commit() instead of IndexWriter.commit() to clear se…

ee5d1e2

…gments reliably

add tests

b53dec1

Use VarHandle instead of reflection in UpgradeCoreIndexActionTest to …

d7faaa4

…keep forbiddenApisTest happy

github-actions bot added the tests label Jan 6, 2026

Add license

b68f587

dsmiley reviewed Jan 7, 2026

View reviewed changes


		private boolean isIndexUpgraded(SolrCore core) throws IOException {

		try (FSDirectory dir = FSDirectory.open(Path.of(core.getIndexDir()));

SOLR-17725: CoreAdmin API to upgrade an index in-place to facilitate Solr upgrade across major versions #3903

Are you sure you want to change the base?

SOLR-17725: CoreAdmin API to upgrade an index in-place to facilitate Solr upgrade across major versions #3903

Conversation

rahulgoswami commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

rahulgoswami commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rahulgoswami commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rahulgoswami commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rahulgoswami commented Jan 2, 2026

Uh oh!

dsmiley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgoswami commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rahulgoswami commented Nov 30, 2025 •

edited

Loading

rahulgoswami commented Nov 30, 2025 •

edited

Loading

rahulgoswami commented Dec 23, 2025 •

edited

Loading

rahulgoswami commented Dec 31, 2025 •

edited

Loading