rbd: implement GetReplicationDestinationInfo for volumes and groups - #6366
Open
rewantsoni wants to merge 7 commits into
Open
rbd: implement GetReplicationDestinationInfo for volumes and groups#6366rewantsoni wants to merge 7 commits into
rewantsoni wants to merge 7 commits into
Conversation
6 tasks
rewantsoni
force-pushed
the
implement-replication
branch
5 times, most recently
from
July 1, 2026 08:02
cef70f8 to
6180c2c
Compare
rewantsoni
marked this pull request as ready for review
July 16, 2026 08:38
Contributor
Author
|
Testing: |
iPraveenParihar
requested changes
Jul 27, 2026
| req *replication.GetReplicationDestinationInfoRequest, | ||
| ) (*replication.GetReplicationDestinationInfoResponse, error) { | ||
| // Validate request | ||
| if req.GetReplicationSource() == nil { |
Contributor
There was a problem hiding this comment.
why not validate for empty secrets too?
rewantsoni
force-pushed
the
implement-replication
branch
4 times, most recently
from
July 28, 2026 11:21
4451914 to
2ffd5f3
Compare
rewantsoni
force-pushed
the
implement-replication
branch
from
July 28, 2026 11:29
2ffd5f3 to
79d9e7a
Compare
Member
|
Maybe rename this PR to something like this? rbd: implement GetReplicationDestinationInfo for volumes and groups |
rewantsoni
changed the title
rewantsoni
force-pushed
the
implement-replication
branch
from
July 28, 2026 16:57
79d9e7a to
486d005
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds support in the RBD CSI-Addons replication service for the new GetReplicationDestinationInfo RPC, enabling DR orchestrators to map source volume/volume-group CSI IDs to their corresponding destination IDs across mirrored clusters (including optional pool ID remapping via ConfigMap).
Changes:
- Vendor-bump
github.com/csi-addons/specand regenerate protobuf/grpc stubs to include the new RPC + capability enum. - Implement
GetReplicationDestinationInfoin the RBD CSI-Addons replication server for both volumes and volume groups, includingreplicationDestinationConfigMap schema support. - Extend cluster mapping utilities/tests and add unit tests for destination-ID mapping and basic RPC request validation.
Reviewed changes
Copilot reviewed 10 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/csi-addons/rbd/replication.go | Implements the new RPC and ID mapping helper for volumes and volume groups. |
| internal/csi-addons/rbd/replication_destination_test.go | Adds unit tests for destination ID mapping logic and request validation. |
| internal/csi-addons/rbd/identity.go | Advertises the new CSI-Addons replication capability. |
| internal/util/csiconfig.go | Adds accessor for replication destination config from CSI config. |
| internal/util/cluster_mapping.go | Adds helper to resolve mapped cluster IDs (cluster-mapping.json). |
| internal/util/cluster_mapping_test.go | Adds unit tests for mapped cluster ID resolution. |
| api/deploy/kubernetes/csi-config-map.go | Extends CSI config schema with replication destination configuration. |
| vendor/github.com/ceph/ceph-csi/api/deploy/kubernetes/csi-config-map.go | Vendored copy of the CSI config schema update. |
| PendingReleaseNotes.md | Documents the new RPC and configuration behavior. |
| docs/design/proposals/replication-destination-info.md | Updates design proposal with capability advertisement details. |
| go.mod / go.sum | Updates dependency version for csi-addons spec. |
| vendor/modules.txt | Updates vendored module versions. |
| vendor/github.com/csi-addons/spec/lib/go/replication/replication.pb.go | Vendored protobuf changes for new messages/types. |
| vendor/github.com/csi-addons/spec/lib/go/replication/replication_grpc.pb.go | Vendored grpc service changes for new RPC. |
| vendor/github.com/csi-addons/spec/lib/go/identity/identity.pb.go | Vendored capability enum update. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rewantsoni
force-pushed
the
implement-replication
branch
4 times, most recently
from
July 30, 2026 05:01
af236c2 to
26e6c47
Compare
iPraveenParihar
requested changes
Aug 3, 2026
| // ReplicationDestination defines the destination cluster for replication. | ||
| // Populated by ceph-csi-operator from ReplicationDestinationConfig CR. | ||
| // +optional | ||
| ReplicationDestination *ReplicationDestinationInfo `json:"replicationDestination,omitempty"` |
Contributor
There was a problem hiding this comment.
This change should be part of csi: add ReplicationDestinationInfo to ConfigMap schema commit
rewantsoni
force-pushed
the
implement-replication
branch
2 times, most recently
from
August 3, 2026 13:44
caae413 to
439a006
Compare
Rakshith-R
requested review from
iPraveenParihar
and
a lite review from Copilot
August 4, 2026 07:15
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 16 changed files in this pull request and generated 1 comment.
Suppressed comments (5)
internal/csi-addons/rbd/replication.go:1143
getVolumeGroupReplicationDestinationInfocan panic when the request sets the ReplicationSource type tovolumegroupbut leaves thevolumegroupmessage unset (nil).volumeGroupSourceis dereferenced without a nil check, so malformed requests can crash the server instead of returningInvalidArgument.
volumeGroupID := volumeGroupSource.GetVolumeGroupId()
if volumeGroupID == "" {
return nil, status.Error(codes.InvalidArgument, "empty volume group ID in request")
}
internal/csi-addons/rbd/replication.go:1202
- Same issue as above: the replicationDestination lookup uses
localClusterID, but the error message reportsclusterID. This makes failures harder to interpret when cluster-mapping.json remaps IDs.
destInfo, err := util.GetReplicationDestinationInfo(util.CsiConfigFile, localClusterID)
if err != nil {
return nil, status.Error(
codes.Internal,
fmt.Sprintf(
"failed to get replication destination info for cluster %s: %s",
clusterID,
err.Error(),
),
internal/csi-addons/rbd/replication.go:1113
- The error message reports
clusterID, but the lookup is performed usinglocalClusterID(after applyingGetMappedClusterID). This can mislead debugging because the reported cluster is not the one that was used to read the replicationDestination config.
This issue also appears on line 1194 of the same file.
destInfo, err := util.GetReplicationDestinationInfo(util.CsiConfigFile, localClusterID)
if err != nil {
return nil, status.Error(
codes.Internal,
fmt.Sprintf(
"failed to get replication destination info for cluster %s: %s",
clusterID,
err.Error(),
),
internal/util/cluster_mapping.go:201
getMappedClusterIDcan incorrectly remap a clusterID even when that clusterID already exists in the CSI config (e.g., when config.json contains both local and remote clusters). That would flip a local clusterID to its remote counterpart, contradicting the function comment and causing callers to use the wrong cluster config.
clusterMappingInfo, err := getClusterMappingInfo(clusterID, clusterMappingFile)
if err != nil {
return "", err
}
internal/util/cluster_mapping.go:220
- When validating whether the mapped cluster exists, errors from
readClusterInfoare silently ignored. This can mask real configuration/read failures (e.g., unreadable/invalid config.json) and fall back to returning the original clusterID, which is likely wrong. OnlyErrConfigNotFoundshould be ignored; other errors should be returned.
if _, err := readClusterInfo(csiConfigFile, mappedClusterID); err == nil {
log.DebugLog(ctx,
"mapped cluster ID %q to local cluster ID %q",
clusterID,
mappedClusterID)
Update the replication destination info proposal to document that the GET_REPLICATION_DESTINATION_INFO capability must be advertised via GetCapabilities RPC. This allows DR orchestrators to discover whether the driver supports the GetReplicationDestinationInfo feature. Signed-off-by: Rewant Soni <resoni@redhat.com>
Add ReplicationDestinationInfo, RemoteRBDDetails, and RemotePoolDetails structures to the ClusterInfo ConfigMap schema. This enables CSI driver to map source volume/group IDs to destination volume/group IDs when pool IDs differ across mirrored clusters. Signed-off-by: Rewant Soni <resoni@redhat.com>
Update csi-addons/spec from b5a7205f6a79 to d4a373713b9a to include GetReplicationDestinationInfo RPC definitions for volume and volume group replication. Signed-off-by: Rewant Soni <resoni@redhat.com>
rewantsoni
force-pushed
the
implement-replication
branch
from
August 4, 2026 10:10
439a006 to
83c5a05
Compare
Contributor
Author
|
Testing: |
rewantsoni
force-pushed
the
implement-replication
branch
from
August 6, 2026 07:27
83c5a05 to
5a2473f
Compare
Madhu-1
previously approved these changes
Aug 6, 2026
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 17 changed files in this pull request and generated no new comments.
Suppressed comments (1)
internal/csi-common/utils.go:262
- The new ReplicationSource oneof cases can panic if the oneof wrapper is present but its payload message is nil (e.g.,
&replication.ReplicationSource_Volume{}), becauser.Volume/r.Volumegroupis dereferenced unconditionally. Also, this switch does not extract a ReqID for the newGetReplicationDestinationInfoRequest, so logs for that RPC won’t include a request/volume identifier.
case *replication.ReplicationSource_Volume:
reqID = r.Volume.GetVolumeId()
case *replication.ReplicationSource_Volumegroup:
reqID = r.Volumegroup.GetVolumeGroupId()
iPraveenParihar
left a comment
iPraveenParihar
left a comment
Contributor
There was a problem hiding this comment.
@rewantsoni few nits.
Rest LGTM.
Thanks!
Tagging @ceph/ceph-csi-contributors for Review
| remotePoolDetails.PoolID, poolName, clusterID, err) | ||
| } | ||
| remotePoolID = parsedPoolID | ||
| log.DebugLog(ctx, "mapped pool %s from ID %d to remote ID %d", poolName, sourceID.LocationID, remotePoolID) |
Contributor
There was a problem hiding this comment.
Not sure if it should be logging concern.
@Madhu-1 @Rakshith-R For VolumeGroup path, this would be called per image and flood log with mapped pool pool-2 from ID 6 to remote ID 5 since groups are scoped at pool and all images will be in same pool unless we plan to support different pool?
I0806 06:59:21.362965 1 replication.go:1331] ID: 77 mapped pool pool-2 from ID 6 to remote ID 5
I0806 06:59:21.362978 1 replication.go:1349] ID: 77 mapped source ID 0001-0024-42ceda1d-b18d-4d63-95ad-708fc0c61744-0000000000000006-7322879e-8909-4cd5-851a-55c4e51c36c1 to destination ID 0001-0024-447af91f-794b-400d-a972-b6c5275c3113-0000000000000005-7322879e-8909-4cd5-851a-55c4e51c36c1
I0806 06:59:21.362984 1 replication.go:1331] ID: 77 mapped pool pool-2 from ID 6 to remote ID 5
I0806 06:59:21.362989 1 replication.go:1349] ID: 77 mapped source ID 0001-0024-42ceda1d-b18d-4d63-95ad-708fc0c61744-0000000000000006-cf7bd188-293d-4d63-8efb-189b0b43aa58 to destination ID 0001-0024-447af91f-794b-400d-a972-b6c5275c3113-0000000000000005-cf7bd188-293d-4d63-8efb-189b0b43aa58
Collaborator
There was a problem hiding this comment.
yes it should be supported no limitation from rbd. i agree with concern but its required for debugging.
Contributor
There was a problem hiding this comment.
🤔 @rewantsoni any ideas to minimize logging for groups to reduce spam ?
Contributor
Author
There was a problem hiding this comment.
@Rakshith-R Since this information is crucial and required for debugging, I would say that we can live with this. When we have groups from different pools enabled this information would be more useful.
Contributor
There was a problem hiding this comment.
@Rakshith-R Since this information is crucial and required for debugging, I would say that we can live with this. When we have groups from different pools enabled this information would be more useful.
I don't think that's anywhere on the roadmap ?
but okay
Contributor
@rewantsoni Can you check this and rule out if not correct? |
rewantsoni
force-pushed
the
implement-replication
branch
from
August 6, 2026 08:56
5a2473f to
9830703
Compare
Add GetReplicationDestinationInfo RPC implementation for both volume and volume group replication. This RPC maps source volume/group IDs to destination IDs by using the replicationDestination configuration from the ConfigMap. Signed-off-by: Rewant Soni <resoni@redhat.com>
Add GET_REPLICATION_DESTINATION_INFO to the list of advertised capabilities in GetCapabilities RPC. The capability is advertised on controller servers and applies to both volume and volume group replication, as both are handled by the same Replication Controller service. Signed-off-by: Rewant Soni <resoni@redhat.com>
Add unit tests for GetReplicationDestinationInfo RPC Signed-off-by: Rewant Soni <resoni@redhat.com>
Document the new GetReplicationDestinationInfo RPC feature in the pending release notes for v3.18. This RPC enables DR orchestrators to map source volume/volume group IDs to destination IDs across mirrored clusters when pools have different IDs, supporting both single volumes and volume groups with pool-based mapping configuration. Signed-off-by: Rewant Soni <resoni@redhat.com>
rewantsoni
force-pushed
the
implement-replication
branch
from
August 6, 2026 09:03
9830703 to
81eaa8f
Compare
Madhu-1
approved these changes
Aug 7, 2026
Rakshith-R
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe what this PR does
Implement the change for the GetReplicationDestinationInfo RPC as per the design proposal in #6316
Is there anything that requires special attention
Provide any external context for the change, if any.
Checklist:
Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:
/retest ci/centos/<job-name>: retest the<job-name>after unrelatedfailure (please report the failure too!)