Add Node-Deployment Mode for High-Performance LocalPV Provisioning #306

laik · 2025-12-10T13:43:48Z

This PR introduces a node-deployment mode for the OpenEBS Dynamic LocalPV
Provisioner, addressing performance bottlenecks identified in openebs/openebs#4050.

Key Changes:

New --node-deployment=true flag to run provisioner as DaemonSet on each node
Direct local volume operations (mkdir, quota, rm) without helper pods
Automatic node filtering using IgnoredError for proper multi-instance coordination
Leader election disabled in node-deployment mode

Benefits:

Eliminates helper pod creation/termination overhead
Reduces PVC binding latency significantly
Decreases API server load
Simpler architecture than external service approach

Solve xfs/ext projId Issues
Solve race conditions Issues

#289
#290

tiagolobocastro

Awesome, just a few questions!

Solve xfs/ext projId Issues

Could you fix this for the helperPod variant as well? And if we could share some of that code that would be great.

Solve race conditions Issues

Is this only solved in new daemonset mode?

tiagolobocastro · 2025-12-11T00:13:06Z

cmd/provisioner-localpv/app/local_volume_manager.go

+        base="%s"
+        # check fs type first
+        fs=$(stat -f -c %%T $base 2>/dev/null)
+        if [ "$fs" = "xfs" ]; then


Is nothing needed for ext4?

Yes, I saw there will be some leftovers, I'll fix it

*** Report for project quotas on device /dev/loop0 Block grace time: 7days; Inode grace time: 7days Block limits File limits Project used soft hard grace used soft hard grace ---------------------------------------------------------------------- #0 -- 20 0 0 3 0 0 #1 -- 4 112640 122880 1 0 0 #2 -- 4 112640 122880 1 0 0 #3 -- 4 112640 122880 1 0 0 #4 -- 4 112640 122880 1 0 0 ➜ pv-test git:(main) ✗ km pod --all && km pvc --all ... ➜ pv-test git:(main) ✗ repquota -P /root/openebs/local *** Report for project quotas on device /dev/loop0 Block grace time: 7days; Inode grace time: 7days Block limits File limits Project used soft hard grace used soft hard grace ---------------------------------------------------------------------- #0 -- 20 0 0 3 0 0 ➜ pv-test git:(main) ✗ repquota -avugP *** Report for project quotas on device /dev/loop0 Block grace time: 7days; Inode grace time: 7days Block limits File limits Project used soft hard grace used soft hard grace ---------------------------------------------------------------------- #0 -- 20 0 0 3 0 0 #1 -- 0 112640 122880 0 0 0 #2 -- 0 112640 122880 0 0 0 #3 -- 0 112640 122880 0 0 0 #4 -- 0 112640 122880 0 0 0

Related issues #155

tiagolobocastro · 2025-12-11T00:14:34Z

cmd/provisioner-localpv/app/local_volume_manager.go

+	script := fmt.Sprintf(`
+		set -e
+		
+		# Find the actual mount point for the path using host's mount info


Do we need this fix on the helper pod variant?

yeah, It will be fixed in the previous commit. I forgot to synchronize it when the commit was rolled back. I will fix it.

tiagolobocastro · 2025-12-11T00:16:04Z

cmd/provisioner-localpv/app/local_volume_manager.go

+func (vm *LocalVolumeManager) applyQuotaByFilesystem(ctx context.Context, parentDir, volumeDir, softLimitGrace, hardLimitGrace string) error {
+	// Create a shell script to detect filesystem and apply quota
+	// We need to find the actual XFS mount point since parentDir might be a bind mount
+	script := fmt.Sprintf(`


Is there some way to share this with the helper pod variant?

This is not suitable for sharing

tiagolobocastro · 2025-12-11T00:19:19Z

@laik please take a look at the test failure
Also probably should add a parallel test run with the daemonset mode?

Signed-off-by: laik <laik.lj@me.com>

laik · 2025-12-11T14:47:54Z

@laik please take a look at the test failure Also probably should add a parallel test run with the daemonset mode?

ok , let me check

laik · 2025-12-11T14:51:06Z

Awesome, just a few questions!

Solve xfs/ext projId Issues

Could you fix this for the helperPod variant as well? And if we could share some of that code that would be great.

Solve race conditions Issues

Is this only solved in new daemonset mode?

yeah, Helper pod belongs to the asynchronous mode at the pod level. If necessary, you may need to create a file to lock it.

tiagolobocastro · 2025-12-11T22:27:12Z

yeah, Helper pod belongs to the asynchronous mode at the pod level. If necessary, you may need to create a file to lock it.

You mean using flock right? We used that on rawfile, so probably could also use it here.

codecov-commenter · 2025-12-11T22:39:10Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 39.18%. Comparing base (cab53c4) to head (7d38c91).
⚠️ Report is 61 commits behind head on develop.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #306      +/-   ##
===========================================
+ Coverage    37.91%   39.18%   +1.26%     
===========================================
  Files           36        1      -35     
  Lines         3373      684    -2689     
===========================================
- Hits          1279      268    -1011     
+ Misses        2012      407    -1605     
+ Partials        82        9      -73

Flag	Coverage Δ
integrationtests	`39.18% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

laik · 2025-12-12T09:30:36Z

yeah, Helper pod belongs to the asynchronous mode at the pod level. If necessary, you may need to create a file to lock it.

You mean using flock right? We used that on rawfile, so probably could also use it here.

Yes, I checked the documentation about flock and it should be possible.

tiagolobocastro

@niladrih do we update the changelog here or would you update it on release?

tiagolobocastro · 2025-12-16T16:32:42Z

cmd/provisioner-localpv/app/helper_hostpath.go

 	config.taints = pOpts.selectedNodeTaints

-	config.pOpts.cmdsForPath = append(config.pOpts.cmdsForPath, filepath.Join("/data/", config.volumeDir))
+	// check if path is xfs quota enabled and remove quota projid


this is the fix for the non-distributed cleanup right? Could you place this on a separate commit?

tiagolobocastro · 2025-12-16T16:38:23Z

cmd/provisioner-localpv/app/local_volume_manager.go

+				setquota -P $ID 0 0 0 0 $base 2>/dev/null || true
+			fi
+		fi
+		rm -rf $d


Is there any safety check against this not being the base path for example?

tiagolobocastro · 2025-12-16T16:46:51Z

buildscripts/provisioner-localpv/Dockerfile

+    e2fsprogs \
+    xfsprogs \
+    xfsprogs-extra \


are all these required?

laik requested a review from a team as a code owner December 10, 2025 13:43

laik force-pushed the develop branch from 71d16ba to 0adeed2 Compare December 10, 2025 13:55

tiagolobocastro reviewed Dec 11, 2025

View reviewed changes

tiagolobocastro requested review from Abhinandan-Purkait and niladrih December 11, 2025 00:19

add distributed-provisioning

7d38c91

Signed-off-by: laik <laik.lj@me.com>

laik force-pushed the develop branch from cf92af8 to 7d38c91 Compare December 11, 2025 14:46

tiagolobocastro reviewed Dec 16, 2025

View reviewed changes

Add Node-Deployment Mode for High-Performance LocalPV Provisioning #306

Are you sure you want to change the base?

Add Node-Deployment Mode for High-Performance LocalPV Provisioning #306

Conversation

laik commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tiagolobocastro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laik Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tiagolobocastro commented Dec 11, 2025

Uh oh!

laik commented Dec 11, 2025

Uh oh!

laik commented Dec 11, 2025

Uh oh!

tiagolobocastro commented Dec 11, 2025

Uh oh!

codecov-commenter commented Dec 11, 2025

Codecov Report

Uh oh!

laik commented Dec 12, 2025

Uh oh!

tiagolobocastro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

laik commented Dec 10, 2025 •

edited

Loading

laik Dec 11, 2025 •

edited

Loading