Skip to content

CSI FUSE driver cannot mount DF 8.0 volume to pod on Ubuntu 24.04 #61

@nightzone

Description

@nightzone

Ubuntu 24.04.4
Tried with different k3s and RKE versions:
K3s: v1.32.12+k3s1, v1.34.4+k3s1
RKE: v1.32.12+rke2r1
MapR CSI deployed from csi-maprkdf-v1.2.15.yaml
Create SC as per documentation.

ISSUE:
PV and PVC are crated as expected, but volume not mounted to pod.

Connectivity checked and is operational.
Same config on k8s v1.31.1-hpe1 on RHEL 8 is operational.

hpadmin@rks02:~$ kubectl get pv
NAME                                           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS    VOLUMEATTRIBUTESCLASS   REASON   AGE
mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3   1Gi        RWO            Delete           Bound    default/my-app-pvc   df-labs-hp-ua   <unset>                          14m
hpadmin@rks02:~$ kubectl get pvc
NAME         STATUS   VOLUME                                         CAPACITY   ACCESS MODES   STORAGECLASS    VOLUMEATTRIBUTESCLASS   AGE
my-app-pvc   Bound    mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3   1Gi        RWO            df-labs-hp-ua   <unset>                 15m

From description of my POD

  Normal   SuccessfulAttachVolume  15m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3"
  Warning  FailedMount             40s (x14 over 15m)  kubelet                  MountVolume.SetUp failed for volume "mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3" : rpc error: code = Internal desc = Fuse mount is not available: /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3. Check CLDB hosts are valid & reachable

mapr-csi driver deployes and operational

hpadmin@rks02:~$ kubectl get pod -n mapr-csi
NAME                       READY   STATUS    RESTARTS      AGE
csi-controller-kdf-0       7/7     Running   4 (21m ago)   23m
csi-nodeplugin-kdf-7x6wh   3/3     Running   1 (22m ago)   23m

hpadmin@rks02:~$ kubectl get sc
NAME                      PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
df-labs-hp-ua (default)   com.mapr.csi-kdf   Delete          Immediate           true                   22m

vi /var/log/csi-maprkdf/csi-plugin.log
....

2026/03/06 19:05:55 start-fuse: INFO Waiting 5 for FUSE to initialize...
2026/03/06 19:06:00 start-fuse: INFO Writing POSIX client pid: 1147 to support directory: /opt/mapr/k8s/support/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3/fusepid
2026/03/06 19:06:00 start-fuse: INFO Checking POSIX client process
2026/03/06 19:06:00 start-fuse: INFO POSIX client process: 1147 is healthy. Check ffs-log.* for more info on running state.
2026/03/06 19:06:00 nodeserver.go:529: INFO [startFuse] Successfully started fuse
2026/03/06 19:06:00 nodeserver.go:141: DEBUG Checking availability of fuse mount : /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3
2026/03/06 19:06:05 utils.go:710: ERROR Fuse mount /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3 is not accessible. Please check if Data Fabric is up & running.
2026/03/06 19:06:05 nodeserver.go:144: ERROR [NodePublishVolume] Fuse mount is not available: /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3. Check CLDB hosts are valid & reachable
2026/03/06 19:06:05 nodeserver.go:543: WARN [removeStaleProcess] Fuse is already running with process id: /opt/mapr/k8s/support/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3, 1147
2026/03/06 19:06:05 nodeserver.go:546: INFO [removeStaleProcess] Mount lost association, check if process is running and remounting attempt
2026/03/06 19:06:05 utils.go:643: DEBUG doUnmount === Stopping fuse mount at: /var/lib/kubelet/pods/ed3223be-cbae-4fd4-a425-6c12f3b7565e/volumes/kubernetes.io~csi/mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3/mount
2026/03/06 19:06:05 utils.go:652: DEBUG doUnmount ===  Failed to unmount mount  /var/lib/kubelet/pods/ed3223be-cbae-4fd4-a425-6c12f3b7565e/volumes/kubernetes.io~csi/mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3/mount : exit status 32
2026/03/06 19:06:05 utils.go:643: DEBUG doUnmount === Stopping fuse mount at: /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3
2026/03/06 19:06:05 utils.go:654: DEBUG doUnmount ===  Unmounted mount /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3
2026/03/06 19:06:05 utils.go:545: DEBUG Killed process pid: 1147
2026/03/06 19:06:05 utils.go:548: DEBUG Found descendant PIDs for 1147: [1148 1177]
2026/03/06 19:06:05 utils.go:555: DEBUG Killed descendant process pid: 1148
2026/03/06 19:06:05 utils.go:555: DEBUG Killed descendant process pid: 1177
2026/03/06 19:06:05 utils.go:495: DEBUG === cleanUp on target /opt/mapr/k8s/support/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3
2026/03/06 19:06:05 utils.go:502: DEBUG === cleanUp on fuse mount /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3
2026/03/06 19:06:05 utils.go:509: DEBUG === Successfully cleaned up /opt/mapr/k8s/support/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3 & /opt/mapr/k8s/mounts/ed3223be-cbae-4fd4-a425-6c12f3b7565e-mapr-pv-04068bde-f19a-495b-b7a2-12b052b2e9d3 ===

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions