[BUG] Operation not permitted building a base image

[treydock]

Describe the bug
"Operation not permitted" when attempting basic base image:

To Reproduce

$ podman run --rm --device /dev/fuse --network host -v $(pwd)/cluster-images/rhel-9-base.yaml:/home/builder/config.yaml -v ~/.config/containers/auth.json:/home/builder/auth.json ghcr.io/openchami/image-build:latest image-build --config config.yaml --log-level DEBUG
INFO - --------------------ARGUEMENTS--------------------
INFO - log_level : DEBUG
INFO - config : config.yaml
INFO - layer_type : base
INFO - pkg_man : dnf
INFO - parent : scratch
INFO - proxy :
INFO - name : rhel-base
INFO - publish_local : False
INFO - publish_s3 : None
INFO - publish_registry : registry.OMIT/cluster-images
INFO - registry_opts_push : ['--authfile=/home/builder/auth.json']
INFO - registry_opts_pull : ['--authfile=/home/builder/auth.json']
INFO - publish_tags : 9.4
INFO - Container: rhel-base20250723173958 mounted at /home/builder/.local/share/containers/storage/overlay/dab45a94bd20ecffe9cd270a6a7ad9d7200dfd0abf2062692c2bf296503b0896/merged
ERROR - Error preparing installer: [Errno 1] Operation not permitted: '/home/builder/.local/share/containers/storage/overlay/dab45a94bd20ecffe9cd270a6a7ad9d7200dfd0abf2062692c2bf296503b0896/merged/tmp'
INFO - f423ca7fd076258d598392fc75748ea1af731d403d62d9f7111175d4bcb82ce3

-------------------BUILD LAYER--------------------
Exiting now ...

Expected behavior
I did the PEARC25 tutorial and saw this work and trying to build images on local system using local $HOME (not NFS) and that's failing.

Additional context

$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.12
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.module+el8.10.0+22931+799fd806.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 9ece2912d3d8b855ab314954a702ea65c5c9db47'
  cpuUtilization:
    idlePercent: 99.31
    systemPercent: 0.17
    userPercent: 0.52
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: rhel
    version: "8.10"
  eventLogger: file
  freeLocks: 2047
  hostname: build-el8.OMIT
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 5509
      size: 1
    - container_id: 1
      host_id: 31671316
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 20821
      size: 1
    - container_id: 1
      host_id: 31671316
      size: 65536
  kernel: 4.18.0-553.60.1.el8_10.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1049473024
  memTotal: 8038006784
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.9.4-20.module+el8.10.0+22931+799fd806.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.4.0-dev
        commit: unknown
        CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
    package: containernetworking-plugins-1.4.0-5.module+el8.10.0+22931+799fd806.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.12-6.module+el8.10.0+22931+799fd806.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.2.0+dev
      go: go1.22.11 (Red Hat 1.22.11-1.module+el8.10.0+22728+ac755c3c)
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/20821/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.module+el8.10.0+22931+799fd806.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4221325312
  swapTotal: 4294963200
  uptime: 315h 20m 19.00s (Approximately 13.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/tdockendorf/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tdockendorf/.local/share/containers/storage
  graphRootAllocated: 158203019264
  graphRootUsed: 112857407488
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 179
  runRoot: /run/user/20821/containers
  transientStore: false
  volumePath: /home/tdockendorf/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1742234826
  BuiltTime: Mon Mar 17 14:07:06 2025
  GitCommit: ""
  GoVersion: go1.22.11 (Red Hat 1.22.11-1.module+el8.10.0+22728+ac755c3c)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

[synackd]

Thanks for trying it out! Can you try with v0.1.0 instead of latest and see if this issue still occurs?

[treydock]

Same error:

$ podman run --rm --device /dev/fuse --network host -v $(pwd)/cluster-images/rhel-9-base.yaml:/home/builder/config.yaml -v ~/.config/containers/auth.json:/home/builder/auth.json ghcr.io/openchami/image-build:v0.1.0 image-build --config config.yaml --log-level DEBUG
Trying to pull ghcr.io/openchami/image-build:v0.1.0...
Getting image source signatures
Copying blob 4927f573082f done   |
Copying blob 960dd2b25e6a done   |
Copying blob 338a80bc21c9 skipped: already exists
Copying blob a40fb56914e1 done   |
Copying blob 32c656e63e6a done   |
Copying blob 19015adf595f done   |
Copying blob 9fa0ce9bbe62 done   |
Copying blob facdc8382cbd done   |
Copying blob deffbae70b7a done   |
Copying blob 4f4fb700ef54 skipped: already exists
Copying config c2fd2b580e done   |
Writing manifest to image destination
INFO - --------------------ARGUEMENTS--------------------
INFO - log_level : DEBUG
INFO - config : config.yaml
INFO - layer_type : base
INFO - pkg_man : dnf
INFO - parent : scratch
INFO - proxy :
INFO - name : rhel-base
INFO - publish_local : False
INFO - publish_s3 : None
INFO - publish_registry : docker-registry.OMIT/osc/cluster-images
INFO - registry_opts_push : ['--authfile=/home/builder/auth.json']
INFO - registry_opts_pull : ['--authfile=/home/builder/auth.json']
INFO - publish_tags : 9.4
INFO - Container: rhel-base20250723174812 mounted at /home/builder/.local/share/containers/storage/overlay/1c2d3b879f35aebce936341f36b4da4f0514aea0d3965519f6a76e6569f1d967/merged
ERROR - Error preparing installer: [Errno 1] Operation not permitted: '/home/builder/.local/share/containers/storage/overlay/1c2d3b879f35aebce936341f36b4da4f0514aea0d3965519f6a76e6569f1d967/merged/tmp'
INFO - 065a64c3448b4af798ba1c4060ec684324f1ec457a03c32e218da0badd5e0749

-------------------BUILD LAYER--------------------
Exiting now ...

When I try to use my own RHEL 9.4 base image instead of scratch, get a little further but still fails:

$ podman run --rm --device /dev/fuse --network host -v $(pwd)/cluster-images/rhel-9-base.yaml:/home/builder/config.yaml -v ~/.config/containers/auth.json:/home/builder/auth.json ghcr.io/openchami/image-build:v0.1.0 image-build --config config.yaml --log-level DEBUG
INFO - --------------------ARGUEMENTS--------------------
INFO - log_level : DEBUG
INFO - config : config.yaml
INFO - layer_type : base
INFO - pkg_man : dnf
INFO - parent : docker-registry.OMIT/osc/rhel-9:9.4
INFO - proxy :
INFO - name : rhel-base
INFO - publish_local : False
INFO - publish_s3 : None
INFO - publish_registry : docker-registry.osc.edu/osc/cluster-images
INFO - registry_opts_push : ['--authfile=/home/builder/auth.json']
INFO - registry_opts_pull : ['--authfile=/home/builder/auth.json']
INFO - publish_tags : 9.4
ERROR - Trying to pull docker-registry.OMIT/osc/rhel-9:9.4...
ERROR - Getting image source signatures
ERROR - Copying blob sha256:d06eb77aff7a4e20844ea412d0e803dc0c9fe629865b06cb0f8697f93644d630
ERROR - Copying blob sha256:a20d5f0bec2bce34d8adb0eb98f3536f214c292b20c6cf1a3a46c802662d912b
ERROR - Copying config sha256:a56a60cee817b230693f860751d8fc325eb8cb427f80f9437559a742a9798655
ERROR - Writing manifest to image destination
INFO - Container: rhel-base20250723174925 mounted at /home/builder/.local/share/containers/storage/overlay/f5ac159dca211d24c099cfd245c9e75f3803b0f26d4de746e12094921ac74d14/merged
INFO - Installer: Temporary directory for dnf created at /var/tmp/image-build-x611hj9l
INFO - REPOS: no repos passed to install

WARNING - PACKAGE MODULES: no modules passed to install

INFO - PACKAGE GROUPS: Installing these package groups to rhel-base20250723174925
INFO - Minimal Install
Development Tools
ERROR - warning: Found SQLITE rpmdb.sqlite database while attempting bdb backend: using sqlite backend.
ERROR - Config error: [Errno 1] Operation not permitted: '/home/builder/.local/share/containers/storage/overlay/f5ac159dca211d24c099cfd245c9e75f3803b0f26d4de746e12094921ac74d14/merged/var/tmp/image-build-x611hj9l': '/home/builder/.local/share/containers/storage/overlay/f5ac159dca211d24c099cfd245c9e75f3803b0f26d4de746e12094921ac74d14/merged/var/tmp/image-build-x611hj9l'
ERROR - Error installing packages: Command '['dnf', '--setopt=reposdir=/home/builder/.local/share/containers/storage/overlay/f5ac159dca211d24c099cfd245c9e75f3803b0f26d4de746e12094921ac74d14/merged/etc/yum.repos.d', '--setopt=logdir=/var/tmp/image-build-x611hj9l/dnf/log', '--setopt=cachedir=/var/tmp/image-build-x611hj9l/dnf/cache', 'groupinstall', '-y', '--nogpgcheck', '--installroot', '/home/builder/.local/share/containers/storage/overlay/f5ac159dca211d24c099cfd245c9e75f3803b0f26d4de746e12094921ac74d14/merged', 'Minimal Install', 'Development Tools']' returned non-zero exit status 1.
INFO - c07bfd1a8d42ebf534a5d5a2413ec44058d92d90cec0310c3120486cc7e9d601

-------------------BUILD LAYER--------------------
Exiting now ...

This is the buildah Dockerfile for this modified base image. The last two ENV items were trying various solutions from Google to attempt to avoid the errors.

$ cat cluster-images/Dockerfile
ARG BASE_IMAGE
FROM $BASE_IMAGE
ARG OS=9.4
ARG OS_MAJOR=9
ENV OS=$OS
ENV OS_MAJOR=$OS_MAJOR

COPY ./rhel$OS_MAJOR/*.repo /etc/yum.repos.d/

ENV _BUILDAH_STARTED_IN_USERNS="" BUILDAH_ISOLATION=chroot

[treydock]

Also the non-scratch image was built this way:

buildah bud --security-opt seccomp=unconfined --layers --ulimit nofile=32000:32000 --format docker \
        --build-arg BASE_IMAGE=docker.io/redhat/ubi9:9.4 --build-arg OS=9.4 --build-arg OS_MAJOR=9 \
        -t docker-registry.OMIT/osc/rhel-9:9.4 -t docker-registry.OMIT/osc/rhel-9:20250723

The --security-opt was also non-standard attempt to avoid these errors.

[davidallendj]

Do you happen to have SELinux enabled?

[treydock]

SELinux is disabled on this host:

$ sudo getenforce
Disabled

These errors were all on our RHEL8 build host. I tried our RHEL9 build host and things are failing on dnf issues that not sure what to make of yet. Related to #32 I'm attempting to make RHEL images.

$ podman run --rm -it docker-registry.OMIT/osc/rhel-9:9.4 /bin/bash
[root@ccd7cf1b7775 /]# dnf makecache

That works as that image has redhat.repo plus the entitlement certs. However when I attempt with image-builder, I get this:

INFO - PACKAGE GROUPS: Installing these package groups to rhel-base20250723190107
INFO - Minimal Install
Development Tools
ERROR - warning: Found SQLITE rpmdb.sqlite database while attempting bdb backend: using sqlite backend.
ERROR - warning: Found SQLITE rpmdb.sqlite database while attempting bdb backend: using sqlite backend.
INFO - RPMS for GPFS                                   3.4 MB/s | 137 kB     00:00
INFO - MLNX_OFED Repository                            3.0 MB/s | 161 kB     00:00
INFO - RPMS for NVIDIA                                 770 kB/s |  42 kB     00:00
INFO - RPMS for SLURM                                  1.1 MB/s |  50 kB     00:00
INFO - RPMS for OSC systems                            7.3 MB/s | 371 kB     00:00
INFO - libnvidia-container                             270 kB/s |  39 kB     00:00
INFO - Red Hat CodeReady Linux Builder for RHEL 9 x86_ 0.0  B/s |   0  B     00:00
ERROR - Errors during downloading metadata for repository 'codeready-builder-for-rhel-9-x86_64-eus-rpms':
ERROR -   - Curl error (58): Problem with the local SSL certificate for https://satellite.OMIT/pulp/content/Ohio_Supercomputer_Center/production/OSC_HPC/content/eus/rhel9/9.4/x86_64/codeready-builder/os/repodata/repomd.xml [could not load PEM client certificate, OpenSSL error error:02001002:system library:fopen:No such file or directory, (no key found, wrong pass phrase, or wrong file format?)]
ERROR - Error: Failed to download metadata for repo 'codeready-builder-for-rhel-9-x86_64-eus-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
ERROR - Error installing packages: Command '['dnf', '--setopt=reposdir=/home/builder/.local/share/containers/storage/overlay/be42af887b2c5631423fd7a593442747b7853d8add4ceb4bd072c8276172aff8/merged/etc/yum.repos.d', '--setopt=logdir=/var/tmp/image-build-trfqtr18/dnf/log', '--setopt=cachedir=/var/tmp/image-build-trfqtr18/dnf/cache', 'groupinstall', '-y', '--nogpgcheck', '--installroot', '/home/builder/.local/share/containers/storage/overlay/be42af887b2c5631423fd7a593442747b7853d8add4ceb4bd072c8276172aff8/merged', 'Minimal Install', 'Development Tools']' returned non-zero exit status 1.
INFO - 45bf89012e87e71fb7abff0c55e45ab746e08105dab0d61b8ac7114d9a08cd3b

I am trying to rely on the repos that I bake into my parent image, since I can't define RedHat Satellite repos in image-builder due to not enough arguments supported for dnf repos.

[treydock]

If I had to guess, the paths don't work with how image-builder is using an overlay to change root path.

[codeready-builder-for-rhel-9-x86_64-eus-rpms]
name = Red Hat CodeReady Linux Builder for RHEL 9 x86_64 - Extended Update Support (RPMs)
baseurl = https://satellite.<OMIT>/pulp/content/Ohio_Supercomputer_Center/production/OSC_HPC/content/eus/rhel9/9.4/x86_64/codeready-builder/os
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify = 1
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
sslclientkey = /etc/pki/entitlement/479350244350876317-key.pem
sslclientcert = /etc/pki/entitlement/479350244350876317.pem
metadata_expire = 1
enabled_metadata = 1

[davidallendj]

@treydock Do you mean your PWD is your home directory? Do you have an minimal example image definition file that also produces the error? I tried running this command in the image-builder repo without mounting the JSON file and I'm not able to reproduce just yet.

podman run --rm --device /dev/fuse --network host -v ./examples/ansible/base.yaml:/home/builder/config.yaml -v  ghcr.io/openchami/image-build:latest image-build --config config.yaml --log-level DEBUG

[treydock]

I'm on RHEL 8.10 host.

[tdockendorf@build-el8 image-builder]$ podman run --rm --device /dev/fuse --network host -v ./examples/ansible/base.yaml:/home/builder/config.yaml ghcr.io/openchami/image-build:latest image-build --config config.yaml --log-level DEBUG
INFO - --------------------ARGUEMENTS--------------------
INFO - log_level : DEBUG
INFO - config : config.yaml
INFO - layer_type : base
INFO - pkg_man : dnf
INFO - parent : scratch
INFO - proxy :
INFO - name : alma-base
INFO - publish_local : True
INFO - publish_s3 : None
INFO - publish_registry : None
INFO - registry_opts_pull : []
INFO - publish_tags : v0.0.1
INFO - Container: alma-base20250723201122 mounted at /home/builder/.local/share/containers/storage/overlay/0ff581475c83aa8e713628b498fc40b9713108d0fc127eaccee86f1e764570dd/merged
ERROR - Error preparing installer: [Errno 1] Operation not permitted: '/home/builder/.local/share/containers/storage/overlay/0ff581475c83aa8e713628b498fc40b9713108d0fc127eaccee86f1e764570dd/merged/tmp'
INFO - a96e1f690d85829060cbd0a9919298ab47931d3e90369f191152e2f4e6b0078d

-------------------BUILD LAYER--------------------
Exiting now ...

In my original examples PWD is just the repo that holds the file I'm using. It's in my ext4 local $HOME.

[tdockendorf@build-el8 container-images]$ pwd
/home/tdockendorf/container-images

I don't get these errors on RHEL 9.x, the operation not permitted only seems to happen with RHEL 8 podman.

[davidallendj]

Ah, let me try running this in a VM and see what I get.

Edit: Let me see if there's somewhere I can test this.

[davidallendj]

I tried something similar on a Rocky 8 install instead of RHEL and can confirm a similar error with both the latest and v0.1.0 tags of the image builder.

[exouser@31-image-builder images]$ podman run --rm --device /dev/fuse --network host -v /opt/workdir/images/rocky-base-9.yaml:/home/builder/config.yaml ghcr.io/openchami/image-build:v0.1.0 image-build --config config.yaml --log-level DEBUG
INFO - --------------------ARGUEMENTS--------------------
INFO - log_level : DEBUG
INFO - config : config.yaml
INFO - layer_type : base
INFO - pkg_man : dnf
INFO - parent : scratch
INFO - proxy : 
INFO - name : rocky-base
INFO - publish_local : False
INFO - publish_s3 : None
INFO - publish_registry : demo.openchami.cluster:5000/demo
INFO - registry_opts_push : ['--tls-verify=false']
INFO - registry_opts_pull : []
INFO - publish_tags : 9
INFO - Container: rocky-base20250724150407 mounted at /home/builder/.local/share/containers/storage/overlay/f5ce50de02f2db0b02c0f7e345e0bf32b4bf6c3e57ab00fdf2b81a1d21b12191/merged
ERROR - Error preparing installer: [Errno 1] Operation not permitted: '/home/builder/.local/share/containers/storage/overlay/f5ce50de02f2db0b02c0f7e345e0bf32b4bf6c3e57ab00fdf2b81a1d21b12191/merged/tmp'
INFO - 367fae2d574dac53ffa1f9b503209e2ef541372b2c783646c6dc26dd5c9a9954

-------------------BUILD LAYER--------------------
Exiting now ...

It's probably worth noting that I chown'd the directory that I'm working in and /tmp before running the above command just to see if it would change anything. It looks like the error message is being printed here in the code.

[travisbcotton]

What's in /etc/sub{uid,gid} on the baremetal host?

[davidallendj]

What's in /etc/sub{uid,gid} on the baremetal host?

[exouser@31-image-builder images]$ cat /etc/subuid
rocky:100000:65536
exouser:165536:65536
[exouser@31-image-builder images]$ cat /etc/subgid
rocky:100000:65536
exouser:165536:65536

[travisbcotton]

What does /opt/workdir/images/rocky-base-9.yaml looks like?