Cannot move mount from /tmp/ocitmp.XXXX to .../merged/run

[aalba6675]

On Fedora 28 /tmp is mounted as shared.

When doing the move mount from /tmp/ocitmp.XXXX to the container overlay it fails with EINVAL.

Steps:

1. Create a systemd-based container with bind mount. (The issue does not happen if the container does not have bind mounts)

podman create --name test_1 --entrypoint /sbin/init -v /volumes/test/home:/home:z --env container=podman fedora:28
podman start test_1
oci-systemd-hook[5870]: systemdhook <error>: 4962ee46e281: Failed to move mount /tmp/ocitmp.jIxv5p to /var/lib/containers/storage/overlay/5348f52873a3f5340e3461d5fb15cbf56acd48a73989673dfd0d1a9107e462b4/merged/run: Invalid argument

2. Setting /tmp to private, but makes this work twice but leads to other problems with containers+bind mounts namely cgroup debris: podman with bind mount leaving cgroup debris and prevents container restart podman-container-tools/podman#730

[aalba6675]

my_mount.c:

## extracted minimal reproducer
#include <stdio.h>
#include <error.h>
#include <sys/mount.h>

int main(int argc, char **argv) {
                if ((mount(argv[1], argv[2], "", MS_MOVE, "") == -1)) {
                        fprintf(stderr, "Failed to move mount %s to %s", argv[1], argv[2]);
                        perror("Error");
                        return -1;
                }
}

./my_mount  /tmp/ocitmp.D6swZ5 /var/lib/containers/storage/overlay/6ab3df626ea9635786d8d5615d41f4c9124c349e42e37f558ecd7235d7bf8c3b/merged/run
Failed to move mount /tmp/ocitmp.D6swZ5 to /var/lib/containers/storage/overlay/6ab3df626ea9635786d8d5615d41f4c9124c349e42e37f558ecd7235d7bf8c3b/merged/runError: Invalid argument

mount | grep oci
tmpfs on /tmp/ocitmp.D6swZ5 type tmpfs (rw,nosuid,nodev,relatime,context="system_u:object_r:container_file_t:s0:c500,c974",size=65536k,mode=755)
tmpfs on /tmp/ocitmp.D6swZ5/.containerenv type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /tmp/ocitmp.D6swZ5/secrets type tmpfs (rw,nosuid,nodev,seclabel,mode=755)

[rhatdan]

@rhvgoyal PTAL

@aalba6675 Do you think this is the comination of the mounts oci-systemd-hook is creating is conflicting with the bind mounted volume? So the kernel does not like it when we do a MS_MOVE?

Did you attempt to change the mount propagation of the internel mounts in oci-systemd-hook?

[rhatdan]

@mrunalp WDYT?

[aalba6675]

@rhatdan I created a submount of /tmp at /tmp/oci and changed the template to /tmp/oci/otctmp.XXXX. For
/tmp/oci I changed the propagation to private. Then it works. The man page states that

Note: moving a mount that resides under a shared mount is invalid.

so the original code would not work if /tmp is shared which is the default on Fedora 28 and RHEL 7.5.

Now I can create a container+bind mount but I will then hit my other issue (cgroup leakage) podman-container-tools/podman#730

[rhatdan]

It will be difficult to enforce this within podman. I guess we could change the default of mounted volumes to be private but that might break users assumptions.

[rhatdan]

@mrunalp I wonder if we could run through the bind mounts and force them to be all mountprivate, and then update the docs to state that if you run with systemd as your pid1, mountpropagation is not supported.

[rhvgoyal]

I vaguely remember this restriction on MS_MOVE as well as pivot_root() that parent mount can not be shared. So this will have to be worked around. That's why we make parent mount of container root private so that pivot_root() can succeed. (in runc)

[houstar]

@rhatdan

1. Appreciated your LWN.net documentation about Systemd vs. Docker

2. From this README.md,

   When oci-systemd-hook detects systemd inside of the container it does the following:

...

• If there is content in the container image's /run and /tmp that content will be copied onto the tmpfs.
  ....

Confusion:
From the code, If there're volume or tmpfs mounted on /tmp, then the original content will be copied onto the /tmp.

But, If there is no volume mounts on /tmp, and there is content in the container image's /run and /tmp,
after prestart functionality, tmpfs mounted on the /tmp and the previously content not existed any more.

[rhatdan]

If you have an image with /run/httpd/httpd.pid in it, and run with oci-systemd-hook, and do not have any volumes mounted on /run or /run/httpd, oci-systemd-hook should mount a tmpfs on /run with the httpd/http.pid file in it.
If this is not happening then it is a bug, perhaps an older version of oci-systemd-hook?

[houstar]

I'm using the latest oci-systemd-hook and have an image with /tmp/tmp.xxxx in it. and run with oci-systemd-hook, and do not have any volumes mounted on /tmp, oci-systemd-hook mount a tmpfs on /tmp without /tmp/tmp.xxx in it.

If this is going to be a bug, could you have a look ?