feat: allow to run T containers as any user, for better volume permission handling (#253)

#### Relevant issue or PR
n/a

#### Description of changes
This enables users to execute Tesseracts as any system user, even in the
presence of volume mounts.

We achieve this in the following ways:
1. Default to running Tesseracts with the same uid as the current host
user. This fixes most permission issues right away (since everything
that's readable from the outside also becomes readable from the inside).
2. Create mount points ahead of time and ensure they have lax
permissions (rw for everyone). This makes it so named and bind-mounted
volumes are mounted with appropriate permissions, even if they're not
owned by the host user.

**Some sharp edges:**
- By default, podman mounts all volumes as owned by root, not the host
user. This needs to be overwritten via `PODMAN_USERNS=keep-id`. We
automatically add this flag if podman is detected.
- There is no reliable way to enable this option in `podman-compose`.
That is, podman users will have to use `--no-compose` for sane volume
permissions. We raise an exception if podman is used with compose +
volume mounts.
- In cases where the Tesseract user is neither `root` nor the local user
/ file owner, [files in bind-mounts need to be world-readable on the
host](https://github.com/pasteurlabs/tesseract-core/pull/253/files#diff-26bec4f76a9f4e334b738405b56d680ad704871cb6233560e9b8e703cd2becf6R494).
- Currently assumes (but not enforces) that users will use
`/tesseract/input_data` and `/tesseract/output_data` as their mount
points. I expect this to be coupled to the `--input-path` and
`--output-path` CLI args in the future (see e.g. #249).

**Main code changes:**
- Do not create a `tesseractor` user in Docker images. Instead, set
global permissions on files so containers can run as any uid / gid.
- Test [all relevant
scenarios](https://github.com/pasteurlabs/tesseract-core/pull/253/files#diff-26bec4f76a9f4e334b738405b56d680ad704871cb6233560e9b8e703cd2becf6R421).
- Add logic to detect podman and automatically inject required flags for
consistent permission handling.
- Introduce `TESSERACT_DOCKER_RUN_ARGS` config which allows users to
pass custom args to `tesseract run`. This isn't used in the final form
of this PR but I figured it's nice to have anyway.

#### Testing done
CI, + manually with Docker Desktop on OSX (where permissions are always
mapped automatically)

Author: dionhaefner

docs/content/examples/building-blocks/arm64.md

@@ -12,4 +12,4 @@ Via `tesseract_config.yaml`, it is possible to somewhat flexibly alter the build
 :language: yaml
 ```
 
-Using the `custom_build_steps` field, we can run arbitrary commands on the image as if they were in a Dockerfile. We start here by temporarily setting the user to `root`, as the default user in the Tesseract build process is `tesseractor` -- which does not have root privileges -- and then switch back to the `tesseractor` user at the very end. We then run commands directly on the shell via `RUN` commands. All these steps specified in `custom_build_steps` are executed at the very end of the build process, followed only by a last execution of `tesseract-runtime check` that checks that the runtime can be launched and the user-defined `tesseract_api` module can be imported.
+Using the `custom_build_steps` field, we can run arbitrary commands on the image as if they were in a Dockerfile. We run commands directly on the shell via `RUN` commands. All these steps specified in `custom_build_steps` are executed at the very end of the build process, followed only by a last execution of `tesseract-runtime check` that checks that the runtime can be launched and the user-defined `tesseract_api` module can be imported.

docs/content/using-tesseracts/advanced.md

@@ -1,6 +1,6 @@
 # Advanced usage
 
-## File system I/O
+## File aliasing
 
 The `tesseract` command can take care of
 passing data from local disk
@@ -16,6 +16,18 @@ target path:
 $ tesseract run vectoradd apply --output-path /tmp/output @inputs.json
 ```
 
+## Volume mounts and user permissions
+
+When mounting a volume into a Tesseract container, default behavior depends on the Docker engine being used. Specifically, Docker Desktop, Docker Engine, and Podman have different ways of handling user permissions for mounted volumes.
+
+Tesseract tries to ensure that the container user has the same permissions as the host user running the `tesseract` command. This is done by setting the user ID and group ID of the container user to match those of the host user.
+
+In cases where this fails or is not desired, you can explicitly set the user ID and group ID of the container user using the `--user` argument. This allows you to specify a different user or group for the container, which can be useful for ensuring proper permissions when accessing mounted volumes.
+
+```{warning}
+In cases where the Tesseract user is neither `root` nor the local user / file owner, you may encounter permission issues when accessing files in mounted volumes. To resolve this, ensure that the user ID and group ID are set correctly using the `--user` argument, or modify the permissions of files to be readable by any user.
+```
+
 ## Passing environment variables to Tesseract containers
 
 Through the optional `--env` argument, you can pass environment variables to Tesseracts.

examples/pyvista-arm64/tesseract_config.yaml

@@ -13,12 +13,10 @@ build_config:
 
   custom_build_steps:
     - |
-      USER root
       # Python bindings into the Python site-packages directory
       RUN python_site=$(python -c "import site; print(site.getsitepackages()[0])") && \
           ln -s /usr/lib/python3/dist-packages/vtk* $python_site && \
           ls -l $python_site/vtk* && \
           python -c "import vtk"
       # Must install pyvista with --no-deps to avoid installing vtk (which we copied from the system)
       RUN pip install matplotlib numpy pillow pooch scooby && pip install --no-deps pyvista==0.44.1
-      USER tesseractor

tesseract_core/sdk/cli.py

@@ -523,7 +523,8 @@ def serve(
         typer.Option(
             "--user",
             help=(
-                "User to run the Tesseracts as e.g. '1000' or '1000:1000' (uid:gid)."
+                "User to run the Tesseracts as e.g. '1000' or '1000:1000' (uid:gid). "
+                "Defaults to the current user."
             ),
         ),
     ] = None,
@@ -876,7 +877,10 @@ def run_container(
         str | None,
         typer.Option(
             "--user",
-            help=("User to run the Tesseract as e.g. '1000' or '1000:1000' (uid:gid)."),
+            help=(
+                "User to run the Tesseract as e.g. '1000' or '1000:1000' (uid:gid). "
+                "Defaults to the current user."
+            ),
         ),
     ] = None,
 ) -> None:

tesseract_core/sdk/config.py

@@ -50,6 +50,7 @@ class RuntimeConfig(BaseModel):
     docker_build_args: Annotated[
         tuple[str, ...], BeforeValidator(maybe_split_args)
     ] = ()
+    docker_run_args: Annotated[tuple[str, ...], BeforeValidator(maybe_split_args)] = ()
 
     model_config = ConfigDict(frozen=True, extra="forbid")
 

tesseract_core/sdk/docker_client.py

@@ -31,6 +31,21 @@ def _get_executable(program: Literal["docker", "docker-compose"]) -> tuple[str,
     raise ValueError(f"Unknown program: {program}")
 
 
+def is_podman() -> bool:
+    """Check if the current environment is using Podman instead of Docker."""
+    docker = _get_executable("docker")
+    try:
+        result = subprocess.run(
+            [*docker, "version"],
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        return "podman" in result.stdout.lower()
+    except subprocess.CalledProcessError:
+        return False
+
+
 @dataclass
 class Image:
     """Image class to wrap Docker image details."""
@@ -514,6 +529,7 @@ def run(
         stdout: bool = True,
         stderr: bool = False,
         user: str | None = None,
+        extra_args: list_[str] | None = None,
     ) -> Container | tuple[bytes, bytes] | bytes:
         """Run a command in a container from an image.
 
@@ -535,16 +551,17 @@ def run(
                    and the values are the container ports.
             stdout: If True, return stdout.
             stderr: If True, return stderr.
+            environment: Environment variables to set in the container.
+            extra_args: Additional arguments to pass to the `docker run` CLI command.
 
         Returns:
             Container object if detach is True, otherwise returns list of stdout and stderr.
         """
+        config = get_config()
         docker = _get_executable("docker")
 
-        # If command is a type string and not list, make list
         if isinstance(command, str):
             command = [command]
-        logger.debug(f"Running command: {command}")
 
         optional_args = []
 
@@ -586,15 +603,21 @@ def run(
             for host_port, container_port in ports.items():
                 optional_args.extend(["-p", f"{host_port}:{container_port}"])
 
-        # Run with detached to get the container id of the running container.
+        if extra_args is None:
+            extra_args = []
+
         full_cmd = [
             *docker,
             "run",
             *optional_args,
+            *config.docker_run_args,
+            *extra_args,
             image,
             *command,
         ]
 
+        logger.debug(f"Running command: {full_cmd}")
+
         result = subprocess.run(
             full_cmd,
             capture_output=True,

tesseract_core/sdk/engine.py

@@ -39,6 +39,7 @@
     ContainerError,
     Image,
     build_docker_image,
+    is_podman,
 )
 from .exceptions import UserError
 
@@ -570,6 +571,7 @@ def serve(
         no_compose: if True, do not use Docker Compose to serve the Tesseracts.
         service_names: list of service names under which to expose each Tesseract container on the shared network.
         user: user to run the Tesseracts as, e.g. '1000' or '1000:1000' (uid:gid).
+            Defaults to the current user.
 
     Returns:
         A string representing the Tesseract project ID.
@@ -597,6 +599,10 @@ def serve(
             )
         _validate_service_names(service_names)
 
+    if user is None:
+        # Use the current user if not specified
+        user = f"{os.getuid()}:{os.getgid()}" if os.name != "nt" else None
+
     if no_compose:
         if len(images) > 1:
             raise ValueError(
@@ -639,15 +645,22 @@ def serve(
         if debug:
             logger.info(f"Debugpy server listening at http://{ping_ip}:{debugpy_port}")
 
+        parsed_volumes = _parse_volumes(volumes) if volumes else {}
+
+        extra_args = []
+        if is_podman():
+            extra_args.extend(["--userns", "keep-id"])
+
         container = docker_client.containers.run(
             image=image_ids[0],
             command=["serve", *args],
             device_requests=gpus,
             ports=port_mappings,
             detach=True,
-            volumes=volumes,
+            volumes=parsed_volumes,
             user=user,
             environment=environment,
+            extra_args=extra_args,
         )
         # wait for server to start
         timeout = 30
@@ -668,6 +681,12 @@ def serve(
 
         return container.name
 
+    if is_podman() and volumes:
+        raise UserError(
+            "Podman does not support volume mounts in Docker Compose. "
+            "Please use --no-compose / no_compose=True instead."
+        )
+
     template = _create_docker_compose_template(
         image_ids,
         host_ip,
@@ -751,13 +770,15 @@ def _create_docker_compose_template(
         else:
             gpu_settings = f"device_ids: {gpus}"
 
+    parsed_volumes = _parse_volumes(volumes) if volumes else {}
+
     for i, image_id in enumerate(image_ids):
         service = {
             "name": service_names[i],
             "user": user,
             "image": image_id,
             "port": f"{ports[i]}:8000",
-            "volumes": volumes,
+            "volumes": parsed_volumes,
             "gpus": gpu_settings,
             "environment": {
                 "TESSERACT_DEBUG": "1" if debug else "0",
@@ -818,8 +839,11 @@ def _parse_option(option: str):
                 f"Invalid mount volume specification {option} "
                 "(must be `/path/to/source:/path/totarget:(ro|rw)`)",
             )
-        # Docker doesn't like paths like ".", so we convert to absolute path here
-        source = str(Path(source).resolve())
+
+        is_local_mount = "/" in source or Path(source).exists()
+        if is_local_mount:
+            # Docker doesn't like paths like ".", so we convert to absolute path here
+            source = str(Path(source).resolve())
         return source, {"bind": target, "mode": mode}
 
     return dict(_parse_option(opt) for opt in options)
@@ -865,6 +889,7 @@ def run_tesseract(
         environment: list of environment variables to set in the container,
             in Docker format: key=value.
         user: user to run the Tesseract as, e.g. '1000' or '1000:1000' (uid:gid).
+            Defaults to the current user.
 
     Returns:
         Tuple with the stdout and stderr of the Tesseract.
@@ -880,6 +905,10 @@ def run_tesseract(
     else:
         parsed_volumes = _parse_volumes(volumes)
 
+    if user is None:
+        # Use the current user if not specified
+        user = f"{os.getuid()}:{os.getgid()}" if os.name != "nt" else None
+
     for arg in args:
         if arg.startswith("-"):
             current_cmd = arg
@@ -901,7 +930,7 @@ def run_tesseract(
                     f"Path {local_path} provided as output is not a directory"
                 )
 
-            path_in_container = "/mnt/output"
+            path_in_container = "/tesseract/output_data"
             arg = path_in_container
 
             # Bind-mount directory
@@ -914,7 +943,9 @@ def run_tesseract(
             if not local_path.is_file():
                 raise RuntimeError(f"Path {local_path} provided as input is not a file")
 
-            path_in_container = os.path.join("/mnt", f"payload{local_path.suffix}")
+            path_in_container = os.path.join(
+                "/tesseract/input_data", f"payload{local_path.suffix}"
+            )
             arg = f"@{path_in_container}"
 
             # Bind-mount file
@@ -923,6 +954,10 @@ def run_tesseract(
         current_cmd = None
         cmd.append(arg)
 
+    extra_args = []
+    if is_podman():
+        extra_args.extend(["--userns", "keep-id"])
+
     # Run the container
     stdout, stderr = docker_client.containers.run(
         image=image,
@@ -935,6 +970,7 @@ def run_tesseract(
         remove=True,
         stderr=True,
         user=user,
+        extra_args=extra_args,
     )
     stdout = stdout.decode("utf-8")
     stderr = stderr.decode("utf-8")

tesseract_core/sdk/templates/Dockerfile.base

@@ -63,12 +63,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && rm -rf /var/lib/apt/lists/*
 {% endif %}
 
-# Drop to a non-root user
-RUN groupadd -o -g 1000 tesseractor && \
-    useradd -o -u 1000 -g 1000 --create-home -s /bin/bash tesseractor
 WORKDIR /tesseract
-RUN chown tesseractor:tesseractor /tesseract
-USER tesseractor
+RUN chmod 755 /tesseract
 
 # Set environment variables
 ENV TESSERACT_NAME="{{ config.name | replace('"', '\\"') | replace('\n', '\\n') }}" \
@@ -78,7 +74,7 @@ ENV TESSERACT_NAME="{{ config.name | replace('"', '\\"') | replace('\n', '\\n')
 
 # Copy only necessary files
 COPY --from=build_stage /python-env /python-env
-COPY --chown=1000:1000 "{{ tesseract_source_directory }}/tesseract_api.py" ${TESSERACT_API_PATH}
+COPY "{{ tesseract_source_directory }}/tesseract_api.py" ${TESSERACT_API_PATH}
 
 ENV PATH="/python-env/bin:$PATH"
 
@@ -94,6 +90,12 @@ COPY ["{{ tesseract_source_directory }}/{{ source_path }}", "{{ target_path }}"]
 {{ config.build_config.custom_build_steps | join("\n") }}
 {% endif %}
 
+RUN mkdir -p /tesseract/input_data /tesseract/output_data && \
+    chmod 755 /tesseract/input_data && \
+    chmod 777 /tesseract/output_data
+
+USER 1000:1000
+
 # Final sanity check to ensure the runtime is installed and tesseract_api.py is valid
 {% if not config.build_config.skip_checks %}
 RUN ["tesseract-runtime", "check"]

tesseract_core/sdk/templates/docker-compose.yml

@@ -14,8 +14,8 @@ services:
     {% endif %}
     {%- if service.volumes %}
     volumes:
-    {%- for volume in service.volumes %}
-      - {{ volume }}
+    {%- for volume_source, volume_data in service.volumes.items() %}
+      - {{ volume_source }}:{{ volume_data["bind"] }}
     {% endfor -%}
     {% endif %}
     healthcheck:
@@ -42,6 +42,7 @@ services:
     {% endif %}
 {% endfor %}
 
+
 {%- if docker_volumes %}
 volumes:
 {%- for name, external in docker_volumes.items() %}

tests/conftest.py

@@ -210,15 +210,12 @@ def docker_client():
 @pytest.fixture
 def docker_volume(docker_client):
     # Create the Docker volume
-    volume = docker_client.volumes.create(name="docker_client_test_volume")
+    volume_name = f"test_volume_{''.join(random.choices(string.ascii_lowercase + string.digits, k=8))}"
+    volume = docker_client.volumes.create(name=volume_name)
     try:
         yield volume
     finally:
-        try:
-            volume.remove()
-        except Exception:
-            # already removed
-            pass
+        volume.remove(force=True)
 
 
 @pytest.fixture(scope="module")
@@ -449,6 +446,7 @@ def exists(project_id: str) -> bool:
 
     mock_instance = MockedDocker()
     monkeypatch.setattr(engine, "docker_client", mock_instance)
+    monkeypatch.setattr(engine, "is_podman", lambda: False)
     monkeypatch.setattr(
         tesseract_core.sdk.docker_client, "CLIDockerClient", MockedDocker
     )