From 195ed6eae6d34fac48350ecb8606980300c2cb81 Mon Sep 17 00:00:00 2001 From: Kurtis Wright Date: Sun, 27 Jul 2025 14:21:49 -0700 Subject: [PATCH 1/2] Updates Podman Instructions, Docker-Compose, & Spark Script docs/contributing/podman.md - Podman instructions have been updated for clarity and to emphasis its usage in a Linux environment. crates/integration_tests/testdata/docker-compose.yaml - Updated to remove deprecated Link feature, which breaks when used with Podman. https://stackoverflow.com/questions/35832095/difference-between-links-and-depends-on-in-docker-compose-yml crates/integration_tests/testdata/spark/provision.py - Added Create Namespace `default` to fix breaking tests. --- CONTRIBUTING.md | 2 +- .../testdata/docker-compose.yaml | 3 --- .../testdata/spark/provision.py | 2 ++ docs/contributing/podman.md | 27 ++++++++++++++----- 4 files changed, 24 insertions(+), 10 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8c143ba905..bef6e24155 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -105,7 +105,7 @@ Currently, iceberg-rust uses Docker to set up environment for integration tests. For MacOS users, you can install [OrbStack as a docker alternative](docs/contributing/orbstack.md). -For podman, refer to [Using Podman instead of Docker](docs/contributing/podman.md) +For Linux users Podman is recommended, refer to [Using Podman instead of Docker](docs/contributing/podman.md) ## Build diff --git a/crates/integration_tests/testdata/docker-compose.yaml b/crates/integration_tests/testdata/docker-compose.yaml index 7b4efebb26..52934dc5af 100644 --- a/crates/integration_tests/testdata/docker-compose.yaml +++ b/crates/integration_tests/testdata/docker-compose.yaml @@ -80,6 +80,3 @@ services: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 - links: - - rest:rest - - minio:minio diff --git a/crates/integration_tests/testdata/spark/provision.py b/crates/integration_tests/testdata/spark/provision.py index 1929266b4a..899a49f6ac 100755 --- a/crates/integration_tests/testdata/spark/provision.py +++ b/crates/integration_tests/testdata/spark/provision.py @@ -30,6 +30,8 @@ .getOrCreate() ) +spark.sql(f"""CREATE NAMESPACE IF NOT EXISTS rest.default""") + spark.sql( f""" CREATE OR REPLACE TABLE rest.default.test_positional_merge_on_read_deletes ( diff --git a/docs/contributing/podman.md b/docs/contributing/podman.md index 3281ad4da5..ef2bca97f9 100644 --- a/docs/contributing/podman.md +++ b/docs/contributing/podman.md @@ -19,15 +19,16 @@ # Using Podman instead of Docker -Iceberg-rust does not require containerization, except for integration tests, where "docker" and "docker-compose" are used to start containers for minio and various catalogs. Below instructions setup "rootful podman" and docker's official docker-compose plugin to run integration tests as an alternative to docker or Orbstack. +Iceberg-rust does not require containerization, except for integration tests, where "docker" and "docker-compose" are used to start containers for minio and various catalogs. Below instructions setup "rootful podman" and docker's official docker-compose plugin to run integration tests as an alternative to docker or Orbstack. It is recommended to setup Podman if you are developing in a Linux or WSL environment. 1. Have podman v4 or newer. - ```console + ```shell $ podman --version podman version 4.9.4-rhel ``` - -2. Open file `/usr/bin/docker` and add the below contents: +2. Create a docker wrapper script: + + * Create a fresh `/usr/bin/docker` file and add the below contents: ```bash #!/bin/sh [ -e /etc/containers/nodocker ] || \ @@ -35,8 +36,13 @@ Iceberg-rust does not require containerization, except for integration tests, wh exec sudo /usr/bin/podman "$@" ``` + * Set new `/usr/bin/docker` file to executable. + ```shell + sudo chmod +x /usr/bin/docker + ``` + 3. Install the [docker compose plugin](https://docs.docker.com/compose/install/linux). Check for successful installation. - ```console + ```shell $ docker compose version Docker Compose version v2.28.1 ``` @@ -53,7 +59,7 @@ Iceberg-rust does not require containerization, except for integration tests, wh ``` 6. Check that the following symlink exists. - ```console + ```shell $ ls -al /var/run/docker.sock lrwxrwxrwx 1 root root 27 Jul 24 12:18 /var/run/docker.sock -> /var/run/podman/podman.sock ``` @@ -83,3 +89,12 @@ As of podman v4, ["To be succinct and simple, when running rootless containers, * * + +# Debugging Note: + - Fix for error: `Error: short-name "apache/iceberg-rest-fixture" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"` + - Add or modify the `/etc/containers/registries.conf` file: + ```toml + [[registry]] + prefix = "docker.io" + location = "docker.io" + ``` From 8682b4dc0572ba5fb5625f79edfd748feac33c4e Mon Sep 17 00:00:00 2001 From: Kurtis Wright Date: Sun, 27 Jul 2025 14:21:49 -0700 Subject: [PATCH 2/2] Updates Podman Instructions, Docker-Compose, & Spark Script docs/contributing/podman.md - Podman instructions have been updated for clarity and to emphasis its usage in a Linux environment. crates/integration_tests/testdata/docker-compose.yaml - Updated to remove deprecated Link feature, which breaks when used with Podman. https://stackoverflow.com/questions/35832095/difference-between-links-and-depends-on-in-docker-compose-yml crates/integration_tests/testdata/spark/provision.py - Added Create Namespace `default` to fix breaking tests. --- CONTRIBUTING.md | 3 +-- docs/contributing/podman.md | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bef6e24155..712128f5a7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -104,8 +104,7 @@ cargo 1.69.0 (6e9a83356 2023-04-12) Currently, iceberg-rust uses Docker to set up environment for integration tests. Native Docker has some limitations, please check (https://github.com/apache/iceberg-rust/pull/748). Please use Orbstack or Podman. For MacOS users, you can install [OrbStack as a docker alternative](docs/contributing/orbstack.md). - -For Linux users Podman is recommended, refer to [Using Podman instead of Docker](docs/contributing/podman.md) +For Podman users, refer to [Using Podman instead of Docker](docs/contributing/podman.md) ## Build diff --git a/docs/contributing/podman.md b/docs/contributing/podman.md index ef2bca97f9..eee93a0b35 100644 --- a/docs/contributing/podman.md +++ b/docs/contributing/podman.md @@ -19,7 +19,7 @@ # Using Podman instead of Docker -Iceberg-rust does not require containerization, except for integration tests, where "docker" and "docker-compose" are used to start containers for minio and various catalogs. Below instructions setup "rootful podman" and docker's official docker-compose plugin to run integration tests as an alternative to docker or Orbstack. It is recommended to setup Podman if you are developing in a Linux or WSL environment. +Iceberg-rust does not require containerization, except for integration tests, where "docker" and "docker-compose" are used to start containers for minio and various catalogs. Below instructions setup "rootful podman" and docker's official docker-compose plugin to run integration tests as an alternative to docker or Orbstack. 1. Have podman v4 or newer. ```shell