diff --git a/.gitignore b/.gitignore index 5bd840e..4019bd6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ db-data/data -s3-data/data +garage-data/meta +garage-data/data .env +config/application.local.yml diff --git a/README.md b/README.md index a70c105..d4cd32b 100644 --- a/README.md +++ b/README.md @@ -7,28 +7,77 @@ If you use or plan to use DSW, please let us know via [info@ds-wizard.org](mailt - Join our [**Discord** server](https://discord.gg/MW3H9tdMcT), where you can be notified about important updates and releases + we can discuss your issues and ideas. - Provide us feedback (what is good and bad, [feature requests](https://ideas.ds-wizard.org/), etc.) -This example is intended for **local setup and testing**. For production use there are many more things to do such as authentication, controlling exposed ports (e.g. do not expose ports of `postgres` and `minio`), data backups, or using proxy (with HTTPS and WebSocket enabled). As it is highly dependent on your use case, consult production deployment with your sysadmin or [contact us](https://ds-wizard.org/contact). +This example is intended for **local setup and testing**. For production use there are many more things to do such as authentication, controlling exposed ports (e.g. do not expose ports of `postgres` and `garage`), data backups, or using proxy (with HTTPS and WebSocket enabled). As it is highly dependent on your use case, consult production deployment with your sysadmin or [contact us](https://ds-wizard.org/contact). -## Usage +## Overview -This is an example deployment of the [Data Stewardship Wizard](https://ds-wizard.org) using [docker-compose](https://docs.docker.com/compose/). You can clone the repository, create `.env` file using `example.env` and run it with: +This is an example deployment of the [Data Stewardship Wizard](https://ds-wizard.org) using [Docker Compose](https://docs.docker.com/compose/) and [Garage](https://garagehq.deuxfleurs.fr/) as the S3-compatible object storage. -``` -$ docker-compose up -d -``` +It is intentionally set up as a **single-node local POC**: -Then visit [localhost:8080/wizard](http://localhost:8080/wizard/) and login as `albert.einstein@example.com` with password `password`. +- Garage runs in Docker on host ports `9000` (S3 API), `9002` (web endpoint), and `9003` (Admin API) +- Garage UI runs on `127.0.0.1:8081` +- Plugin files are served locally through `127.0.0.1:9004` +- DSW points to `http://host.docker.internal:9000` so presigned URLs are reachable from the browser +- `create-bucket.sh` performs the one-time Garage bootstrap for this example For information on how to use Data Stewardship Wizard, visit our [guide](https://guide.ds-wizard.org). -## Important notes +## Required Steps + +These are the only steps needed to run the local example. + +1. Create the local environment file: + + ```bash + cp example.env .env + ``` + +2. The tracked `config/application.yml` already contains demo application values for this example. + +3. Start the stack: + + ```bash + docker compose up -d + ``` + +4. Bootstrap Garage: + + ```bash + ./create-bucket.sh + ``` + +5. Open DSW: + + [http://localhost:8080/wizard](http://localhost:8080/wizard/) + +6. Log in with: + + - Email: `albert.einstein@example.com` + - Password: `password` + +## Notes + +- Garage UI is available on [http://localhost:8081](http://localhost:8081) with username `admin` and password `admin` +- `create-bucket.sh` prepares both the private `engine-wizard` bucket and the public `plugins` bucket +- For local plugin testing, upload plugin files to the `plugins` bucket under `//` +- The plugin file should be stored as `//plugin.js` +- The plugin URL in the DSW database should point to `http://localhost:9004///` +- Example DB value: `http://localhost:9004/6534f8f7-0d43-4c4d-9157-15af17f37649/1.2.0/` +- Usable plugin examples: [replies-importer-plugin](https://github.com/ds-wizard/replies-importer-plugin) and [madmp-importer-plugin](https://github.com/ds-wizard/madmp-importer-plugin) +- This repository is a local example only; if you self-host Garage publicly, place it behind HTTPS reverse proxy or another equivalent security layer + +## Important Notes * Use `docker compose pull` to get newest image (hotfixes) before starting -* **Do not expose** PostgreSQL and MinIO to the internet (MinIO should be exposed only via proxy) +* **Do not expose** PostgreSQL or raw Garage ports directly to the internet in a public deployment +* If you self-host Garage publicly, place it behind an HTTPS reverse proxy or another equivalent security layer and protect admin access * When you want to use DSW publicly, **set up HTTPS proxy** (e.g. Nginx) with a certificate for your domain and change default accounts -* Set up volume mounted to PostgreSQL and MinIO containers for persistent data -* You have to create S3 bucket, either using Web UI (for MinIO, you can expose and use `http://localhost:9000`) or via client: https://docs.min.io/docs/minio-client-complete-guide.html#mb, e.g. use `create-bucket.sh` script -* Always use **strong passwords** and never use default values, **change the secrets** in `config/application.yml` (32 character string in `general.secret` and RSA private key in `general.rsaPrivateKey` via `ssh-keygen -t rsa -b 4096 -m PEM -f jwtRS256.key`) +* Set up volume mounted to PostgreSQL and Garage containers for persistent data +* Garage needs a one-time bootstrap after the stack starts. `create-bucket.sh` assigns the single-node layout, creates the `engine-wizard` and `plugins` buckets, imports the configured S3 key, and grants bucket permissions +* DSW uses `http://host.docker.internal:9000` as the S3 endpoint so both the DSW containers and the browser can reach the same local Garage endpoint +* Garage UI is configured with local basic auth defaults for this POC; change them before sharing the setup +* Always use **strong passwords** and never use default values, **change the demo secrets** in `config/application.yml` and `.env` before using this anywhere except local testing ## Security Audit diff --git a/config/application.yml b/config/application.yml index d5831b9..80383a7 100644 --- a/config/application.yml +++ b/config/application.yml @@ -62,14 +62,17 @@ database: # (!!) Change default password s3: url: http://host.docker.internal:9000 - username: minio - password: minioPassword + # Garage access key ID must start with GK and contain 24 hex characters after it. + username: GK111111111111111111111111 + # Garage secret key must be 64 hex characters. + password: "1111111111111111111111111111111111111111111111111111111111111111" bucket: engine-wizard + region: garage # (!!) Configure SMTP mail: enabled: false - name: + name: email: provider: smtp smtp: diff --git a/config/garage-ui.yaml b/config/garage-ui.yaml new file mode 100644 index 0000000..c9eb7cc --- /dev/null +++ b/config/garage-ui.yaml @@ -0,0 +1,27 @@ +server: + host: "::" + port: 8080 + environment: "production" + domain: "localhost" + protocol: "http" + root_url: "http://localhost:8081" + +garage: + endpoint: "http://garage:3900" + admin_endpoint: "http://garage:3903" + region: "garage" + +auth: + admin: + enabled: true + token: + enabled: false + session_max_age: 86400 + cookie_name: "garage_session" + cookie_secure: false + cookie_http_only: true + cookie_same_site: "lax" + +logging: + level: "info" + format: "text" diff --git a/config/garage.toml b/config/garage.toml new file mode 100644 index 0000000..5f0e6d5 --- /dev/null +++ b/config/garage.toml @@ -0,0 +1,22 @@ +metadata_dir = "/var/lib/garage/meta" +data_dir = "/var/lib/garage/data" +db_engine = "sqlite" + +# Single-node local POC settings. +replication_factor = 1 +rpc_bind_addr = "[::]:3901" +rpc_public_addr = "127.0.0.1:3901" + +[s3_api] +# Path-style requests work without extra DNS; root_domain only enables optional vhost-style access. +s3_region = "garage" +api_bind_addr = "[::]:3900" +root_domain = ".s3.garage.localhost" + +[s3_web] +bind_addr = "[::]:3902" +root_domain = ".web.garage.localhost" +index = "index.html" + +[admin] +api_bind_addr = "[::]:3903" diff --git a/config/plugin-proxy.conf b/config/plugin-proxy.conf new file mode 100644 index 0000000..11e1c02 --- /dev/null +++ b/config/plugin-proxy.conf @@ -0,0 +1,37 @@ +server { + listen 8080; + server_name _; + + location /plugins/ { + rewrite ^/plugins/(.*)$ /$1 break; + proxy_pass http://garage:3902; + proxy_set_header Host plugins.web.garage.localhost; + + add_header Access-Control-Allow-Origin "*" always; + add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always; + add_header Access-Control-Allow-Headers "*" always; + add_header Access-Control-Expose-Headers "Content-Length, Content-Type, ETag" always; + + if ($request_method = OPTIONS) { + add_header Content-Length 0; + add_header Content-Type text/plain; + return 204; + } + } + + location / { + proxy_pass http://garage:3902; + proxy_set_header Host plugins.web.garage.localhost; + + add_header Access-Control-Allow-Origin "*" always; + add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always; + add_header Access-Control-Allow-Headers "*" always; + add_header Access-Control-Expose-Headers "Content-Length, Content-Type, ETag" always; + + if ($request_method = OPTIONS) { + add_header Content-Length 0; + add_header Content-Type text/plain; + return 204; + } + } +} diff --git a/create-bucket.sh b/create-bucket.sh index 33238a0..dd653ef 100755 --- a/create-bucket.sh +++ b/create-bucket.sh @@ -1,18 +1,93 @@ #!/bin/sh -set -a; source .env; set +a - -# (!!) Change default password -MINIO_NET="dsw-deployment-example_default" -MINIO_BUCKET="engine-wizard" -MINIO_USER="minio" -MINIO_PASS="minioPassword" - -docker run --rm --net $MINIO_NET \ - -e MINIO_BUCKET=$MINIO_BUCKET \ - -e MINIO_USER=$MINIO_USER \ - -e MINIO_PASS=$MINIO_PASS \ - --entrypoint sh minio/mc:$MINIO_VERSION -c "\ - mc alias set dswminio http://minio:9000 $MINIO_USER $MINIO_PASS && \ - mc mb dswminio/\$MINIO_BUCKET -" +set -eu + +ENV_FILE="${ENV_FILE:-.env}" + +if [ -f "$ENV_FILE" ]; then + set -a + . "./$ENV_FILE" + set +a +fi + +GARAGE_SERVICE="${GARAGE_SERVICE:-garage}" +GARAGE_ZONE="${GARAGE_ZONE:-dc1}" +GARAGE_CAPACITY="${GARAGE_CAPACITY:-1G}" +GARAGE_KEY_NAME="${GARAGE_KEY_NAME:-dsw-engine-wizard}" +S3_URL="${S3_URL:-http://host.docker.internal:9000}" +S3_BUCKET="${S3_BUCKET:-engine-wizard}" +PLUGINS_BUCKET="${PLUGINS_BUCKET:-plugins}" +# Garage access key ID must start with GK and contain 24 hex characters after it. +S3_USERNAME="${S3_USERNAME:-GK111111111111111111111111}" +# Garage secret key must be 64 hex characters. +S3_PASSWORD="${S3_PASSWORD:-1111111111111111111111111111111111111111111111111111111111111111}" + +garage_exec() { + if [ -f "$ENV_FILE" ]; then + docker compose --env-file "$ENV_FILE" exec -T "$GARAGE_SERVICE" /garage "$@" + else + docker compose exec -T "$GARAGE_SERVICE" /garage "$@" + fi +} + +compose_ps() { + if [ -f "$ENV_FILE" ]; then + docker compose --env-file "$ENV_FILE" ps -q --status running "$GARAGE_SERVICE" + else + docker compose ps -q --status running "$GARAGE_SERVICE" + fi +} + +garage_exec_quiet() { + output_file="$(mktemp)" + if garage_exec "$@" >"$output_file" 2>&1; then + rm -f "$output_file" + return 0 + fi + + cat "$output_file" >&2 + rm -f "$output_file" + return 1 +} + +ensure_bucket() { + bucket_name="$1" + website_public="${2:-false}" + + if ! garage_exec bucket info "$bucket_name" >/dev/null 2>&1; then + garage_exec_quiet bucket create "$bucket_name" + fi + + garage_exec_quiet bucket allow --read --write --owner "$bucket_name" --key "$S3_USERNAME" + + if [ "$website_public" = "true" ]; then + garage_exec_quiet bucket website --allow "$bucket_name" + fi +} + +if [ -z "$(compose_ps 2>/dev/null)" ]; then + echo "Garage service '$GARAGE_SERVICE' is not running. Start the stack first with: docker compose up -d" >&2 + exit 1 +fi + +STATUS="$(garage_exec status 2>/dev/null)" +NODE_ID="$(printf '%s\n' "$STATUS" | awk '/==== HEALTHY NODES ====/{flag=1; next} flag && $1 ~ /^[0-9a-f]+$/ {print $1; exit}')" + +if [ -z "$NODE_ID" ]; then + echo "Unable to determine the Garage node ID from 'garage status'." >&2 + exit 1 +fi + +if printf '%s\n' "$STATUS" | grep -q "NO ROLE ASSIGNED"; then + garage_exec_quiet layout assign -z "$GARAGE_ZONE" -c "$GARAGE_CAPACITY" "$NODE_ID" + garage_exec_quiet layout apply --version 1 +fi + +if ! garage_exec key info "$S3_USERNAME" >/dev/null 2>&1; then + garage_exec_quiet key import "$S3_USERNAME" "$S3_PASSWORD" -n "$GARAGE_KEY_NAME" --yes +fi + +ensure_bucket "$S3_BUCKET" false +ensure_bucket "$PLUGINS_BUCKET" true + +echo "Garage is ready." diff --git a/docker-compose.yml b/docker-compose.yml index 01078d5..c06006b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -4,37 +4,77 @@ services: image: datastewardshipwizard/wizard-server:${DSW_VERSION} platform: linux/amd64 restart: always + # Local demo only; put an HTTPS proxy in front for shared deployments. ports: - # (!!) Expose only for local deployment, externally use HTTPS proxy - 127.0.0.1:3000:3000 depends_on: - postgres - - minio + - garage volumes: - ./config/application.yml:/app/config/application.yml:ro extra_hosts: - host.docker.internal:host-gateway + # Keep runtime S3 values in .env so they stay aligned with Garage bootstrap. + environment: + S3_URL: ${S3_URL} + S3_USERNAME: ${S3_USERNAME} + S3_PASSWORD: ${S3_PASSWORD} + S3_BUCKET: ${S3_BUCKET} + S3_REGION: ${S3_REGION} client: image: datastewardshipwizard/wizard-client:${DSW_VERSION} restart: always + # Local demo only; put an HTTPS proxy in front for shared deployments. ports: - # (!!) Expose only for local deployment, externally use HTTPS proxy - 127.0.0.1:8080:8080 environment: API_URL: http://localhost:3000/wizard-api + garage-ui: + image: noooste/garage-ui:${GARAGE_UI_VERSION} + restart: always + # Local admin UI only; do not expose this directly in a public deployment. + ports: + - 127.0.0.1:${GARAGE_UI_PORT}:8080 + depends_on: + - garage + volumes: + - ./config/garage-ui.yaml:/app/config.yaml:ro + environment: + GARAGE_UI_SERVER_ROOT_URL: http://localhost:${GARAGE_UI_PORT} + GARAGE_UI_GARAGE_ADMIN_TOKEN: ${GARAGE_ADMIN_TOKEN} + GARAGE_UI_AUTH_ADMIN_USERNAME: ${GARAGE_UI_ADMIN_USERNAME} + GARAGE_UI_AUTH_ADMIN_PASSWORD: ${GARAGE_UI_ADMIN_PASSWORD} + + plugin-proxy: + image: nginx:1.27-alpine + restart: always + ports: + - 127.0.0.1:9004:8080 + depends_on: + - garage + volumes: + - ./config/plugin-proxy.conf:/etc/nginx/conf.d/default.conf:ro + docworker: image: datastewardshipwizard/document-worker:${DSW_VERSION} restart: always depends_on: - postgres - - minio + - garage - server volumes: - ./config/application.yml:/app/config/application.yml:ro extra_hosts: - host.docker.internal:host-gateway + # Keep runtime S3 values in .env so they stay aligned with Garage bootstrap. + environment: + S3_URL: ${S3_URL} + S3_USERNAME: ${S3_USERNAME} + S3_PASSWORD: ${S3_PASSWORD} + S3_BUCKET: ${S3_BUCKET} + S3_REGION: ${S3_REGION} mailer: image: datastewardshipwizard/mailer:${DSW_VERSION} @@ -48,39 +88,38 @@ services: postgres: image: postgres:${POSTGRES_VERSION} restart: always - # (!!) Expose only for debugging locally + # Local debug access only. ports: - 127.0.0.1:5432:5432 - # (!!) Change default password + # Change defaults before using this outside a local POC. environment: - POSTGRES_DB: engine-wizard - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - # (!!) Mount for persistent data + POSTGRES_DB: ${POSTGRES_DB} + POSTGRES_USER: ${POSTGRES_USER} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + # Uncomment one option below to persist PostgreSQL data locally. # volumes: # - db-data:/var/lib/postgresql/data # OR # - ./db-data/data:/var/lib/postgresql/data - minio: - image: minio/minio:${MINIO_VERSION} + garage: + image: dxflrs/garage:${GARAGE_VERSION} restart: always - command: server /data --console-address ":9001" - # (!!) Expose only for debugging locally, externally use HTTPS proxy (see MinIO docs) + # Publish Garage on the host so both containers and the browser can reach the same endpoint. + # Do not expose this directly in a public deployment. ports: - - 9000:9000 - - 9001:9001 - # (!!) Change default password + - 9000:3900 + - 9002:3902 + - 9003:3903 + # Change these defaults before sharing or reusing this setup. environment: - MINIO_ROOT_USER: minio - MINIO_ROOT_PASSWORD: minioPassword - MINIO_VOLUMES: /data - # (!!) Mount and backup for persistent data - # volumes: - # - s3-data:/data - # OR - # - ./s3-data/data:/data + GARAGE_RPC_SECRET: ${GARAGE_RPC_SECRET} + GARAGE_ADMIN_TOKEN: ${GARAGE_ADMIN_TOKEN} + GARAGE_METRICS_TOKEN: ${GARAGE_METRICS_TOKEN} + volumes: + - ./config/garage.toml:/etc/garage.toml:ro + # Uncomment to persist Garage state between local restarts. + # - ./garage-data:/var/lib/garage # volumes: # db-data: -# s3-data: diff --git a/example.env b/example.env index 41064b8..ac55991 100644 --- a/example.env +++ b/example.env @@ -1,6 +1,35 @@ DSW_VERSION=4.31 # Versions of dependencies (see their docs) -# - possibly a migration may be needed when upgrading +# Upgrades may require migrations. POSTGRES_VERSION=17.5 -MINIO_VERSION=RELEASE.2025-05-24T17-08-30Z +GARAGE_VERSION=v2.2.0 +GARAGE_UI_VERSION=v0.8.4 +GARAGE_UI_PORT=8081 + +# DSW DB configuration. +POSTGRES_USER=postgres +POSTGRES_PASSWORD=postgres +POSTGRES_DB=engine-wizard + +# DSW S3 configuration. +# Keep these aligned with the credentials imported by create-bucket.sh. +# Garage access key ID must start with GK and contain 24 hex characters after it. +# Garage secret key must be 64 hex characters. +S3_URL=http://host.docker.internal:9000 +S3_USERNAME=GK111111111111111111111111 +S3_PASSWORD=1111111111111111111111111111111111111111111111111111111111111111 +S3_BUCKET=engine-wizard +PLUGINS_BUCKET=plugins +S3_REGION=garage + +# Garage daemon secrets. +# Change these defaults before sharing or reusing this setup. +# GARAGE_RPC_SECRET must be 64 hex characters. +GARAGE_RPC_SECRET=1111111111111111111111111111111111111111111111111111111111111111 +GARAGE_ADMIN_TOKEN=garageAdminToken +GARAGE_METRICS_TOKEN=garageMetricsToken + +# Garage UI local admin login. +GARAGE_UI_ADMIN_USERNAME=admin +GARAGE_UI_ADMIN_PASSWORD=admin diff --git a/garage-data/.gitkeep b/garage-data/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/garage-data/.gitkeep @@ -0,0 +1 @@ + diff --git a/s3-data/.gitkeep b/s3-data/.gitkeep deleted file mode 100644 index e69de29..0000000