diff --git a/docs/assets/images/admin/trino/catalogs-pending-restart.png b/docs/assets/images/admin/trino/catalogs-pending-restart.png new file mode 100644 index 0000000000..c6074607f1 Binary files /dev/null and b/docs/assets/images/admin/trino/catalogs-pending-restart.png differ diff --git a/docs/assets/images/admin/trino/catalogs-pending.png b/docs/assets/images/admin/trino/catalogs-pending.png new file mode 100644 index 0000000000..7b286580fc Binary files /dev/null and b/docs/assets/images/admin/trino/catalogs-pending.png differ diff --git a/docs/assets/images/admin/trino/restart-confirm.png b/docs/assets/images/admin/trino/restart-confirm.png new file mode 100644 index 0000000000..a3bec46a58 Binary files /dev/null and b/docs/assets/images/admin/trino/restart-confirm.png differ diff --git a/docs/assets/images/guides/trino/catalog-pending-sync.png b/docs/assets/images/guides/trino/catalog-pending-sync.png new file mode 100644 index 0000000000..9da146be09 Binary files /dev/null and b/docs/assets/images/guides/trino/catalog-pending-sync.png differ diff --git a/docs/assets/images/guides/trino/catalogs-list.png b/docs/assets/images/guides/trino/catalogs-list.png new file mode 100644 index 0000000000..77d37507ce Binary files /dev/null and b/docs/assets/images/guides/trino/catalogs-list.png differ diff --git a/docs/assets/images/guides/trino/create-catalog.png b/docs/assets/images/guides/trino/create-catalog.png new file mode 100644 index 0000000000..8747191636 Binary files /dev/null and b/docs/assets/images/guides/trino/create-catalog.png differ diff --git a/docs/assets/images/guides/trino/test-connection-failed.png b/docs/assets/images/guides/trino/test-connection-failed.png new file mode 100644 index 0000000000..83667d753c Binary files /dev/null and b/docs/assets/images/guides/trino/test-connection-failed.png differ diff --git a/docs/assets/images/guides/trino/test-connection-success.png b/docs/assets/images/guides/trino/test-connection-success.png new file mode 100644 index 0000000000..30c1ae1a4f Binary files /dev/null and b/docs/assets/images/guides/trino/test-connection-success.png differ diff --git a/docs/setup_installation/admin/trino.md b/docs/setup_installation/admin/trino.md index a6bd60e9a6..821a589352 100644 --- a/docs/setup_installation/admin/trino.md +++ b/docs/setup_installation/admin/trino.md @@ -80,6 +80,43 @@ Use this detailed view to diagnose worker-specific issues and optimize resource
Trino worker status
+## Managing Catalogs + +Catalogs created by project Data Owners are saved to the database but are not loaded by the running cluster until an administrator applies them. +Applying a catalog is a two-step, admin-gated workflow on the Catalogs tab under Cluster Settings, Query Engine: sync the pending changes, then restart Trino. + +The tab lists every catalog waiting to be applied, along with its status and the operation to apply (create, update, or remove). + +
+ Pending catalogs +
Catalogs awaiting sync and restart
+
+ +### Syncing + +Select the catalogs to apply and click "Sync selected". +This writes the catalog definitions into the backend-owned ConfigMap that the cluster mounts. +Any `${HOPSWORKS_SECRET:...}` reference is resolved to its value only at this step, so secrets are never stored in the catalog database rows. +After syncing, a catalog moves to Pending restart, meaning it is present in the ConfigMap but not yet loaded by the running cluster. + +
+ Catalogs pending restart +
Synced catalogs wait in Pending restart until the next restart
+
+ +### Restarting + +Trino reads catalogs only at startup, so synced changes take effect on the next restart. +Click "Restart Trino" to roll out the coordinator and workers. +The confirmation dialog reports how many queries are currently running or queued, because the restart interrupts them, so you can choose a low-traffic window before confirming. + +
+ Restart confirmation +
The restart confirmation reports the running queries the restart will interrupt
+
+ +A restart is refused while another sync or restart is already running, and while a rollout is still in progress, so concurrent actions by different administrators cannot collide or trigger redundant restarts. + ## Configuration Trino behavior can be customized through cluster configuration variables. To modify these settings, navigate to **Cluster Settings** → **Configuration** and search for the variable name. @@ -88,6 +125,7 @@ Trino behavior can be customized through cluster configuration variables. To mod - **trino_enabled**: Enable or disable Trino cluster-wide (default: `true`) - **trino_default_catalog**: Default catalog used for Superset queries (default: `hive`) +- **trino_test_coordinator_enabled**: Enable the optional test coordinator that backs the "Test connection" action for user-created catalogs (default: `true`) These settings control the availability and default behavior of the Trino query engine across your Hopsworks cluster. diff --git a/docs/user_guides/projects/trino/query_engine.md b/docs/user_guides/projects/trino/query_engine.md index d47a50da00..fc50c8dbf8 100644 --- a/docs/user_guides/projects/trino/query_engine.md +++ b/docs/user_guides/projects/trino/query_engine.md @@ -58,6 +58,69 @@ This information helps you understand cluster performance and capacity.
Query Engine cluster overview
+## Managing Catalogs + +Catalogs connect the Query Engine to external data sources such as PostgreSQL, MySQL, BigQuery, or MongoDB. +Project Data Owners can create, edit, and delete catalogs for their project from the Catalogs tab. +A catalog you create is named `__` and is only queryable within your own project. + +
+ Catalogs list +
The Catalogs tab lists the shared default catalogs and your project's catalogs
+
+ +### Creating a Catalog + +Click "Create catalog", then provide the catalog details: + +1. Enter a name. The `__` prefix is added automatically, so you type only the short name. +2. Choose the connector type, for example `postgresql`, `mysql`, `bigquery`, or `mongodb`. +3. Enter the connector properties, one `key=value` per line. + +Reference a Hopsworks secret with `${HOPSWORKS_SECRET:}` instead of typing the value inline to keep the secret out of the stored catalog definition. +Type `${HOPSWORKS_SECRET:` in the properties editor to pick from your own secrets. + +
+ Create catalog +
Creating a catalog with the auto-prefixed name and connector properties
+
+ +### Testing the Connection + +Click "Test connection" to validate the configuration against the backing system before you save. +The test creates a temporary catalog, lists its schemas, and reports the result, so you catch a wrong host, port, or credential immediately. + +When the connection cannot be established, the connector's own error is shown. + +
+ Test connection failed +
A failed test reports the underlying connector error
+
+ +When the configuration is correct, the test confirms that the catalog connects. + +
+ Test connection succeeded +
A successful connection test
+
+ +### Availability After Creation + +A newly created catalog has the status Pending sync, meaning it is saved but not yet loaded by the running Query Engine. +It becomes queryable only after an administrator syncs it and restarts Trino, because Trino reads catalogs only at startup. +Until then the catalog is listed with its pending status and does not appear as a target in the SQL runner. + +
+ Catalog pending sync +
A created catalog waits in Pending sync until an administrator applies it
+
+ +### Access to Catalog Tables + +Access to a user-created catalog is granted at the catalog level per project: a project's Data Owners can read and write, and Data Scientists can read. +Trino's file-based access control provides no per-schema or per-table restriction for these catalogs, so access is all-or-nothing within the catalog. +To limit what a catalog exposes, scope the database user in the connection credentials at the source database, since Trino reads the external system as that user and can only ever see what those credentials allow. + ## Queries The Queries tab displays a history of all executed queries. For each query, you can see: