Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
38 changes: 38 additions & 0 deletions docs/setup_installation/admin/trino.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,43 @@ Use this detailed view to diagnose worker-specific issues and optimize resource
<figcaption>Trino worker status</figcaption>
</figure>

## Managing Catalogs

Catalogs created by project Data Owners are saved to the database but are not loaded by the running cluster until an administrator applies them.
Applying a catalog is a two-step, admin-gated workflow on the Catalogs tab under Cluster Settings, Query Engine: sync the pending changes, then restart Trino.

The tab lists every catalog waiting to be applied, along with its status and the operation to apply (create, update, or remove).

<figure>
<img src="../../../assets/images/admin/trino/catalogs-pending.png" alt="Pending catalogs" />
<figcaption>Catalogs awaiting sync and restart</figcaption>
</figure>

### Syncing

Select the catalogs to apply and click "Sync selected".
This writes the catalog definitions into the backend-owned ConfigMap that the cluster mounts.
Any `${HOPSWORKS_SECRET:...}` reference is resolved to its value only at this step, so secrets are never stored in the catalog database rows.
After syncing, a catalog moves to Pending restart, meaning it is present in the ConfigMap but not yet loaded by the running cluster.

<figure>
<img src="../../../assets/images/admin/trino/catalogs-pending-restart.png" alt="Catalogs pending restart" />
<figcaption>Synced catalogs wait in Pending restart until the next restart</figcaption>
</figure>

### Restarting

Trino reads catalogs only at startup, so synced changes take effect on the next restart.
Click "Restart Trino" to roll out the coordinator and workers.
The confirmation dialog reports how many queries are currently running or queued, because the restart interrupts them, so you can choose a low-traffic window before confirming.

<figure>
<img src="../../../assets/images/admin/trino/restart-confirm.png" alt="Restart confirmation" />
<figcaption>The restart confirmation reports the running queries the restart will interrupt</figcaption>
</figure>

A restart is refused while another sync or restart is already running, and while a rollout is still in progress, so concurrent actions by different administrators cannot collide or trigger redundant restarts.

## Configuration

Trino behavior can be customized through cluster configuration variables. To modify these settings, navigate to **Cluster Settings** → **Configuration** and search for the variable name.
Expand All @@ -88,6 +125,7 @@ Trino behavior can be customized through cluster configuration variables. To mod

- **trino_enabled**: Enable or disable Trino cluster-wide (default: `true`)
- **trino_default_catalog**: Default catalog used for Superset queries (default: `hive`)
- **trino_test_coordinator_enabled**: Enable the optional test coordinator that backs the "Test connection" action for user-created catalogs (default: `true`)

These settings control the availability and default behavior of the Trino query engine across your Hopsworks cluster.

Expand Down
63 changes: 63 additions & 0 deletions docs/user_guides/projects/trino/query_engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,69 @@ This information helps you understand cluster performance and capacity.
<figcaption>Query Engine cluster overview</figcaption>
</figure>

## Managing Catalogs

Catalogs connect the Query Engine to external data sources such as PostgreSQL, MySQL, BigQuery, or MongoDB.
Project Data Owners can create, edit, and delete catalogs for their project from the Catalogs tab.
A catalog you create is named `<project>__<name>` and is only queryable within your own project.

<figure>
<img src="../../../../assets/images/guides/trino/catalogs-list.png" alt="Catalogs list" />
<figcaption>The Catalogs tab lists the shared default catalogs and your project's catalogs</figcaption>
</figure>

### Creating a Catalog

Click "Create catalog", then provide the catalog details:

1. Enter a name. The `<project>__` prefix is added automatically, so you type only the short name.
2. Choose the connector type, for example `postgresql`, `mysql`, `bigquery`, or `mongodb`.
3. Enter the connector properties, one `key=value` per line.

Reference a Hopsworks secret with `${HOPSWORKS_SECRET:<name>}` instead of typing the value inline to keep the secret out of the stored catalog definition.
Type `${HOPSWORKS_SECRET:` in the properties editor to pick from your own secrets.

<figure>
<img src="../../../../assets/images/guides/trino/create-catalog.png" alt="Create catalog" />
<figcaption>Creating a catalog with the auto-prefixed name and connector properties</figcaption>
</figure>

### Testing the Connection

Click "Test connection" to validate the configuration against the backing system before you save.
The test creates a temporary catalog, lists its schemas, and reports the result, so you catch a wrong host, port, or credential immediately.

When the connection cannot be established, the connector's own error is shown.

<figure>
<img src="../../../../assets/images/guides/trino/test-connection-failed.png" alt="Test connection failed" />
<figcaption>A failed test reports the underlying connector error</figcaption>
</figure>

When the configuration is correct, the test confirms that the catalog connects.

<figure>
<img src="../../../../assets/images/guides/trino/test-connection-success.png" alt="Test connection succeeded" />
<figcaption>A successful connection test</figcaption>
</figure>

### Availability After Creation

A newly created catalog has the status Pending sync, meaning it is saved but not yet loaded by the running Query Engine.
It becomes queryable only after an administrator syncs it and restarts Trino, because Trino reads catalogs only at startup.
Until then the catalog is listed with its pending status and does not appear as a target in the SQL runner.

<figure>
<img src="../../../../assets/images/guides/trino/catalog-pending-sync.png" alt="Catalog pending sync" />
<figcaption>A created catalog waits in Pending sync until an administrator applies it</figcaption>
</figure>

### Access to Catalog Tables

Access to a user-created catalog is granted at the catalog level per project: a project's Data Owners can read and write, and Data Scientists can read.
Trino's file-based access control provides no per-schema or per-table restriction for these catalogs, so access is all-or-nothing within the catalog.
To limit what a catalog exposes, scope the database user in the connection credentials at the source database, since Trino reads the external system as that user and can only ever see what those credentials allow.

## Queries

The Queries tab displays a history of all executed queries. For each query, you can see:
Expand Down
Loading