diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index d44d9e46c..8287cfc82 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -63,6 +63,8 @@ * [Solana - SPL Initialized Account](how-to-guides/composing-substreams/foundational-stores/solana/spl-initialized-account.md) * [Published Packages](how-to-guides/composing-substreams/published-packages.md) * [Consuming Substreams](how-to-guides/sinks/sinks.md) + * [Hosted Sinks](how-to-guides/sinks/hosted-sinks/hosted-sinks.md) + * [Managing Your Sink](how-to-guides/sinks/hosted-sinks/manage-your-sink.md) * [Substreams:SQL](how-to-guides/sinks/sql/sql.md) * [Using Relational Mappings](how-to-guides/sinks/sql/relational-mappings.md) * [Using Database Changes](how-to-guides/sinks/sql/db_out.md) diff --git a/docs/assets/add-sinks-graph-market.png b/docs/assets/add-sinks-graph-market.png new file mode 100644 index 000000000..de78435a7 Binary files /dev/null and b/docs/assets/add-sinks-graph-market.png differ diff --git a/docs/assets/sink-details-graph-market.png b/docs/assets/sink-details-graph-market.png new file mode 100644 index 000000000..0d8b2d4d4 Binary files /dev/null and b/docs/assets/sink-details-graph-market.png differ diff --git a/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md b/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md new file mode 100644 index 000000000..271fc7436 --- /dev/null +++ b/docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md @@ -0,0 +1,92 @@ +# Hosted Sinks + +**Hosted Sinks** is a managed service on [The Graph Market](https://thegraph.market) that runs your Substreams sink for you. Instead of provisioning servers, managing deployments, or operating sink processes yourself, you configure a sink in the portal and StreamingFast handles the infrastructure. + +A hosted sink continuously reads data from your Substreams package and writes it to your own Postgres or ClickHouse database. + +{% hint style="info" %} +Hosted Sinks is available to organizations on The Graph Market. Your database must be publicly reachable from StreamingFast's infrastructure. +{% endhint %} + +## Requirements + +Before creating a sink, you need: + +- An account on [The Graph Market](https://thegraph.market) with an active organization. +- A Substreams package (`.spkg`) that outputs data using a `db_out` module or relational mappings. See [Substreams:SQL](../sql/sql.md) for how to build one. +- A running Postgres (default port `5432`) or ClickHouse (default port `9000`) database that is network-accessible from the internet. +- Your database schema already applied, or a schema that your Substreams package will create automatically. See [Sink Config](../../../references/sql/sink-config.md). + +## Creating a Sink + +Navigate to [**Sinks**](https://thegraph.market/sinks) in the sidebar and click **New Sink**. + +![](<../../../assets/add-sinks-graph-market.png>) + +### 1. Basics + +Enter a **Sink name**. This is a human-readable label for the deployment (e.g., `mainnet-erc20-transfers`). + +### 2. Substreams Package + +Choose how you want to supply the `.spkg`: + +| Source | When to use | +|---|---| +| **URL** | The package is hosted at a public URL (e.g., GitHub releases, IPFS). | +| **Substreams.dev** | The package is published on [the Substreams Registry](https://substreams.dev). Enter its ID (e.g., `ethereum_common@v0.3.3`). | +| **Binary upload** | Upload a `.spkg` file directly from your machine. | + +### 3. Output + +Choose **Postgres** or **ClickHouse** and fill in the connection details. + +**Postgres fields:** + +| Field | Description | +|---|---| +| Host | Database hostname or IP address. | +| Port | Default `5432`. | +| Database | Database name. | +| Schema | Target schema. Default `public`. | +| User | Database user with write access. | +| Password | Password for the user. | +| SSL Mode | One of `disable`, `require`, `verify-ca`, or `verify-full`. | + +**ClickHouse fields:** + +| Field | Description | +|---|---| +| Host | ClickHouse hostname or IP address. | +| Port | Default `9000` (native protocol). | +| Database | Database name. Default `default`. | +| User | ClickHouse user. | +| Password | Password for the user. | +| Secure (TLS) | Enable TLS for the connection. | + +{% hint style="warning" %} +Credentials are stored securely and used only to connect the sink process to your database. Use a dedicated database user with write access restricted to the target schema. +{% endhint %} + +### 4. Execution + +| Field | Description | +|---|---| +| Start block | Block number to start processing from. Use `0` to start from genesis. | +| Stop block | Block number to stop at. Use `0` to follow the chain head indefinitely. | +| Output module | The name of the Substreams module whose output is written to the database (e.g., `db_out`). | + +**Advanced options** (optional): + +| Field | Description | +|---|---| +| Filters | A filter expression passed to the Substreams execution. | +| Parameters | Module parameters passed at execution time. | + +### 5. Deploy + +Click **Deploy sink**. The sink enters the `Deploying` state while the infrastructure provisions. Once pods are ready it transitions to `Deployed` and begins indexing. + +## Next Steps + +- [Monitor and manage your sink](./manage-your-sink.md) — check status, view logs, edit config, stop, or delete. diff --git a/docs/how-to-guides/sinks/hosted-sinks/manage-your-sink.md b/docs/how-to-guides/sinks/hosted-sinks/manage-your-sink.md new file mode 100644 index 000000000..dff7cf926 --- /dev/null +++ b/docs/how-to-guides/sinks/hosted-sinks/manage-your-sink.md @@ -0,0 +1,93 @@ +# Managing Your Hosted Sink + +Once a sink is deployed, the sink detail page gives you visibility into its runtime state and controls to manage its lifecycle. + +## Sink Detail Page + +Navigate to **Sinks** and click a sink to open its detail page. The page has three main sections: the runtime summary, the pod list, and the deployment event log. + +![](<../../../assets/sink-details-graph-market.png>) + +### Runtime Summary + +The summary strip shows four metrics at a glance: + +| Metric | Meaning | +|---|---| +| **Replicas** | `ready / total` — how many replicas are ready vs. requested. | +| **Instances** | Number of active pods. | +| **Restarts** | Total container restarts across all pods. Elevated counts indicate instability. | +| **Output module** | The Substreams module being executed, and the package source URL. | + +### Pod List + +Each pod row shows: + +- **Status badge** — the execution state of that pod (see [Execution States](#execution-states) below). +- **Current block / Head block** — how far behind the chain head the pod is. +- **Head block time drift** — seconds behind real-time. Near zero means the pod is live. +- **Restart count** — restarts for this individual pod. +- **Logs** — click to view the latest container logs for that pod. + +### Deployment Events + +The events log records lifecycle changes to the deployment (e.g., created, config updated, stopped). Use it to audit changes or troubleshoot issues. + +## Status Reference + +### Deployment Status + +The status badge in the page header reflects the overall deployment state. + +| Status | Meaning | +|---|---| +| **Deploying** | Pods are starting up; replicas requested but not yet ready. | +| **Deployed** | All requested replicas are running. | +| **Stopped** | Replica count is set to 0; no pods are running. | +| **Error** | The deployment encountered a hard failure. | +| **Deleting** | The deployment is being torn down. | + +### Execution States + +Pod-level execution states show how far along the indexing process a running pod is. + +| State | Meaning | +|---|---| +| **Initiating** | Pod started; sink process is initializing. | +| **Catching up** | Actively processing historical blocks behind the chain head. | +| **Live** | Caught up to the chain head; processing new blocks as they arrive. | +| **Failing** | Pod encountered an error during processing. Check logs. | + +{% hint style="info" %} +A `CrashLoopBackOff` banner appears at the top of the detail page when one or more pods are repeatedly crashing. This usually indicates a database connection issue or a schema mismatch. Check pod logs for the root cause. +{% endhint %} + +## Editing a Sink + +Click **Edit** in the top-right toolbar to open the edit modal. Changes take effect after you save — the sink process restarts automatically. + +### Deployment tab + +Update the Substreams package source (URL, Substreams.dev ID, or upload a new binary) and the execution config (start block, stop block, output module, filters, parameters). + +### Output tab + +Update the database connection details (host, port, credentials, SSL mode). Useful when your database credentials rotate or you migrate to a new host. + +## Stopping and Starting a Sink + +Click **Stop** to scale the deployment to zero replicas. The sink process halts and no data is written to your database, but the deployment config is preserved. + +To restart, click **Start**. The sink resumes from its last cursor position — no data is re-indexed from scratch unless you explicitly reset (see below). + +## Resetting a Sink + +To re-index from a specific block, edit the sink and set a new **Start block** with the **Restart from scratch** option enabled. This clears the cursor and drops or truncates the existing data depending on the reset mode chosen. + +{% hint style="warning" %} +Resetting is irreversible. All previously indexed data in the target schema will be removed. +{% endhint %} + +## Deleting a Sink + +Click **Delete** in the toolbar and confirm. The deployment and all associated pods are permanently removed. Your database and its data are not affected — only the hosted sink process is deleted. diff --git a/docs/how-to-guides/sinks/sinks.md b/docs/how-to-guides/sinks/sinks.md index bdf53a3db..1e43667e6 100644 --- a/docs/how-to-guides/sinks/sinks.md +++ b/docs/how-to-guides/sinks/sinks.md @@ -1,18 +1,15 @@ Once you find a package that fits your needs, you can choose how you want to consume the data. Sinks are integrations that allow you to send the extracted data to different destinations, such as a SQL database, or a file. {% hint style="info" %} -**Note**: Some of the sinks are officially supported by StreamingFast (i.e. active support is provided), but other sinks are community-driven and support can't be guaranteed. +**Note**: Some of the sinks are officially supported by StreamingFast (i.e. active support is provided), but other sinks are community-driven and support can’t be guaranteed. {% endhint %} +- [Hosted Sinks](./hosted-sinks/hosted-sinks.md): Let StreamingFast run your sink for you — no infrastructure to manage. - [SQL Database](./sql/sql.md): Send the data to a database. - [Direct Streaming](./stream/stream.md): Stream data directly from your application. - [PubSub](./pubsub.md): Send data to a PubSub topic. - [Community Sinks](../sinks/community): Explore quality community maintained sinks. -{% hint style="success" %} -**Deployable Service**: If you’d like your sink (e.g., SQL or PubSub) to be hosted for you, reach out to the StreamingFast team [here](mailto:sales@streamingfast.io). -{% endhint %} - ## Navigating Sink Repos ### Official