Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@
* [Solana - SPL Initialized Account](how-to-guides/composing-substreams/foundational-stores/solana/spl-initialized-account.md)
* [Published Packages](how-to-guides/composing-substreams/published-packages.md)
* [Consuming Substreams](how-to-guides/sinks/sinks.md)
* [Hosted Sinks](how-to-guides/sinks/hosted-sinks/hosted-sinks.md)
* [Managing Your Sink](how-to-guides/sinks/hosted-sinks/manage-your-sink.md)
* [Substreams:SQL](how-to-guides/sinks/sql/sql.md)
* [Using Relational Mappings](how-to-guides/sinks/sql/relational-mappings.md)
* [Using Database Changes](how-to-guides/sinks/sql/db_out.md)
Expand Down
Binary file added docs/assets/add-sinks-graph-market.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/sink-details-graph-market.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
92 changes: 92 additions & 0 deletions docs/how-to-guides/sinks/hosted-sinks/hosted-sinks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Hosted Sinks

**Hosted Sinks** is a managed service on [The Graph Market](https://thegraph.market) that runs your Substreams sink for you. Instead of provisioning servers, managing deployments, or operating sink processes yourself, you configure a sink in the portal and StreamingFast handles the infrastructure.

A hosted sink continuously reads data from your Substreams package and writes it to your own Postgres or ClickHouse database.

{% hint style="info" %}
Hosted Sinks is available to organizations on The Graph Market. Your database must be publicly reachable from StreamingFast's infrastructure.
{% endhint %}

## Requirements

Before creating a sink, you need:

- An account on [The Graph Market](https://thegraph.market) with an active organization.
- A Substreams package (`.spkg`) that outputs data using a `db_out` module or relational mappings. See [Substreams:SQL](../sql/sql.md) for how to build one.
- A running Postgres (default port `5432`) or ClickHouse (default port `9000`) database that is network-accessible from the internet.
- Your database schema already applied, or a schema that your Substreams package will create automatically. See [Sink Config](../../../references/sql/sink-config.md).

## Creating a Sink

Navigate to [**Sinks**](https://thegraph.market/sinks) in the sidebar and click **New Sink**.

![](<../../../assets/add-sinks-graph-market.png>)

### 1. Basics

Enter a **Sink name**. This is a human-readable label for the deployment (e.g., `mainnet-erc20-transfers`).

### 2. Substreams Package

Choose how you want to supply the `.spkg`:

| Source | When to use |
|---|---|
| **URL** | The package is hosted at a public URL (e.g., GitHub releases, IPFS). |
| **Substreams.dev** | The package is published on [the Substreams Registry](https://substreams.dev). Enter its ID (e.g., `ethereum_common@v0.3.3`). |
| **Binary upload** | Upload a `.spkg` file directly from your machine. |

### 3. Output

Choose **Postgres** or **ClickHouse** and fill in the connection details.

**Postgres fields:**

| Field | Description |
|---|---|
| Host | Database hostname or IP address. |
| Port | Default `5432`. |
| Database | Database name. |
| Schema | Target schema. Default `public`. |
| User | Database user with write access. |
| Password | Password for the user. |
| SSL Mode | One of `disable`, `require`, `verify-ca`, or `verify-full`. |

**ClickHouse fields:**

| Field | Description |
|---|---|
| Host | ClickHouse hostname or IP address. |
| Port | Default `9000` (native protocol). |
| Database | Database name. Default `default`. |
| User | ClickHouse user. |
| Password | Password for the user. |
| Secure (TLS) | Enable TLS for the connection. |

{% hint style="warning" %}
Credentials are stored securely and used only to connect the sink process to your database. Use a dedicated database user with write access restricted to the target schema.
{% endhint %}

### 4. Execution

| Field | Description |
|---|---|
| Start block | Block number to start processing from. Use `0` to start from genesis. |
| Stop block | Block number to stop at. Use `0` to follow the chain head indefinitely. |
| Output module | The name of the Substreams module whose output is written to the database (e.g., `db_out`). |

**Advanced options** (optional):

| Field | Description |
|---|---|
| Filters | A filter expression passed to the Substreams execution. |
| Parameters | Module parameters passed at execution time. |

### 5. Deploy

Click **Deploy sink**. The sink enters the `Deploying` state while the infrastructure provisions. Once pods are ready it transitions to `Deployed` and begins indexing.

## Next Steps

- [Monitor and manage your sink](./manage-your-sink.md) — check status, view logs, edit config, stop, or delete.
93 changes: 93 additions & 0 deletions docs/how-to-guides/sinks/hosted-sinks/manage-your-sink.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Managing Your Hosted Sink

Once a sink is deployed, the sink detail page gives you visibility into its runtime state and controls to manage its lifecycle.

## Sink Detail Page

Navigate to **Sinks** and click a sink to open its detail page. The page has three main sections: the runtime summary, the pod list, and the deployment event log.

![](<../../../assets/sink-details-graph-market.png>)

### Runtime Summary

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, screenshots or URLs minimally


The summary strip shows four metrics at a glance:

| Metric | Meaning |
|---|---|
| **Replicas** | `ready / total` — how many replicas are ready vs. requested. |
| **Instances** | Number of active pods. |
| **Restarts** | Total container restarts across all pods. Elevated counts indicate instability. |
| **Output module** | The Substreams module being executed, and the package source URL. |

### Pod List

Each pod row shows:

- **Status badge** — the execution state of that pod (see [Execution States](#execution-states) below).
- **Current block / Head block** — how far behind the chain head the pod is.
- **Head block time drift** — seconds behind real-time. Near zero means the pod is live.
- **Restart count** — restarts for this individual pod.
- **Logs** — click to view the latest container logs for that pod.

### Deployment Events

The events log records lifecycle changes to the deployment (e.g., created, config updated, stopped). Use it to audit changes or troubleshoot issues.

## Status Reference

### Deployment Status

The status badge in the page header reflects the overall deployment state.

| Status | Meaning |
|---|---|
| **Deploying** | Pods are starting up; replicas requested but not yet ready. |
| **Deployed** | All requested replicas are running. |
| **Stopped** | Replica count is set to 0; no pods are running. |
| **Error** | The deployment encountered a hard failure. |
| **Deleting** | The deployment is being torn down. |

### Execution States

Pod-level execution states show how far along the indexing process a running pod is.

| State | Meaning |
|---|---|
| **Initiating** | Pod started; sink process is initializing. |
| **Catching up** | Actively processing historical blocks behind the chain head. |
| **Live** | Caught up to the chain head; processing new blocks as they arrive. |
| **Failing** | Pod encountered an error during processing. Check logs. |

{% hint style="info" %}
A `CrashLoopBackOff` banner appears at the top of the detail page when one or more pods are repeatedly crashing. This usually indicates a database connection issue or a schema mismatch. Check pod logs for the root cause.
{% endhint %}

## Editing a Sink

Click **Edit** in the top-right toolbar to open the edit modal. Changes take effect after you save — the sink process restarts automatically.

### Deployment tab

Update the Substreams package source (URL, Substreams.dev ID, or upload a new binary) and the execution config (start block, stop block, output module, filters, parameters).

### Output tab

Update the database connection details (host, port, credentials, SSL mode). Useful when your database credentials rotate or you migrate to a new host.

## Stopping and Starting a Sink

Click **Stop** to scale the deployment to zero replicas. The sink process halts and no data is written to your database, but the deployment config is preserved.

To restart, click **Start**. The sink resumes from its last cursor position — no data is re-indexed from scratch unless you explicitly reset (see below).

## Resetting a Sink

To re-index from a specific block, edit the sink and set a new **Start block** with the **Restart from scratch** option enabled. This clears the cursor and drops or truncates the existing data depending on the reset mode chosen.

{% hint style="warning" %}
Resetting is irreversible. All previously indexed data in the target schema will be removed.
{% endhint %}

## Deleting a Sink

Click **Delete** in the toolbar and confirm. The deployment and all associated pods are permanently removed. Your database and its data are not affected — only the hosted sink process is deleted.
7 changes: 2 additions & 5 deletions docs/how-to-guides/sinks/sinks.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
Once you find a package that fits your needs, you can choose how you want to consume the data. Sinks are integrations that allow you to send the extracted data to different destinations, such as a SQL database, or a file.

{% hint style="info" %}
**Note**: Some of the sinks are officially supported by StreamingFast (i.e. active support is provided), but other sinks are community-driven and support can't be guaranteed.
**Note**: Some of the sinks are officially supported by StreamingFast (i.e. active support is provided), but other sinks are community-driven and support cant be guaranteed.
{% endhint %}

- [Hosted Sinks](./hosted-sinks/hosted-sinks.md): Let StreamingFast run your sink for you — no infrastructure to manage.
- [SQL Database](./sql/sql.md): Send the data to a database.
- [Direct Streaming](./stream/stream.md): Stream data directly from your application.
- [PubSub](./pubsub.md): Send data to a PubSub topic.
- [Community Sinks](../sinks/community): Explore quality community maintained sinks.

{% hint style="success" %}
**Deployable Service**: If you’d like your sink (e.g., SQL or PubSub) to be hosted for you, reach out to the StreamingFast team [here](mailto:sales@streamingfast.io).
{% endhint %}

## Navigating Sink Repos

### Official
Expand Down
Loading