-
Notifications
You must be signed in to change notification settings - Fork 90
docs: add Hosted Sinks documentation #804
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Hosted Sinks | ||
|
|
||
| **Hosted Sinks** is a managed service on [The Graph Market](https://thegraph.market) that runs your Substreams sink for you. Instead of provisioning servers, managing deployments, or operating sink processes yourself, you configure a sink in the portal and StreamingFast handles the infrastructure. | ||
|
|
||
| A hosted sink continuously reads data from your Substreams package and writes it to your own Postgres or ClickHouse database. | ||
|
|
||
| {% hint style="info" %} | ||
| Hosted Sinks is available to organizations on The Graph Market. Your database must be publicly reachable from StreamingFast's infrastructure. | ||
| {% endhint %} | ||
|
|
||
| ## Requirements | ||
|
|
||
| Before creating a sink, you need: | ||
|
|
||
| - An account on [The Graph Market](https://thegraph.market) with an active organization. | ||
| - A Substreams package (`.spkg`) that outputs data using a `db_out` module or relational mappings. See [Substreams:SQL](../sql/sql.md) for how to build one. | ||
| - A running Postgres (default port `5432`) or ClickHouse (default port `9000`) database that is network-accessible from the internet. | ||
| - Your database schema already applied, or a schema that your Substreams package will create automatically. See [Sink Config](../../../references/sql/sink-config.md). | ||
|
|
||
| ## Creating a Sink | ||
|
|
||
| Navigate to [**Sinks**](https://thegraph.market/sinks) in the sidebar and click **New Sink**. | ||
|
|
||
|  | ||
|
|
||
| ### 1. Basics | ||
|
|
||
| Enter a **Sink name**. This is a human-readable label for the deployment (e.g., `mainnet-erc20-transfers`). | ||
|
|
||
| ### 2. Substreams Package | ||
|
|
||
| Choose how you want to supply the `.spkg`: | ||
|
|
||
| | Source | When to use | | ||
| |---|---| | ||
| | **URL** | The package is hosted at a public URL (e.g., GitHub releases, IPFS). | | ||
| | **Substreams.dev** | The package is published on [the Substreams Registry](https://substreams.dev). Enter its ID (e.g., `ethereum_common@v0.3.3`). | | ||
| | **Binary upload** | Upload a `.spkg` file directly from your machine. | | ||
|
|
||
| ### 3. Output | ||
|
|
||
| Choose **Postgres** or **ClickHouse** and fill in the connection details. | ||
|
|
||
| **Postgres fields:** | ||
|
|
||
| | Field | Description | | ||
| |---|---| | ||
| | Host | Database hostname or IP address. | | ||
| | Port | Default `5432`. | | ||
| | Database | Database name. | | ||
| | Schema | Target schema. Default `public`. | | ||
| | User | Database user with write access. | | ||
| | Password | Password for the user. | | ||
| | SSL Mode | One of `disable`, `require`, `verify-ca`, or `verify-full`. | | ||
|
|
||
| **ClickHouse fields:** | ||
|
|
||
| | Field | Description | | ||
| |---|---| | ||
| | Host | ClickHouse hostname or IP address. | | ||
| | Port | Default `9000` (native protocol). | | ||
| | Database | Database name. Default `default`. | | ||
| | User | ClickHouse user. | | ||
| | Password | Password for the user. | | ||
| | Secure (TLS) | Enable TLS for the connection. | | ||
|
|
||
| {% hint style="warning" %} | ||
| Credentials are stored securely and used only to connect the sink process to your database. Use a dedicated database user with write access restricted to the target schema. | ||
| {% endhint %} | ||
|
|
||
| ### 4. Execution | ||
|
|
||
| | Field | Description | | ||
| |---|---| | ||
| | Start block | Block number to start processing from. Use `0` to start from genesis. | | ||
| | Stop block | Block number to stop at. Use `0` to follow the chain head indefinitely. | | ||
| | Output module | The name of the Substreams module whose output is written to the database (e.g., `db_out`). | | ||
|
|
||
| **Advanced options** (optional): | ||
|
|
||
| | Field | Description | | ||
| |---|---| | ||
| | Filters | A filter expression passed to the Substreams execution. | | ||
| | Parameters | Module parameters passed at execution time. | | ||
|
|
||
| ### 5. Deploy | ||
|
|
||
| Click **Deploy sink**. The sink enters the `Deploying` state while the infrastructure provisions. Once pods are ready it transitions to `Deployed` and begins indexing. | ||
|
|
||
| ## Next Steps | ||
|
|
||
| - [Monitor and manage your sink](./manage-your-sink.md) — check status, view logs, edit config, stop, or delete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| # Managing Your Hosted Sink | ||
|
|
||
| Once a sink is deployed, the sink detail page gives you visibility into its runtime state and controls to manage its lifecycle. | ||
|
|
||
| ## Sink Detail Page | ||
|
|
||
| Navigate to **Sinks** and click a sink to open its detail page. The page has three main sections: the runtime summary, the pod list, and the deployment event log. | ||
|
|
||
|  | ||
|
|
||
| ### Runtime Summary | ||
|
|
||
| The summary strip shows four metrics at a glance: | ||
|
|
||
| | Metric | Meaning | | ||
| |---|---| | ||
| | **Replicas** | `ready / total` — how many replicas are ready vs. requested. | | ||
| | **Instances** | Number of active pods. | | ||
| | **Restarts** | Total container restarts across all pods. Elevated counts indicate instability. | | ||
| | **Output module** | The Substreams module being executed, and the package source URL. | | ||
|
|
||
| ### Pod List | ||
|
|
||
| Each pod row shows: | ||
|
|
||
| - **Status badge** — the execution state of that pod (see [Execution States](#execution-states) below). | ||
| - **Current block / Head block** — how far behind the chain head the pod is. | ||
| - **Head block time drift** — seconds behind real-time. Near zero means the pod is live. | ||
| - **Restart count** — restarts for this individual pod. | ||
| - **Logs** — click to view the latest container logs for that pod. | ||
|
|
||
| ### Deployment Events | ||
|
|
||
| The events log records lifecycle changes to the deployment (e.g., created, config updated, stopped). Use it to audit changes or troubleshoot issues. | ||
|
|
||
| ## Status Reference | ||
|
|
||
| ### Deployment Status | ||
|
|
||
| The status badge in the page header reflects the overall deployment state. | ||
|
|
||
| | Status | Meaning | | ||
| |---|---| | ||
| | **Deploying** | Pods are starting up; replicas requested but not yet ready. | | ||
| | **Deployed** | All requested replicas are running. | | ||
| | **Stopped** | Replica count is set to 0; no pods are running. | | ||
| | **Error** | The deployment encountered a hard failure. | | ||
| | **Deleting** | The deployment is being torn down. | | ||
|
|
||
| ### Execution States | ||
|
|
||
| Pod-level execution states show how far along the indexing process a running pod is. | ||
|
|
||
| | State | Meaning | | ||
| |---|---| | ||
| | **Initiating** | Pod started; sink process is initializing. | | ||
| | **Catching up** | Actively processing historical blocks behind the chain head. | | ||
| | **Live** | Caught up to the chain head; processing new blocks as they arrive. | | ||
| | **Failing** | Pod encountered an error during processing. Check logs. | | ||
|
|
||
| {% hint style="info" %} | ||
| A `CrashLoopBackOff` banner appears at the top of the detail page when one or more pods are repeatedly crashing. This usually indicates a database connection issue or a schema mismatch. Check pod logs for the root cause. | ||
| {% endhint %} | ||
|
|
||
| ## Editing a Sink | ||
|
|
||
| Click **Edit** in the top-right toolbar to open the edit modal. Changes take effect after you save — the sink process restarts automatically. | ||
|
|
||
| ### Deployment tab | ||
|
|
||
| Update the Substreams package source (URL, Substreams.dev ID, or upload a new binary) and the execution config (start block, stop block, output module, filters, parameters). | ||
|
|
||
| ### Output tab | ||
|
|
||
| Update the database connection details (host, port, credentials, SSL mode). Useful when your database credentials rotate or you migrate to a new host. | ||
|
|
||
| ## Stopping and Starting a Sink | ||
|
|
||
| Click **Stop** to scale the deployment to zero replicas. The sink process halts and no data is written to your database, but the deployment config is preserved. | ||
|
|
||
| To restart, click **Start**. The sink resumes from its last cursor position — no data is re-indexed from scratch unless you explicitly reset (see below). | ||
|
|
||
| ## Resetting a Sink | ||
|
|
||
| To re-index from a specific block, edit the sink and set a new **Start block** with the **Restart from scratch** option enabled. This clears the cursor and drops or truncates the existing data depending on the reset mode chosen. | ||
|
|
||
| {% hint style="warning" %} | ||
| Resetting is irreversible. All previously indexed data in the target schema will be removed. | ||
| {% endhint %} | ||
|
|
||
| ## Deleting a Sink | ||
|
|
||
| Click **Delete** in the toolbar and confirm. The deployment and all associated pods are permanently removed. Your database and its data are not affected — only the hosted sink process is deleted. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here, screenshots or URLs minimally