README.md

ETL

Build real-time Postgres replication applications in Rust
Documentation · Examples · Issues

ETL is a Rust framework by Supabase for building high‑performance, real‑time data replication apps on Postgres. It sits on top of Postgres logical replication and gives you a clean, Rust‑native API for streaming changes to your own destinations.

Highlights

• Real‑time replication: stream changes in real time to your own destinations.
• High performance: configurable batching and parallelism to maximize throughput.
• Fault-tolerant: robust error handling and retry logic built-in.
• Extensible: implement your own custom destinations and state/schema stores.
• Rust native: typed and ergonomic Rust API.

Requirements

PostgreSQL Version: ETL officially supports and tests against PostgreSQL 14, 15, 16, and 17.

• PostgreSQL 15+ is recommended for access to advanced publication features including:
  • Column-level filtering
  • Row-level filtering with WHERE clauses
  • FOR ALL TABLES IN SCHEMA syntax

For detailed configuration instructions, see the Configure Postgres documentation.

Get Started

Install via Git while we prepare for a crates.io release:

[dependencies]
etl = { git = "https://github.com/supabase/etl" }

Quick example using the in‑memory destination:

use etl::{
    config::{BatchConfig, PgConnectionConfig, PipelineConfig, TlsConfig},
    destination::memory::MemoryDestination,
    pipeline::Pipeline,
    store::both::memory::MemoryStore,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pg = PgConnectionConfig {
        host: "localhost".into(),
        port: 5432,
        name: "mydb".into(),
        username: "postgres".into(),
        password: Some("password".into()),
        tls: TlsConfig { enabled: false, trusted_root_certs: String::new() },
    };

    let store = MemoryStore::new();
    let destination = MemoryDestination::new();

    let config = PipelineConfig {
        id: 1,
        publication_name: "my_publication".into(),
        pg_connection: pg,
        batch: BatchConfig { max_size: 1000, max_fill_ms: 5000 },
        table_error_retry_delay_ms: 10_000,
        table_error_retry_max_attempts: 5,
        max_table_sync_workers: 4,
    };

    let mut pipeline = Pipeline::new(config, store, destination);
    pipeline.start().await?;
    // pipeline.wait().await?; // Optional: block until completion

    Ok(())
}

For tutorials and deeper guidance, see the Documentation or jump into the examples.

Destinations

ETL is designed to be extensible. You can implement your own destinations to send data to any destination you like, however it comes with a few built in destinations:

• BigQuery

Out-of-the-box destinations are available in the etl-destinations crate:

[dependencies]
etl = { git = "https://github.com/supabase/etl" }
etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] }

License

Apache‑2.0. See LICENSE for details.

---

Made with ❤️ by the Supabase team