SQLFlow is an open-source, metadata-driven automation framework designed to eliminate the complexities of modern data warehouse management, pipeline automation, and data governance. Unlike traditional ETL frameworks, SQLFlow features built-in schema evolution, meaning it automatically adapts to changing data structures—ensuring a hassle-free, zero-maintenance data delivery experience.
With SQLFlow, data engineers and architects can build scalable, resilient, and future-proof data pipelines without the constant overhead of manual schema updates or pipeline disruptions. SQLFlow is fully containerized and available on Docker Hub, making deployment seamless across environments.
🔹 Automatic Schema Evolution – Unlike other frameworks, SQLFlow dynamically adjusts to incoming schema changes, eliminating the need for manual intervention.
🔹 Zero Maintenance Pipelines – Say goodbye to pipeline failures caused by evolving data structures.
🔹 Metadata-Driven Simplicity – Automate everything—from data ingestion to governance—without writing custom scripts for every change.
🔹 Scalability at Its Core – SQLFlow grows with your data, ensuring seamless performance at any scale.
SQLFlow is actively used by organizations optimizing large-scale data operations:
- Kolumbus AS – Norway’s public transportation leader, managing data for buses, boats, and trams.
- Nortura SA – Norway’s largest food supplier (23.545B NOK revenue), where SQLFlow supported critical data operations.
Whether you have questions, issues, or success stories, feel free to reach out. Together, let's redefine data warehousing.
This Slack workspace allows developers, data engineers, and contributors to discuss SQLFlow, ask questions, and share insights.
Click the link to join the workspace: Join SQLFlow Community Slack
- Email: tahir@sqlflow.io
- LinkedIn: Contact
Eliminate manual tasks by leveraging metadata to automate the end-to-end data warehouse processes, significantly increasing operational efficiency.
Automatically generate and manage data pipelines from defined metadata, ensuring seamless and error-free data flow from source systems to data targets.
Automate schema creation, updates, and versioning dynamically, allowing teams to focus on data strategy rather than operational details.
Utilize built-in transformation capabilities defined via metadata to perform complex, consistent, and maintainable data cleansing, enrichment, and standardization.
Gain comprehensive visibility into data flow, origin, and transformations with automated lineage tracking. This capability significantly improves data governance, regulatory compliance, and enables proactive impact analysis.
Leverage Azure Container Instances and Ingress for responsive vertical scaling of SQLFlow APIs, optimizing performance and resource allocation during peak demand.
Integrate machine learning techniques for proactive monitoring of data quality, anomaly detection, and historical trend analysis, ensuring high data integrity and reliability.
Precisely monitor changes, including deletions, in source systems through the powerful MatchKey feature, offering customizable grace periods and flexible data lifecycle management aligned with business rules and governance policies.
Facilitate data pipeline development and collaboration by integrating seamlessly with popular source control systems, streamlining pipeline versioning and change tracking.
Ensure fast, reliable performance through pipelines built upon industry best practices, complemented by comprehensive logging for efficient debugging and tuning.
Easily onboard and maximize SQLFlow with detailed documentation covering everything from basic setup to advanced features.
Download and run the powershell script to get SQLFlow up and running with zero manual configuration. Access SQLFlow at http://localhost:8110 or https://localhost:8111 using the default credentials (demo@sqlflow.io/@Demo123).
.\setup.ps1chmod +x setup.sh
./setup.shThis guide will walk you through setting up SQLFlow using Docker containers.
Your contributions help us grow. You can:
- Blog about it
- Comment on issues
- Fork the repository
- Star the project
- Share with peers
- Report bugs and feature requests
SQLFlow is licensed under the GNU General Public License v3.0. For detailed terms, refer to the LICENSE file or visit gnu.org/licenses/gpl-3.0.
An alternative version under the permissive MIT License is available upon request. This license allows reuse in proprietary applications. Please contact us for more details.