2.0.0b1 - Beta Release for Spark Connect and a new DuckDB engine

PyDeequ 2.0.0b1 - Beta Release

We're excited to announce the first beta release of PyDeequ 2.0, a major architectural overhaul that introduces Spark Connect support!

What's New

• PyDeequ 2.0 moves away from the Py4J-based JVM bridge to a modern client-server architecture using Spark Connect (introduced in Spark 3.4+). See #254 for the code changes.
• Multi-engine support with DuckDB backend. This is #255.

Call for Testing

This is a beta release intended for getting early feedback and is NOT production ready.

• Please follow the testing setup for local testing.

• End-to-end examples:

  • https://github.com/awslabs/python-deequ/blob/v2_engine/tutorials/data_quality_example_duckdb.py
  • https://github.com/awslabs/python-deequ/blob/v2_engine/tutorials/data_quality_example_v2.py

• Please create issues using the v2_beta label.

v1.4.0

What's Changed

• Adding support for Spark 3.5 by @rdsharma26 in #210

Full Changelog: v1.3.0...v1.4.0

v1.3.0

What's Changed

Upgrades

• Upgraded the Deequ version to 2.0.7 for Spark versions 3.1 to 3.3 by @rdsharma26 in #200

Fixes

• Fix default assertion for satisfies check. by @poolis in #183

Other Changes

• [Testing] Added a dockerfile for building and testing the package by @rdsharma26 in #195
• [Documentation] FileSystemMetricsRepository sync on DBFS tutorial by @WiktorMadejski in #187

New Contributors

• @poolis made their first contribution in #183
• @WiktorMadejski made their first contribution in #187

Full Changelog: v1.2.0...v1.3.0

Release 1.2.0

Added where filter and fix some minor bugs. See #177

https://pypi.org/project/pydeequ/1.2.0/

Patch release 1.1.1

Bugfix release to fix multiple issues related to hasPattern. Closing #152.

1.1.0

We are pleased to announce a release for python-deequ 1.1.0.

This release candidate brings long-waited Spark and Deequ recency updates (support 3.3.0) and many dependency upgrade.

1.1.0 RC0

We are pleased to announce a release candidate for python-deequ 1.1.0. If all goes well, we'll release 1.1.0 in about two weeks. This release candidate brings long-waited Spark and Deequ recency updates (support 3.3.0) and many dependency upgrade.

The release can be installed from PyPI

python -m pip install --upgrade --pre pydeequ==1.1.0rc0

See #106 for more context and what's new.

PyDeequ: Spark3 support!!!

Features:

• Spark 3 support!
• #49
• #34

PyDeequ

Features

• Implemented hasPattern for regex rules verification
• Implemented alternate method of passing list of constraints for verification