Skip to content

[FEATURE] databricks.sdk.runtime should provide a pipelines wrapper that exposes the differences from Apache Spark and LDP #1491

Description

@tigerhawkvok

The use case is straightforward enough -- if I want to develop locally to take advantage of local speed/improved introspection/other agents/different keyboard commands/you name it, the minor API differences are difficult to introspect and raise static linter errors.

A trivial example would be the private kwarg in materialized views for Lakeflow Declarative Pipelines which is not part of Apache Spark.

Suppression of the local error with, say, # ty:ignore[no-matching-overload] can obscure actual errors.

The low-hanging-fruit version of this would be a single stub function that just exposes the correct argument availability and passes them through blindly, which would still be an improvement over the current state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions