Skip to content

Docs: clarify SparkSessionCatalog function limitations #15736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevinjqliu merged 3 commits into from
Mar 26, 2026
+18 −0
Merged

Docs: clarify SparkSessionCatalog function limitations #15736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c4da376
Docs: clarify SparkSessionCatalog function limitations
jackylee-ch Mar 23, 2026
b8d41a7
Docs: explain SparkSessionCatalog V2Function limitation
jackylee-ch Mar 24, 2026
ccb9dae
Docs: fix comments
jackylee-ch Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/docs/spark-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,16 @@ Spark's built-in catalog supports existing v1 and v2 tables tracked in a Hive Me

This configuration can use same Hive Metastore for both Iceberg and non-Iceberg tables.

`SparkSessionCatalog` is useful when you want `spark_catalog` to work with both Iceberg and non-Iceberg
tables in the same metastore.

!!! note
Spark before 4.2.0 does not support `V2Function` in the session catalog. See
[SPARK-54760](https://issues.apache.org/jira/browse/SPARK-54760) ([apache/spark#53531](https://github.com/apache/spark/pull/53531)) for details. As a result,
catalog-scoped SQL functions such as `system.bucket`, `system.days`, and `system.iceberg_version`
are not available through `spark_catalog`. To work around this limitation, configure a separate
Iceberg catalog with `org.apache.iceberg.spark.SparkCatalog` and call them through that catalog.

### Using catalog specific Hadoop configuration values

Similar to configuring Hadoop properties by using `spark.hadoop.*`, it's possible to set per-catalog Hadoop configuration values when using Spark by adding the property for the catalog with the prefix `spark.sql.catalog.(catalog-name).hadoop.*`. These properties will take precedence over values configured globally using `spark.hadoop.*` and will only affect Iceberg tables.
Expand Down
8 changes: 8 additions & 0 deletions docs/docs/spark-queries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,14 @@ writing filters that match Iceberg partition transforms. These functions are ava
[Iceberg catalog](spark-configuration.md#catalog-configuration); they are not registered in Spark's
built-in catalog.

!!! note
Copy link
Copy Markdown
Contributor

@kevinjqliu kevinjqliu Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2026-03-25 at 10 49 47 AM

Spark before 4.2.0 does not support `V2Function` in the session catalog.
Queries such as `SELECT spark_catalog.system.bucket(16, id)` fail even when
`spark_catalog` is configured with `org.apache.iceberg.spark.SparkSessionCatalog`.
See [SPARK-54760](https://issues.apache.org/jira/browse/SPARK-54760) ([apache/spark#53531](https://github.com/apache/spark/pull/53531)) for details.
To use Iceberg SQL functions, call them through a catalog configured with
`org.apache.iceberg.spark.SparkCatalog`.

Use the `system` namespace when calling these functions:

```sql
Expand Down
Loading