Skip to content

Snowflake: Avoid unbounded FileIO accumulation in SnowflakeCatalog #17095

Description

@thswlsqls

Apache Iceberg version
main @ 3038fde

Query engine
N/A (catalog-level, engine-agnostic)

Please describe the bug
SnowflakeCatalog.newTableOps(TableIdentifier) creates a new FileIO on every table load and unconditionally registers it with closeableGroup (snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java line 255-256, before this fix). CloseableGroup backs its closeable list with a Deque that only grows until the whole group is closed, so a long-lived SnowflakeCatalog instance that repeatedly loads or refreshes tables (a REST catalog server, a long-running Spark session) accumulates one FileIO per load, holding native resources (S3/Azure/GCS clients, connection pools) until the catalog itself is closed.

GlueCatalog.newTableOps(TableIdentifier) (aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java) has the same "new FileIO per table load" design but already solves this with core's FileIOTracker (core/src/main/java/org/apache/iceberg/io/FileIOTracker.java), which tracks each FileIO against its TableOperations with a weak-keyed cache and closes it once the TableOperations is no longer reachable.

Steps to reproduce

  1. Create one SnowflakeCatalog instance.
  2. Call loadTable() (or newTableOps()) repeatedly, including reloading the same table.
  3. Expected: unreferenced FileIO instances get reclaimed independently of catalog lifetime, matching GlueCatalog. Actual: every call adds a FileIO to closeableGroup that is never released until catalog.close().

Additional context
N/A, covered above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions