Apache Iceberg version
main @ 3038fde
Query engine
N/A (catalog-level, engine-agnostic)
Please describe the bug
SnowflakeCatalog.newTableOps(TableIdentifier) creates a new FileIO on every table load and unconditionally registers it with closeableGroup (snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java line 255-256, before this fix). CloseableGroup backs its closeable list with a Deque that only grows until the whole group is closed, so a long-lived SnowflakeCatalog instance that repeatedly loads or refreshes tables (a REST catalog server, a long-running Spark session) accumulates one FileIO per load, holding native resources (S3/Azure/GCS clients, connection pools) until the catalog itself is closed.
GlueCatalog.newTableOps(TableIdentifier) (aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java) has the same "new FileIO per table load" design but already solves this with core's FileIOTracker (core/src/main/java/org/apache/iceberg/io/FileIOTracker.java), which tracks each FileIO against its TableOperations with a weak-keyed cache and closes it once the TableOperations is no longer reachable.
Steps to reproduce
- Create one
SnowflakeCatalog instance.
- Call
loadTable() (or newTableOps()) repeatedly, including reloading the same table.
- Expected: unreferenced
FileIO instances get reclaimed independently of catalog lifetime, matching GlueCatalog. Actual: every call adds a FileIO to closeableGroup that is never released until catalog.close().
Additional context
N/A, covered above.
Apache Iceberg version
main @ 3038fde
Query engine
N/A (catalog-level, engine-agnostic)
Please describe the bug
SnowflakeCatalog.newTableOps(TableIdentifier)creates a newFileIOon every table load and unconditionally registers it withcloseableGroup(snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.javaline 255-256, before this fix).CloseableGroupbacks its closeable list with aDequethat only grows until the whole group is closed, so a long-livedSnowflakeCataloginstance that repeatedly loads or refreshes tables (a REST catalog server, a long-running Spark session) accumulates oneFileIOper load, holding native resources (S3/Azure/GCS clients, connection pools) until the catalog itself is closed.GlueCatalog.newTableOps(TableIdentifier)(aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java) has the same "new FileIO per table load" design but already solves this with core'sFileIOTracker(core/src/main/java/org/apache/iceberg/io/FileIOTracker.java), which tracks eachFileIOagainst itsTableOperationswith a weak-keyed cache and closes it once theTableOperationsis no longer reachable.Steps to reproduce
SnowflakeCataloginstance.loadTable()(ornewTableOps()) repeatedly, including reloading the same table.FileIOinstances get reclaimed independently of catalog lifetime, matchingGlueCatalog. Actual: every call adds aFileIOtocloseableGroupthat is never released untilcatalog.close().Additional context
N/A, covered above.