Core: Add metadata table for discovering metadata tables by yangshangqing95 · Pull Request #17134 · apache/iceberg

yangshangqing95 · 2026-07-07T20:26:26Z

Summary

This PR adds a new Iceberg metadata table, metadata_tables, to make available metadata tables discoverable through the existing metadata table mechanism.

For example, in Spark users can query:

SELECT * FROM catalog.db.table.metadata_tables;

Result:

entries
files
data_files
delete_files
history
metadata_log_entries
snapshots
refs
manifests
partitions
all_data_files
all_delete_files
all_files
all_manifests
all_entries
position_deletes
metadata_tables

The new metadata table returns the supported metadata table names

This avoids requiring users to know all metadata table names ahead of time from documentation or source code.

Also, replace selected hard-coded metadata table suffixes in tests and metadata table implementations with MetadataTableType#tableName, enhance code robustness.

gaborkaszab · 2026-07-09T13:53:16Z


  AllDataFilesTable(Table table) {
-    this(table, table.name() + ".all_data_files");
+    this(table, table.name() + "." + MetadataTableType.ALL_DATA_FILES.tableName());


To reduce the size of this PR, would you mind moving this refactor and all of the similar replacements into a separate PR?

Yes, sure. These refactors are related to and required by the changes in this PR, though. Another option is that I can wait until this PR is merged, then submit the refactors separately in a follow-up PR. Would that work for you?

gaborkaszab

Hi @yangshangqing95 ,
I took a Quick Look and left some comments wrt refactors and the new metadata scan implementation.

A general note: querying static data such as metadata table names through SELECT * FROM catalog.db.table.metadata_tables seems somewhat odd to me. We'd expect the same answer regardless the table we use for the query, right?
Anyway, I recall for Iceberg V4 there are consideration to not allow some of the current metadata table and have new ones because the physical representation of a table changes. So this might make sense to have different answers based on the table's version.
Let's see what others think!

gaborkaszab · 2026-07-09T14:07:46Z

+  }
+
+  private static final Schema METADATA_TABLES_SCHEMA =
+      new Schema(Types.NestedField.required(1, "metadata_table", Types.StringType.get()));


"metadata_table_name" ?

gaborkaszab · 2026-07-09T14:10:30Z

+    return new StaticDataTask(metadataFile, schema(), scan.schema(), rows);
+  }
+
+  private class MetadataTableScan extends BaseMetadataTableScan {


Instead of MetadataTableScan maybe MetadataTablesScan or MetadataTableNameScan

gaborkaszab · 2026-07-09T14:24:31Z

+            .map(metadataType -> StaticDataTask.Row.of(metadataType.tableName()))
+            .toArray(StaticDataTask.Row[]::new);
+
+    DataFile metadataFile =


I randomly checked other metadata tables and they seem to use the following InputFile:
table().io().newInputFile(table().operations().current().metadataFileLocation()),

That’s a good point. For listing the available metadata tables, at least at this stage, we don’t actually need to read any physical file.

Using a directly constructed empty DataFile has a couple of benefits here:

It avoids unnecessary file operations.

It makes the intent clearer in the code, since otherwise it may look like the metadata table information is stored in or read from the table metadata file.

So I think using an empty DataFile is more explicit for this particular metadata table.

gaborkaszab · 2026-07-09T14:25:08Z

+            .withFileSizeInBytes(0)
+            .build();
+
+    return new StaticDataTask(metadataFile, schema(), scan.schema(), rows);


Other metadata tables use StaticDataTask.of()

same reason as the above one

gaborkaszab · 2026-07-09T14:30:14Z

+  }
+
+  private DataTask task(BaseTableScan scan) {
+    StaticDataTask.Row[] rows =


Checking RefsTable instead of constructing StaticDataTask.Row[]', we can simply use Collectionwith a transform function to convert it to row viaStaticDataTask.Row.of()`

Hi @gaborkaszab same reason with the usage of empty Datafile, as constructor accept parameter Row[]

StaticDataTask( DataFile metadataFile, Schema tableSchema, Schema projectedSchema, StructLike[] rows)

gaborkaszab · 2026-07-09T14:34:08Z

+  }
+
+  private class MetadataTableScan extends BaseMetadataTableScan {
+    MetadataTableScan(Table table, Schema schema) {


I don't think any of these constructors need the Schema parameter. See RefsTable.RefsTableScan

make sense, fixed

gaborkaszab · 2026-07-09T14:41:20Z

+    protected CloseableIterable<FileScanTask> doPlanFiles() {
+      return CloseableIterable.withNoopClose(task(this));
+    }
+  }


After reding this Scan class and comparing to others can't we simply do what PartitionsScan does?

PartitionsScan(Table table) { super( table, PartitionsTable.this.schema(), MetadataTableType.PARTITIONS, PartitionsTable.this::task);

make sense, fixed

gaborkaszab · 2026-07-09T14:42:43Z


      // Expect that the new data is changed in the meantime
-      assertThat(deserializedFiles).isNotEqualTo(newFiles);
+      if (type == MetadataTableType.METADATA_TABLES) {


not entirely sure why the new table is any different than the other in this test.

Since reading the metadata tables does not require reading any actual files, the file path would remain unchanged regardless of how many commits are made. Therefore, this case needs to be handled separately.

gaborkaszab · 2026-07-09T14:43:43Z

            Set.of(DataFile.FIRST_ROW_ID.fieldId()));
    Schema filesTableSchema =
-        Spark3Util.loadIcebergTable(spark, tableName + ".all_data_files").schema();
+        Spark3Util.loadIcebergTable(


These changes are pure refactors. Move to separate PR to reduce the noise as advised above.

sure for the refactors I can submit a separate one when this PR get merged.

yangshangqing95 · 2026-07-09T18:57:17Z

Hi @yangshangqing95 , I took a Quick Look and left some comments wrt refactors and the new metadata scan implementation.

A general note: querying static data such as metadata table names through SELECT * FROM catalog.db.table.metadata_tables seems somewhat odd to me. We'd expect the same answer regardless the table we use for the query, right? Anyway, I recall for Iceberg V4 there are consideration to not allow some of the current metadata table and have new ones because the physical representation of a table changes. So this might make sense to have different answers based on the table's version. Let's see what others think!

Hi @gaborkaszab thanks for taking a look and leaving the comments!

Regarding the metadata table query, I did consider this initially as well. From both an implementation perspective and the current project structure, I’m still leaning toward exposing it under each table, mainly for a few reasons:

We don’t currently have a default metadata catalog to store this kind of information, and introducing one just for this metadata feels a bit heavy.
Given the current code structure, putting this in MetadataTableType keeps the change smaller and less intrusive.
There may be more metadata table types in the future, and the supported set could differ across Iceberg versions such as V1, V2, V3, and V4. Keeping this under each table leaves room for future extension, where each table version can expose the metadata tables that are meaningful for it.

So from that perspective, it may not necessarily be true that all tables should always return the exact same metadata tables.

github-actions Bot added spark core docs labels Jul 7, 2026

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch 6 times, most recently from 44676e5 to 4c79db9 Compare July 8, 2026 19:59

gaborkaszab reviewed Jul 9, 2026

View reviewed changes

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch from 4c79db9 to fbd4c32 Compare July 9, 2026 20:17

Core: Add metadata table for discovering metadata tables

ef31403

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch from fbd4c32 to ef31403 Compare July 9, 2026 20:23

Uh oh!

Conversation

yangshangqing95 commented Jul 7, 2026

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yangshangqing95 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants