loader) {
+ return loader.get();
+ }
+ };
+}
diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java
index 7846b13bd19ef0..1ba13db82c1b64 100644
--- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java
+++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java
@@ -169,16 +169,18 @@ private static String rootCauseMessage(Throwable t) {
*/
HiveConnectorMetadata newMetadata(HmsClient client) {
return new HiveConnectorMetadata(client, properties, context,
- this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwner,
+ this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwnerLabeled,
fileListingCache);
}
/**
- * Resolves which embedded sibling connector OWNS a foreign (non-hive) table handle, for the per-handle
- * gateway seams (the connector-level {@code get*Provider(handle)} below and the ~34 guard-and-forward
- * methods in {@link HiveConnectorMetadata}). Asks each sibling's {@link Connector#ownsHandle} — the sibling
- * tests its OWN in-loader handle type, which is invisible to the hive loader across the plugin split, so the
- * gateway can never {@code instanceof} the foreign handle itself.
+ * Resolves which embedded sibling connector OWNS a foreign (non-hive) table handle AND its stable owner label,
+ * for the per-handle gateway seams (the connector-level {@code get*Provider(handle)} below and the
+ * guard-and-forward methods in {@link HiveConnectorMetadata}). Asks each sibling's {@link Connector#ownsHandle}
+ * — the sibling tests its OWN in-loader handle type, which is invisible to the hive loader across the plugin
+ * split, so the gateway can never {@code instanceof} the foreign handle itself. The MATCHED ARM supplies the
+ * label ({@link SiblingOwner#ICEBERG_LABEL} / {@link SiblingOwner#HUDI_LABEL}), so a per-handle forward keys the
+ * per-statement metadata funnel by owner without a force-build or an identity comparison against the suppliers.
*
* Consults only ALREADY-BUILT siblings (a plain field read, never {@code getOrCreate*}). The owning
* sibling is always already built: a foreign handle can only originate from {@code getTableHandle}'s divert,
@@ -187,19 +189,28 @@ HiveConnectorMetadata newMetadata(HmsClient client) {
* plugin must still route its hudi handles without building an iceberg sibling. Fails loud when no built
* sibling owns the handle (an orphan handle is a bug, not a route), naming the catalog.
*/
- private Connector resolveSiblingOwner(ConnectorTableHandle handle) {
+ SiblingOwner resolveSiblingOwnerLabeled(ConnectorTableHandle handle) {
Connector iceberg = icebergSibling;
if (iceberg != null && iceberg.ownsHandle(handle)) {
- return iceberg;
+ return new SiblingOwner(iceberg, SiblingOwner.ICEBERG_LABEL);
}
Connector hudi = hudiSibling;
if (hudi != null && hudi.ownsHandle(handle)) {
- return hudi;
+ return new SiblingOwner(hudi, SiblingOwner.HUDI_LABEL);
}
throw new DorisConnectorException("Cannot route a foreign table handle in catalog '"
+ context.getCatalogName() + "': no embedded sibling connector owns it");
}
+ /**
+ * The owning sibling {@link Connector} alone (label dropped) for the connector-level per-handle provider seams
+ * ({@code get*Provider(handle)} below), which route by owner but never touch the metadata funnel. Delegates to
+ * {@link #resolveSiblingOwnerLabeled} so the 3-way ownsHandle dispatch has a single implementation.
+ */
+ private Connector resolveSiblingOwner(ConnectorTableHandle handle) {
+ return resolveSiblingOwnerLabeled(handle).connector();
+ }
+
@Override
public ConnectorScanPlanProvider getScanPlanProvider() {
return new HiveScanPlanProvider(getOrCreateClient(), properties, context, readTxnManager, fileListingCache);
@@ -496,6 +507,21 @@ Connector getOrCreateIcebergSibling() {
"Cannot serve iceberg-on-HMS tables in catalog '" + context.getCatalogName()
+ "': the iceberg connector plugin is not available");
}
+ // Fail-loud invariant guard for the cache-isolation security track: the hive gateway FRONT
+ // DOOR never declares SUPPORTS_USER_SESSION, so fe-core keys its per-user schema/name cache
+ // bypass off THIS (front-door) connector's capabilities and would NOT bypass for a delegated
+ // sibling. The iceberg sibling is forced iceberg.catalog.type=hms (IcebergSiblingProperties
+ // .synthesize) and can never be REST session=user, so this must hold today. If a future change
+ // ever let the sibling be session=user, the front-door-only bypass would silently leak
+ // cross-user metadata — fail here instead.
+ if (sibling.getCapabilities().contains(ConnectorCapability.SUPPORTS_USER_SESSION)) {
+ throw new DorisConnectorException(
+ "iceberg-on-HMS sibling in catalog '" + context.getCatalogName()
+ + "' unexpectedly declares SUPPORTS_USER_SESSION: the hive gateway front "
+ + "door is not session=user, so fe-core's per-user schema/name cache bypass "
+ + "would not trigger and cross-user metadata would leak. The sibling must "
+ + "stay iceberg.catalog.type=hms (never REST session=user).");
+ }
icebergSibling = sibling;
}
}
diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
index 9e8ea66551561b..07f50ca76ca4a2 100644
--- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
+++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
@@ -192,7 +192,7 @@ public class HiveConnectorMetadata implements ConnectorMetadata {
// The by-handle owner resolver installed by the 3-arg constructor (hive-only construction). Invoked only when
// a NON-hive handle reaches a per-handle guard-and-forward site — which such a construction never does — so it
// fails loud instead of NPEing.
- private static final Function NO_SIBLING_OWNER = handle -> {
+ private static final Function NO_SIBLING_OWNER = handle -> {
throw new DorisConnectorException("no sibling connector configured for this hive metadata");
};
@@ -218,7 +218,7 @@ public class HiveConnectorMetadata implements ConnectorMetadata {
// guard-and-forward methods below. Backed by HiveConnector.resolveSiblingOwner (a 3-way ownsHandle dispatch
// over the ALREADY-BUILT iceberg / hudi siblings). Used ONLY through the parent-first Connector /
// ConnectorMetadata interfaces — the owning sibling's concrete types are never cast here (cross-loader CCE).
- private final Function siblingOwnerResolver;
+ private final Function siblingOwnerResolver;
// Connector-owned directory-listing cache, shared with the scan provider so estimateDataSizeByListingFiles
// (the periodic ExternalRowCountCache refresh source) reuses listings a scan warmed and vice versa. Injected
// by HiveConnector.newMetadata as the SAME instance; the convenience constructors below build a private
@@ -232,7 +232,7 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties
public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context,
Supplier icebergSiblingSupplier,
Supplier hudiSiblingSupplier,
- Function siblingOwnerResolver) {
+ Function siblingOwnerResolver) {
this(hmsClient, properties, context, icebergSiblingSupplier, hudiSiblingSupplier, siblingOwnerResolver,
new HiveFileListingCache(properties));
}
@@ -240,7 +240,7 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties
public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context,
Supplier icebergSiblingSupplier,
Supplier hudiSiblingSupplier,
- Function siblingOwnerResolver,
+ Function siblingOwnerResolver,
HiveFileListingCache fileListingCache) {
this.hmsClient = hmsClient;
this.properties = properties;
@@ -251,46 +251,69 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties
this.fileListingCache = fileListingCache;
}
+ /**
+ * Obtains the sibling's {@link ConnectorMetadata} through the per-statement funnel: within one statement, the
+ * first forward for a given owner runs {@code owner.getMetadata(session)} and every later forward (read / scan
+ * / DDL / MVCC / the per-handle {@code beginTransaction} open) reuses that ONE memoized instance — mirroring
+ * fe-core's own {@code PluginDrivenMetadata} funnel for a plain connector (the sibling's heavy catalog/caches
+ * live on the single memoized sibling CONNECTOR regardless of this). The key is
+ * {@code "metadata:" + catalogId + ":" + ownerLabel}: the three connectors of a heterogeneous gateway
+ * (hive + iceberg + hudi) share ONE catalogId, so the owner label keeps their metadata entries distinct — a
+ * plain catalogId key would collapse them onto one metadata and misroute. Under a
+ * {@link org.apache.doris.connector.api.ConnectorStatementScope#NONE NONE} scope (offline / no statement) the
+ * factory runs on every call — byte-identical to the pre-funnel behavior. Only fe-connector-api types are
+ * touched, so no fe-core dependency is introduced. The returned metadata and any handle it produces are used
+ * ONLY through the parent-first SPI interfaces and MUST NOT be cast (the sibling's concrete iceberg/hudi types
+ * would CCE across the loader split).
+ */
+ private ConnectorMetadata memoizedSiblingMetadata(ConnectorSession session, Connector owner, String ownerLabel) {
+ String key = "metadata:" + session.getCatalogId() + ":" + ownerLabel;
+ return session.getStatementScope().getOrCreateMetadata(key, () -> owner.getMetadata(session));
+ }
+
/**
* The embedded iceberg sibling's metadata resolved BY TYPE, for the getTableHandle ICEBERG divert only (an
- * iceberg-detected table has no handle yet, so the sibling is force-built and asked directly). Obtained fresh
- * per call — parity with fe-core, which acquires a ConnectorMetadata per operation; the heavy catalog/caches
- * live on the single (memoized) sibling connector, so this is cheap. The returned metadata and any handle it
- * produces are used ONLY through the parent-first SPI interfaces and MUST NOT be cast (the sibling's concrete
- * iceberg types would CCE across the loader split).
+ * iceberg-detected table has no handle yet, so the sibling is force-built and asked directly). Routed through
+ * {@link #memoizedSiblingMetadata} under {@link SiblingOwner#ICEBERG_LABEL}, so the divert and the same
+ * statement's later per-handle forwards share ONE iceberg metadata instance. The returned metadata and any
+ * handle it produces are used ONLY through the parent-first SPI interfaces and MUST NOT be cast.
*
* Package-private (not private) so HiveConnectorThreeWayRoutingTest can assert that
* {@link HiveConnector#getMetadata} wires the iceberg by-TYPE supplier to THIS arm (the two same-typed
* supplier args are otherwise transposable at that sole production wiring point).
*/
ConnectorMetadata icebergSiblingMetadata(ConnectorSession session) {
- return icebergSiblingSupplier.get().getMetadata(session);
+ return memoizedSiblingMetadata(session, icebergSiblingSupplier.get(), SiblingOwner.ICEBERG_LABEL);
}
/**
* The embedded hudi sibling's metadata resolved BY TYPE, for the getTableHandle HUDI divert only (a
* hudi-detected table has no handle yet, so the sibling is force-built and asked directly). Same lifecycle and
- * casting contract as {@link #icebergSiblingMetadata}: obtained fresh per call, used ONLY through the
- * parent-first SPI interfaces, and never cast (the sibling's concrete hudi types would CCE across the loader
- * split).
+ * casting contract as {@link #icebergSiblingMetadata}: routed through {@link #memoizedSiblingMetadata} under
+ * {@link SiblingOwner#HUDI_LABEL}, used ONLY through the parent-first SPI interfaces, and never cast.
*
*
Package-private (not private) so HiveConnectorThreeWayRoutingTest can assert that
* {@link HiveConnector#getMetadata} wires the hudi by-TYPE supplier to THIS arm (see
* {@link #icebergSiblingMetadata}).
*/
ConnectorMetadata hudiSiblingMetadata(ConnectorSession session) {
- return hudiSiblingSupplier.get().getMetadata(session);
+ return memoizedSiblingMetadata(session, hudiSiblingSupplier.get(), SiblingOwner.HUDI_LABEL);
}
/**
* The OWNING sibling's metadata for a foreign (non-hive) table handle, resolved BY HANDLE (3-way ownsHandle
- * dispatch over the already-built iceberg / hudi siblings — see HiveConnector.resolveSiblingOwner). Every
- * per-handle guard-and-forward method routes through here so a hudi handle reaches the hudi sibling and an
- * iceberg handle the iceberg sibling. Obtained fresh per call; the handle is used ONLY through the
- * parent-first SPI interfaces and MUST NOT be cast (cross-loader CCE).
+ * dispatch over the already-built iceberg / hudi siblings — see HiveConnector.resolveSiblingOwnerLabeled).
+ * Every per-handle guard-and-forward method (and the per-handle {@code beginTransaction} open) routes through
+ * here, so a hudi handle reaches the hudi sibling and an iceberg handle the iceberg sibling. The resolver
+ * supplies the owner label from its matched arm, and {@link #memoizedSiblingMetadata} keys the funnel by it —
+ * so a statement's forwards for one owner share ONE metadata instance, and the by-HANDLE label matches the
+ * by-TYPE label above (same owner → same key → the getTableHandle divert and these forwards reuse
+ * one instance). The handle is used ONLY through the parent-first SPI interfaces and MUST NOT be cast
+ * (cross-loader CCE).
*/
private ConnectorMetadata siblingMetadata(ConnectorSession session, ConnectorTableHandle handle) {
- return siblingOwnerResolver.apply(handle).getMetadata(session);
+ SiblingOwner owner = siblingOwnerResolver.apply(handle);
+ return memoizedSiblingMetadata(session, owner.connector(), owner.label());
}
// ========== ConnectorSchemaOps ==========
@@ -425,9 +448,10 @@ public ConnectorTableSchema getTableSchema(
// capabilities govern the table). Only hasScanCapability consumers read the marker, so a capability
// that is not per-table-refinable (view / show-create / mvcc) is inert here. Resolve the owner ONCE
// (getMetadata is not free) and reuse it for the schema build and the capability read.
- Connector owner = siblingOwnerResolver.apply(handle);
- ConnectorTableSchema siblingSchema = owner.getMetadata(session).getTableSchema(session, handle);
- return reflectSiblingScanCapabilities(owner, siblingSchema);
+ SiblingOwner owner = siblingOwnerResolver.apply(handle);
+ ConnectorTableSchema siblingSchema = memoizedSiblingMetadata(session, owner.connector(), owner.label())
+ .getTableSchema(session, handle);
+ return reflectSiblingScanCapabilities(owner.connector(), siblingSchema);
}
HiveTableHandle hiveHandle = (HiveTableHandle) handle;
String dbName = hiveHandle.getDbName();
diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java
new file mode 100644
index 00000000000000..890b5e9b8f712e
--- /dev/null
+++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.hive;
+
+import org.apache.doris.connector.api.Connector;
+
+/**
+ * The embedded sibling connector that owns a foreign (iceberg/hudi-on-HMS) table, paired with its stable owner
+ * label. Produced by the 3-way ownsHandle dispatch ({@code HiveConnector.resolveSiblingOwnerLabeled}) so the
+ * per-handle gateway seams learn WHICH sibling owns a handle AND its label in a single peek (no force-build, no
+ * identity comparison).
+ *
+ *
The label is the single source of truth for the per-statement metadata funnel key
+ * ({@code "metadata:" + catalogId + ":" + label}) that lets a statement reuse ONE sibling metadata across all its
+ * forwards. It MUST be identical on the by-TYPE divert path ({@code getTableHandle}, which has no handle yet and
+ * asks a sibling by type) and the by-HANDLE forward path (every per-handle guard-and-forward) for the same owner
+ * — otherwise the two paths would memoize two sibling metadata instances under two keys within one statement.
+ * Both routes therefore key off {@link #ICEBERG_LABEL} / {@link #HUDI_LABEL} here.
+ *
+ *
The connector is held ONLY as the parent-first {@link Connector} interface — never cast (its concrete
+ * iceberg/hudi types are invisible across the plugin classloader split; a cast would CCE).
+ */
+final class SiblingOwner {
+
+ /** Owner label for the embedded iceberg sibling — the funnel-key discriminator, shared by both routes. */
+ static final String ICEBERG_LABEL = "iceberg";
+ /** Owner label for the embedded hudi sibling — the funnel-key discriminator, shared by both routes. */
+ static final String HUDI_LABEL = "hudi";
+
+ private final Connector connector;
+ private final String label;
+
+ SiblingOwner(Connector connector, String label) {
+ this.connector = connector;
+ this.label = label;
+ }
+
+ Connector connector() {
+ return connector;
+ }
+
+ String label() {
+ return label;
+ }
+}
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
index 91089872bfc484..c0121cb70fbba0 100644
--- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
@@ -24,6 +24,7 @@
import org.apache.doris.connector.api.ConnectorMetadata;
import org.apache.doris.connector.api.ConnectorPartitionInfo;
import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorStatementScope;
import org.apache.doris.connector.api.ConnectorTableSchema;
import org.apache.doris.connector.api.ConnectorTableStatistics;
import org.apache.doris.connector.api.ConnectorType;
@@ -82,6 +83,12 @@ private static final class ForeignHandle implements ConnectorTableHandle {
private final RecordingSiblingMetadata siblingMetadata = new RecordingSiblingMetadata();
private final RecordingSiblingConnector siblingConnector = new RecordingSiblingConnector(siblingMetadata);
+ // A session whose per-statement scope is NONE (offline): the sibling-metadata funnel runs its factory on every
+ // forward, so the sibling is consulted per call exactly as before the funnel — these forwarding assertions are
+ // byte-equivalent to the pre-funnel behavior. Per-statement REUSE (one sibling metadata across many forwards)
+ // is pinned by the "per-statement sibling-metadata funnel" tests below, which run under a live TestStatementScope.
+ private final ConnectorSession session = new ScopeSession(1L, "q1", ConnectorStatementScope.NONE);
+
/**
* The by-TYPE force-build supplier constructor arg. This suite exercises only per-handle (by-handle) sites —
* which must ALL route via the peek resolver — and never calls getTableHandle (the only by-type site), so the
@@ -103,7 +110,8 @@ private static final class ForeignHandle implements ConnectorTableHandle {
*/
private HiveConnectorMetadata withSibling() {
return new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(),
- SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector);
+ SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
+ handle -> new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL));
}
private HiveTableHandle hiveHandle() {
@@ -115,34 +123,34 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() {
HiveConnectorMetadata md = withSibling();
// ---- set (a): methods hive overrides — a foreign handle must NOT run hive logic, it must divert ----
- md.getTableSchema(null, foreignHandle);
- md.getColumnHandles(null, foreignHandle);
- md.getTableStatistics(null, foreignHandle);
- md.getColumnStatistics(null, foreignHandle, "c");
- long size = md.estimateDataSizeByListingFiles(null, foreignHandle);
- List> timelineRows = md.getMetadataTableRows(null, foreignHandle, "timeline");
- Optional> filter = md.applyFilter(null, foreignHandle, null);
- List partNames = md.listPartitionNames(null, foreignHandle);
- md.listPartitions(null, foreignHandle, Optional.empty());
- ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, foreignHandle).orElse(null);
- md.getTableFreshness(null, foreignHandle);
- md.getPartitionFreshnessMillis(null, foreignHandle, "p");
- md.dropTable(null, foreignHandle);
- md.truncateTable(null, foreignHandle, Collections.emptyList());
+ md.getTableSchema(session, foreignHandle);
+ md.getColumnHandles(session, foreignHandle);
+ md.getTableStatistics(session, foreignHandle);
+ md.getColumnStatistics(session, foreignHandle, "c");
+ long size = md.estimateDataSizeByListingFiles(session, foreignHandle);
+ List> timelineRows = md.getMetadataTableRows(session, foreignHandle, "timeline");
+ Optional> filter = md.applyFilter(session, foreignHandle, null);
+ List partNames = md.listPartitionNames(session, foreignHandle);
+ md.listPartitions(session, foreignHandle, Optional.empty());
+ ConnectorMvccSnapshot pin = md.beginQuerySnapshot(session, foreignHandle).orElse(null);
+ md.getTableFreshness(session, foreignHandle);
+ md.getPartitionFreshnessMillis(session, foreignHandle, "p");
+ md.dropTable(session, foreignHandle);
+ md.truncateTable(session, foreignHandle, Collections.emptyList());
// ---- set (b): methods hive does NOT override — the silent gaps that must be filled by forwarding ----
- md.getTableSchema(null, foreignHandle, null);
- md.getMvccPartitionView(null, foreignHandle);
- md.resolveTimeTravel(null, foreignHandle, null);
- ConnectorTableHandle afterSnapshot = md.applySnapshot(null, foreignHandle, null);
- List predicates = md.getSyntheticScanPredicates(null, foreignHandle, null);
- ConnectorTableHandle afterScope = md.applyRewriteFileScope(null, foreignHandle, Collections.emptySet());
- ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(null, foreignHandle);
- List sysTables = md.listSupportedSysTables(null, foreignHandle);
- Optional sysHandle = md.getSysTableHandle(null, foreignHandle, "snapshots");
+ md.getTableSchema(session, foreignHandle, null);
+ md.getMvccPartitionView(session, foreignHandle);
+ md.resolveTimeTravel(session, foreignHandle, null);
+ ConnectorTableHandle afterSnapshot = md.applySnapshot(session, foreignHandle, null);
+ List predicates = md.getSyntheticScanPredicates(session, foreignHandle, null);
+ ConnectorTableHandle afterScope = md.applyRewriteFileScope(session, foreignHandle, Collections.emptySet());
+ ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(session, foreignHandle);
+ List sysTables = md.listSupportedSysTables(session, foreignHandle);
+ Optional sysHandle = md.getSysTableHandle(session, foreignHandle, "snapshots");
// "snapshots" (not "partitions"): hive's own logic returns false for it, so a true answer proves the
// reply came from the sibling, not hive.
- boolean sysIsTvf = md.isPartitionValuesSysTable(null, foreignHandle, "snapshots");
+ boolean sysIsTvf = md.isPartitionValuesSysTable(session, foreignHandle, "snapshots");
// Every per-handle method reached the sibling (proves the divert covers the whole surface).
Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_METHODS, siblingMetadata.calls,
@@ -186,24 +194,24 @@ public void hiveHandleRunsHiveBranchAndNeverConsultsSibling() {
// The set-(b) + beginQuerySnapshot branches reproduce the SPI default / hive pin WITHOUT the sibling and
// without touching the (null) hmsClient — proving the guard falls through to the hive path for a hive handle.
- Assertions.assertFalse(md.getMvccPartitionView(null, hive).isPresent(), "hive has no range partition view");
- Assertions.assertFalse(md.resolveTimeTravel(null, hive, null).isPresent(), "hive has no time travel");
- Assertions.assertSame(hive, md.applySnapshot(null, hive, null), "hive applySnapshot returns the handle");
- Assertions.assertTrue(md.getSyntheticScanPredicates(null, hive, null).isEmpty(),
+ Assertions.assertFalse(md.getMvccPartitionView(session, hive).isPresent(), "hive has no range partition view");
+ Assertions.assertFalse(md.resolveTimeTravel(session, hive, null).isPresent(), "hive has no time travel");
+ Assertions.assertSame(hive, md.applySnapshot(session, hive, null), "hive applySnapshot returns the handle");
+ Assertions.assertTrue(md.getSyntheticScanPredicates(session, hive, null).isEmpty(),
"plain hive has no synthetic scan predicate");
- Assertions.assertSame(hive, md.applyRewriteFileScope(null, hive, Collections.emptySet()),
+ Assertions.assertSame(hive, md.applyRewriteFileScope(session, hive, Collections.emptySet()),
"hive applyRewriteFileScope returns the handle");
- Assertions.assertSame(hive, md.applyTopnLazyMaterialization(null, hive),
+ Assertions.assertSame(hive, md.applyTopnLazyMaterialization(session, hive),
"hive applyTopnLazyMaterialization returns the handle");
- Assertions.assertEquals(Collections.singletonList("partitions"), md.listSupportedSysTables(null, hive),
+ Assertions.assertEquals(Collections.singletonList("partitions"), md.listSupportedSysTables(session, hive),
"hive exposes the partitions sys table (t$partitions), served by the partition_values TVF");
- Assertions.assertTrue(md.isPartitionValuesSysTable(null, hive, "partitions"),
+ Assertions.assertTrue(md.isPartitionValuesSysTable(session, hive, "partitions"),
"hive's partitions sys table is TVF-backed");
- Assertions.assertFalse(md.isPartitionValuesSysTable(null, hive, "snapshots"),
+ Assertions.assertFalse(md.isPartitionValuesSysTable(session, hive, "snapshots"),
"hive exposes no sys table other than partitions");
- Assertions.assertFalse(md.getSysTableHandle(null, hive, "snapshots").isPresent(),
+ Assertions.assertFalse(md.getSysTableHandle(session, hive, "snapshots").isPresent(),
"hive's TVF-backed sys table has no native handle");
- ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, hive).orElse(null);
+ ConnectorMvccSnapshot pin = md.beginQuerySnapshot(session, hive).orElse(null);
Assertions.assertNotNull(pin);
Assertions.assertEquals(-1L, pin.getSnapshotId(), "hive's pin is the empty (-1) last-modified pin");
Assertions.assertTrue(pin.isLastModifiedFreshness(), "hive's pin flags last-modified freshness");
@@ -219,7 +227,7 @@ public void foreignHandleFailsLoudWhenNoSiblingConfigured() {
// raise a clear error, not NPE deep in a forward.
HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(),
new FakeConnectorContext());
- Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableSchema(null, foreignHandle),
+ Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableSchema(session, foreignHandle),
"a foreign handle with no sibling configured must fail loud");
}
@@ -230,25 +238,25 @@ public void everyAlterDdlAndValidateMethodForwardsAForeignHandleToTheSibling() {
// The 14 ALTER-DDL mutators + 2 write validators: a foreign (iceberg-on-HMS) handle must divert, never
// run the hive branch and never be cast. Change objects are null — the guard fires on the handle type
// before any param is touched.
- md.renameTable(null, foreignHandle, "new");
- md.addColumn(null, foreignHandle, null, null);
- md.addColumns(null, foreignHandle, Collections.emptyList());
- md.dropColumn(null, foreignHandle, "c");
- md.renameColumn(null, foreignHandle, "a", "b");
- md.modifyColumn(null, foreignHandle, null, null);
- md.reorderColumns(null, foreignHandle, Collections.emptyList());
- md.createOrReplaceBranch(null, foreignHandle, null);
- md.createOrReplaceTag(null, foreignHandle, null);
- md.dropBranch(null, foreignHandle, null);
- md.dropTag(null, foreignHandle, null);
- md.addPartitionField(null, foreignHandle, null);
- md.dropPartitionField(null, foreignHandle, null);
- md.replacePartitionField(null, foreignHandle, null);
- md.validateRowLevelDmlMode(null, foreignHandle, null);
- md.validateStaticPartitionColumns(null, foreignHandle, Collections.emptyList());
+ md.renameTable(session, foreignHandle, "new");
+ md.addColumn(session, foreignHandle, null, null);
+ md.addColumns(session, foreignHandle, Collections.emptyList());
+ md.dropColumn(session, foreignHandle, "c");
+ md.renameColumn(session, foreignHandle, "a", "b");
+ md.modifyColumn(session, foreignHandle, null, null);
+ md.reorderColumns(session, foreignHandle, Collections.emptyList());
+ md.createOrReplaceBranch(session, foreignHandle, null);
+ md.createOrReplaceTag(session, foreignHandle, null);
+ md.dropBranch(session, foreignHandle, null);
+ md.dropTag(session, foreignHandle, null);
+ md.addPartitionField(session, foreignHandle, null);
+ md.dropPartitionField(session, foreignHandle, null);
+ md.replacePartitionField(session, foreignHandle, null);
+ md.validateRowLevelDmlMode(session, foreignHandle, null);
+ md.validateStaticPartitionColumns(session, foreignHandle, Collections.emptyList());
// Empty list on purpose: a foreign handle must forward REGARDLESS of emptiness (the empty-early-return is
// hive-only) — this would fail if the empty check were placed before the foreign-handle divert.
- md.validateWritePartitionNames(null, foreignHandle, Collections.emptyList());
+ md.validateWritePartitionNames(session, foreignHandle, Collections.emptyList());
Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_WRITE_METHODS, siblingMetadata.calls,
"every ALTER-DDL mutator + write validator must forward a foreign handle to the sibling");
@@ -263,12 +271,12 @@ public void hiveHandleRejectsNonEmptyPartitionNamesWithLegacyMessage() {
// partition-NAME list form INSERT ... PARTITION(p1, p2) is unsupported on a hive table. UNLIKE the two
// permissive validators, a hive handle here THROWS the EXACT legacy message on a non-empty list. The e2e
// test_hive_write_type.groovy asserts on this literal substring, so it must stay byte-identical.
- assertThrowsMessage(() -> md.validateWritePartitionNames(null, hive, Arrays.asList("p1", "p2")),
+ assertThrowsMessage(() -> md.validateWritePartitionNames(session, hive, Arrays.asList("p1", "p2")),
"Not support insert with partition spec in hive catalog.");
// An empty list (a plain INSERT ... SELECT or a static PARTITION(col='val') INSERT) is legal plain-hive
// and MUST return silently — a throw here would newly reject legal writes.
- md.validateWritePartitionNames(null, hive, Collections.emptyList());
+ md.validateWritePartitionNames(session, hive, Collections.emptyList());
Assertions.assertEquals(0, siblingConnector.getMetadataCount,
"a hive handle must never build/consult the iceberg sibling to validate partition names");
@@ -282,26 +290,26 @@ public void hiveHandleAlterDdlThrowsAndValidateIsNoopAndNeverConsultsSibling() {
// Group-1: ALTER-DDL for a hive handle throws the EXACT inherited SPI-default message (byte-parity with
// pre-override behavior) without building or consulting the sibling.
- assertThrowsMessage(() -> md.renameTable(null, hive, "n"), "RENAME TABLE not supported");
- assertThrowsMessage(() -> md.addColumn(null, hive, null, null), "ADD COLUMN not supported");
- assertThrowsMessage(() -> md.addColumns(null, hive, Collections.emptyList()), "ADD COLUMNS not supported");
- assertThrowsMessage(() -> md.dropColumn(null, hive, "c"), "DROP COLUMN not supported");
- assertThrowsMessage(() -> md.renameColumn(null, hive, "a", "b"), "RENAME COLUMN not supported");
- assertThrowsMessage(() -> md.modifyColumn(null, hive, null, null), "MODIFY COLUMN not supported");
- assertThrowsMessage(() -> md.reorderColumns(null, hive, Collections.emptyList()),
+ assertThrowsMessage(() -> md.renameTable(session, hive, "n"), "RENAME TABLE not supported");
+ assertThrowsMessage(() -> md.addColumn(session, hive, null, null), "ADD COLUMN not supported");
+ assertThrowsMessage(() -> md.addColumns(session, hive, Collections.emptyList()), "ADD COLUMNS not supported");
+ assertThrowsMessage(() -> md.dropColumn(session, hive, "c"), "DROP COLUMN not supported");
+ assertThrowsMessage(() -> md.renameColumn(session, hive, "a", "b"), "RENAME COLUMN not supported");
+ assertThrowsMessage(() -> md.modifyColumn(session, hive, null, null), "MODIFY COLUMN not supported");
+ assertThrowsMessage(() -> md.reorderColumns(session, hive, Collections.emptyList()),
"REORDER COLUMNS not supported");
- assertThrowsMessage(() -> md.createOrReplaceBranch(null, hive, null), "CREATE/REPLACE BRANCH not supported");
- assertThrowsMessage(() -> md.createOrReplaceTag(null, hive, null), "CREATE/REPLACE TAG not supported");
- assertThrowsMessage(() -> md.dropBranch(null, hive, null), "DROP BRANCH not supported");
- assertThrowsMessage(() -> md.dropTag(null, hive, null), "DROP TAG not supported");
- assertThrowsMessage(() -> md.addPartitionField(null, hive, null), "ADD PARTITION FIELD not supported");
- assertThrowsMessage(() -> md.dropPartitionField(null, hive, null), "DROP PARTITION FIELD not supported");
- assertThrowsMessage(() -> md.replacePartitionField(null, hive, null), "REPLACE PARTITION FIELD not supported");
+ assertThrowsMessage(() -> md.createOrReplaceBranch(session, hive, null), "CREATE/REPLACE BRANCH not supported");
+ assertThrowsMessage(() -> md.createOrReplaceTag(session, hive, null), "CREATE/REPLACE TAG not supported");
+ assertThrowsMessage(() -> md.dropBranch(session, hive, null), "DROP BRANCH not supported");
+ assertThrowsMessage(() -> md.dropTag(session, hive, null), "DROP TAG not supported");
+ assertThrowsMessage(() -> md.addPartitionField(session, hive, null), "ADD PARTITION FIELD not supported");
+ assertThrowsMessage(() -> md.dropPartitionField(session, hive, null), "DROP PARTITION FIELD not supported");
+ assertThrowsMessage(() -> md.replacePartitionField(session, hive, null), "REPLACE PARTITION FIELD not supported");
// Group-2: validate* for a hive handle MUST return silently — a throw here would newly reject legal
// plain-hive row-level DML / static-partition INSERTs.
- md.validateRowLevelDmlMode(null, hive, null);
- md.validateStaticPartitionColumns(null, hive, Collections.emptyList());
+ md.validateRowLevelDmlMode(session, hive, null);
+ md.validateStaticPartitionColumns(session, hive, Collections.emptyList());
Assertions.assertEquals(0, siblingConnector.getMetadataCount,
"a hive handle must never build/consult the iceberg sibling for ALTER-DDL / validate");
@@ -315,7 +323,7 @@ public void beginTransactionForwardsAForeignHandleToTheSibling() {
// A foreign (iceberg-on-HMS) write must open the SIBLING's transaction, so iceberg's write plan can
// downcast the session-bound transaction to IcebergConnectorTransaction — a HiveConnectorTransaction
// (what the unconditional open would bind) would ClassCastException there.
- ConnectorTransaction txn = md.beginTransaction(null, foreignHandle);
+ ConnectorTransaction txn = md.beginTransaction(session, foreignHandle);
Assertions.assertSame(RecordingSiblingMetadata.SIBLING_TXN, txn,
"a foreign handle must open the sibling's transaction, not a hive one");
@@ -332,14 +340,15 @@ public void beginTransactionForHiveHandleOpensHiveTxnAndNeverConsultsSibling() {
// sibling. The selection must be symmetric — hive and iceberg write plans downcast to different types.
ConnectorTransaction hiveTxn = new NoOpConnectorTransaction(70099L, "HIVE");
HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(),
- SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector) {
+ SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
+ handle -> new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL)) {
@Override
public ConnectorTransaction beginTransaction(ConnectorSession session) {
return hiveTxn;
}
};
- Assertions.assertSame(hiveTxn, md.beginTransaction(null, hiveHandle()),
+ Assertions.assertSame(hiveTxn, md.beginTransaction(session, hiveHandle()),
"a hive handle must open the connector-level (hive) transaction, not the sibling's");
Assertions.assertEquals(0, siblingConnector.getMetadataCount,
"a hive handle must never build/consult the iceberg sibling to open a transaction");
@@ -360,9 +369,10 @@ public void foreignHandleSchemaReflectsSiblingScanCapabilitiesAsPerTableMarker()
ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE);
HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(),
SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
- handle -> new CapabilityDeclaringSiblingConnector(siblingCaps));
+ handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector(siblingCaps),
+ SiblingOwner.ICEBERG_LABEL));
- ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle);
+ ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle);
String csv = schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY);
Assertions.assertNotNull(csv, "the delegated schema must carry the reflected per-table capability marker");
List names = Arrays.asList(csv.split(","));
@@ -382,7 +392,7 @@ public void foreignHandleSchemaUnchangedWhenSiblingDeclaresNoCapabilities() {
// lacks auto-analyze) is pinned by foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling below.
// MUTATION: dropping the isEmpty() early-return and stamping an (empty) marker unconditionally -> red here.
HiveConnectorMetadata md = withSibling(); // RecordingSiblingConnector declares no capabilities
- ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle);
+ ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle);
Assertions.assertNull(schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY),
"no marker when the sibling declares no capabilities");
}
@@ -399,10 +409,10 @@ public void foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling() {
// flag) -> the marker would contain SUPPORTS_COLUMN_AUTO_ANALYZE -> red here.
HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(),
SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
- handle -> new CapabilityDeclaringSiblingConnector(
- EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE)));
+ handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector(
+ EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE)), SiblingOwner.ICEBERG_LABEL));
- ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle);
+ ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle);
String csv = schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY);
Assertions.assertNotNull(csv, "a non-empty hudi sibling must still stamp its declared capabilities");
List names = Arrays.asList(csv.split(","));
@@ -416,6 +426,102 @@ public void foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling() {
"hudi-on-HMS gains no nested-column prune from a sibling that does not declare it");
}
+ // ============== per-statement sibling-metadata funnel (HMS heterogeneous gateway) ==============
+ // Within one statement, the gateway obtains ONE sibling ConnectorMetadata per (catalogId, owner) and reuses it
+ // across every forward (read / scan / DDL / MVCC / the write-transaction open), keyed
+ // "metadata::" on the session's per-statement scope — mirroring fe-core's own funnel for
+ // a plain connector. RecordingSiblingConnector.getMetadataCount is the load count.
+
+ @Test
+ public void liveScopeSharesOneSiblingMetadataAcrossEveryForwardIncludingTheWriteTxn() {
+ // Many read forwards (including the getTableSchema stray) + the per-handle beginTransaction open, all in one
+ // statement -> the sibling is built ONCE and every forward (reads AND the write transaction) reuses it.
+ // MUTATION: not memoizing -> a build per forward -> count > 1 -> red.
+ HiveConnectorMetadata md = withSibling();
+ ConnectorSession live = new ScopeSession(1L, "q1", new TestStatementScope());
+
+ md.getColumnHandles(live, foreignHandle);
+ md.listPartitionNames(live, foreignHandle);
+ md.getMetadataTableRows(live, foreignHandle, "timeline");
+ md.getTableSchema(live, foreignHandle);
+ md.beginTransaction(live, foreignHandle);
+
+ Assertions.assertEquals(1, siblingConnector.getMetadataCount,
+ "one sibling metadata per (catalog, owner) per statement — reads and the write txn share it");
+ }
+
+ @Test
+ public void noneScopeBuildsAFreshSiblingMetadataEachForward() {
+ // No live statement scope (offline / no ConnectContext): the funnel factory runs on every forward, exactly
+ // as before the funnel existed. This is the byte-equivalence guard for the NONE path.
+ HiveConnectorMetadata md = withSibling();
+
+ md.getColumnHandles(session, foreignHandle);
+ md.listPartitionNames(session, foreignHandle);
+ md.getColumnHandles(session, foreignHandle);
+
+ Assertions.assertEquals(3, siblingConnector.getMetadataCount,
+ "NONE scope -> the sibling metadata factory runs on every forward (pre-funnel behavior)");
+ }
+
+ @Test
+ public void byTypeDivertAndByHandleForwardShareOneSiblingMetadata() {
+ // The getTableHandle divert asks the sibling BY TYPE (icebergSiblingMetadata) before any handle exists; the
+ // later per-handle forwards resolve BY HANDLE. Both must mint the SAME funnel key for the same owner (the
+ // by-TYPE literal label == the by-HANDLE resolver-arm label), or the statement would hold two sibling
+ // metadata instances. Wire the iceberg by-TYPE supplier to the SAME recording connector the resolver
+ // returns, then drive both paths under one live scope. MUTATION: mismatched labels -> two builds -> red.
+ HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(),
+ () -> siblingConnector, SUPPLIER_MUST_NOT_BE_USED,
+ handle -> new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL));
+ ConnectorSession live = new ScopeSession(1L, "q1", new TestStatementScope());
+
+ md.icebergSiblingMetadata(live); // by-TYPE (getTableHandle divert path)
+ md.getColumnHandles(live, foreignHandle); // by-HANDLE (per-handle forward)
+
+ Assertions.assertEquals(1, siblingConnector.getMetadataCount,
+ "the by-TYPE divert and the by-HANDLE forward key the same owner -> one shared sibling metadata");
+ }
+
+ @Test
+ public void differentCatalogIdIsolatesTheSiblingMetadata() {
+ // A statement joining two heterogeneous HMS catalogs shares one scope map; the catalog id in the key keeps
+ // each catalog's sibling metadata isolated. MUTATION: dropping catalogId from the key -> the two collide.
+ HiveConnectorMetadata md = withSibling();
+ TestStatementScope scope = new TestStatementScope();
+
+ md.getColumnHandles(new ScopeSession(1L, "q1", scope), foreignHandle);
+ md.getColumnHandles(new ScopeSession(2L, "q1", scope), foreignHandle);
+
+ Assertions.assertEquals(2, siblingConnector.getMetadataCount,
+ "a different catalog id keys a different sibling metadata (no cross-catalog collapse)");
+ }
+
+ @Test
+ public void icebergAndHudiSiblingsAreIsolatedWithinAStatement() {
+ // The whole point of the owner label: iceberg-on-HMS and hudi-on-HMS tables of ONE gateway share a catalog
+ // id, so ONLY the label ("iceberg" vs "hudi") keeps their metadata entries apart. Each owner is built once
+ // and never collapses onto the other. MUTATION: a shared (label-less) key -> one owner's metadata serves
+ // the other -> a count is 0 while the other is 2 -> red.
+ RecordingSiblingConnector hudiConnector = new RecordingSiblingConnector(new RecordingSiblingMetadata());
+ ForeignHandle hudiHandle = new ForeignHandle();
+ HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(),
+ SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
+ handle -> handle == hudiHandle
+ ? new SiblingOwner(hudiConnector, SiblingOwner.HUDI_LABEL)
+ : new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL));
+ ConnectorSession live = new ScopeSession(1L, "q1", new TestStatementScope());
+
+ md.getColumnHandles(live, foreignHandle); // iceberg owner
+ md.getColumnHandles(live, hudiHandle); // hudi owner
+ md.getColumnHandles(live, foreignHandle); // iceberg owner again -> reuse
+
+ Assertions.assertEquals(1, siblingConnector.getMetadataCount,
+ "the iceberg owner is built once under its label");
+ Assertions.assertEquals(1, hudiConnector.getMetadataCount,
+ "the hudi owner is isolated under its own label and built once (never collapsed onto iceberg)");
+ }
+
private static void assertThrowsMessage(Executable exec, String expectedMessage) {
DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, exec);
Assertions.assertEquals(expectedMessage, e.getMessage(),
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java
index a8996a9ca9c20c..a7aa4830fb39b2 100644
--- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java
@@ -20,6 +20,7 @@
import org.apache.doris.connector.api.Connector;
import org.apache.doris.connector.api.ConnectorMetadata;
import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorStatementScope;
import org.apache.doris.connector.api.DorisConnectorException;
import org.apache.doris.connector.api.handle.ConnectorTableHandle;
import org.apache.doris.connector.hms.HmsClient;
@@ -68,10 +69,15 @@ public class HiveConnectorMetadataTableHandleDivertTest {
// getTableHandle routes BY TYPE (the two by-TYPE suppliers), NEVER via the by-handle owner resolver — so wire
// the owner resolver to fail loud if anything reaches for it.
- private static final Function OWNER_RESOLVER_UNUSED = handle -> {
+ private static final Function OWNER_RESOLVER_UNUSED = handle -> {
throw new AssertionError("getTableHandle must divert BY TYPE, never via the by-handle owner resolver");
};
+ // getTableHandle's by-TYPE divert routes through the per-statement sibling-metadata funnel, which reads
+ // session.getStatementScope() + getCatalogId(); a NONE scope makes the factory run every call (byte-equivalent
+ // to the pre-funnel divert), so these by-type routing assertions are unchanged.
+ private final ConnectorSession session = new ScopeSession(1L, "q1", ConnectorStatementScope.NONE);
+
/** The foreign (non-hive) handle a sibling's getTableHandle produces post-flip. */
private static final class ForeignHandle implements ConnectorTableHandle {
}
@@ -85,7 +91,7 @@ private static final class ForeignHandle implements ConnectorTableHandle {
public void icebergTableDivertsToIcebergSiblingNotHudi() {
HiveConnectorMetadata md = withSiblings(icebergTable());
- Optional handle = md.getTableHandle(null, "db", "t");
+ Optional handle = md.getTableHandle(session, "db", "t");
Assertions.assertTrue(handle.isPresent(), "an existing iceberg-on-HMS table must resolve a handle");
Assertions.assertSame(icebergHandle, handle.get(),
@@ -106,7 +112,7 @@ public void icebergDivertPropagatesSiblingEmpty() {
icebergSibling.metadata.returnHandle = null;
HiveConnectorMetadata md = withSiblings(icebergTable());
- Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(),
+ Assertions.assertFalse(md.getTableHandle(session, "db", "t").isPresent(),
"an empty from the sibling must pass through unchanged");
// Prove the empty was FORWARDED from the sibling (its getTableHandle was consulted), not short-circuited
// to empty by the gateway — otherwise a broken `if (ICEBERG) return Optional.empty()` would pass this test.
@@ -120,7 +126,7 @@ public void hudiTableDivertsToHudiSiblingNotIceberg() {
// OWN foreign handle verbatim — NOT a HiveTableHandle stamped HUDI, and NOT routed to the iceberg sibling.
HiveConnectorMetadata md = withSiblings(hiveTable(HUDI));
- Optional handle = md.getTableHandle(null, "db", "t");
+ Optional handle = md.getTableHandle(session, "db", "t");
Assertions.assertTrue(handle.isPresent(), "an existing hudi-on-HMS table must resolve a handle");
Assertions.assertSame(hudiHandle, handle.get(),
@@ -141,7 +147,7 @@ public void hudiDivertPropagatesSiblingEmpty() {
hudiSibling.metadata.returnHandle = null;
HiveConnectorMetadata md = withSiblings(hiveTable(HUDI));
- Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(),
+ Assertions.assertFalse(md.getTableHandle(session, "db", "t").isPresent(),
"an empty from the hudi sibling must pass through unchanged");
Assertions.assertEquals(1, hudiSibling.metadata.getTableHandleCalls,
"the hudi sibling is authoritative for hudi existence: its getTableHandle must be the source of empty");
@@ -151,7 +157,7 @@ public void hudiDivertPropagatesSiblingEmpty() {
public void hiveTableBuildsHiveHandleWithoutConsultingSibling() {
HiveConnectorMetadata md = withSiblings(hiveTable(PARQUET));
- Optional handle = md.getTableHandle(null, "db", "t");
+ Optional handle = md.getTableHandle(session, "db", "t");
Assertions.assertTrue(handle.get() instanceof HiveTableHandle, "a hive table resolves a HiveTableHandle");
Assertions.assertEquals(HiveTableType.HIVE, ((HiveTableHandle) handle.get()).getTableType());
@@ -167,7 +173,7 @@ public void unknownNonViewTableStillFailsLoud() {
// non-view table is still rejected (not swallowed into a hudi divert).
HiveConnectorMetadata md = withSiblings(hiveTable(UNKNOWN_FORMAT));
- Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"),
+ Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(session, "db", "t"),
"an unsupported non-view input format must fail loud, not be diverted to a sibling");
Assertions.assertEquals(0, hudiSibling.getMetadataCalls,
"the UNKNOWN fail-loud must not consult the hudi sibling");
@@ -181,7 +187,7 @@ public void missingTableReturnsEmptyWithoutConsultingSibling() {
new FakeHmsClient(icebergTable(), false), Collections.emptyMap(), new FakeConnectorContext(),
() -> icebergSibling, () -> hudiSibling, OWNER_RESOLVER_UNUSED);
- Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(),
+ Assertions.assertFalse(md.getTableHandle(session, "db", "t").isPresent(),
"a non-existent table short-circuits to empty before any format detection or divert");
Assertions.assertEquals(0, icebergSibling.getMetadataCalls, "a missing table must not build the iceberg sibling");
Assertions.assertEquals(0, hudiSibling.getMetadataCalls, "a missing table must not build the hudi sibling");
@@ -194,7 +200,7 @@ public void icebergTableFailsLoudWhenNoSiblingConfigured() {
HiveConnectorMetadata md = new HiveConnectorMetadata(
new FakeHmsClient(icebergTable(), true), Collections.emptyMap(), new FakeConnectorContext());
- Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"),
+ Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(session, "db", "t"),
"an iceberg table with no sibling configured must fail loud");
}
@@ -205,7 +211,7 @@ public void hudiTableFailsLoudWhenNoSiblingConfigured() {
HiveConnectorMetadata md = new HiveConnectorMetadata(
new FakeHmsClient(hiveTable(HUDI), true), Collections.emptyMap(), new FakeConnectorContext());
- Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"),
+ Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(session, "db", "t"),
"a hudi table with no sibling configured must fail loud");
}
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java
index 38f7b8bac79a45..0ff3bd02108a0f 100644
--- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java
@@ -18,6 +18,7 @@
package org.apache.doris.connector.hive;
import org.apache.doris.connector.api.Connector;
+import org.apache.doris.connector.api.ConnectorCapability;
import org.apache.doris.connector.api.ConnectorMetadata;
import org.apache.doris.connector.api.ConnectorSession;
import org.apache.doris.connector.api.DorisConnectorException;
@@ -25,8 +26,11 @@
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.Collections;
+import java.util.EnumSet;
import java.util.HashMap;
import java.util.Map;
+import java.util.Set;
/**
* Pins the HMS-cutover embedded-sibling holders: a flipped hms gateway lazily builds ONE embedded iceberg
@@ -298,6 +302,36 @@ public void refreshHooksNeverForceBuildSiblings() {
"invalidate hooks must only forward to ALREADY-BUILT siblings, never force-build one");
}
+ @Test
+ public void icebergSiblingWithoutUserSessionCapabilityIsAccepted() {
+ // The normal case: an hms-flavor sibling declares no SUPPORTS_USER_SESSION, so the fail-loud guard is
+ // inert and the sibling is returned as-is.
+ FakeSibling plainSibling = new FakeSibling();
+ HiveConnector connector =
+ new HiveConnector(new HashMap<>(), new RecordingSiblingContext(plainSibling));
+
+ Assertions.assertSame(plainSibling, connector.getOrCreateIcebergSibling(),
+ "a sibling without SUPPORTS_USER_SESSION must be accepted");
+ }
+
+ @Test
+ public void icebergSiblingDeclaringUserSessionFailsLoud() {
+ // Cache-isolation security invariant: the hive gateway FRONT DOOR is never session=user, so fe-core keys
+ // its per-user schema/name cache bypass off the front door and would NOT bypass for a delegated sibling.
+ // The iceberg sibling is forced iceberg.catalog.type=hms and can never be session=user; if a future change
+ // ever broke that, building the sibling must FAIL LOUD rather than silently leak cross-user metadata.
+ // MUTATION: dropping the guard -> the session=user sibling is accepted -> no throw -> red.
+ FakeSibling userSessionSibling =
+ new FakeSibling(EnumSet.of(ConnectorCapability.SUPPORTS_USER_SESSION));
+ HiveConnector connector =
+ new HiveConnector(new HashMap<>(), new RecordingSiblingContext(userSessionSibling));
+
+ DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class,
+ connector::getOrCreateIcebergSibling);
+ Assertions.assertTrue(ex.getMessage().contains("SUPPORTS_USER_SESSION"),
+ "the failure must name the broken session=user invariant");
+ }
+
/**
* A bare {@link Connector} stand-in for the cross-loader iceberg/hudi sibling; records lifecycle
* ({@code close}) and invalidation forwarding.
@@ -307,6 +341,20 @@ private static final class FakeSibling implements Connector {
int invalidateAllCount;
String lastInvalidatedTable;
String lastInvalidatedDb;
+ private final Set capabilities;
+
+ FakeSibling() {
+ this(Collections.emptySet());
+ }
+
+ FakeSibling(Set capabilities) {
+ this.capabilities = capabilities;
+ }
+
+ @Override
+ public Set getCapabilities() {
+ return capabilities;
+ }
@Override
public ConnectorMetadata getMetadata(ConnectorSession session) {
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java
index e127cac59229c2..e8a85c1830a416 100644
--- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java
@@ -20,6 +20,7 @@
import org.apache.doris.connector.api.Connector;
import org.apache.doris.connector.api.ConnectorMetadata;
import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorStatementScope;
import org.apache.doris.connector.api.DorisConnectorException;
import org.apache.doris.connector.api.handle.ConnectorColumnHandle;
import org.apache.doris.connector.api.handle.ConnectorTableHandle;
@@ -192,15 +193,19 @@ public void getMetadataWiresEachByTypeSupplierToItsOwnSibling() {
HiveConnector connector = new HiveConnector(props(), ctx);
HiveConnectorMetadata md = connector.newMetadata(null);
+ // A NONE-scope session: the by-TYPE helpers now route through the per-statement metadata funnel (which
+ // reads session.getStatementScope()/getCatalogId()), so a null session would NPE after the force-build we
+ // assert. NONE makes the funnel run its factory straight through — the force-build order is unchanged.
+ ConnectorSession session = new ScopeSession(1L, "q1", ConnectorStatementScope.NONE);
// The getTableHandle ICEBERG arm resolves BY TYPE via icebergSiblingMetadata, which force-builds the
// iceberg sibling (createSiblingConnector("iceberg")). A transposed getMetadata would build hudi here.
- md.icebergSiblingMetadata(null);
+ md.icebergSiblingMetadata(session);
Assertions.assertEquals(1, ctx.icebergBuilds, "the iceberg by-TYPE arm must force-build the iceberg sibling");
Assertions.assertEquals(0, ctx.hudiBuilds, "the iceberg by-TYPE arm must NOT build the hudi sibling");
// Symmetrically the HUDI arm must resolve the hudi sibling, not rebuild iceberg.
- md.hudiSiblingMetadata(null);
+ md.hudiSiblingMetadata(session);
Assertions.assertEquals(1, ctx.hudiBuilds, "the hudi by-TYPE arm must force-build the hudi sibling");
Assertions.assertEquals(1, ctx.icebergBuilds, "the hudi by-TYPE arm must not rebuild the iceberg sibling");
}
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java
new file mode 100644
index 00000000000000..f2e1d6d7c29ff8
--- /dev/null
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java
@@ -0,0 +1,89 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.hive;
+
+import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorStatementScope;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Minimal {@link ConnectorSession} for hive-connector unit tests, carrying a catalog id and a per-statement
+ * scope. The heterogeneous gateway keys its per-statement sibling-metadata funnel on
+ * {@code "metadata:" + getCatalogId() + ":" + ownerLabel} and reads the scope via {@link #getStatementScope()},
+ * so a test wires a live {@link TestStatementScope} to prove sharing (or {@link ConnectorStatementScope#NONE} to
+ * prove the pre-funnel load-every-time behavior). Mirrors the iceberg connector's test session.
+ */
+final class ScopeSession implements ConnectorSession {
+
+ private final long catalogId;
+ private final String queryId;
+ private final ConnectorStatementScope scope;
+
+ ScopeSession(long catalogId, String queryId, ConnectorStatementScope scope) {
+ this.catalogId = catalogId;
+ this.queryId = queryId;
+ this.scope = scope;
+ }
+
+ @Override
+ public long getCatalogId() {
+ return catalogId;
+ }
+
+ @Override
+ public String getQueryId() {
+ return queryId;
+ }
+
+ @Override
+ public ConnectorStatementScope getStatementScope() {
+ return scope;
+ }
+
+ @Override
+ public String getUser() {
+ return "u";
+ }
+
+ @Override
+ public String getTimeZone() {
+ return "UTC";
+ }
+
+ @Override
+ public String getLocale() {
+ return "en_US";
+ }
+
+ @Override
+ public String getCatalogName() {
+ return "c";
+ }
+
+ @Override
+ public T getProperty(String name, Class type) {
+ return null;
+ }
+
+ @Override
+ public Map getCatalogProperties() {
+ return Collections.emptyMap();
+ }
+}
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java
new file mode 100644
index 00000000000000..7d52c162a12dac
--- /dev/null
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java
@@ -0,0 +1,40 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.hive;
+
+import org.apache.doris.connector.api.ConnectorStatementScope;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Supplier;
+
+/**
+ * A memoizing {@link ConnectorStatementScope} for hive-connector unit tests. The hive connector module does not
+ * depend on fe-core, so tests cannot use the engine's {@code ConnectorStatementScopeImpl}; this is a faithful
+ * copy of it (mirrors the iceberg connector's test copy). Sharing one instance across a statement's forwards lets
+ * a test prove that the heterogeneous gateway reuses ONE sibling metadata per owner per statement.
+ */
+final class TestStatementScope implements ConnectorStatementScope {
+
+ private final ConcurrentHashMap cache = new ConcurrentHashMap<>();
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T computeIfAbsent(String key, Supplier loader) {
+ return (T) cache.computeIfAbsent(key, k -> loader.get());
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java
new file mode 100644
index 00000000000000..aa179082d3d234
--- /dev/null
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.cache.CacheSpec;
+import org.apache.doris.connector.cache.MetaCacheEntry;
+
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+import java.util.concurrent.ForkJoinPool;
+import java.util.function.Supplier;
+
+/**
+ * Per-catalog cache of an iceberg table's {@code comment} property (PERF-05), keyed by {@link TableIdentifier}.
+ * Kills the per-table remote {@code loadTable} that {@code information_schema.tables} / {@code SHOW TABLE STATUS}
+ * pays for EVERY table just to read its comment ({@code FrontendServiceImpl.listTableStatus} loops per table and
+ * unconditionally calls {@code getComment} -> {@link IcebergConnectorMetadata#getTableComment} ->
+ * {@code loadTable}).
+ *
+ * Scope: REST vended-credentials catalogs ONLY (see {@link IcebergConnector}). Plain catalogs already
+ * reuse the raw {@link IcebergTableCache} (PERF-01) for the comment path, so this cache is not built for them (no
+ * redundancy). A {@code iceberg.rest.session=user} catalog is DELIBERATELY excluded: there the {@code loadTable}
+ * call itself carries the querying user's per-request authorization, and a connector-shared comment cache would
+ * serve one user's comment to another whose delegated token was never validated for that table (a metadata
+ * disclosure). Vended-credentials uses a single static catalog identity, so every user loads the same table and
+ * sharing a comment is safe.
+ *
+ *
No credential gate (unlike {@link IcebergTableCache}, like {@link IcebergPartitionCache} /
+ * {@link IcebergFormatCache}): the cached value is a bare comment {@link String} with no {@code FileIO} /
+ * credential. A comment changes only via external DDL, picked up through the REFRESH invalidate hooks. TTL is
+ * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live), so a no-cache catalog serves a
+ * fresh comment even when this object is built. Backed identically to {@link IcebergTableCache}: a contextual,
+ * access-TTL {@link MetaCacheEntry} with manual miss-load, so the remote load runs OUTSIDE Caffeine's compute
+ * lock and its exception (e.g. the view-handle {@code NoSuchTableException}) propagates verbatim and a failed
+ * load is not cached. Lives on the long-lived per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds it.
+ */
+final class IcebergCommentCache {
+
+ private final MetaCacheEntry entry;
+
+ IcebergCommentCache(long ttlSeconds, int maxSize) {
+ // Mirror IcebergTableCache: translate the connector's "<= 0 disables" contract to CacheSpec's ttl == 0
+ // (disabled) rather than passing a negative value through, which CacheSpec reads as ttl == -1 "no
+ // expiration (enabled)". This ttl<=0 -> disabled mapping is load-bearing: a vended no-cache catalog builds
+ // this object but must NOT cache comments (operator "no meta cache" intent).
+ CacheSpec spec = ttlSeconds > 0
+ ? CacheSpec.of(true, ttlSeconds, maxSize)
+ : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize);
+ this.entry = new MetaCacheEntry<>("iceberg-comment", null, spec,
+ ForkJoinPool.commonPool(), false, true, 0L, true);
+ }
+
+ /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always read the comment live". */
+ boolean isEnabled() {
+ return entry.stats().isEffectiveEnabled();
+ }
+
+ /**
+ * Returns the cached comment for {@code identifier} if present and unexpired, else runs {@code loader} (the
+ * live remote {@code loadTable} + property read), caches and returns it. Disabled cache -> {@code loader}
+ * every call. The loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception
+ * propagates unwrapped and is NOT cached (the caller degrades a thrown comment load to {@code ""}).
+ */
+ String getOrLoad(TableIdentifier identifier, Supplier loader) {
+ return entry.get(identifier, ignored -> loader.get());
+ }
+
+ /** Drops the cached comment for one table so the next read goes live (REFRESH TABLE). */
+ void invalidate(TableIdentifier identifier) {
+ entry.invalidateKey(identifier);
+ }
+
+ /** Drops every cached comment for one database (REFRESH DATABASE / DROP DATABASE); match = namespace equality. */
+ void invalidateDb(String dbName) {
+ Namespace ns = Namespace.of(dbName);
+ entry.invalidateIf(id -> id.namespace().equals(ns));
+ }
+
+ /** Drops all cached comments (REFRESH CATALOG). */
+ void invalidateAll() {
+ entry.invalidateAll();
+ }
+
+ /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */
+ int size() {
+ int[] count = {0};
+ entry.forEach((key, value) -> count[0]++);
+ return count[0];
+ }
+
+ /** Test-only: how many times the live loader (the remote comment load) actually ran — the metric gate. */
+ long loadCountForTest() {
+ return entry.stats().getLoadSuccessCount();
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
index 08a1670544a285..47fb3dd5abd2ae 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
@@ -158,13 +158,32 @@ public class IcebergConnector implements Connector {
// connector (PluginDrivenExternalCatalog.onClose nulls + recreates it) and thus drops both caches. The
// manifest cache is path-keyed, no-TTL, capacity-bounded; it is consumed only when
// meta.cache.iceberg.manifest.enable is set (default off → scan uses the SDK planFiles path).
- private final IcebergLatestSnapshotCache latestSnapshotCache;
+ // Each cross-query cache below carries an isolation-discipline marker enforced by
+ // tools/check-authz-cache-sharding.sh: under iceberg.rest.session=user a shared, un-partitioned cache would
+ // bypass the per-user loadTable authorization (a metadata disclosure), so every cache field must declare
+ // either 'authz-cache-session-user-disabled' (null under isUserSessionEnabled()) or 'authz-cache-exempt'.
+ private final IcebergLatestSnapshotCache latestSnapshotCache; // authz-cache-session-user-disabled
+ // PERF-01: cross-query cache of the RAW iceberg Table (restores the legacy IcebergExternalMetaCache table
+ // cache that the SPI cutover dropped). null when the catalog's credentials are query-dependent
+ // (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The per-statement scope
+ // (ConnectorStatementScope) shares one loaded table across a statement's read/scan/write regardless of this
+ // field, and is what a credential-gated catalog (this field null) relies on within a statement.
+ private final IcebergTableCache tableCache; // authz-cache-session-user-disabled
+ // PERF-02: cross-query partition-view cache (the raw PARTITIONS-scan result, keyed by (table, snapshotId)).
+ // The value is pure metadata (no FileIO/credential), but under session=user it is an authorization-sensitive
+ // projection (a shared hit would disclose one user's partitions), so it is disabled there (see constructor).
+ private final IcebergPartitionCache partitionCache; // authz-cache-session-user-disabled
+ // PERF-03: cross-query inferred-file-format cache (the whole-table planFiles() fallback result, keyed by
+ // (table, snapshotId)). Same authorization-sensitive treatment as partitionCache: disabled under session=user.
+ private final IcebergFormatCache formatCache; // authz-cache-session-user-disabled
+ // PERF-05: cross-query table-comment cache (value = the 'comment' property string). Built ONLY for a REST
+ // vended-credentials catalog that is NOT session=user -- plain catalogs already reuse tableCache (PERF-01) for
+ // the comment path, and session=user must stay live because the loadTable itself carries per-user
+ // authorization a shared cache would bypass. null for every other flavor.
+ private final IcebergCommentCache commentCache; // authz-cache-session-user-disabled
+ // Manifest content cache — pure metadata, default-off (meta.cache.iceberg.manifest.enable), and consumed
+ // ONLY after a per-user resolveTable(ForRead). authz-cache-exempt (no read path without a per-user load).
private final IcebergManifestCache manifestCache = new IcebergManifestCache();
- // commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply
- // across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it
- // into rewritable_delete_file_sets. Like the caches above, a REFRESH CATALOG rebuilds the connector and thus
- // drops it. Inert pre-cutover (iceberg scans/writes do not route through the providers until P6.6).
- private final IcebergRewritableDeleteStash rewritableDeleteStash = new IcebergRewritableDeleteStash();
// Lazily-built plugin-side Kerberos authenticator (single-owner auth; see TcclPinningConnectorContext).
// null for a non-Kerberos catalog. Its doAs acts on the PLUGIN's UserGroupInformation copy — the one the
@@ -185,8 +204,56 @@ public IcebergConnector(Map properties, ConnectorContext context
// authenticator never logs in — so without this the DDL/read hits secured HDFS as SIMPLE auth.
this.context = new TcclPinningConnectorContext(context, getClass().getClassLoader(),
this::pluginAuthenticator);
- this.latestSnapshotCache = new IcebergLatestSnapshotCache(
- resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ // Authorization-sensitive projection (snapshotId/schemaId). Under iceberg.rest.session=user the value is
+ // per-user AUTHORIZED metadata that a "can-list-cannot-load" principal must not see. beginQuerySnapshot
+ // reads this cache WITHOUT a preceding per-user loadTable, so a shared (table-keyed, no user dimension)
+ // hit would bypass the per-user authorization that lives inside loadTable (a metadata disclosure).
+ // Disabled (null) for session=user so beginQuerySnapshot re-loads live per-user every call (no stale-authz
+ // window); kept for every other flavor (single static identity, no cross-user axis). Mirrors the
+ // tableCache/commentCache discipline: session=user => no LIVE cross-query metadata cache.
+ this.latestSnapshotCache = isUserSessionEnabled()
+ ? null
+ : new IcebergLatestSnapshotCache(
+ resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ // PERF-01 cross-query RAW-table cache. Disabled (null) when the catalog's credentials are
+ // query-dependent, because a cached raw Table carries its FileIO's credentials:
+ // - iceberg.rest.session=user: per-user delegated FileIO -> sharing across users leaks credentials.
+ // - REST vended-credentials: the FileIO carries a server-vended token that expires within ~an hour;
+ // iceberg keeps it fresh by reloading the table each query, so a 24h-TTL hit would hand BE an
+ // expired token (403 mid-scan). Both gates are independent; either one disables this layer.
+ // The query-scoped fat handle stays on in all cases (its token is fresh within the one query). Same
+ // TTL/capacity as the snapshot cache (the single meta.cache.iceberg.table.ttl-second knob).
+ this.tableCache = (isUserSessionEnabled()
+ || IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties))
+ ? null
+ : new IcebergTableCache(
+ resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ // PERF-02: partition-view cache. Authorization-sensitive projection: a shared (table+snapshot-keyed, no
+ // user dimension) hit would disclose one user's partition list. Its readers are all downstream of a
+ // per-user resolveTableForRead today (so a hit cannot precede authz), but that safety rests entirely on
+ // tableCache being null under session=user. Disabled (null) under session=user makes it safe by
+ // construction and holds the "session=user => no live cross-query metadata cache" invariant; kept
+ // otherwise (single static identity). Readers already tolerate a null cache (loadRawPartitions).
+ this.partitionCache = isUserSessionEnabled()
+ ? null
+ : new IcebergPartitionCache(
+ resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ // PERF-03: inferred-file-format cache. Same authorization-sensitive treatment as partitionCache (disabled
+ // under session=user, kept otherwise); readers already tolerate a null cache (resolveFileFormatName).
+ this.formatCache = isUserSessionEnabled()
+ ? null
+ : new IcebergFormatCache(
+ resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
+ // PERF-05: table-comment cache, built ONLY for a REST vended-credentials catalog that is NOT session=user.
+ // Plain catalogs (tableCache on) already serve the comment path from tableCache; session=user is excluded
+ // because a shared comment cache would bypass the per-user loadTable authorization (a metadata disclosure).
+ // Comment is pure metadata (no credential) so no gate on the VALUE -- the flavor gate is an authorization
+ // decision, not a credential-leak one. Same TTL/capacity (ttl<=0 still disables internally).
+ this.commentCache = (IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties)
+ && !isUserSessionEnabled())
+ ? new IcebergCommentCache(
+ resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY)
+ : null;
}
/**
@@ -210,8 +277,8 @@ static long resolveTableCacheTtlSecond(Map properties) {
@Override
public ConnectorMetadata getMetadata(ConnectorSession session) {
- return new IcebergConnectorMetadata(
- newCatalogBackedOps(session), properties, context, latestSnapshotCache);
+ return new IcebergConnectorMetadata(newCatalogBackedOps(session), properties, context,
+ latestSnapshotCache, tableCache, partitionCache, commentCache);
}
/**
@@ -521,7 +588,21 @@ private IcebergCatalogOps newCatalogBackedOps(ConnectorSession session) {
*/
@Override
public void invalidateTable(String dbName, String tableName) {
- latestSnapshotCache.invalidate(TableIdentifier.of(dbName, tableName));
+ if (latestSnapshotCache != null) {
+ latestSnapshotCache.invalidate(TableIdentifier.of(dbName, tableName));
+ }
+ if (tableCache != null) {
+ tableCache.invalidate(TableIdentifier.of(dbName, tableName));
+ }
+ if (partitionCache != null) {
+ partitionCache.invalidate(TableIdentifier.of(dbName, tableName));
+ }
+ if (formatCache != null) {
+ formatCache.invalidate(TableIdentifier.of(dbName, tableName));
+ }
+ if (commentCache != null) {
+ commentCache.invalidate(TableIdentifier.of(dbName, tableName));
+ }
}
/**
@@ -537,7 +618,21 @@ public void invalidateTable(String dbName, String tableName) {
*/
@Override
public void invalidateDb(String dbName) {
- latestSnapshotCache.invalidateDb(dbName);
+ if (latestSnapshotCache != null) {
+ latestSnapshotCache.invalidateDb(dbName);
+ }
+ if (tableCache != null) {
+ tableCache.invalidateDb(dbName);
+ }
+ if (partitionCache != null) {
+ partitionCache.invalidateDb(dbName);
+ }
+ if (formatCache != null) {
+ formatCache.invalidateDb(dbName);
+ }
+ if (commentCache != null) {
+ commentCache.invalidateDb(dbName);
+ }
}
/**
@@ -549,7 +644,21 @@ public void invalidateDb(String dbName) {
*/
@Override
public void invalidateAll() {
- latestSnapshotCache.invalidateAll();
+ if (latestSnapshotCache != null) {
+ latestSnapshotCache.invalidateAll();
+ }
+ if (tableCache != null) {
+ tableCache.invalidateAll();
+ }
+ if (partitionCache != null) {
+ partitionCache.invalidateAll();
+ }
+ if (formatCache != null) {
+ formatCache.invalidateAll();
+ }
+ if (commentCache != null) {
+ commentCache.invalidateAll();
+ }
manifestCache.invalidateAll();
}
@@ -579,6 +688,31 @@ IcebergManifestCache manifestCacheForTest() {
return manifestCache;
}
+ /** Test-only: the cross-query table cache, or {@code null} when disabled by the credential gate (PERF-01). */
+ IcebergTableCache tableCacheForTest() {
+ return tableCache;
+ }
+
+ /** Test-only: the latest-snapshot cache, or {@code null} when disabled for a session=user catalog. */
+ IcebergLatestSnapshotCache latestSnapshotCacheForTest() {
+ return latestSnapshotCache;
+ }
+
+ /** Test-only: the cross-query partition-view cache (PERF-02), or {@code null} for a session=user catalog. */
+ IcebergPartitionCache partitionCacheForTest() {
+ return partitionCache;
+ }
+
+ /** Test-only: the cross-query inferred-file-format cache (PERF-03), or {@code null} for a session=user catalog. */
+ IcebergFormatCache formatCacheForTest() {
+ return formatCache;
+ }
+
+ /** Test-only: the table-comment cache (PERF-05), or {@code null} unless vended-credentials and non-session. */
+ IcebergCommentCache commentCacheForTest() {
+ return commentCache;
+ }
+
@Override
public ConnectorScanPlanProvider getScanPlanProvider() {
// Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built
@@ -588,7 +722,7 @@ public ConnectorScanPlanProvider getScanPlanProvider() {
// threaded for parity with the legacy single per-catalog IcebergMetadataOps.
return new IcebergScanPlanProvider(properties,
this::newCatalogBackedOps, context, manifestCache,
- rewritableDeleteStash);
+ tableCache, formatCache);
}
@Override
@@ -598,8 +732,7 @@ public ConnectorWritePlanProvider getWritePlanProvider() {
// IcebergConnectorTransaction. It resolves the target via catalogOps.loadTable, so it shares the
// fully-threaded ops (newCatalogBackedOps) — external_catalog.name must apply to INSERT/DELETE/MERGE.
return new IcebergWritePlanProvider(properties,
- this::newCatalogBackedOps, context,
- rewritableDeleteStash);
+ this::newCatalogBackedOps, context);
}
@Override
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
index 35027f5578f1dd..70f5d0a146e056 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
@@ -144,18 +144,54 @@ public class IcebergConnectorMetadata implements ConnectorMetadata {
// IcebergExternalMetaCache parity, mirrors paimon). The 3-arg ctor (direct-construction tests) passes a
// DISABLED cache so those reads stay always-live.
private final IcebergLatestSnapshotCache latestSnapshotCache;
+ // PERF-01: cross-query RAW-table cache (null = no cross-query layer). The 3-arg direct-construction tests
+ // and the credential-gated catalogs pass null; the query-scoped fat handle (IcebergTableHandle) works
+ // regardless. Consumed only by resolveTableForRead.
+ private final IcebergTableCache tableCache;
+ // PERF-02: cross-query partition-view cache (null = no cross-query layer; the convenience ctors used by
+ // direct-construction tests pass null). Consumed by getMvccPartitionView / listPartitions / listPartitionNames.
+ private final IcebergPartitionCache partitionCache;
+ // PERF-05: cross-query table-comment cache (null = no cross-query layer). Non-null only when the owning
+ // connector is a REST vended-credentials, non-session catalog (see IcebergConnector); the convenience ctors
+ // used by direct-construction tests pass null. Consumed only by getTableComment.
+ private final IcebergCommentCache commentCache;
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties,
ConnectorContext context) {
- this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1));
+ this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null, null);
}
+ /** Convenience ctor without a cross-query table cache (tableCache null); used by MVCC/statistics tests. */
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties,
ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache) {
+ this(catalogOps, properties, context, latestSnapshotCache, null, null);
+ }
+
+ /** Convenience ctor without a partition-view cache (partitionCache null). */
+ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties,
+ ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache,
+ IcebergTableCache tableCache) {
+ this(catalogOps, properties, context, latestSnapshotCache, tableCache, null);
+ }
+
+ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties,
+ ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache,
+ IcebergTableCache tableCache, IcebergPartitionCache partitionCache) {
+ this(catalogOps, properties, context, latestSnapshotCache, tableCache, partitionCache, null);
+ }
+
+ /** Full ctor used by {@link IcebergConnector#getMetadata}, adding the PERF-05 table-comment cache. */
+ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties,
+ ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache,
+ IcebergTableCache tableCache, IcebergPartitionCache partitionCache,
+ IcebergCommentCache commentCache) {
this.catalogOps = catalogOps;
this.properties = properties;
this.context = context;
this.latestSnapshotCache = latestSnapshotCache;
+ this.tableCache = tableCache;
+ this.partitionCache = partitionCache;
+ this.commentCache = commentCache;
}
// ========== ConnectorSchemaOps ==========
@@ -310,7 +346,20 @@ public String getTableComment(ConnectorSession session, String dbName, String ta
// Comment column would all be blank even though the raw comment key still appears in the SHOW CREATE
// PROPERTIES(...) block (F9/F12). Views render their comment through getViewDefinition / the view
// SHOW CREATE arm, so a view handle here (loadTable throws) falls back to "" via the caller's catch.
- Table table = loadTable(new IcebergTableHandle(dbName, tableName));
+ // PERF-05: on a vended-credentials (non-session) catalog, memoize the comment per table across queries so
+ // the per-table loadTable that information_schema.tables / SHOW TABLE STATUS pays for EVERY table collapses
+ // on repeats. commentCache is null for every other flavor (plain catalogs reuse tableCache via loadTable;
+ // session=user must stay live to preserve per-user authorization) -> resolve directly. A thrown load (view
+ // handle) is not cached and propagates to the caller's catch (still ""), so behavior is unchanged.
+ if (commentCache != null) {
+ return commentCache.getOrLoad(TableIdentifier.of(dbName, tableName),
+ () -> loadTableComment(session, dbName, tableName));
+ }
+ return loadTableComment(session, dbName, tableName);
+ }
+
+ private String loadTableComment(ConnectorSession session, String dbName, String tableName) {
+ Table table = loadTable(session, new IcebergTableHandle(dbName, tableName));
return table.properties().getOrDefault(TABLE_COMMENT_PROP, "");
}
@@ -345,7 +394,7 @@ public ConnectorTableSchema getTableSchema(
}
// Mirror legacy IcebergMetadataOps.loadTable: wrap the remote load in the auth context. The schema
// + table-property assembly is pure (operates on the already-loaded Table).
- Table table = loadTable(iceHandle);
+ Table table = loadTable(session, iceHandle);
return buildTableSchema(iceHandle.getTableName(), table, table.schema());
}
@@ -370,7 +419,7 @@ public ConnectorTableSchema getTableSchema(
if (snapshot == null || snapshot.getSchemaId() < 0) {
return getTableSchema(session, handle);
}
- Table table = loadTable(iceHandle);
+ Table table = loadTable(session, iceHandle);
Schema schema;
if (table.currentSnapshot() == null) {
// Empty table: legacy getSchema falls back to the latest schema (NEWEST_SCHEMA_ID path).
@@ -536,16 +585,38 @@ static String buildShowSortClause(Table table) {
return "ORDER BY (" + String.join(", ", sortItems) + ")";
}
- /** Loads the iceberg {@link Table} through the seam, wrapped in the FE-injected auth context (Kerberos UGI). */
- private Table loadTable(IcebergTableHandle handle) {
+ /**
+ * Loads the iceberg {@link Table} for {@code handle}, wrapped in the FE-injected auth context (Kerberos
+ * UGI). Resolution goes through {@link #resolveTableForRead} (fat handle -> cross-query cache ->
+ * remote), so the many reads sharing one handle in a planning/analysis pass collapse onto a single remote
+ * {@code loadTable} (PERF-01); a fat-handle hit returns without any remote call.
+ */
+ private Table loadTable(ConnectorSession session, IcebergTableHandle handle) {
try {
- return context.executeAuthenticated(
- () -> catalogOps.loadTable(handle.getDbName(), handle.getTableName()));
+ return context.executeAuthenticated(() -> resolveTableForRead(session, handle));
} catch (Exception e) {
throw new RuntimeException("Failed to load table, error message is:" + e.getMessage(), e);
}
}
+ /**
+ * Resolves the RAW iceberg {@link Table} for {@code handle}, WITHOUT opening an auth scope or wrapping
+ * exceptions — callers own both. The per-statement scope ({@link IcebergStatementScope#sharedTable}) comes
+ * first, so the statement's read metadata, scan planning and write all resolve the SAME one loaded object;
+ * on a scope miss the loader consults the cross-query {@link IcebergTableCache} when enabled (else a direct
+ * remote load). The remote loader's exception propagates verbatim (the cache re-throws it unwrapped), so a
+ * caller's own {@code NoSuchTableException} degradation (the partition-view readers) still fires. Callers
+ * needing the auth scope wrap the call in {@code executeAuthenticated} (see {@link #loadTable}). NOT used by
+ * the sys-table path ({@link #loadSysTable}), which takes a fresh remote base by design.
+ */
+ private Table resolveTableForRead(ConnectorSession session, IcebergTableHandle handle) {
+ return IcebergStatementScope.sharedTable(session, handle.getDbName(), handle.getTableName(),
+ () -> tableCache != null
+ ? tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()),
+ () -> catalogOps.loadTable(handle.getDbName(), handle.getTableName()))
+ : catalogOps.loadTable(handle.getDbName(), handle.getTableName()));
+ }
+
/**
* Loads the iceberg metadata (system) table for {@code handle} through the seam, wrapped in the
* FE-injected auth context (Kerberos UGI). Mirrors legacy
@@ -584,7 +655,7 @@ public Map getColumnHandles(
// Mirror getTableSchema: wrap the remote load in the auth context. A sys handle resolves the
// metadata-table columns (t$snapshots -> committed_at/...) so the generic scan node can look up
// its pruned sys-table slots by name; a data handle resolves the base table's columns.
- Table table = iceHandle.isSystemTable() ? loadSysTable(iceHandle) : loadTable(iceHandle);
+ Table table = iceHandle.isSystemTable() ? loadSysTable(iceHandle) : loadTable(session, iceHandle);
List fields = table.schema().columns();
Map handles = new LinkedHashMap<>(fields.size());
for (Types.NestedField field : fields) {
@@ -621,7 +692,7 @@ public Optional getTableStatistics(
}
long rowCount;
try {
- rowCount = computeRowCount(loadTable(iceHandle));
+ rowCount = computeRowCount(loadTable(session, iceHandle));
} catch (Exception e) {
LOG.warn("Failed to compute Iceberg row count for {}.{}",
iceHandle.getDbName(), iceHandle.getTableName(), e);
@@ -1372,7 +1443,7 @@ public void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHand
default:
return;
}
- Table table = loadTable((IcebergTableHandle) handle);
+ Table table = loadTable(session, (IcebergTableHandle) handle);
String mode = table.properties().getOrDefault(modeProperty, defaultMode);
if (RowLevelOperationMode.COPY_ON_WRITE.modeName().equalsIgnoreCase(mode)) {
throw new DorisConnectorException(String.format(
@@ -1397,7 +1468,7 @@ public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTa
return;
}
IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
- Table table = loadTable(iceHandle);
+ Table table = loadTable(session, iceHandle);
PartitionSpec spec = table.spec();
String tableName = iceHandle.getTableName();
if (!spec.isPartitioned()) {
@@ -1450,8 +1521,9 @@ public Optional getMvccPartitionView(
IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
try {
return context.executeAuthenticated(() -> {
- Table table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName());
- return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId()));
+ Table table = resolveTableForRead(session, iceHandle);
+ return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId(),
+ TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache));
});
} catch (Exception e) {
throw new RuntimeException("Failed to build iceberg MVCC partition view, error message is:"
@@ -1473,13 +1545,14 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH
return context.executeAuthenticated(() -> {
Table table;
try {
- table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName());
+ table = resolveTableForRead(session, iceHandle);
} catch (NoSuchTableException e) {
LOG.warn("Iceberg table not found while listing partitions: {}.{}",
iceHandle.getDbName(), iceHandle.getTableName(), e);
return Collections.emptyList();
}
- return IcebergPartitionUtils.listPartitionNames(table);
+ return IcebergPartitionUtils.listPartitionNames(table,
+ TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache);
});
} catch (Exception e) {
throw new RuntimeException("Failed to list iceberg partition names, error message is:"
@@ -1504,13 +1577,14 @@ public List listPartitions(ConnectorSession session,
return context.executeAuthenticated(() -> {
Table table;
try {
- table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName());
+ table = resolveTableForRead(session, iceHandle);
} catch (NoSuchTableException e) {
LOG.warn("Iceberg table not found while listing partitions: {}.{}",
iceHandle.getDbName(), iceHandle.getTableName(), e);
return Collections.emptyList();
}
- return IcebergPartitionUtils.listPartitions(table);
+ return IcebergPartitionUtils.listPartitions(table,
+ TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache);
});
} catch (Exception e) {
throw new RuntimeException("Failed to list iceberg partitions, error message is:"
@@ -1537,16 +1611,30 @@ public Optional beginQuerySnapshot(
ConnectorSession session, ConnectorTableHandle handle) {
IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
TableIdentifier id = TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName());
- IcebergLatestSnapshotCache.CachedSnapshot pin = latestSnapshotCache.getOrLoad(id, () -> {
- Table table = loadTable(iceHandle);
- Snapshot current = table.currentSnapshot();
- return new IcebergLatestSnapshotCache.CachedSnapshot(
- current == null ? -1L : current.snapshotId(), table.schema().schemaId());
- });
+ // A null latestSnapshotCache (iceberg.rest.session=user, where a shared table-keyed hit would bypass the
+ // per-user loadTable authorization) reads live per-user every call, so authorization runs every time (no
+ // stale-authz window); mirrors resolveTableForRead's tableCache null-fallback. A disabled cache (ttl<=0)
+ // is a non-null cache that reads live internally.
+ IcebergLatestSnapshotCache.CachedSnapshot pin = latestSnapshotCache != null
+ ? latestSnapshotCache.getOrLoad(id, () -> loadLatestSnapshotPin(session, iceHandle))
+ : loadLatestSnapshotPin(session, iceHandle);
return Optional.of(
ConnectorMvccSnapshot.builder().snapshotId(pin.snapshotId).schemaId(pin.schemaId).build());
}
+ /**
+ * Loads the table live and pins its latest snapshot id (or {@code -1} for an empty table) plus its LATEST
+ * schema id — the {@link #beginQuerySnapshot} loader, extracted so the per-user (null-cache) path can reuse
+ * the exact same pin logic without a cache round-trip.
+ */
+ private IcebergLatestSnapshotCache.CachedSnapshot loadLatestSnapshotPin(
+ ConnectorSession session, IcebergTableHandle iceHandle) {
+ Table table = loadTable(session, iceHandle);
+ Snapshot current = table.currentSnapshot();
+ return new IcebergLatestSnapshotCache.CachedSnapshot(
+ current == null ? -1L : current.snapshotId(), table.schema().schemaId());
+ }
+
/**
* Resolves an explicit time-travel {@code spec} into a pinned {@link ConnectorMvccSnapshot}, owning ALL
* iceberg-specific parsing. Mirrors legacy {@code IcebergUtils.getQuerySpecSnapshot}, returning
@@ -1571,7 +1659,7 @@ public Optional beginQuerySnapshot(
@Override
public Optional resolveTimeTravel(
ConnectorSession session, ConnectorTableHandle handle, ConnectorTimeTravelSpec spec) {
- Table table = loadTable((IcebergTableHandle) handle);
+ Table table = loadTable(session, (IcebergTableHandle) handle);
switch (spec.getKind()) {
case SNAPSHOT_ID: {
long id = Long.parseLong(spec.getStringValue());
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java
index dc6342b0fef3c3..b850f84c63fdc4 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java
@@ -205,7 +205,11 @@ public void beginWrite(ConnectorSession session, String db, String tableName, Ic
this.zone = IcebergTimeUtils.resolveSessionZone(session);
try {
context.executeAuthenticated(() -> {
- Table loaded = catalogOps.loadTable(db, tableName);
+ // PERF-07: resolve the table through the per-statement scope so a row-level DML reuses the SAME
+ // one object the scan already loaded (a write-only INSERT loads once here). openTransaction still
+ // issues newTransaction()'s refresh, giving the commit a fresh OCC base off that shared object.
+ Table loaded = IcebergStatementScope.sharedTable(session, db, tableName,
+ () -> catalogOps.loadTable(db, tableName));
this.table = loaded;
applyBeginGuards(ctx, tableName);
this.transaction = openTransaction(loaded);
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java
new file mode 100644
index 00000000000000..294b29b3738e8c
--- /dev/null
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java
@@ -0,0 +1,145 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.cache.CacheSpec;
+import org.apache.doris.connector.cache.MetaCacheEntry;
+
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+import java.util.Objects;
+import java.util.concurrent.ForkJoinPool;
+import java.util.function.Supplier;
+
+/**
+ * Per-catalog cache of an iceberg table's inferred write file-format name (PERF-03), keyed by
+ * {@code (TableIdentifier, snapshotId)}. Kills the per-query revival of the #64134 heavy fallback: when a table
+ * carries neither the {@code write-format} nickname nor {@code write.format.default} (migrated tables, and any
+ * table whose writer never set the property — parquet is an implicit default, not stored),
+ * {@link IcebergScanPlanProvider#getScanNodeProperties} resolves the scan-level {@code file_format_type} through
+ * {@link IcebergWriterHelper#getFileFormat}, which falls back to an unfiltered {@code table.newScan().planFiles()}
+ * — a whole-table manifest scan (remote FileIO) — just to read the FIRST data file's format. That fallback was
+ * re-run on every query/EXPLAIN with no cross-query reuse; this cache collapses it to one scan per
+ * {@code (table, snapshot)}.
+ *
+ * Snapshot-keyed, so always correct. A snapshot is immutable, so its first data file's format is a
+ * pure function of the key; a new commit yields a new snapshot id (a new key -> a live scan). Within the TTL
+ * the snapshot id itself is held stable by {@link IcebergLatestSnapshotCache}, which is what makes the key stable
+ * across queries. The key snapshot is the table's {@code currentSnapshot().snapshotId()} — exactly what the
+ * inference reads (it scans {@code table.newScan()}, never the handle's time-travel pin), matching legacy
+ * {@code IcebergUtils.getFileFormat} and PERF-02's partition cache verbatim.
+ *
+ *
Only the inference fallback is cached. The two cheap property probes ({@code write-format},
+ * {@code write.format.default}) stay outside this cache — the caller consults it only when both miss.
+ *
+ *
No credential gate (unlike {@link IcebergTableCache}, like {@link IcebergPartitionCache}): the cached
+ * value is a bare format-name {@link String} with no {@code FileIO} / credential, so it is safe to share across
+ * users and is built unconditionally (only the TTL knob disables it).
+ *
+ *
Backed identically to {@link IcebergPartitionCache}: a contextual, access-TTL {@link MetaCacheEntry} with
+ * manual miss-load, so the inference runs OUTSIDE Caffeine's compute lock and a failed scan's exception
+ * propagates verbatim and is NOT cached (the next query retries — legacy parity). TTL is
+ * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live). Lives on the long-lived
+ * per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache.
+ */
+final class IcebergFormatCache {
+
+ /** Immutable composite key: a table's inferred format is distinct per pinned snapshot id. */
+ static final class Key {
+ final TableIdentifier id;
+ final long snapshotId;
+
+ Key(TableIdentifier id, long snapshotId) {
+ this.id = id;
+ this.snapshotId = snapshotId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof Key)) {
+ return false;
+ }
+ Key that = (Key) o;
+ return snapshotId == that.snapshotId && Objects.equals(id, that.id);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(id, snapshotId);
+ }
+ }
+
+ private final MetaCacheEntry entry;
+
+ IcebergFormatCache(long ttlSeconds, int maxSize) {
+ // Mirror IcebergPartitionCache: translate the connector's "<= 0 disables" contract to CacheSpec's ttl == 0
+ // (disabled) rather than passing a negative value through (which CacheSpec reads as "no expiration").
+ CacheSpec spec = ttlSeconds > 0
+ ? CacheSpec.of(true, ttlSeconds, maxSize)
+ : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize);
+ this.entry = new MetaCacheEntry<>("iceberg-format", null, spec,
+ ForkJoinPool.commonPool(), false, true, 0L, true);
+ }
+
+ /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always infer live". */
+ boolean isEnabled() {
+ return entry.stats().isEffectiveEnabled();
+ }
+
+ /**
+ * Returns the cached format name for {@code key} if present and unexpired, else runs {@code loader} (the live
+ * whole-table format inference), caches and returns it. Disabled cache -> {@code loader} every call. The
+ * loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception propagates unwrapped
+ * and is NOT cached.
+ */
+ String getOrLoad(Key key, Supplier loader) {
+ return entry.get(key, ignored -> loader.get());
+ }
+
+ /** Drops every cached snapshot entry for one table so the next read infers live (REFRESH TABLE). */
+ void invalidate(TableIdentifier id) {
+ entry.invalidateIf(key -> key.id.equals(id));
+ }
+
+ /** Drops every cached entry for one database (REFRESH DATABASE / DROP DATABASE); db match = namespace equality. */
+ void invalidateDb(String dbName) {
+ Namespace ns = Namespace.of(dbName);
+ entry.invalidateIf(key -> key.id.namespace().equals(ns));
+ }
+
+ /** Drops all cached entries (REFRESH CATALOG). */
+ void invalidateAll() {
+ entry.invalidateAll();
+ }
+
+ /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */
+ int size() {
+ int[] count = {0};
+ entry.forEach((key, value) -> count[0]++);
+ return count[0];
+ }
+
+ /** Test-only: how many times the live loader (the whole-table format inference) actually ran — the metric gate. */
+ long loadCountForTest() {
+ return entry.stats().getLoadSuccessCount();
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java
index 689f1e7722dad4..f3c46732fb2fc1 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java
@@ -73,7 +73,7 @@ final class IcebergManifestCache {
// (ConnectorSession.getQueryId()), so VERBOSE EXPLAIN can report THIS scan's hits/misses/failures (the
// "manifest cache:" line). The provider that PLANS the scan and the (transient, fresh-per-call) provider that
// renders EXPLAIN are different instances, so per-scan state cannot live on the provider; the long-lived
- // per-catalog cache is their only shared survivor — same rationale as IcebergRewritableDeleteStash.
+ // per-catalog cache is their only shared survivor (the queryId keys this per-scan tally onto it).
// {@link #takeStats} is the primary eviction (EXPLAIN drains its entry); a non-EXPLAIN query records but
// never drains, so its leaked entry is aged out by the lazy TTL sweep in {@link #touch}.
private final Map statsByQuery = new ConcurrentHashMap<>();
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java
new file mode 100644
index 00000000000000..3d68ab664cfb87
--- /dev/null
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.cache.CacheSpec;
+import org.apache.doris.connector.cache.MetaCacheEntry;
+import org.apache.doris.connector.iceberg.IcebergPartitionUtils.IcebergRawPartition;
+
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.ForkJoinPool;
+import java.util.function.Supplier;
+
+/**
+ * Per-catalog cache of an iceberg table's raw partition list (PERF-02), keyed by {@code (TableIdentifier,
+ * snapshotId)}. Restores the partition-info half of the legacy {@code IcebergExternalMetaCache} that the SPI
+ * cutover dropped: the analysis-phase PARTITIONS metadata-table scan
+ * ({@link IcebergPartitionUtils#loadRawPartitionsUncached}, which the iceberg SDK materializes by reading EVERY
+ * data+delete manifest of the snapshot) was re-run per query and re-run 4~6 times per MTMV refresh, with no
+ * cross-query reuse. The three consumers ({@code buildMvccPartitionView} for the MVCC/MTMV partition view,
+ * {@code listPartitions} for {@code selectedPartitionNum}, {@code listPartitionNames} for SHOW PARTITIONS) all
+ * funnel through it, so they share a single scan per {@code (table, snapshot)}.
+ *
+ * Snapshot-keyed, so always correct. A snapshot is immutable, so the derived partitions are a pure
+ * function of the key; a new commit yields a new snapshot id (a new key -> a live scan). Within the TTL the
+ * snapshot id itself is held stable by {@link IcebergLatestSnapshotCache}, which is what makes the key stable
+ * across queries and across the enumeration points of one MTMV refresh.
+ *
+ *
No credential gate (unlike {@link IcebergTableCache}): the cached value is pure metadata (partition
+ * names, values, transforms, timestamps, snapshot ids) and carries no {@code FileIO} / credential, so it is
+ * safe to share across users and is built unconditionally (only the TTL knob disables it).
+ *
+ *
Backed identically to {@link IcebergLatestSnapshotCache}: a contextual, access-TTL {@link MetaCacheEntry}
+ * with manual miss-load, so the scan runs OUTSIDE Caffeine's compute lock and its exception (e.g. the
+ * dropped-partition-source-column {@link org.apache.iceberg.exceptions.ValidationException} that
+ * {@code listPartitions} degrades on) propagates verbatim and a failed scan is not cached. TTL is
+ * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live). Lives on the long-lived
+ * per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache.
+ */
+final class IcebergPartitionCache {
+
+ /** Immutable composite key: a table's partition list is distinct per pinned snapshot id. */
+ static final class Key {
+ final TableIdentifier id;
+ final long snapshotId;
+
+ Key(TableIdentifier id, long snapshotId) {
+ this.id = id;
+ this.snapshotId = snapshotId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof Key)) {
+ return false;
+ }
+ Key that = (Key) o;
+ return snapshotId == that.snapshotId && Objects.equals(id, that.id);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(id, snapshotId);
+ }
+ }
+
+ private final MetaCacheEntry> entry;
+
+ IcebergPartitionCache(long ttlSeconds, int maxSize) {
+ // Mirror IcebergLatestSnapshotCache: translate the connector's "<= 0 disables" contract to CacheSpec's
+ // ttl == 0 (disabled) rather than passing a negative value through (which CacheSpec reads as "no
+ // expiration").
+ CacheSpec spec = ttlSeconds > 0
+ ? CacheSpec.of(true, ttlSeconds, maxSize)
+ : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize);
+ this.entry = new MetaCacheEntry<>("iceberg-partition", null, spec,
+ ForkJoinPool.commonPool(), false, true, 0L, true);
+ }
+
+ /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always scan live". */
+ boolean isEnabled() {
+ return entry.stats().isEffectiveEnabled();
+ }
+
+ /**
+ * Returns the cached raw partition list for {@code key} if present and unexpired, else runs {@code loader}
+ * (the live PARTITIONS scan), caches and returns it. Disabled cache -> {@code loader} every call. The
+ * loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception propagates unwrapped.
+ */
+ List getOrLoad(Key key, Supplier> loader) {
+ return entry.get(key, ignored -> loader.get());
+ }
+
+ /** Drops every cached snapshot entry for one table so the next read scans live (REFRESH TABLE). */
+ void invalidate(TableIdentifier id) {
+ entry.invalidateIf(key -> key.id.equals(id));
+ }
+
+ /** Drops every cached entry for one database (REFRESH DATABASE / DROP DATABASE); db match = namespace equality. */
+ void invalidateDb(String dbName) {
+ Namespace ns = Namespace.of(dbName);
+ entry.invalidateIf(key -> key.id.namespace().equals(ns));
+ }
+
+ /** Drops all cached entries (REFRESH CATALOG). */
+ void invalidateAll() {
+ entry.invalidateAll();
+ }
+
+ /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */
+ int size() {
+ int[] count = {0};
+ entry.forEach((key, value) -> count[0]++);
+ return count[0];
+ }
+
+ /** Test-only: how many times the live loader (the PARTITIONS scan) actually ran — the metric gate. */
+ long loadCountForTest() {
+ return entry.stats().getLoadSuccessCount();
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java
index 45b191180670c1..31767a03c4aef2 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java
@@ -36,6 +36,7 @@
import org.apache.iceberg.Snapshot;
import org.apache.iceberg.StructLike;
import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.types.Type;
@@ -530,6 +531,16 @@ static List parsePartitionValuesFromJson(String partitionDataJson) {
* handle), or {@code < 0} to enumerate at the table's current (latest) snapshot.
*/
static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinnedSnapshotId) {
+ return buildMvccPartitionView(table, pinnedSnapshotId, null, null);
+ }
+
+ /**
+ * Cache-aware overload (PERF-02): {@code id} + {@code cache} route the PARTITIONS scan through the
+ * per-catalog {@link IcebergPartitionCache} keyed by {@code (id, resolvedSnapshotId)}. {@code cache == null}
+ * reads live (offline tests / no-cache catalog).
+ */
+ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinnedSnapshotId,
+ TableIdentifier id, IcebergPartitionCache cache) {
if (!isValidRelatedTable(table)) {
return ConnectorMvccPartitionView.unpartitioned();
}
@@ -562,7 +573,7 @@ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinne
// two rows rendering the same field=value name have byte-identical ranges, and feeding both to the merge
// would make the name self-enclose and be dropped (0 partitions where master keeps 1).
Map allByName = new LinkedHashMap<>();
- for (IcebergRawPartition raw : loadRawPartitions(table, snapshotId)) {
+ for (IcebergRawPartition raw : loadRawPartitions(id, table, snapshotId, cache)) {
RangeBuild rb = buildRange(raw.name, raw.values.get(0), raw.transforms.get(0), sourceType,
raw.lastUpdateTime, raw.lastSnapshotId);
allByName.put(rb.name, rb);
@@ -602,6 +613,12 @@ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinne
* unpartitioned or empty table yields an empty list. Must run inside {@code context.executeAuthenticated}.
*/
static List listPartitionNames(Table table) {
+ return listPartitionNames(table, null, null);
+ }
+
+ /** Cache-aware overload (PERF-02): see {@link #buildMvccPartitionView(Table, long, TableIdentifier,
+ * IcebergPartitionCache)}. Keyed by the table's CURRENT snapshot id. */
+ static List listPartitionNames(Table table, TableIdentifier id, IcebergPartitionCache cache) {
if (table.spec().isUnpartitioned()) {
return Collections.emptyList();
}
@@ -609,7 +626,7 @@ static List listPartitionNames(Table table) {
if (current == null) {
return Collections.emptyList();
}
- List raws = loadRawPartitions(table, current.snapshotId());
+ List raws = loadRawPartitions(id, table, current.snapshotId(), cache);
List names = new ArrayList<>(raws.size());
for (IcebergRawPartition raw : raws) {
names.add(raw.name);
@@ -629,6 +646,12 @@ static List listPartitionNames(Table table) {
* {@code context.executeAuthenticated}.
*/
static List listPartitions(Table table) {
+ return listPartitions(table, null, null);
+ }
+
+ /** Cache-aware overload (PERF-02): see {@link #buildMvccPartitionView(Table, long, TableIdentifier,
+ * IcebergPartitionCache)}. Keyed by the table's CURRENT snapshot id. */
+ static List listPartitions(Table table, TableIdentifier id, IcebergPartitionCache cache) {
if (table.spec().isUnpartitioned()) {
return Collections.emptyList();
}
@@ -638,7 +661,7 @@ static List listPartitions(Table table) {
}
List raws;
try {
- raws = loadRawPartitions(table, current.snapshotId());
+ raws = loadRawPartitions(id, table, current.snapshotId(), cache);
} catch (ValidationException e) {
if (!isDroppedPartitionSourceColumn(e)) {
// A different iceberg validation error (not the dropped-partition-source-column case) — fail loud.
@@ -706,7 +729,26 @@ static boolean isValidRelatedTable(Table table) {
* {@code last_updated_at} (row 9) / {@code last_updated_snapshot_id} (row 10) are optional, so a missing
* value (NPE on the typed getter) degrades to {@code 0} / {@code UNKNOWN_SNAPSHOT_ID}, exactly like master.
*/
- private static List loadRawPartitions(Table table, long snapshotId) {
+ /**
+ * The cross-query PARTITIONS-scan de-duplication seam (PERF-02): when {@code cache} is non-null the raw
+ * partition list is served from / populated into the per-catalog {@link IcebergPartitionCache} keyed by
+ * {@code (id, snapshotId)} — a snapshot is immutable, so the derived partitions are a pure function of that
+ * key and safe to reuse across queries (restoring the legacy IcebergExternalMetaCache partition-info cache).
+ * A {@code null} cache (offline unit tests / the no-cache catalog) reads live every call. The cached list is
+ * unmodifiable so a shared entry cannot be mutated by a concurrent reader; the loader's exception (e.g. the
+ * dropped-partition-source-column {@link ValidationException}) propagates verbatim so callers keep their own
+ * degradation, and a failed scan is not cached.
+ */
+ private static List loadRawPartitions(TableIdentifier id, Table table, long snapshotId,
+ IcebergPartitionCache cache) {
+ if (cache == null) {
+ return loadRawPartitionsUncached(table, snapshotId);
+ }
+ return cache.getOrLoad(new IcebergPartitionCache.Key(id, snapshotId),
+ () -> Collections.unmodifiableList(loadRawPartitionsUncached(table, snapshotId)));
+ }
+
+ private static List loadRawPartitionsUncached(Table table, long snapshotId) {
Table partitionsTable = MetadataTableUtils.createMetadataTableInstance(table, MetadataTableType.PARTITIONS);
List partitions = new ArrayList<>();
try (CloseableIterable tasks = partitionsTable.newScan().useSnapshot(snapshotId).planFiles()) {
@@ -902,7 +944,7 @@ static long latestSnapshotId(String name, Map> mergeMap,
}
/** One PARTITIONS-metadata-table row reduced to what the MTMV partition view needs (port of IcebergPartition). */
- private static final class IcebergRawPartition {
+ static final class IcebergRawPartition {
private final String name;
// Partition-field SOURCE column names (lowercased), parallel to {@link #values}, so listPartitions can
// build a value map keyed by the generic partition-column remote name (see IcebergConnectorMetadata
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java
deleted file mode 100644
index 3f153371b8bb95..00000000000000
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java
+++ /dev/null
@@ -1,140 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.connector.iceberg;
-
-import org.apache.doris.thrift.TIcebergDeleteFileDesc;
-
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.TimeUnit;
-import java.util.function.LongSupplier;
-
-/**
- * Per-catalog stash that carries the merge-on-read "rewritable delete" supply across the scan→write seam of a
- * single row-level DML (DELETE / UPDATE / MERGE) on a format-version≥3 iceberg table.
- *
- * Why a singleton stash. When a DML rewrites the deletes of a data file, the BE writes a NEW deletion
- * vector that must OR-merge that file's still-live old deletes (old DVs + old position deletes) — otherwise the
- * previously-deleted rows resurrect. The BE does not read iceberg metadata; the FE must hand it, per touched
- * data file, the list of old non-equality delete files via {@code rewritable_delete_file_sets}
- * ({@code TIcebergDeleteSink}/{@code TIcebergMergeSink}). The legacy fe-core path collected this from the
- * {@code IcebergScanNode}'s own fields at sink-finalize time (plan-scoped, GC'd with the plan). In the plugin
- * SPI the scan-plan provider and the write-plan provider are mutually-unaware, used-once objects; the only
- * cross-scan→write survivor is the long-lived per-catalog {@link IcebergConnector}. So the scan provider stashes
- * the supply here keyed by the statement's stable {@code queryId} ({@code ConnectorSession.getQueryId()} =
- * {@code DebugUtil.printId(ctx.queryId())}, identical across the scan and write sessions of one statement), and
- * the write provider retrieves it by the same key.
- *
- *
Key = RAW data-file path. The map is keyed on the un-normalized {@code dataFile.path()} string
- * ({@link IcebergScanRange#getOriginalPath()}), because the BE matches a rewritable set to the file it is writing
- * a DV for by exact string equality against that raw path (mirrors legacy
- * {@code IcebergScanNode.deleteFilesByReferencedDataFile} keyed on {@code getOriginalPath()}). Keying on the
- * scheme-normalized open path would silently miss every lookup → resurrection.
- *
- *
Eviction. The primary removal is the write provider's eager {@link #retrieveAndRemove} per statement
- * (covers every DML write — DELETE/MERGE consume the supply, INSERT/OVERWRITE simply discard it). A statement
- * that scans a v3-delete table but never writes (a plain {@code SELECT}, or a DML that errors before the write
- * is planned) leaves a leaked entry that {@link #retrieveAndRemove} never reaches; a lazy TTL sweep (run when a
- * new query is first seen) ages those out. Unlike a value cache, the sweep removes ONLY entries untouched for
- * longer than the TTL — never a live entry (a statement's scan→write gap is milliseconds, far below the TTL), so
- * a live supply is never dropped (which would itself resurrect rows). With a unique per-statement queryId a
- * leaked entry is unreachable (no later statement reuses the key), so a leak is bounded memory, not a stale read.
- */
-final class IcebergRewritableDeleteStash {
-
- // Leak backstop only: a statement's scan→write gap is milliseconds (both happen in one planning pass, before
- // BE execution), so this TTL is orders of magnitude larger than any live entry's lifetime — it ages out only
- // leaked plain-SELECT / aborted-DML entries.
- private static final long DEFAULT_TTL_SECONDS = 300L;
-
- /** One in-flight statement's supply: rawDataFilePath -> its non-equality delete descs, plus a touch stamp. */
- private static final class Entry {
- // Concurrent because, defensively, a single statement could plan more than one iceberg scan (MERGE scans
- // both the target and the source table) — data-file paths are globally unique, so accumulating distinct
- // keys (and idempotently overwriting a split data file's identical list) is always safe.
- final Map> sets = new ConcurrentHashMap<>();
- volatile long lastTouchNanos;
-
- Entry(long nowNanos) {
- this.lastTouchNanos = nowNanos;
- }
- }
-
- private final Map stash = new ConcurrentHashMap<>();
- private final long ttlNanos;
- private final LongSupplier nanoClock;
-
- IcebergRewritableDeleteStash() {
- this(DEFAULT_TTL_SECONDS, System::nanoTime);
- }
-
- /** Visible for testing: injectable TTL + clock so sweep is deterministic without sleeping. */
- IcebergRewritableDeleteStash(long ttlSeconds, LongSupplier nanoClock) {
- this.ttlNanos = TimeUnit.SECONDS.toNanos(Math.max(1L, ttlSeconds));
- this.nanoClock = nanoClock;
- }
-
- /**
- * Records, for {@code queryId}, the non-equality delete descs of one touched data file (keyed on its RAW
- * path). No-op when the queryId is blank (a null/absent {@code ConnectContext} coerces the queryId to "",
- * which would collide across concurrent statements — such a statement gets no supply, which is safe because a
- * real row-level DML always carries a non-null query id) or when there is nothing to supply. Splits of one
- * data file carry an identical delete list, so the per-path {@code put} is idempotent; two tables in one
- * statement contribute distinct paths.
- */
- void accumulate(String queryId, String rawDataFilePath, List deleteDescs) {
- if (queryId == null || queryId.isEmpty() || rawDataFilePath == null
- || deleteDescs == null || deleteDescs.isEmpty()) {
- return;
- }
- long now = nanoClock.getAsLong();
- Entry entry = stash.get(queryId);
- if (entry == null) {
- // First range of a not-yet-seen query: opportunistically age out leaked entries. Done OUTSIDE the
- // computeIfAbsent mapping function (ConcurrentHashMap forbids mutating other mappings from within it).
- sweepExpired(now);
- entry = stash.computeIfAbsent(queryId, k -> new Entry(now));
- }
- entry.lastTouchNanos = now;
- entry.sets.put(rawDataFilePath, deleteDescs);
- }
-
- /**
- * Returns and removes the supply map for {@code queryId} (the rewritable delete sets keyed by raw data-file
- * path), or {@code null} when none was stashed / the queryId is blank. The write provider calls this once per
- * statement; the remove is the primary eviction.
- */
- Map> retrieveAndRemove(String queryId) {
- if (queryId == null || queryId.isEmpty()) {
- return null;
- }
- Entry entry = stash.remove(queryId);
- return entry == null ? null : entry.sets;
- }
-
- /** Drops entries untouched for longer than the TTL — leaked plain-SELECT / aborted-DML supplies only. */
- private void sweepExpired(long nowNanos) {
- stash.entrySet().removeIf(e -> nowNanos - e.getValue().lastTouchNanos >= ttlNanos);
- }
-
- /** Test-only: current number of in-flight stashed statements. */
- int size() {
- return stash.size();
- }
-}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
index 70cd39f5f5e076..b53465392350e5 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
@@ -37,6 +37,7 @@
import org.apache.doris.thrift.TIcebergFileDesc;
import org.apache.doris.thrift.TTableFormatFileDesc;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.iceberg.BaseFileScanTask;
import org.apache.iceberg.BaseTable;
import org.apache.iceberg.BatchScan;
@@ -65,6 +66,7 @@
import org.apache.iceberg.TableOperations;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
+import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.expressions.Expression;
import org.apache.iceberg.expressions.Expressions;
@@ -94,6 +96,7 @@
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
@@ -104,6 +107,7 @@
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Function;
+import java.util.function.UnaryOperator;
/**
* {@link ConnectorScanPlanProvider} for Iceberg tables, mirroring the paimon connector's
@@ -216,12 +220,15 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider {
// Nullable — null via the 2-/3-arg ctors (offline tests, default-disabled gate); when null the gate is
// forced off and planScan uses the SDK splitFiles path.
private final IcebergManifestCache manifestCache;
- // commit-bridge supply (S4 part 2): owned by the long-lived IcebergConnector, shared with the write provider.
- // A format-version>=3 DELETE/MERGE scan stashes its non-equality delete supply here keyed by queryId; the
- // write provider retrieves it to fill rewritable_delete_file_sets. Nullable — null via the 2-/3-/4-arg ctors
- // (offline tests), in which case stashing is skipped (the supply is exercised only on the post-cutover write
- // path; pre-flip the provider never runs at all).
- private final IcebergRewritableDeleteStash rewritableDeleteStash;
+ // PERF-01: cross-query RAW-table cache shared with the metadata layer, owned by the long-lived
+ // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors and
+ // when the connector's credential gate disables the cross-query layer; when null resolveTable still uses the
+ // per-statement scope and falls back to a direct remote load.
+ private final IcebergTableCache tableCache;
+ // PERF-03: cross-query inferred-file-format cache shared with the connector, owned by the long-lived
+ // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors; when null
+ // getScanNodeProperties resolves file_format_type live (matching pre-PERF-03 behaviour, node-memoized per query).
+ private final IcebergFormatCache formatCache;
// FIX-SCAN-METRICS: per-query stash of the iceberg SDK scan diagnostics captured by the attached
// IcebergScanProfileReporter during planScan, keyed by session queryId. fe-core drains it
@@ -230,26 +237,37 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider {
// system-table, which fe-core never drains), so the value list is appended single-threaded.
private final ConcurrentHashMap> scanProfileStash = new ConcurrentHashMap<>();
+ // Test-only gate for the PERF-11 per-file memo: how many times computePerFileInvariants actually ran across
+ // this provider's scans. A file split into k byte-slices must increment it ONCE (not k times), proving the
+ // per-slice recompute collapsed to per-file. Not reset per scan — a test uses a fresh provider.
+ @VisibleForTesting
+ int perFileInvariantComputeCount;
+
public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps) {
- this(properties, catalogOps, null, null, null);
+ this(properties, catalogOps, null, null);
}
public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps,
ConnectorContext context) {
- this(properties, catalogOps, context, null, null);
+ this(properties, catalogOps, context, null);
}
public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps,
ConnectorContext context, IcebergManifestCache manifestCache) {
- this(properties, catalogOps, context, manifestCache, null);
+ // Constant resolver: these ctors (offline tests + the pre-session connector paths) bind a single ops that
+ // ignores the session, so existing behaviour/tests are byte-identical. No cross-query cache (tableCache
+ // null) — the per-statement scope still dedups within a statement.
+ this(properties, session -> catalogOps, context, manifestCache, null);
}
- public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps,
- ConnectorContext context, IcebergManifestCache manifestCache,
- IcebergRewritableDeleteStash rewritableDeleteStash) {
- // Constant resolver: these ctors (offline tests + the pre-session connector paths) bind a single ops that
- // ignores the session, so existing behaviour/tests are byte-identical.
- this(properties, session -> catalogOps, context, manifestCache, rewritableDeleteStash);
+ /**
+ * Session-aware convenience ctor without a cross-query table cache (tableCache null); used by the offline
+ * session-routing tests. The per-statement scope still dedups within a statement.
+ */
+ public IcebergScanPlanProvider(Map properties,
+ Function catalogOpsResolver,
+ ConnectorContext context, IcebergManifestCache manifestCache) {
+ this(properties, catalogOpsResolver, context, manifestCache, null);
}
/**
@@ -260,13 +278,25 @@ public IcebergScanPlanProvider(Map properties, IcebergCatalogOps
*/
public IcebergScanPlanProvider(Map properties,
Function catalogOpsResolver,
- ConnectorContext context, IcebergManifestCache manifestCache,
- IcebergRewritableDeleteStash rewritableDeleteStash) {
+ ConnectorContext context, IcebergManifestCache manifestCache, IcebergTableCache tableCache) {
+ this(properties, catalogOpsResolver, context, manifestCache, tableCache, null);
+ }
+
+ /**
+ * Full ctor used by {@link IcebergConnector#getScanPlanProvider()}, adding the PERF-03 cross-query
+ * inferred-file-format cache ({@code formatCache}). The 5-arg ctor delegates here with a null format cache
+ * (offline tests + pre-cache paths resolve {@code file_format_type} live).
+ */
+ public IcebergScanPlanProvider(Map properties,
+ Function catalogOpsResolver,
+ ConnectorContext context, IcebergManifestCache manifestCache, IcebergTableCache tableCache,
+ IcebergFormatCache formatCache) {
this.properties = properties;
this.catalogOpsResolver = catalogOpsResolver;
this.context = context;
this.manifestCache = manifestCache;
- this.rewritableDeleteStash = rewritableDeleteStash;
+ this.tableCache = tableCache;
+ this.formatCache = formatCache;
}
/**
@@ -457,12 +487,36 @@ public ConnectorSplitSource streamSplits(ConnectorSession session, ConnectorTabl
boolean partitioned = table.spec().isPartitioned();
Map vendedToken = context != null
? extractVendedToken(table, restVendedCredentialsEnabled()) : Collections.emptyMap();
+ UnaryOperator uriNormalizer = newUriNormalizer(vendedToken);
long fileSplitSize = sessionLong(session, FILE_SPLIT_SIZE, 0L);
long sliceSize = fileSplitSize > 0 ? fileSplitSize
: sessionLong(session, MAX_FILE_SPLIT_SIZE, DEFAULT_MAX_FILE_SPLIT_SIZE);
- CloseableIterable tasks = TableScanUtil.splitFiles(scan.planFiles(), sliceSize);
+ CloseableIterable tasks = streamingFileScanTasks(scan, session, table, filter, sliceSize);
return new IcebergStreamingSplitSource(tasks, table, formatVersion, partitioned,
- orderedPartitionKeys, zone, vendedToken, sliceSize, iceHandle.getRewriteFileScope());
+ orderedPartitionKeys, zone, uriNormalizer, sliceSize, iceHandle.getRewriteFileScope());
+ }
+
+ /**
+ * The streaming source's whole-file enumeration, byte-offset-split at {@code sliceSize}. PERF-04 (C17): when
+ * the manifest cache is enabled, read manifests THROUGH THE CACHE via the lazy {@link #cacheBackedFileScanTasks}
+ * (no-stats overload — Phase 2 iterates on the engine pump thread, see there) so a big streaming scan finally
+ * hits the cache while staying lazy (OOM-safe). An eager Phase-1 cache failure records it and falls back to the
+ * SDK {@code planFiles()} path (mirrors {@link #planFileScanTask}); a later lazy failure surfaces through the
+ * streaming source's {@code hasNext}. Cache disabled -> the SDK path, byte-unchanged.
+ */
+ private CloseableIterable streamingFileScanTasks(TableScan scan, ConnectorSession session,
+ Table table, Optional filter, long sliceSize) {
+ if (isManifestCacheEnabled()) {
+ try {
+ return TableScanUtil.splitFiles(
+ cacheBackedFileScanTasks(scan, session, table, filter, null), sliceSize);
+ } catch (Exception e) {
+ LOG.warn("Iceberg streaming plan with manifest cache failed, falling back to SDK scan: {}",
+ e.getMessage(), e);
+ manifestCache.recordFailure(session.getQueryId());
+ }
+ }
+ return TableScanUtil.splitFiles(scan.planFiles(), sliceSize);
}
/**
@@ -479,7 +533,7 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource
private final boolean partitioned;
private final List orderedPartitionKeys;
private final ZoneId zone;
- private final Map vendedToken;
+ private final UnaryOperator uriNormalizer;
private final long sliceSize;
private final Set rewriteScope;
// Lazily opened on first hasNext() so the ctor never throws — iceberg's ParallelIterable submits
@@ -490,17 +544,20 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource
private CloseableIterator iterator;
// Look-ahead buffer so hasNext() can skip data files filtered out by the rewrite scope.
private IcebergScanRange buffered;
+ // Per-file invariant cache (PERF-11): per split-source = per scan; the pump is single-threaded, so the
+ // 1-entry cache stays O(1) memory (never accumulates), preserving the streaming path's OOM safety.
+ private final PerFileScratch scratch = new PerFileScratch();
IcebergStreamingSplitSource(CloseableIterable tasks, Table table, int formatVersion,
boolean partitioned, List orderedPartitionKeys, ZoneId zone,
- Map vendedToken, long sliceSize, Set rewriteScope) {
+ UnaryOperator uriNormalizer, long sliceSize, Set rewriteScope) {
this.tasks = tasks;
this.table = table;
this.formatVersion = formatVersion;
this.partitioned = partitioned;
this.orderedPartitionKeys = orderedPartitionKeys;
this.zone = zone;
- this.vendedToken = vendedToken;
+ this.uriNormalizer = uriNormalizer;
this.sliceSize = sliceSize;
this.rewriteScope = rewriteScope;
}
@@ -515,7 +572,7 @@ public boolean hasNext() {
}
while (iterator.hasNext()) {
IcebergScanRange range = buildRangeForTask(iterator.next(), table, formatVersion, partitioned,
- orderedPartitionKeys, zone, vendedToken, sliceSize, rewriteScope, false, null);
+ orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, null, scratch);
if (range != null) {
buffered = range;
return true;
@@ -585,6 +642,9 @@ private List planScanInternal(
// BE-credential overlay is emitted separately by getScanNodeProperties.
Map vendedToken = context != null
? extractVendedToken(table, restVendedCredentialsEnabled()) : Collections.emptyMap();
+ // Derive the vended storage config ONCE per scan (the token is scan-invariant) and reuse it for every
+ // per-file path normalization below, instead of rebuilding it per data/delete file (C3).
+ UnaryOperator uriNormalizer = newUriNormalizer(vendedToken);
// COUNT(*) pushdown (T05): when the count is servable from the snapshot summary, collapse the scan to
// a single whole-file range carrying the full count (mirrors paimon's collapse + legacy's <=10000
@@ -595,7 +655,7 @@ private List planScanInternal(
long realCount = getCountFromSnapshot(scan, session);
if (realCount >= 0) {
return planCountPushdown(table, scan, realCount, formatVersion, partitioned,
- orderedPartitionKeys, zone, vendedToken);
+ orderedPartitionKeys, zone, uriNormalizer, session, filter);
}
}
@@ -603,15 +663,15 @@ private List planScanInternal(
// and emit one BE-ready IcebergScanRange per task, populating the typed iceberg carriers — incl. the
// merge-on-read delete files (T04) — mirroring legacy IcebergScanNode.createIcebergSplit. The field-id
// history dict (T06, scan-level), MVCC pin, and vended credentials (T09) land later.
- // commit-bridge supply (S4 part 2): for a format-version>=3 scan, stash each data file's non-equality
- // delete supply (old DVs + old position deletes) keyed by the statement queryId, so a DELETE/MERGE write
- // on the same statement can fill rewritable_delete_file_sets and the BE OR-merges those old deletes into
- // the new deletion vector — a missing supply silently resurrects previously-deleted rows. queryId is read
- // once (stable across this statement's scan and write sessions). Skipped pre-v3 and when the stash is
- // absent (offline tests / pre-cutover the provider never runs); a non-DML scan just leaves a leaked entry
- // the stash ages out, and accumulate() itself no-ops a blank queryId or an empty (no non-eq delete) list.
- boolean stashRewritableDeletes = rewritableDeleteStash != null && formatVersion >= 3;
- String stashQueryId = stashRewritableDeletes ? session.getQueryId() : null;
+ // commit-bridge supply (S4 part 2): for a format-version>=3 scan, accumulate each data file's non-equality
+ // delete supply (old DVs + old position deletes) into the per-statement scope, so a DELETE/MERGE write on
+ // the same statement can fill rewritable_delete_file_sets and the BE OR-merges those old deletes into the
+ // new deletion vector — a missing supply silently resurrects previously-deleted rows. The scope is keyed by
+ // catalog id + queryId (shared across this statement's scan and write, isolated per catalog for a
+ // cross-catalog MERGE). Skipped pre-v3; a non-DML scan just leaves an entry GC'd with the statement, and an
+ // absent scope (offline) yields a throwaway map that the write seam guards against (fail loud on v3 DML).
+ Map> rewritableDeleteSupply = formatVersion >= 3
+ ? IcebergStatementScope.rewritableDeleteSupply(session) : null;
// WS-REWRITE R2 per-group scope: when the handle carries a rewrite file scope (the engine
// rewrite_data_files driver sets it before each group's INSERT-SELECT), keep ONLY the data files in
@@ -623,14 +683,18 @@ private List planScanInternal(
// (buildRange re-attaches task.deletes()), so scoping never drops a delete binding.
Set rewriteScope = iceHandle.getRewriteFileScope();
+ // Per-file invariant cache (PERF-11): ONE instance per scan, never shared across scans. Reused across
+ // a data file's consecutive byte-slices so partition JSON / identity map / delete carriers are computed
+ // once per file, not per slice. The streaming source below holds its own.
+ PerFileScratch scratch = new PerFileScratch();
List ranges = new ArrayList<>();
try (SplitPlan plan = planFileScanTask(scan, session, table, filter)) {
for (FileScanTask task : plan.tasks) {
// Shared per-task mapping (rewrite-scope skip + M-2 weight denominator + v3 stash side-effect),
// identical to the streaming path's IcebergStreamingSplitSource so both produce the same ranges.
IcebergScanRange range = buildRangeForTask(task, table, formatVersion, partitioned,
- orderedPartitionKeys, zone, vendedToken, plan.targetSplitSize, rewriteScope,
- stashRewritableDeletes, stashQueryId);
+ orderedPartitionKeys, zone, uriNormalizer, plan.targetSplitSize, rewriteScope,
+ rewritableDeleteSupply, scratch);
if (range != null) {
ranges.add(range);
}
@@ -681,26 +745,35 @@ private static String rootCauseMessage(Throwable t) {
/**
* Map one {@link FileScanTask} to its BE-ready {@link IcebergScanRange}, applying the rewrite-scope filter
* (returns {@code null} to skip a data file outside the scope) and the v3 commit-bridge rewritable-delete
- * stash side-effect. Shared by the synchronous {@link #planScanInternal} loop and the streaming
+ * accumulation. Shared by the synchronous {@link #planScanInternal} loop and the streaming
* {@code IcebergStreamingSplitSource} so both paths produce byte-identical ranges and never drop a
- * side-effect. The streaming path passes {@code stashRewritableDeletes=false} (v3 is gated onto the eager
- * path — see {@link #streamingSplitEstimate}), so the stash is inert there.
+ * side-effect. The streaming path passes {@code rewritableDeleteSupply=null} (v3 is gated onto the eager
+ * path — see {@link #streamingSplitEstimate}), so the accumulation is inert there.
*/
private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, int formatVersion,
boolean partitioned, List orderedPartitionKeys, ZoneId zone,
- Map vendedToken, long targetSplitSize, Set rewriteScope,
- boolean stashRewritableDeletes, String stashQueryId) {
+ UnaryOperator uriNormalizer, long targetSplitSize, Set rewriteScope,
+ Map> rewritableDeleteSupply, PerFileScratch scratch) {
DataFile dataFile = task.file();
if (rewriteScope != null && !rewriteScope.contains(dataFile.path().toString())) {
return null;
}
+ // First byte-slice of a new data file? (scratch still holds the previous file until buildRange refreshes
+ // it below — so capture this BEFORE the buildRange call.) The v3 rewritable-delete supply is identical
+ // for every slice of a file, so record it exactly ONCE per file — on the file's first slice, keyed to
+ // the new file, so the LAST file in the stream is never dropped.
+ boolean firstSliceOfFile = scratch.file != dataFile;
// targetSplitSize is the scan-level weight denominator (M-2): each data-file range carries a
// size-proportional BE scheduling weight (selfSplitWeight computed inside buildRange).
IcebergScanRange range = buildRange(table, dataFile, task, formatVersion, partitioned,
- orderedPartitionKeys, zone, vendedToken, -1, targetSplitSize);
- if (stashRewritableDeletes) {
- rewritableDeleteStash.accumulate(stashQueryId, range.getOriginalPath(),
- range.rewritableDeleteDescs());
+ orderedPartitionKeys, zone, uriNormalizer, -1, targetSplitSize, scratch);
+ if (rewritableDeleteSupply != null && firstSliceOfFile) {
+ // Record this data file's non-equality delete supply keyed on its RAW path (the exact string the BE
+ // matches a rewritable set against). An empty list (no old non-eq deletes) contributes nothing.
+ List descs = range.rewritableDeleteDescs();
+ if (range.getOriginalPath() != null && descs != null && !descs.isEmpty()) {
+ rewritableDeleteSupply.put(range.getOriginalPath(), descs);
+ }
}
return range;
}
@@ -873,12 +946,13 @@ private List doPlanPositionDeletesSystemTableScan(IcebergTab
ZoneId zone = resolveSessionZone(session);
Map vendedToken = context != null
? extractVendedToken(metadataTable, restVendedCredentialsEnabled()) : Collections.emptyMap();
+ UnaryOperator uriNormalizer = newUriNormalizer(vendedToken);
List ranges = new ArrayList<>();
for (PositionDeletesScanTask task : tasks) {
for (PositionDeletesScanTask splitTask : splitPositionDeleteScanTask(task, targetSplitSize)) {
ranges.add(buildPositionDeleteRange(splitTask, metadataTable, outputPartitionFields,
- enableMappingVarbinary, zone, vendedToken));
+ enableMappingVarbinary, zone, uriNormalizer));
}
}
LOG.debug("Iceberg planScan produced {} position_deletes splits for {}.{}", ranges.size(),
@@ -898,11 +972,11 @@ private static Iterable splitPositionDeleteScanTask(Pos
*/
private IcebergScanRange buildPositionDeleteRange(PositionDeletesScanTask task, Table metadataTable,
List outputPartitionFields, boolean enableMappingVarbinary, ZoneId zone,
- Map vendedToken) {
+ UnaryOperator uriNormalizer) {
DeleteFile deleteFile = task.file();
String originalPath = deleteFile.path().toString();
IcebergScanRange.Builder builder = new IcebergScanRange.Builder()
- .path(normalizeUri(originalPath, vendedToken))
+ .path(uriNormalizer.apply(originalPath))
.start(task.start())
.length(task.length())
.fileSize(deleteFile.fileSizeInBytes())
@@ -1090,13 +1164,13 @@ private static Schema pinnedSchema(Table table, IcebergTableHandle handle) {
*/
private List planCountPushdown(Table table, TableScan scan, long realCount,
int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone,
- Map vendedToken) {
- try (CloseableIterable tasks = scan.planFiles()) {
+ UnaryOperator uriNormalizer, ConnectorSession session, Optional filter) {
+ try (CloseableIterable tasks = countPushdownFileScanTasks(scan, session, table, filter)) {
for (FileScanTask task : tasks) {
// targetSplitSize = -1: the count-pushdown collapse emits a single range, so its scheduling
// weight is irrelevant → PluginDrivenSplit keeps SplitWeight.standard().
return Collections.singletonList(buildRange(table, task.file(), task, formatVersion,
- partitioned, orderedPartitionKeys, zone, vendedToken, realCount, -1));
+ partitioned, orderedPartitionKeys, zone, uriNormalizer, realCount, -1, null));
}
} catch (IOException e) {
throw new RuntimeException("Failed to plan iceberg count-pushdown file, error message is:"
@@ -1105,16 +1179,117 @@ private List planCountPushdown(Table table, TableScan scan,
return Collections.emptyList();
}
+ /**
+ * The COUNT(*)-pushdown placeholder enumeration: only the FIRST surviving file is consumed (BE serves the
+ * count from {@code table_level_row_count} and never reads the file). PERF-04 (C18): when the manifest cache is
+ * enabled, read through the lazy {@link #cacheBackedFileScanTasks} (stats overload — this runs on the single
+ * planning thread) so the manifest reads are cache hits and, being lazy, stop at the first file's manifest
+ * instead of the SDK {@code planFiles()}'s {@code ParallelIterable} eagerly submitting every manifest reader.
+ * An eager cache failure falls back to the SDK path (mirrors {@link #planFileScanTask}). The first surviving
+ * (pruned) file may differ from the SDK path's first file (its {@code ParallelIterable} order is
+ * non-deterministic), but the count is identical (from the snapshot summary) and BE ignores the file. Cache
+ * disabled -> the SDK path, byte-unchanged.
+ */
+ private CloseableIterable countPushdownFileScanTasks(TableScan scan, ConnectorSession session,
+ Table table, Optional filter) {
+ if (isManifestCacheEnabled()) {
+ try {
+ return cacheBackedFileScanTasks(scan, session, table, filter, session.getQueryId());
+ } catch (Exception e) {
+ LOG.warn("Iceberg count-pushdown plan with manifest cache failed, falling back to SDK scan: {}",
+ e.getMessage(), e);
+ manifestCache.recordFailure(session.getQueryId());
+ }
+ }
+ return scan.planFiles();
+ }
+
+ /**
+ * Per-file scratch for {@link #buildRange}: the values identical for every byte-slice of one data file
+ * ({@code TableScanUtil.splitFiles} cuts a file into k slices whose {@code FileScanTask}s all return the
+ * SAME {@code DataFile} instance from {@code file()}, and emits them consecutively). Computed once on a
+ * file change and reused across the file's slices, collapsing the per-slice partition-JSON / identity-map /
+ * delete-carrier recompute to per-file (PERF-11 / C12); the k ranges then share the same immutable
+ * {@code partitionValues} / {@code deleteCarriers} instances (C15a). The eager loop and the streaming
+ * source each own ONE instance, never shared across scans. {@code file == null} = fresh / unused.
+ */
+ private static final class PerFileScratch {
+ private DataFile file;
+ private Integer partitionSpecId;
+ private String partitionDataJson;
+ private Map partitionValues = Collections.emptyMap();
+ private List deleteCarriers = Collections.emptyList();
+ private String fileFormat;
+ private Long firstRowId;
+ private Long lastUpdatedSequenceNumber;
+ private String rawDataPath;
+ private String normalizedPath;
+ }
+
/**
* Build the BE-ready {@link IcebergScanRange} for one {@link FileScanTask}, mirroring legacy
* {@code IcebergScanNode.createIcebergSplit} + {@code setIcebergParams}: the file path/offset/size, the
* per-file format (native parquet/orc), the table format version, the v3 row-lineage fields, and — for a
* partitioned table — the partition spec-id, the all-fields {@code partition_data_json}, and the ordered
* identity {@code partitionValues} that become columns-from-path.
+ *
+ * The per-file-invariant work is memoized through {@code scratch} (keyed by the shared {@code DataFile}
+ * instance) and reused across the file's byte-slices; only {@code start} / {@code length} / the
+ * size-proportional {@code selfSplitWeight} are per-slice. A {@code null} scratch (the single
+ * count-pushdown range) computes without caching. Byte-identical to the former per-slice recompute.
*/
private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask task, int formatVersion,
boolean partitioned, List orderedPartitionKeys, ZoneId zone,
- Map vendedToken, long pushDownRowCount, long targetSplitSize) {
+ UnaryOperator uriNormalizer, long pushDownRowCount, long targetSplitSize,
+ PerFileScratch scratch) {
+ PerFileScratch file = (scratch != null && scratch.file == dataFile)
+ ? scratch
+ : computePerFileInvariants(table, dataFile, task, formatVersion, partitioned,
+ orderedPartitionKeys, zone, uriNormalizer, scratch);
+ // M-2 size-proportional weight numerator = this split's byte length + the byte size of every
+ // merge-on-read delete file applying to it, mirroring legacy IcebergSplit.selfSplitWeight (ctor sets
+ // = length; setDeleteFileFilters adds Σ delete fileSizeInBytes). The delete-size sum is file-invariant
+ // but the task.length() term is per byte-slice, so the whole weight stays per-slice (NEVER memoized).
+ // The denominator (targetSplitSize) is passed in; for the single count-pushdown range it is -1 →
+ // PluginDrivenSplit keeps SplitWeight.standard() (one range, weight irrelevant).
+ long selfSplitWeight = task.length();
+ if (task.deletes() != null) {
+ for (DeleteFile delete : task.deletes()) {
+ selfSplitWeight += delete.fileSizeInBytes();
+ }
+ }
+ return new IcebergScanRange.Builder()
+ .path(file.normalizedPath)
+ .originalPath(file.rawDataPath)
+ .start(task.start())
+ .length(task.length())
+ .fileSize(dataFile.fileSizeInBytes())
+ .fileFormat(file.fileFormat)
+ .formatVersion(formatVersion)
+ .partitionSpecId(file.partitionSpecId)
+ .partitionDataJson(file.partitionDataJson)
+ .firstRowId(file.firstRowId)
+ .lastUpdatedSequenceNumber(file.lastUpdatedSequenceNumber)
+ .partitionValues(file.partitionValues)
+ .deleteFiles(file.deleteCarriers)
+ .pushDownRowCount(pushDownRowCount)
+ .selfSplitWeight(selfSplitWeight)
+ .targetSplitSize(targetSplitSize)
+ .build();
+ }
+
+ /**
+ * Compute {@link #buildRange}'s per-file invariants and, when {@code scratch} is non-null, store them into
+ * it so the file's remaining byte-slices reuse them. Uses {@code task.deletes()} (identical across a
+ * file's slices) for the delete carriers. Mirrors the former inline per-slice computation exactly — same
+ * partition-JSON / identity-map ordering, same fail-loud on a non-parquet/orc file, same v3 row-lineage —
+ * so the memoized range is byte-identical to the per-slice recompute. The scratch fields are assigned only
+ * after the fail-loud check, so a rejected file never leaves a half-populated scratch.
+ */
+ private PerFileScratch computePerFileInvariants(Table table, DataFile dataFile, FileScanTask task,
+ int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone,
+ UnaryOperator uriNormalizer, PerFileScratch scratch) {
+ perFileInvariantComputeCount++;
Integer partitionSpecId = null;
String partitionDataJson = null;
Map partitionValues = Collections.emptyMap();
@@ -1155,40 +1330,22 @@ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask
dataFile.fileSequenceNumber() != null && dataFile.firstRowId() != null
? dataFile.fileSequenceNumber() : -1L;
}
- // M-2 size-proportional weight numerator = this split's byte length + the byte size of every
- // merge-on-read delete file applying to it, mirroring legacy IcebergSplit.selfSplitWeight (ctor sets
- // = length; setDeleteFileFilters adds Σ delete fileSizeInBytes). task.deletes() is a cached list (no
- // extra I/O; buildDeleteFiles reads the same one). The denominator (targetSplitSize) is passed in; for
- // the single count-pushdown range it is -1 → PluginDrivenSplit keeps SplitWeight.standard() (one range,
- // weight irrelevant), matching the normal-data-only weighting.
- long selfSplitWeight = task.length();
- if (task.deletes() != null) {
- for (DeleteFile delete : task.deletes()) {
- selfSplitWeight += delete.fileSizeInBytes();
- }
- }
// The range path BE opens is scheme-normalized (legacy createIcebergSplit:852 normalizes via the
// 2-arg LocationPath.of(path, storagePropertiesMap)); original_file_path stays raw so BE can match
// position-delete entries against the raw iceberg path (legacy setOriginalFilePath:304).
String rawDataPath = dataFile.path().toString();
- return new IcebergScanRange.Builder()
- .path(normalizeUri(rawDataPath, vendedToken))
- .originalPath(rawDataPath)
- .start(task.start())
- .length(task.length())
- .fileSize(dataFile.fileSizeInBytes())
- .fileFormat(fileFormat)
- .formatVersion(formatVersion)
- .partitionSpecId(partitionSpecId)
- .partitionDataJson(partitionDataJson)
- .firstRowId(firstRowId)
- .lastUpdatedSequenceNumber(lastUpdatedSequenceNumber)
- .partitionValues(partitionValues)
- .deleteFiles(buildDeleteFiles(task, vendedToken))
- .pushDownRowCount(pushDownRowCount)
- .selfSplitWeight(selfSplitWeight)
- .targetSplitSize(targetSplitSize)
- .build();
+ PerFileScratch file = scratch != null ? scratch : new PerFileScratch();
+ file.file = dataFile;
+ file.partitionSpecId = partitionSpecId;
+ file.partitionDataJson = partitionDataJson;
+ file.partitionValues = partitionValues;
+ file.fileFormat = fileFormat;
+ file.firstRowId = firstRowId;
+ file.lastUpdatedSequenceNumber = lastUpdatedSequenceNumber;
+ file.rawDataPath = rawDataPath;
+ file.normalizedPath = uriNormalizer.apply(rawDataPath);
+ file.deleteCarriers = buildDeleteFiles(task, uriNormalizer);
+ return file;
}
/**
@@ -1196,14 +1353,15 @@ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask
* mirroring legacy {@code IcebergScanNode.getDeleteFileFilters} + {@code IcebergDeleteFileFilter}. Empty
* for v1 / no-delete files (v1 has no delete files, so {@code task.deletes()} is always empty there).
*/
- private List buildDeleteFiles(FileScanTask task, Map vendedToken) {
+ private List buildDeleteFiles(FileScanTask task,
+ UnaryOperator uriNormalizer) {
List deletes = task.deletes();
if (deletes == null || deletes.isEmpty()) {
return Collections.emptyList();
}
List result = new ArrayList<>(deletes.size());
for (DeleteFile delete : deletes) {
- result.add(convertDelete(delete, vendedToken));
+ result.add(convertDelete(delete, uriNormalizer));
}
return result;
}
@@ -1218,13 +1376,13 @@ private List buildDeleteFiles(FileScanTask task, Ma
* {@code EQUALITY_DELETES} → an equality delete (content 2) with the delete-file's equality
* field-ids (read straight from delete metadata — correct independent of the T06 data dictionary).
*
- * The delete path is normalized through the engine seam (legacy
- * {@code LocationPath.of(path,config).toStorageLocation()}), threading the per-table vended token (empty
- * for non-REST) so a REST object-store deletion path normalizes via the vended map (T09). Package-private
- * for direct unit testing.
+ * The delete path is normalized through the scan-scoped {@code uriNormalizer} (legacy
+ * {@code LocationPath.of(path,config).toStorageLocation()}), which bakes in the per-table vended token
+ * (empty for non-REST) so a REST object-store deletion path normalizes via the vended map (T09).
+ * Package-private for direct unit testing.
*/
- IcebergScanRange.DeleteFile convertDelete(DeleteFile delete, Map vendedToken) {
- String path = normalizeUri(delete.path().toString(), vendedToken);
+ IcebergScanRange.DeleteFile convertDelete(DeleteFile delete, UnaryOperator uriNormalizer) {
+ String path = uriNormalizer.apply(delete.path().toString());
FileContent content = delete.content();
if (content == FileContent.POSITION_DELETES) {
Long lowerBound = readPositionBound(delete.lowerBounds());
@@ -1281,20 +1439,22 @@ private static TFileFormatType deleteFileFormat(FileFormat format) {
}
/**
- * Normalize a raw iceberg storage path (the data file BE opens, or a delete file) to BE's canonical
- * scheme via the engine seam (legacy goes through {@code LocationPath.of(path, storagePropertiesMap)
- * .toStorageLocation()}; the connector cannot import fe-core's {@code LocationPath}). BE's
- * scheme-dispatched S3 factory only opens {@code s3://}, so an un-normalized {@code oss://}/{@code cos://}
- * /{@code obs://}/{@code s3a://} path fails the native read (data file) or silently drops the deletes
+ * Build the scan-scoped URI normalizer once (where the per-table vended token is extracted) and thread it
+ * through the per-file range builders, instead of re-deriving the vended storage config per data/delete
+ * file. Each application normalizes a raw iceberg storage path (the data file BE opens, or a delete file)
+ * to BE's canonical scheme via the engine seam (legacy goes through {@code LocationPath.of(path,
+ * storagePropertiesMap).toStorageLocation()}; the connector cannot import fe-core's {@code LocationPath}).
+ * BE's scheme-dispatched S3 factory only opens {@code s3://}, so an un-normalized {@code oss://}/{@code
+ * cos://}/{@code obs://}/{@code s3a://} path fails the native read (data file) or silently drops the deletes
* (merge-on-read wrong rows). Mirrors paimon's {@code normalizeUri} (FIX-URI-NORMALIZE), which normalizes
* both the data-file and deletion-vector paths. The {@code vendedToken} (empty for non-REST / no context)
- * is the per-table vended credential map, routed into normalization so a REST object-store path normalizes
- * via the vended map (T09); when empty the 2-arg seam folds to the catalog's static storage map, byte-
- * equivalent to legacy for non-vended catalogs. A {@code null} context (offline unit tests) preserves the
- * raw path (paimon parity).
+ * is the per-table vended credential map, baked into the normalizer so a REST object-store path normalizes
+ * via the vended map (T09); when empty the seam folds to the catalog's static storage map, byte-equivalent
+ * to legacy for non-vended catalogs. A {@code null} context (offline unit tests) yields an identity
+ * normalizer that preserves the raw path (paimon parity).
*/
- private String normalizeUri(String rawPath, Map vendedToken) {
- return context != null ? context.normalizeStorageUri(rawPath, vendedToken) : rawPath;
+ UnaryOperator newUriNormalizer(Map vendedToken) {
+ return context != null ? context.newStorageUriNormalizer(vendedToken) : UnaryOperator.identity();
}
/**
@@ -1381,8 +1541,14 @@ public Map getScanNodeProperties(
// V1-only reader bug. Emit the table's real data format (legacy IcebergScanNode.getFileFormatType
// parity, same IcebergUtils.getFileFormat resolution, which throws on a non-parquet/orc table); the
// per-file format still travels per range, since one table may mix parquet and orc data files.
+ // PERF-03: the non-system format resolution falls back to an unfiltered whole-table planFiles() when the
+ // table sets neither write-format nor write.format.default; memoize that inference per (table, snapshot)
+ // across queries via formatCache (pure metadata, no credential gate). Null cache (offline) resolves live.
props.put("file_format_type",
- systemTable ? "jni" : IcebergWriterHelper.getFileFormat(table).name().toLowerCase(Locale.ROOT));
+ systemTable ? "jni"
+ : IcebergWriterHelper.getFileFormat(table,
+ TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), formatCache)
+ .name().toLowerCase(Locale.ROOT));
// [D-065] System (metadata) tables ($snapshots/$files/...) read via the JNI serialized-split path
// (planSystemTableScan): the metadata-table schema travels INSIDE the serialized FileScanTask, so BE
// needs neither the base-table path_partition_keys (a metadata table is not base-spec partitioned ->
@@ -1768,24 +1934,62 @@ private SplitPlan planFileScanTask(TableScan scan, ConnectorSession session, Tab
}
/**
- * Manifest-level planning that consumes {@link IcebergManifestCache}, ported faithfully from legacy
- * {@code IcebergScanNode.planFileScanTaskWithManifestCache}. It reconstructs iceberg's own planning:
- * partition-prune manifests with a {@link ManifestEvaluator}, read each surviving manifest's data/delete
- * files THROUGH THE CACHE, then per data file apply the {@link InclusiveMetricsEvaluator} (file-stats
- * pruning) + {@link ResidualEvaluator} (partition residual) and attach its deletes via a
- * {@link DeleteFileIndex}. The resulting {@link FileScanTask}s are byte-offset-split exactly like
- * {@link #splitFiles}, so the downstream {@code buildRange} (T03-T07) is unchanged. The predicate /
- * metrics / schema use the table's CURRENT schema (legacy parity).
+ * Synchronous (below-batch-threshold) manifest-cache planning: materialize the shared lazy
+ * {@link #cacheBackedFileScanTasks} enumeration into a list, because {@link #determineTargetFileSplitSize}
+ * (the per-table heuristic that gives small tables good BE parallelism) needs the whole task list. Byte
+ * identical to the pre-PERF-04 body; the streaming ({@link #streamSplits}) and COUNT(*)
+ * ({@link #planCountPushdown}) paths consume the same enumeration LAZILY instead (bounded FE heap). Uses the
+ * stats-tallying cache overload — this path runs on the single planning thread. The predicate / metrics /
+ * schema use the table's CURRENT schema (legacy parity).
*/
private SplitPlan planFileScanTaskWithManifestCache(TableScan scan,
ConnectorSession session, Table table, Optional filter) throws IOException {
+ // Null-safe queryId (offline tests pass a null session): a null id selects the no-stats cache overload,
+ // matching the pre-PERF-04 body, which read scan.snapshot() and returned early for an empty table BEFORE
+ // ever calling session.getQueryId().
+ String statsQueryId = session != null ? session.getQueryId() : null;
+ List tasks = new ArrayList<>();
+ try (CloseableIterable whole =
+ cacheBackedFileScanTasks(scan, session, table, filter, statsQueryId)) {
+ for (FileScanTask task : whole) {
+ tasks.add(task);
+ }
+ }
+ long targetSplitSize = determineTargetFileSplitSize(tasks, session);
+ return new SplitPlan(
+ TableScanUtil.splitFiles(CloseableIterable.withNoopClose(tasks), targetSplitSize), targetSplitSize);
+ }
+
+ /**
+ * PERF-04: the lazy, cache-backed {@link FileScanTask} enumeration shared by the synchronous
+ * ({@link #planFileScanTaskWithManifestCache}), streaming ({@link #streamSplits}), and COUNT(*)
+ * ({@link #planCountPushdown}) paths whenever {@link #isManifestCacheEnabled()}. Ported faithfully from legacy
+ * {@code IcebergScanNode.planFileScanTaskWithManifestCache}: partition-prune manifests with a
+ * {@link ManifestEvaluator}, read each surviving manifest's data/delete files THROUGH THE CACHE, then per data
+ * file apply the {@link InclusiveMetricsEvaluator} (file-stats prune) + {@link ResidualEvaluator} (partition
+ * residual) and attach its deletes via a {@link DeleteFileIndex}. Produces WHOLE-FILE
+ * {@link BaseFileScanTask}s (callers byte-offset-split via {@link TableScanUtil#splitFiles}).
+ *
+ * Phase 1 is eager (delete-manifest reads + {@link DeleteFileIndex} build): a data file's deletes
+ * need the full index before any data task can be produced, and running it here (on the caller's thread) lets
+ * the streaming/count callers catch a cache failure and fall back to the SDK path. Bounded like the SDK
+ * {@code planFiles()} (which also reads all delete manifests up front); delete manifests are far fewer than
+ * data manifests. Phase 2 is lazy: the returned iterable's iterator flat-maps the matching data
+ * manifests, yielding one surviving task at a time WITHOUT materializing the list, so a million-file streaming
+ * scan keeps FE heap bounded (peak = the delete index + one manifest's files + the split queue).
+ *
+ *
{@code statsQueryId} is nullable: a non-null id tallies cache hits/misses under that query (the
+ * single-threaded synchronous + COUNT paths); {@code null} selects the no-stats overload for the STREAMING
+ * path, whose Phase 2 runs on the engine pump thread while Phase 1 ran on the calling thread — tallying the
+ * per-query {@code ScanStats} counters from two threads would race (the cache documents them as
+ * single-thread-per-query), and streaming reports no manifest-cache stats today anyway.
+ */
+ private CloseableIterable cacheBackedFileScanTasks(TableScan scan,
+ ConnectorSession session, Table table, Optional filter, String statsQueryId) {
Snapshot snapshot = scan.snapshot();
if (snapshot == null) {
- return new SplitPlan(CloseableIterable.withNoopClose(Collections.emptyList()), -1);
+ return CloseableIterable.withNoopClose(Collections.emptyList());
}
- // Stable per-statement key so VERBOSE EXPLAIN (rendered on a different, transient provider instance) can
- // report THIS scan's manifest-cache hits/misses via the shared per-catalog cache.
- String queryId = session.getQueryId();
Expression filterExpr = combineFilter(filter, table, session);
Map specsById = table.specs();
boolean caseSensitive = true;
@@ -1795,8 +1999,9 @@ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan,
ResidualEvaluator.of(spec, filterExpr, caseSensitive)));
InclusiveMetricsEvaluator metricsEvaluator =
new InclusiveMetricsEvaluator(table.schema(), filterExpr, caseSensitive);
+ String schemaJson = SchemaParser.toJson(table.schema());
- // Phase 1: partition-prune + cache-load delete manifests into a flat delete-file list.
+ // Phase 1 (eager): partition-prune + cache-load delete manifests into the delete-file index.
List deleteFiles = new ArrayList<>();
for (ManifestFile manifest : snapshot.deleteManifests(table.io())) {
if (manifest.content() != ManifestContent.DELETES) {
@@ -1809,50 +2014,131 @@ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan,
if (!ManifestEvaluator.forPartitionFilter(filterExpr, spec, caseSensitive).eval(manifest)) {
continue;
}
- deleteFiles.addAll(manifestCache.getManifestCacheValue(manifest, table, queryId).getDeleteFiles());
+ deleteFiles.addAll(manifestCacheGet(manifest, table, statsQueryId).getDeleteFiles());
}
DeleteFileIndex deleteIndex = DeleteFileIndex.builderFor(deleteFiles)
.specsById(specsById)
.caseSensitive(caseSensitive)
.build();
- // Phase 2: partition-prune + cache-load data manifests, then file-level prune + attach deletes.
- List tasks = new ArrayList<>();
- try (CloseableIterable dataManifests =
- getMatchingManifest(snapshot.dataManifests(table.io()), specsById, filterExpr)) {
- for (ManifestFile manifest : dataManifests) {
- if (manifest.content() != ManifestContent.DATA) {
- continue;
- }
- PartitionSpec spec = specsById.get(manifest.partitionSpecId());
- if (spec == null) {
- continue;
- }
- ResidualEvaluator residualEvaluator = residualEvaluators.get(manifest.partitionSpecId());
- if (residualEvaluator == null) {
- continue;
- }
- ManifestCacheValue value = manifestCache.getManifestCacheValue(manifest, table, queryId);
- for (DataFile dataFile : value.getDataFiles()) {
+ // Phase 2 (lazy): flat-map the matching data manifests, read through the cache on demand.
+ CloseableIterable dataManifests =
+ getMatchingManifest(snapshot.dataManifests(table.io()), specsById, filterExpr);
+ return new CloseableIterable() {
+ @Override
+ public CloseableIterator iterator() {
+ return new ManifestCacheFileScanTaskIterator(dataManifests.iterator(), table, specsById,
+ residualEvaluators, metricsEvaluator, deleteIndex, schemaJson, statsQueryId);
+ }
+
+ @Override
+ public void close() throws IOException {
+ dataManifests.close();
+ }
+ };
+ }
+
+ /** Dispatch a manifest read to the stats-tallying or no-stats cache overload (see cacheBackedFileScanTasks). */
+ private ManifestCacheValue manifestCacheGet(ManifestFile manifest, Table table, String statsQueryId) {
+ return statsQueryId != null
+ ? manifestCache.getManifestCacheValue(manifest, table, statsQueryId)
+ : manifestCache.getManifestCacheValue(manifest, table);
+ }
+
+ /**
+ * Lazy flat-map iterator behind {@link #cacheBackedFileScanTasks}: walks the matching data manifests, reads
+ * each through the manifest cache on demand, and yields the surviving (metrics + residual pruned)
+ * {@link BaseFileScanTask}s one file at a time so the consumer never holds the whole table's task list. Not
+ * thread-safe: single-pass, driven by ONE consumer (the sync materialize loop, the streaming pump, or the
+ * COUNT take-first). {@code schemaJson}/{@code currentSpecJson} are hoisted loop invariants (constant per
+ * table / per manifest spec), byte-identical to the pre-PERF-04 per-file computation.
+ */
+ private final class ManifestCacheFileScanTaskIterator implements CloseableIterator {
+ private final CloseableIterator manifestIt;
+ private final Table table;
+ private final Map specsById;
+ private final Map residualEvaluators;
+ private final InclusiveMetricsEvaluator metricsEvaluator;
+ private final DeleteFileIndex deleteIndex;
+ private final String schemaJson;
+ private final String statsQueryId;
+
+ private Iterator dataFileIt;
+ private ResidualEvaluator currentResidual;
+ private String currentSpecJson;
+ private FileScanTask next;
+
+ ManifestCacheFileScanTaskIterator(CloseableIterator manifestIt, Table table,
+ Map specsById, Map residualEvaluators,
+ InclusiveMetricsEvaluator metricsEvaluator, DeleteFileIndex deleteIndex, String schemaJson,
+ String statsQueryId) {
+ this.manifestIt = manifestIt;
+ this.table = table;
+ this.specsById = specsById;
+ this.residualEvaluators = residualEvaluators;
+ this.metricsEvaluator = metricsEvaluator;
+ this.deleteIndex = deleteIndex;
+ this.schemaJson = schemaJson;
+ this.statsQueryId = statsQueryId;
+ }
+
+ @Override
+ public boolean hasNext() {
+ advance();
+ return next != null;
+ }
+
+ @Override
+ public FileScanTask next() {
+ advance();
+ if (next == null) {
+ throw new NoSuchElementException();
+ }
+ FileScanTask result = next;
+ next = null;
+ return result;
+ }
+
+ // Fill `next` with the next surviving task, draining the current data manifest's files and advancing to
+ // further data manifests as needed. Per-file logic mirrors the pre-PERF-04 materialized Phase 2 exactly.
+ private void advance() {
+ while (next == null) {
+ if (dataFileIt != null && dataFileIt.hasNext()) {
+ DataFile dataFile = dataFileIt.next();
if (!metricsEvaluator.eval(dataFile)) {
continue;
}
- if (residualEvaluator.residualFor(dataFile.partition()).equals(Expressions.alwaysFalse())) {
+ if (currentResidual.residualFor(dataFile.partition()).equals(Expressions.alwaysFalse())) {
continue;
}
DeleteFile[] deletes = deleteIndex.forDataFile(dataFile.dataSequenceNumber(), dataFile);
- tasks.add(new BaseFileScanTask(
- dataFile,
- deletes,
- SchemaParser.toJson(table.schema()),
- PartitionSpecParser.toJson(spec),
- residualEvaluator));
+ next = new BaseFileScanTask(dataFile, deletes, schemaJson, currentSpecJson, currentResidual);
+ return;
+ }
+ if (!manifestIt.hasNext()) {
+ return;
+ }
+ ManifestFile manifest = manifestIt.next();
+ if (manifest.content() != ManifestContent.DATA) {
+ dataFileIt = null;
+ continue;
+ }
+ PartitionSpec spec = specsById.get(manifest.partitionSpecId());
+ ResidualEvaluator residual = residualEvaluators.get(manifest.partitionSpecId());
+ if (spec == null || residual == null) {
+ dataFileIt = null;
+ continue;
}
+ currentResidual = residual;
+ currentSpecJson = PartitionSpecParser.toJson(spec);
+ dataFileIt = manifestCacheGet(manifest, table, statsQueryId).getDataFiles().iterator();
}
}
- long targetSplitSize = determineTargetFileSplitSize(tasks, session);
- return new SplitPlan(
- TableScanUtil.splitFiles(CloseableIterable.withNoopClose(tasks), targetSplitSize), targetSplitSize);
+
+ @Override
+ public void close() throws IOException {
+ manifestIt.close();
+ }
}
/**
@@ -2053,17 +2339,36 @@ static ZoneId resolveSessionZone(ConnectorSession session) {
* {@code null} context (offline unit tests / simple-auth) resolves directly.
*/
private Table resolveTable(ConnectorSession session, IcebergTableHandle handle) {
- // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim.
+ // Per-statement scope (PERF-07): the statement's read metadata, scan planning and write all resolve the
+ // SAME one loaded RAW table. The scope holds the RAW table; wrapTableForScan (the Kerberos doAs FileIO) is
+ // re-applied per call below so no per-request authenticator is ever frozen into the shared object.
+ // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim (it
+ // re-validates the credential even on a scope hit).
IcebergCatalogOps ops = catalogOpsResolver.apply(session);
- if (context == null) {
- return ops.loadTable(handle.getDbName(), handle.getTableName());
- }
- try {
- return wrapTableForScan(context.executeAuthenticated(
- () -> ops.loadTable(handle.getDbName(), handle.getTableName())));
- } catch (Exception e) {
- throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e);
+ Table raw = IcebergStatementScope.sharedTable(session, handle.getDbName(), handle.getTableName(), () -> {
+ if (context == null) {
+ return loadRawTable(ops, handle);
+ }
+ try {
+ return context.executeAuthenticated(() -> loadRawTable(ops, handle));
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e);
+ }
+ });
+ return wrapTableForScan(raw);
+ }
+
+ /**
+ * Loads the RAW iceberg table for {@code handle} through the cross-query {@link IcebergTableCache} when
+ * enabled (the connector disables it for credential-dependent catalogs), else a direct remote
+ * {@code loadTable}. No wrap and no auth scope here — {@link #resolveTable} owns both.
+ */
+ private Table loadRawTable(IcebergCatalogOps ops, IcebergTableHandle handle) {
+ if (tableCache != null) {
+ return tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()),
+ () -> ops.loadTable(handle.getDbName(), handle.getTableName()));
}
+ return ops.loadTable(handle.getDbName(), handle.getTableName());
}
/**
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java
new file mode 100644
index 00000000000000..28b0e28fe41d49
--- /dev/null
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorStatementScope;
+import org.apache.doris.thrift.TIcebergDeleteFileDesc;
+
+import org.apache.iceberg.Table;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Supplier;
+
+/**
+ * Connector-private helpers over the neutral {@link ConnectorStatementScope} (reached via
+ * {@link ConnectorSession#getStatementScope()}), giving iceberg one place to key its per-statement state.
+ *
+ * The scope is the per-statement table-load owner: the read metadata path, scan planning, write shaping
+ * and {@code beginWrite} all resolve one table through {@link #sharedTable} so a single statement loads each
+ * table once and every resolver shares that one RAW object (snapshot pins and auth wraps are applied per
+ * consumer, never frozen into the shared object). It also carries the merge-on-read rewritable-delete supply
+ * from the scan seam to the write seam ({@link #rewritableDeleteSupply}), replacing the former per-catalog
+ * singleton stash — the scope is per-statement, so a statement's supply is GC'd with it and a reused
+ * prepared-statement scope is reset per execution (see {@code ExecuteCommand}).
+ *
+ * Under {@link ConnectorStatementScope#NONE} (offline planning / no live statement) {@link #sharedTable}
+ * loads every time (byte-identical to the pre-scope behavior) and {@link #rewritableDeleteSupply} returns a
+ * throwaway map that does NOT bridge scan→write — so a format-version≥3 row-level DML under NONE fails
+ * loud at the write seam rather than silently resurrecting rows.
+ */
+final class IcebergStatementScope {
+
+ private IcebergStatementScope() {}
+
+ /**
+ * Loads the RAW iceberg {@link Table} for {@code db.tbl} once per statement and shares it across every
+ * resolver. The key includes the catalog id (cross-catalog MERGE isolation) and the statement's queryId
+ * (a reused prepared context sees each execution's own table). {@code loader} runs at most once per
+ * statement — callers pass the raw load (cross-query cache or direct remote) and own the auth scope
+ * (the caller wraps this in {@code executeAuthenticated}).
+ */
+ static Table sharedTable(ConnectorSession session, String dbName, String tableName, Supplier loader) {
+ if (session == null) {
+ // No session (offline / direct-construction tests): load every time, like ConnectorStatementScope.NONE.
+ return loader.get();
+ }
+ String key = "iceberg.table:" + session.getCatalogId() + ":" + dbName + ":" + tableName
+ + ":" + session.getQueryId();
+ return session.getStatementScope().computeIfAbsent(key, loader);
+ }
+
+ /**
+ * Returns this statement's rewritable-delete supply map (RAW data-file path → its non-equality delete
+ * descs), creating it empty on first use. The scan seam accumulates into it (per touched data file) and the
+ * write seam drains it; keyed by catalog id + queryId so a cross-catalog MERGE keeps each table's supply
+ * isolated. Under {@link ConnectorStatementScope#NONE} each call returns a fresh throwaway map, so scan and
+ * write do NOT share — the write seam guards format-version≥3 DML against that (fail loud).
+ */
+ static Map> rewritableDeleteSupply(ConnectorSession session) {
+ if (session == null) {
+ // No session: a throwaway map that does NOT bridge scan->write (same as NONE).
+ return new ConcurrentHashMap<>();
+ }
+ String key = "iceberg.rewritable-delete-supply:" + session.getCatalogId() + ":" + session.getQueryId();
+ return session.getStatementScope().computeIfAbsent(key, ConcurrentHashMap::new);
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java
new file mode 100644
index 00000000000000..8a6b53ef599787
--- /dev/null
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.doris.connector.cache.CacheSpec;
+import org.apache.doris.connector.cache.MetaCacheEntry;
+
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+import java.util.concurrent.ForkJoinPool;
+import java.util.function.Supplier;
+
+/**
+ * Per-catalog cache of the RAW iceberg {@link Table} object, keyed by {@link TableIdentifier} (db.table)
+ * (PERF-01). This restores the OTHER half of the legacy {@code IcebergExternalMetaCache} that the SPI cutover
+ * dropped: {@link IcebergLatestSnapshotCache} kept only the {@code (snapshotId, schemaId)} pin, so every SPI
+ * read entry ({@code getColumnHandles}, {@code getTableStatistics}, the scan provider's {@code resolveTable},
+ * ...) re-loaded the table from the remote catalog (a metastore RPC + a {@code metadata.json} read). This cache
+ * lets consecutive queries — and the analysis/planning phases of one query, whose handles have distinct memo
+ * lineages — reuse a single loaded table, exactly as the legacy with-cache catalog did.
+ *
+ * Backing. Reuses the shared {@link MetaCacheEntry} framework identically to
+ * {@link IcebergLatestSnapshotCache}: a contextual, access-TTL entry whose per-key loader is supplied at
+ * {@link #getOrLoad}, with manual miss-load on so the loader runs OUTSIDE Caffeine's compute lock
+ * (single-flight per key) and propagates its exception verbatim (a concurrent-drop
+ * {@code NoSuchTableException} reaches the caller unwrapped, preserving each read entry's own degradation).
+ * TTL is {@code meta.cache.iceberg.table.ttl-second} — the same knob that governs the snapshot cache: a
+ * value {@code <= 0} disables caching (every read goes live), a positive value is Caffeine
+ * {@code expireAfterAccess} with a {@code maxSize} capacity. Lives on the long-lived per-catalog
+ * {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache.
+ *
+ *
Values are RAW tables. The scan provider applies {@code wrapTableForScan} (the Kerberos
+ * {@code doAs} FileIO wrap) per call on the way out, so no per-request authenticator is ever frozen into a
+ * shared entry.
+ *
+ *
Credential isolation. A raw table carries its FileIO's credentials, so this cross-query layer is
+ * built ONLY when the connector's credentials are query-independent — it is left disabled (the connector
+ * passes {@code null}) for {@code iceberg.rest.session=user} (per-user delegated FileIO) and REST
+ * vended-credentials (server-vended tokens expire within the query, and iceberg keeps them fresh by reloading
+ * the table each query). See {@code IcebergConnector}.
+ */
+final class IcebergTableCache {
+
+ private final MetaCacheEntry entry;
+
+ IcebergTableCache(long ttlSeconds, int maxSize) {
+ // Mirror IcebergLatestSnapshotCache: translate the connector's "<= 0 disables" contract to CacheSpec's
+ // ttl == 0 (disabled) rather than passing a negative value through, which CacheSpec reads as ttl == -1
+ // "no expiration (enabled)".
+ CacheSpec spec = ttlSeconds > 0
+ ? CacheSpec.of(true, ttlSeconds, maxSize)
+ : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize);
+ this.entry = new MetaCacheEntry<>("iceberg-table", null, spec,
+ ForkJoinPool.commonPool(), false, true, 0L, true);
+ }
+
+ /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always read live". */
+ boolean isEnabled() {
+ return entry.stats().isEffectiveEnabled();
+ }
+
+ /**
+ * Returns the cached table for {@code identifier} if present and unexpired, else runs {@code loader} (the
+ * live remote {@code loadTable}), caches and returns it. When caching is disabled ({@link #isEnabled()} is
+ * false) {@code loader} runs every call and nothing is cached. A hit refreshes the entry's expiry
+ * (access-based). The loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception
+ * propagates unwrapped.
+ */
+ Table getOrLoad(TableIdentifier identifier, Supplier loader) {
+ return entry.get(identifier, ignored -> loader.get());
+ }
+
+ /** Drops the cached entry for one table so the next read goes live (REFRESH TABLE). */
+ void invalidate(TableIdentifier identifier) {
+ entry.invalidateKey(identifier);
+ }
+
+ /**
+ * Drops every cached entry for one database so the next read of any of its tables goes live
+ * (REFRESH DATABASE / a Doris-issued DROP DATABASE). Entries are keyed by
+ * {@code TableIdentifier.of(db, table)} (single-level namespace = {@code [db]}), so a db match is
+ * namespace equality — mirroring {@link IcebergLatestSnapshotCache#invalidateDb}.
+ */
+ void invalidateDb(String dbName) {
+ Namespace ns = Namespace.of(dbName);
+ entry.invalidateIf(id -> id.namespace().equals(ns));
+ }
+
+ /** Drops all cached entries. */
+ void invalidateAll() {
+ entry.invalidateAll();
+ }
+
+ /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */
+ int size() {
+ int[] count = {0};
+ entry.forEach((key, value) -> count[0]++);
+ return count[0];
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java
index aaa3e1ff050c76..4ad5d97c5fd9a6 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java
@@ -19,6 +19,7 @@
import org.apache.doris.connector.api.ConnectorColumn;
import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorStatementScope;
import org.apache.doris.connector.api.ConnectorType;
import org.apache.doris.connector.api.DorisConnectorException;
import org.apache.doris.connector.api.handle.ConnectorTableHandle;
@@ -91,7 +92,7 @@
* write distribution and the vended-credentials overlay of the hadoop config are registered deviations
* (DV-T0x-vended / -broker / -materialize) closed at the P6.6 cutover. At format-version≥3 the DELETE /
* MERGE sink's {@code rewritable_delete_file_sets} is filled here from the scan-time supply the
- * {@link IcebergRewritableDeleteStash} carried across the scan→write seam (commit-bridge S4 part 2),
+ * per-statement {@link IcebergStatementScope} carried across the scan→write seam (commit-bridge S4 part 2),
* replacing the legacy fe-resident rewritable-delete planner.
*
* Gate-closed / dormant. Iceberg is not in {@code SPI_READY_TYPES} until P6.6, so nothing
@@ -137,22 +138,12 @@ private static ConnectorColumn buildRowIdColumn() {
// single shared ops regardless of session (constant s -> catalogOps).
private final Function catalogOpsResolver;
private final ConnectorContext context;
- // commit-bridge supply (S4 part 2): the per-catalog stash the scan provider filled with each touched data
- // file's non-equality delete supply. planWrite retrieves (and evicts) it by queryId to fill the v3
- // rewritable_delete_file_sets. Nullable — null via the 3-arg ctor (offline tests), in which case no supply is
- // attached (and the BE would resurrect rows, which is why the cutover wiring injects the real stash).
- private final IcebergRewritableDeleteStash rewritableDeleteStash;
public IcebergWritePlanProvider(Map properties, IcebergCatalogOps catalogOps,
ConnectorContext context) {
- this(properties, catalogOps, context, null);
- }
-
- public IcebergWritePlanProvider(Map properties, IcebergCatalogOps catalogOps,
- ConnectorContext context, IcebergRewritableDeleteStash rewritableDeleteStash) {
// Constant resolver: these ctors (offline tests) bind a single ops that ignores the session, so existing
// behaviour/tests are byte-identical.
- this(properties, session -> catalogOps, context, rewritableDeleteStash);
+ this(properties, session -> catalogOps, context);
}
/**
@@ -163,11 +154,10 @@ public IcebergWritePlanProvider(Map properties, IcebergCatalogOp
*/
public IcebergWritePlanProvider(Map properties,
Function catalogOpsResolver,
- ConnectorContext context, IcebergRewritableDeleteStash rewritableDeleteStash) {
+ ConnectorContext context) {
this.properties = properties;
this.catalogOpsResolver = catalogOpsResolver;
this.context = context;
- this.rewritableDeleteStash = rewritableDeleteStash;
}
@Override
@@ -183,13 +173,24 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl
transaction.beginWrite(session, tableHandle.getDbName(), tableHandle.getTableName(), writeContext);
Table table = transaction.getTable();
- // commit-bridge supply (S4 part 2): retrieve (and evict) the non-equality delete supply the scan provider
- // stashed for this statement. Done once for every write op — DELETE/MERGE attach it to the sink so the BE
- // OR-merges old deletes into the new deletion vector (a missing supply silently resurrects deleted rows);
- // INSERT/OVERWRITE discard it, but the retrieve still evicts the stash entry (e.g. an INSERT ... SELECT
- // FROM an iceberg source). Null when the stash is absent (offline tests) or nothing was stashed.
- Map> rewritableDeletes = rewritableDeleteStash != null
- ? rewritableDeleteStash.retrieveAndRemove(session.getQueryId()) : null;
+ // commit-bridge supply (S4 part 2): read the non-equality delete supply the scan seam accumulated into the
+ // per-statement scope. DELETE/MERGE attach it to the sink so the BE OR-merges old deletes into the new
+ // deletion vector (a missing supply silently resurrects deleted rows); INSERT/OVERWRITE/REWRITE ignore it
+ // (buildRewritableDeleteFileSets no-ops an empty map, same as the former null).
+ // Fail loud (never silently resurrect): a format-version>=3 row-level DML under an absent statement scope
+ // (ConnectorStatementScope.NONE — offline / no live statement) cannot have received the scan's supply (each
+ // NONE lookup is a throwaway map), so reject it rather than write a deletion vector that drops old deletes.
+ WriteOperation writeOp = writeContext.getWriteOperation();
+ if ((writeOp == WriteOperation.DELETE || writeOp == WriteOperation.UPDATE || writeOp == WriteOperation.MERGE)
+ && session.getStatementScope() == ConnectorStatementScope.NONE
+ && IcebergWriterHelper.getFormatVersion(table) >= 3) {
+ throw new DorisConnectorException("Iceberg row-level " + writeOp + " on a format-version>=3 table requires "
+ + "a per-statement scope to carry the rewritable-delete supply from scan to write; none is present "
+ + "(ConnectorStatementScope.NONE). Refusing to write a deletion vector that would drop old deletes "
+ + "and resurrect previously-deleted rows.");
+ }
+ Map> rewritableDeletes =
+ IcebergStatementScope.rewritableDeleteSupply(session);
// Dispatch on the write operation to the matching BE sink dialect (each is a distinct TDataSinkType,
// byte-identical to the legacy fe-core planner sink). OVERWRITE shares TIcebergTableSink with INSERT
@@ -687,18 +688,24 @@ private IcebergConnectorTransaction currentTransaction(ConnectorSession session)
}
private Table resolveTable(ConnectorSession session, IcebergTableHandle handle) {
- // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim.
+ // Per-statement scope (PERF-07): the write-shaping derivations (sort columns / partitioning / explain)
+ // share the SAME one loaded RAW table as the statement's read + scan resolvers. For a row-level DML the
+ // scan has already populated the scope, so this is a hit (no write-side load); a write-only INSERT loads
+ // once here. Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces
+ // verbatim (it re-validates the credential even on a scope hit).
IcebergCatalogOps ops = catalogOpsResolver.apply(session);
- if (context == null) {
- return ops.loadTable(handle.getDbName(), handle.getTableName());
- }
- try {
- return context.executeAuthenticated(
- () -> ops.loadTable(handle.getDbName(), handle.getTableName()));
- } catch (Exception e) {
- throw new DorisConnectorException("Failed to load iceberg table "
- + handle.getDbName() + "." + handle.getTableName() + ": " + e.getMessage(), e);
- }
+ return IcebergStatementScope.sharedTable(session, handle.getDbName(), handle.getTableName(), () -> {
+ if (context == null) {
+ return ops.loadTable(handle.getDbName(), handle.getTableName());
+ }
+ try {
+ return context.executeAuthenticated(
+ () -> ops.loadTable(handle.getDbName(), handle.getTableName()));
+ } catch (Exception e) {
+ throw new DorisConnectorException("Failed to load iceberg table "
+ + handle.getDbName() + "." + handle.getTableName() + ": " + e.getMessage(), e);
+ }
+ });
}
private static TFileFormatType toTFileFormatType(FileFormat format) {
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java
index 3ac890e7282650..a069a29921b610 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java
@@ -34,9 +34,11 @@
import org.apache.iceberg.PartitionData;
import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
import org.apache.iceberg.SortOrder;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.io.WriteResult;
import org.apache.iceberg.types.Type;
@@ -45,6 +47,8 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.io.IOException;
+import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.time.ZoneId;
import java.util.ArrayList;
@@ -263,8 +267,21 @@ static List convertToDeleteFiles(FileFormat format, PartitionSpec sp
* from the current snapshot's data files (defaulting to parquet). Throws on a non-orc/parquet format.
*/
static FileFormat getFileFormat(Table table) {
- Map properties = table.properties();
- String fileFormatName = resolveFileFormatName(table, properties);
+ return toFileFormat(resolveFileFormatName(table, table.properties()));
+ }
+
+ /**
+ * PERF-03 cache-aware overload used by the scan path ({@code IcebergScanPlanProvider.getScanNodeProperties}).
+ * Identical resolution to {@link #getFileFormat(Table)}, except the whole-table {@code planFiles()} inference
+ * fallback (fired only when the table sets neither {@code write-format} nor {@code write.format.default}) is
+ * memoized per {@code (id, currentSnapshotId)} through {@code cache}. The two cheap property probes stay
+ * uncached. A {@code null} cache/id (offline tests) or an empty table (no current snapshot) resolves live.
+ */
+ static FileFormat getFileFormat(Table table, TableIdentifier id, IcebergFormatCache cache) {
+ return toFileFormat(resolveFileFormatName(table, table.properties(), id, cache));
+ }
+
+ private static FileFormat toFileFormat(String fileFormatName) {
if (fileFormatName.toLowerCase().contains(ORC_NAME)) {
return FileFormat.ORC;
} else if (fileFormatName.toLowerCase().contains(PARQUET_NAME)) {
@@ -284,20 +301,70 @@ private static String resolveFileFormatName(Table table, Map pro
return inferFileFormatFromDataFiles(table);
}
+ /**
+ * PERF-03: like {@link #resolveFileFormatName(Table, Map)} but routes the inference fallback (the heavy
+ * unfiltered {@code planFiles()}) through the cross-query {@code (id, snapshotId)} cache. Only the inference is
+ * cached; the property probes are cheap. A failed inference (the loader throws) is NOT cached, so the next
+ * query retries — for this query it degrades to the parquet default, mirroring the swallow of the live path.
+ */
+ private static String resolveFileFormatName(Table table, Map properties,
+ TableIdentifier id, IcebergFormatCache cache) {
+ if (properties.containsKey(WRITE_FORMAT)) {
+ return properties.get(WRITE_FORMAT);
+ }
+ if (properties.containsKey(TableProperties.DEFAULT_FILE_FORMAT)) {
+ return properties.get(TableProperties.DEFAULT_FILE_FORMAT);
+ }
+ Snapshot snapshot = table.currentSnapshot();
+ if (cache == null || id == null || snapshot == null) {
+ return inferFileFormatFromDataFiles(table);
+ }
+ try {
+ return cache.getOrLoad(new IcebergFormatCache.Key(id, snapshot.snapshotId()),
+ () -> inferFirstDataFileFormat(table));
+ } catch (RuntimeException e) {
+ LOG.warn("Failed to infer file format from data files for table {}, defaulting to {}",
+ table.name(), PARQUET_NAME, e);
+ return PARQUET_NAME;
+ }
+ }
+
+ /**
+ * The whole-table format inference (the #64134 heavy op): reads the FIRST data file's format from the current
+ * snapshot via an unfiltered {@code planFiles()}. The swallowing wrapper used by the write path and the direct
+ * live path (a failure degrades to parquet). The cache loader instead calls {@link #inferFirstDataFileFormat}
+ * so a transient failure is not memoized.
+ */
private static String inferFileFormatFromDataFiles(Table table) {
if (table.currentSnapshot() == null) {
return PARQUET_NAME;
}
+ try {
+ return inferFirstDataFileFormat(table);
+ } catch (RuntimeException e) {
+ LOG.warn("Failed to infer file format from data files for table {}, defaulting to {}",
+ table.name(), PARQUET_NAME, e);
+ return PARQUET_NAME;
+ }
+ }
+
+ /**
+ * Raw first-data-file format inference that PROPAGATES failures (unchecked) so the cache loader does not
+ * memoize a transient remote-IO error. The caller guarantees a non-null current snapshot; an empty snapshot
+ * (no data files) yields the deterministic parquet default (cacheable). The try-with-resources
+ * {@code CloseableIterable.close()} checked {@link IOException} is wrapped as {@link UncheckedIOException} so
+ * the method stays unchecked (a {@code Supplier} loader cannot declare checked throws).
+ */
+ private static String inferFirstDataFileFormat(Table table) {
try (CloseableIterable files = table.newScan().planFiles()) {
Iterator it = files.iterator();
if (it.hasNext()) {
return it.next().file().format().name().toLowerCase();
}
- } catch (Exception e) {
- LOG.warn("Failed to infer file format from data files for table {}, defaulting to {}",
- table.name(), PARQUET_NAME, e);
+ return PARQUET_NAME;
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
}
- return PARQUET_NAME;
}
/**
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java
index e6e70530cfef0a..380f77ec65b915 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java
@@ -30,6 +30,7 @@
import java.util.Objects;
import java.util.concurrent.Callable;
import java.util.function.Supplier;
+import java.util.function.UnaryOperator;
/**
* A {@link ConnectorContext} decorator that pins the thread-context classloader (TCCL) to the iceberg plugin
@@ -167,6 +168,15 @@ public String normalizeStorageUri(String rawUri, Map rawVendedCr
return delegate.normalizeStorageUri(rawUri, rawVendedCredentials);
}
+ @Override
+ public UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) {
+ // Delegate to the raw engine context so the connector gets DefaultConnectorContext's once-per-scan
+ // hoist. Without this override the SPI default would fold back to THIS wrapper's per-call
+ // normalizeStorageUri, silently defeating the optimization. Like normalizeStorageUri, this path runs
+ // entirely in fe-core (LocationPath/StorageProperties, no plugin reflection), so no TCCL pin is needed.
+ return delegate.newStorageUriNormalizer(rawVendedCredentials);
+ }
+
@Override
public String getBackendFileType(String rawUri, Map rawVendedCredentials) {
return delegate.getBackendFileType(rawUri, rawVendedCredentials);
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java
index 4a3e63b0bc1abf..a10363ccc4c28c 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java
@@ -24,6 +24,7 @@
import org.apache.doris.connector.api.pushdown.ConnectorPredicate;
import org.apache.doris.foundation.util.ArgumentParsers;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Lists;
import org.apache.iceberg.DeleteFile;
import org.apache.iceberg.ExpireSnapshots;
@@ -42,8 +43,10 @@
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
@@ -67,6 +70,11 @@ public class IcebergExpireSnapshotsAction extends BaseIcebergAction {
public static final String SNAPSHOT_IDS = "snapshot_ids";
public static final String CLEAN_EXPIRED_METADATA = "clean_expired_metadata";
+ // Test-only gate for the delete-manifest dedup: the number of DISTINCT delete manifests read by the most
+ // recent buildDeleteFileContentMap call. Asserts each manifest is read once, not once per referencing snapshot.
+ @VisibleForTesting
+ int lastDeleteManifestReadCount;
+
public IcebergExpireSnapshotsAction(Map properties, List partitionNames,
ConnectorPredicate whereCondition) {
super("expire_snapshots", properties, partitionNames, whereCondition);
@@ -268,8 +276,17 @@ private long parseTimestamp(String timestamp) {
}
}
- private Map buildDeleteFileContentMap(Table icebergTable) {
+ @VisibleForTesting
+ Map buildDeleteFileContentMap(Table icebergTable) {
Map deleteFileContentByPath = new HashMap<>();
+ // Dedup delete-manifest reads across snapshots. Iceberg manifests are immutable and adjacent snapshots
+ // carry the same delete manifests forward unchanged, so re-reading one yields the identical DeleteFile
+ // set (putIfAbsent already made the re-read a no-op). Reading each DISTINCT manifest exactly once
+ // collapses the O(snapshots x manifests) remote reads to O(distinct manifests) with a byte-identical map.
+ // NOTE: visited MUST live at method scope (outside the snapshot loop) — inside it, it would reset every
+ // snapshot and defeat the cross-snapshot dedup this fix targets.
+ Set visitedDeleteManifests = new HashSet<>();
+ int reads = 0;
try {
for (Snapshot snapshot : icebergTable.snapshots()) {
List deleteManifests = snapshot.deleteManifests(icebergTable.io());
@@ -277,6 +294,10 @@ private Map buildDeleteFileContentMap(Table icebergTable) {
continue;
}
for (ManifestFile manifest : deleteManifests) {
+ if (!visitedDeleteManifests.add(manifest.path())) {
+ continue;
+ }
+ reads++;
try (CloseableIterable deleteFiles = ManifestFiles.readDeleteManifest(
manifest, icebergTable.io(), icebergTable.specs())) {
for (DeleteFile deleteFile : deleteFiles) {
@@ -289,6 +310,7 @@ private Map buildDeleteFileContentMap(Table icebergTable) {
} catch (Exception e) {
throw new DorisConnectorException("Failed to build delete file content map: " + e.getMessage(), e);
}
+ lastDeleteManifestReadCount = reads;
return deleteFileContentByPath;
}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java
new file mode 100644
index 00000000000000..e8a6f21119e95e
--- /dev/null
+++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Unit tests for {@link IcebergCommentCache} (PERF-05). Mirrors {@link IcebergTableCacheTest} but stores the
+ * {@code comment} string keyed by {@link TableIdentifier}. Covers within-TTL stability, the {@code ttl <= 0}
+ * disable (load-bearing: a vended no-cache catalog builds this object but must not cache), invalidation, and the
+ * not-cached-on-failure guarantee that keeps the view-handle {@code NoSuchTableException} degradation to "".
+ */
+public class IcebergCommentCacheTest {
+
+ private static TableIdentifier id(String db, String tbl) {
+ return TableIdentifier.of(db, tbl);
+ }
+
+ @Test
+ public void cachesWithinTtlAndServesTheSameComment() {
+ AtomicInteger loads = new AtomicInteger();
+ IcebergCommentCache c = new IcebergCommentCache(100, 1000);
+
+ String first = c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "sales fact";
+ });
+ // Second read of the SAME table within TTL returns the cached comment, NOT a fresh remote loadTable -> this
+ // is what collapses the per-table information_schema loads on repeat queries. MUTATION: loading live every
+ // call -> returns "other" / loads==2 -> red.
+ String second = c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "other";
+ });
+ Assertions.assertEquals("sales fact", first);
+ Assertions.assertEquals("sales fact", second, "within TTL the cached comment must be served");
+ Assertions.assertEquals(1, loads.get(), "the live loadTable must run exactly once within TTL");
+ Assertions.assertEquals(1, c.loadCountForTest(), "the metric-gate load count must be 1");
+ Assertions.assertTrue(c.isEnabled());
+ }
+
+ @Test
+ public void differentTableIsADifferentKey() {
+ AtomicInteger loads = new AtomicInteger();
+ IcebergCommentCache c = new IcebergCommentCache(100, 1000);
+ c.getOrLoad(id("db", "t1"), () -> {
+ loads.incrementAndGet();
+ return "c1";
+ });
+ String t2 = c.getOrLoad(id("db", "t2"), () -> {
+ loads.incrementAndGet();
+ return "c2";
+ });
+ Assertions.assertEquals("c2", t2);
+ Assertions.assertEquals(2, loads.get());
+ Assertions.assertEquals(2, c.size());
+ }
+
+ @Test
+ public void ttlZeroDisablesCachingAlwaysLive() {
+ AtomicInteger loads = new AtomicInteger();
+ IcebergCommentCache c = new IcebergCommentCache(0, 1000);
+ c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "a";
+ });
+ String second = c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "b";
+ });
+ // ttl-second=0 (a vended no-cache catalog) loads live every time -> operator "no meta cache" intent honored
+ // even though the connector built this object. MUTATION: caching despite ttl<=0 -> second=="a" / loads==1.
+ Assertions.assertEquals("b", second, "ttl-second=0 must always load live");
+ Assertions.assertEquals(2, loads.get());
+ Assertions.assertFalse(c.isEnabled());
+ Assertions.assertEquals(0, c.size(), "ttl-second=0 must not store anything");
+ }
+
+ @Test
+ public void negativeTtlDisablesCachingAlwaysLive() {
+ AtomicInteger loads = new AtomicInteger();
+ IcebergCommentCache c = new IcebergCommentCache(-1, 1000);
+ c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "a";
+ });
+ String second = c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "b";
+ });
+ Assertions.assertEquals("b", second, "ttl-second=-1 must always load live");
+ Assertions.assertEquals(2, loads.get());
+ Assertions.assertFalse(c.isEnabled());
+ }
+
+ @Test
+ public void invalidateForcesReload() {
+ AtomicInteger loads = new AtomicInteger();
+ IcebergCommentCache c = new IcebergCommentCache(100, 1000);
+ c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "old";
+ });
+ // REFRESH TABLE db.t drops the entry so an external ALTER ... SET TBLPROPERTIES('comment'=...) is picked up.
+ // MUTATION: invalidate not clearing the key -> "old" survives -> red.
+ c.invalidate(id("db", "t"));
+ String after = c.getOrLoad(id("db", "t"), () -> {
+ loads.incrementAndGet();
+ return "new";
+ });
+ Assertions.assertEquals("new", after);
+ Assertions.assertEquals(2, loads.get());
+ }
+
+ @Test
+ public void invalidateDbClearsOnlyThatDbsTables() {
+ IcebergCommentCache c = new IcebergCommentCache(100, 1000);
+ c.getOrLoad(id("db1", "t1"), () -> "a");
+ c.getOrLoad(id("db1", "t2"), () -> "b");
+ c.getOrLoad(id("db2", "t1"), () -> "c");
+ Assertions.assertEquals(3, c.size());
+ c.invalidateDb("db1");
+ Assertions.assertEquals(1, c.size(), "only db2's entry must survive REFRESH DATABASE db1");
+ }
+
+ @Test
+ public void invalidateAllClearsEverything() {
+ IcebergCommentCache c = new IcebergCommentCache(100, 1000);
+ c.getOrLoad(id("db", "t1"), () -> "a");
+ c.getOrLoad(id("db", "t2"), () -> "b");
+ Assertions.assertEquals(2, c.size());
+ c.invalidateAll();
+ Assertions.assertEquals(0, c.size());
+ }
+
+ @Test
+ public void loaderFailureIsNotCachedSoNextQueryRetries() {
+ // A view handle (loadTable throws NoSuchTableException) must NOT be cached, so getTableComment keeps
+ // degrading to "" via the caller's catch on every call (and a real table appearing later loads fresh). The
+ // MetaCacheEntry manual-miss-load path re-throws the loader's exception verbatim and does not store it.
+ // MUTATION: caching on failure -> the second call would not re-run the loader / size==1 -> red.
+ IcebergCommentCache c = new IcebergCommentCache(100, 1000);
+ AtomicInteger loads = new AtomicInteger();
+ Assertions.assertThrows(NoSuchTableException.class, () -> c.getOrLoad(id("db", "v"), () -> {
+ loads.incrementAndGet();
+ throw new NoSuchTableException("not a table: db.v");
+ }));
+ Assertions.assertEquals(0, c.size(), "a thrown comment load must not be cached");
+ String ok = c.getOrLoad(id("db", "v"), () -> {
+ loads.incrementAndGet();
+ return "now a table";
+ });
+ Assertions.assertEquals("now a table", ok);
+ Assertions.assertEquals(2, loads.get(), "the loader must re-run after a failure (no sticky failure)");
+ Assertions.assertEquals(1, c.size());
+ }
+}
diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
index bc816df8c48e22..0a22ffff2fb6e4 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java
@@ -128,6 +128,48 @@ public void invalidateHooksAreNoThrowOnFreshConnector() {
Assertions.assertDoesNotThrow(connector::invalidateAll);
}
+ @Test
+ public void latestSnapshotCacheDisabledForSessionUser() {
+ // The latest-snapshot cache is an AUTHORIZATION-sensitive projection (snapshotId/schemaId) that
+ // beginQuerySnapshot reads WITHOUT a preceding per-user loadTable, so a shared hit would bypass the
+ // per-user authorization. It is disabled (null) under iceberg.rest.session=user (kept otherwise, incl.
+ // vended-credentials, since a snapshot id carries no token). MUTATION: dropping the session=user gate ->
+ // non-null for session -> red.
+ Assertions.assertNotNull(
+ new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext())
+ .latestSnapshotCacheForTest(),
+ "a plain catalog builds the latest-snapshot cache");
+ Map vended = new HashMap<>();
+ vended.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST);
+ vended.put(IcebergConnectorProperties.REST_VENDED_CREDENTIALS_ENABLED, "true");
+ Assertions.assertNotNull(
+ new IcebergConnector(vended, new RecordingConnectorContext()).latestSnapshotCacheForTest(),
+ "a vended-credentials catalog still builds the latest-snapshot cache (an id carries no token)");
+ Map session = new HashMap<>();
+ session.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST);
+ session.put(IcebergConnectorProperties.REST_SESSION, IcebergConnectorProperties.SESSION_USER);
+ Assertions.assertNull(
+ new IcebergConnector(session, new RecordingConnectorContext()).latestSnapshotCacheForTest(),
+ "a session=user catalog must NOT build the latest-snapshot cache (per-user authz bypass)");
+ }
+
+ @Test
+ public void invalidateHooksAreNoThrowForSessionUserWithNulledCaches() {
+ // Under session=user the latest-snapshot / partition / format caches are all null. The REFRESH hooks must
+ // still be no-throw (the invalidate* methods null-guard each cache). MUTATION: an unguarded invalidate call
+ // on a nulled cache -> NPE -> red.
+ Map session = new HashMap<>();
+ session.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST);
+ session.put(IcebergConnectorProperties.REST_SESSION, IcebergConnectorProperties.SESSION_USER);
+ IcebergConnector connector = new IcebergConnector(session, new RecordingConnectorContext());
+ Assertions.assertNull(connector.latestSnapshotCacheForTest());
+ Assertions.assertNull(connector.partitionCacheForTest());
+ Assertions.assertNull(connector.formatCacheForTest());
+ Assertions.assertDoesNotThrow(() -> connector.invalidateTable("db1", "t1"));
+ Assertions.assertDoesNotThrow(() -> connector.invalidateDb("db1"));
+ Assertions.assertDoesNotThrow(connector::invalidateAll);
+ }
+
@Test
public void refreshCatalogInvalidateAllDropsManifestCache() {
// H-5: REFRESH CATALOG -> Connector.invalidateAll() must drop the connector's OWN manifest cache too
@@ -165,4 +207,248 @@ private static Table tableWithOneManifest() {
.commit();
return table;
}
+
+ // ==================== PERF-01: cross-query table cache gate + invalidation ====================
+
+ private static Table fakeTable(String name) {
+ return new FakeIcebergTable(name,
+ new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())),
+ PartitionSpec.unpartitioned(), "s3://b/" + name, Collections.emptyMap());
+ }
+
+ @Test
+ public void crossQueryTableCacheEnabledForPlainCatalog() {
+ // A plain catalog (no per-user session, no REST vended credentials) has query-independent credentials,
+ // so the cross-query RAW-table cache is built and enabled at the default 24h TTL — restoring the legacy
+ // IcebergExternalMetaCache table cache. MUTATION: leaving it null/disabled for a plain catalog -> the
+ // 3~7x remote loadTable amplification is not collapsed across queries -> assert below red.
+ IcebergTableCache cache =
+ new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()).tableCacheForTest();
+ Assertions.assertNotNull(cache, "a plain catalog must build the cross-query table cache");
+ Assertions.assertTrue(cache.isEnabled(), "the default 24h TTL enables the cache");
+ }
+
+ @Test
+ public void crossQueryTableCacheDisabledForVendedCredentials() {
+ // REST vended-credentials: the cached raw table's FileIO carries a server-vended token that expires
+ // within the query (iceberg keeps it fresh by reloading the table each query). A 24h-TTL cross-query hit
+ // would hand BE an expired token (403 mid-scan), so this layer MUST be off (null); the query-scoped fat
+ // handle still dedups within one query. MUTATION: building the cache for a vended catalog -> non-null -> red.
+ Map vended = new HashMap<>();
+ vended.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST);
+ vended.put(IcebergConnectorProperties.REST_VENDED_CREDENTIALS_ENABLED, "true");
+ Assertions.assertNull(
+ new IcebergConnector(vended, new RecordingConnectorContext()).tableCacheForTest(),
+ "a REST vended-credentials catalog must NOT build the cross-query table cache");
+ }
+
+ @Test
+ public void crossQueryTableCacheDisabledForPerUserSession() {
+ // iceberg.rest.session=user: the cached raw table carries per-user delegated FileIO, so sharing it
+ // across users would leak credentials. This layer MUST be off (null) — the fat handle keeps within-query
+ // dedup. MUTATION: building the cache for a session=user catalog -> tableCacheForTest non-null -> red.
+ Map