Implement Library Instrumentation for Apache Iceberg

[ghareeb-falazi]

This PR aims at creating library instrumentation for Apache Iceberg scan metrics. The figure above summarizes the planned solution (the green classes constitute the instrumentation).

The Apache Iceberg API currently emits two types of metrics: ScanMetrics that are emitted when a table scan is executed, and CommitMetrics that are emitted when modifications to the data or metadata are executed, e.g., insert, update, delete, drop column, time travel, etc.
ScanMetrics are straightforward to report using a custom reporter. We can programmatically inject such a reporter into an exiting Scan object as described here. On the contrary, CommitMetrics are difficult to report using a custom reporter due to the absence of a programmatical interface to inject such a reporter. A configuration-based approach exists for this purpose as described here. This PR focuses on ScanMetrics only and creates a library-based instrumentation for them.

CommitMetrics and agent-based instrumentation will be the subject of future PRs. Therefore, this PR partially solves #15113

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ghareeb-falazi / name: Ghareeb Falazi (8ccea5c)

[jaydeluca]

why is this second implementation needed?

[ghareeb-falazi]

@jaydeluca In the abstract unit test class AbstractIcebergTest, I am using the public classes TestTables and TestTable that are defined in the src/test/java directory of the iceberg core project, so the second implementation is meant to give me access to these classes in the unit tests of the testing project.
As a side note: I am not really an expert in Gradle, so it could really be that a better approach exists...

[jaydeluca]

this might be ok, but it might be brittle for us to have a dependency on an artifact that they are not publicly publishing. I don't know anything about iceberg, but what if we replaced the test setup with hadoop? if we add the implementation dependency here:

implementation("org.apache.hadoop:hadoop-common:3.4.2")

and then instead of using the test table, you do:

    Configuration conf = new Configuration();
    this.tables = new HadoopTables(conf);
    String tableLocation = new File(tableDir, "test").toString();
    this.table =
        tables.create(
            SCHEMA,
            SPEC,
            Collections.singletonMap("format-version", String.valueOf(FORMAT_VERSION)),
            tableLocation);

[ghareeb-falazi]

@jaydeluca yes this works, very good idea! thanks!

[ghareeb-falazi]

@jaydeluca It turns out that hadoop won't work with a high JDK level without modifying some security settings (see screenshot for concrete error). Therefore, I added maxJavaVersionForTests.set(JavaVersion.VERSION_21) to build.gradke.kts, and with this, the tests work fine.
Alternatively, we can go back to depending on test library of Iceberg if you want. I don't see where the problem is since the dependency is only for testing, and we control the exact iceberg version that we are using for testing. What do you think?

[jaydeluca]

i think using the test dependency is probably preferable compared to not testing against newer versions of java, so It's probably best to switch back. Can you just include a small comment above the gradle config saying something like "Test classes are not published by default" or something similar that explains why it is necessary?

[ghareeb-falazi]

@jaydeluca I reverted back to using the Iceberg test classes and added the required comment in the gradle file

[Copilot]

Pull request overview

This PR introduces library instrumentation for Apache Iceberg scan metrics, enabling OpenTelemetry-based monitoring of table scan operations. The implementation focuses on ScanMetrics only, with CommitMetrics and agent-based instrumentation planned for future work.

Key changes:

• Adds IcebergTelemetry API that wraps Scan instances with OpenTelemetry metrics reporting
• Implements custom MetricsReporter to translate Iceberg scan metrics to OpenTelemetry metrics
• Provides comprehensive test coverage through a shared test base class

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
settings.gradle.kts	Adds new iceberg-1.8 library and testing modules to the build
.fossa.yml	Registers the new library module for dependency scanning
instrumentation/iceberg-1.8/metadata.yaml	Provides metadata description and library link for the instrumentation
instrumentation/iceberg-1.8/README.md	Documents usage patterns and dependency configuration for the library
instrumentation/iceberg-1.8/library/build.gradle.kts	Configures dependencies and Java 11 minimum version requirement
instrumentation/iceberg-1.8/library/src/main/java/.../IcebergTelemetry.java	Public API entry point for wrapping Scan instances with telemetry
instrumentation/iceberg-1.8/library/src/main/java/.../IcebergMetricsReporter.java	Core implementation that converts Iceberg metrics to OpenTelemetry metrics
instrumentation/iceberg-1.8/library/src/main/java/.../ScanMetricsBuilderFactory.java	Factory methods for creating metric builders with proper names, descriptions, and units
instrumentation/iceberg-1.8/library/src/test/java/.../IcebergTest.java	Concrete test implementation for library instrumentation
instrumentation/iceberg-1.8/testing/build.gradle.kts	Configures test dependencies including Iceberg test utilities
instrumentation/iceberg-1.8/testing/src/main/java/.../AbstractIcebergTest.java	Shared test base class with comprehensive metric assertions
docs/supported-libraries.md	Adds Apache Iceberg to the list of supported libraries

[trask]

@ghareeb-falazi don't worry if there's a sporadically failing test, we can re-run those

[ghareeb-falazi]

@ghareeb-falazi don't worry if there's a sporadically failing test, we can re-run those

@trask Thanks for letting me know, and sorry if you have been email notifications about all the merges I did 😅