[FLINK-39674][formats] Upgrade Confluent Schema Registry client

[qiuyanjun888]

Summary

This PR updates flink-avro-confluent-registry to a newer Confluent Schema Registry client and adds a focused regression test for Kafka 3.9.x OAuth compatibility.

Root Cause

flink-avro-confluent-registry still used io.confluent:kafka-schema-registry-client:7.5.3. With Kafka client 3.9.x on the classpath, the Schema Registry OAuth bearer path can fail during client initialization because Confluent 7.5.3 calls the old HttpAccessTokenRetriever constructor signature that is no longer present in Kafka 3.9.x.

Changes

• Upgrade io.confluent:kafka-schema-registry-client from 7.5.3 to 7.9.8.
• Add a test-scoped Kafka 3.9.2 dependency for compatibility validation.
• Add a focused regression test that initializes the Schema Registry OAuth bearer credential path with Kafka 3.9.2 on the test classpath.

Validation

• Added RED regression coverage:
  • ./mvnw -pl flink-formats/flink-avro-confluent-registry -Dtest=CachedSchemaCoderProviderTest#testThatOauthBearerAuthIsCompatibleWithKafkaClient39 test -DskipITs -Drat.skip=true -Dcheckstyle.skip=true
  • Fails before the dependency upgrade with NoSuchMethodError from OauthCredentialProvider.getTokenRetriever(...).
• Verified GREEN after the upgrade:
  • ./mvnw -pl flink-formats/flink-avro-confluent-registry -Dtest=CachedSchemaCoderProviderTest#testThatOauthBearerAuthIsCompatibleWithKafkaClient39 test -DskipITs -Drat.skip=true -Dcheckstyle.skip=true
• Ran the module test suite:
  • ./mvnw -pl flink-formats/flink-avro-confluent-registry test -DskipITs
  • Result: 28 tests, 0 failures, 0 errors; checkstyle and spotless clean.
• Verified the final dependency tree includes:
  • io.confluent:kafka-schema-registry-client:7.9.8
  • org.apache.kafka:kafka-clients:3.9.2 in test scope.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

---

Was generative AI tooling used to co-author this PR?

• Yes, Hermes Agent with OpenAI GPT-5.5

Generated-by: Hermes Agent (OpenAI GPT-5.5)

[flinkbot]

CI report:

• 99d3a87 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[spuru9]

META-INF/NOTICE still declares the old versions, can you update them as well.