adding dataproc labels to bigquery jobs

[tnazarew]

Adding labels containing information about Spark and GCP environment to BigQuery job will help link it's lineage information with Dataplex and make navigation easier for users. The code adds lables to BQ jobs containing information similar to what is collected by OpenLineage.

[gemini-code-assist]

Summary of Changes

Hello @tnazarew, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the Spark BigQuery connector by automatically adding detailed Spark and Google Cloud Platform (GCP) environment labels to BigQuery jobs. This initiative aims to improve data lineage tracking within Dataplex and streamline user navigation by providing richer metadata, aligning with information typically collected by OpenLineage.

Highlights

• Enhanced BigQuery Job Labeling: BigQuery jobs will now automatically include detailed labels derived from the Spark and GCP environment, such as application ID, application name, project ID, Dataproc cluster/batch/session IDs, and region. This enriches the metadata associated with jobs, improving traceability.
• New GCP Metadata Utility: A new utility class, GCPLabelUtils, has been introduced to programmatically fetch relevant metadata from the GCP environment and Spark configurations, serving as the core logic for generating the new labels.
• Dependency Updates: New dependencies, httpclient5 and mockserver-netty, were added to support the fetching of GCP metadata and to facilitate robust testing of the new utility class.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds functionality to include Dataproc environment labels in BigQuery jobs, which is a great enhancement for lineage tracking. The implementation in GCPLabelUtils is mostly solid, but I've identified a few critical issues. There's a potential for a StringIndexOutOfBoundsException when parsing cluster names and a dependency version for httpclient5 that doesn't exist, which will break the build. Additionally, there's a high-severity issue with how job labels are merged, which could lead to runtime exceptions on key collision. I've also included some medium-severity suggestions to improve performance by avoiding redundant network calls, increase robustness in tag parsing, and adjust an overly aggressive HTTP timeout. Please address the critical and high severity issues before merging.

[BoqianShi]

/gcbrun

[codelixir]

/gcbrun

[codelixir]

The presubmit check that fails for this PR is defined here: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/blob/master/scripts/verify-shading.sh

In the build logs, I see a lot of classes of these patterns:

org/apache/hc/client5/**
org/apache/hc/core5/**
...
Found unshaded classes, please fix above findings

[codelixir]

/gcbrun