feat(clickpipes): BigQuery source support #413
Conversation
This PR enables creating BigQuery ClickPipe. This is in private preview.
jkaflik
force-pushed
the
kk/clickpipes-bigquery
branch
from
7ae8fca to
c40f64f
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds support for BigQuery as a source for ClickPipe snapshots, currently in private preview. The implementation includes schema definitions, resource handling, and state synchronization for BigQuery configurations.
Key changes:
- Adds BigQuery source models and schema definitions with support for credentials, settings, and table mappings
- Updates source type detection and handling logic to treat BigQuery as a database pipe alongside Postgres
- Provides a complete example with GCP infrastructure setup for BigQuery ClickPipe integration
Reviewed changes
Copilot reviewed 11 out of 31 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/resource/models/clickpipe_resource.go | Defines model structures for BigQuery source, settings, table mappings, and service account credentials |
| pkg/resource/clickpipe.go | Implements BigQuery schema, extraction logic, and integrates it into the ClickPipe resource lifecycle |
| pkg/internal/api/clickpipe_models.go | Adds API model definitions for BigQuery source configuration and credentials |
| pkg/internal/api/clickpipe.go | Normalizes replication mode and table engine constants to be shared between Postgres and BigQuery |
| examples/resources/clickhouse_clickpipe/resource_bigquery_snapshot.tf | Provides inline documentation example for BigQuery ClickPipe configuration |
| examples/clickpipe/bigquery_snapshot/vars.tf | Defines input variables for the BigQuery example |
| examples/clickpipe/bigquery_snapshot/variables.tfvars.sample | Sample variable values for the BigQuery example |
| examples/clickpipe/bigquery_snapshot/provider.tf.template.alpha | Provider configuration template for the example |
| examples/clickpipe/bigquery_snapshot/provider.tf | Generated provider configuration with version pinning |
| examples/clickpipe/bigquery_snapshot/gcp.tf | GCP infrastructure setup including service account, IAM roles, and staging bucket |
| examples/clickpipe/bigquery_snapshot/clickpipe.tf | Complete example of BigQuery ClickPipe resource configuration |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if isDBPipe { | ||
| destinationModel.Table = types.StringNull() | ||
| destinationModel.ManagedTable = types.BoolValue(false) // Always false for Postgres pipes | ||
| destinationModel.ManagedTable = types.BoolValue(true) // Always true for DB pipes - matches default behavior |
There was a problem hiding this comment.
The comment states this 'matches default behavior' but the previous code set this to false for Postgres pipes. This change appears to reverse the behavior for existing Postgres pipes. If this is intentional, the comment should clarify why the behavior changed from false to true.
| destinationModel.ManagedTable = types.BoolValue(true) // Always true for DB pipes - matches default behavior | |
| destinationModel.ManagedTable = types.BoolValue(true) // Always true for DB pipes. Note: This reverses previous behavior for Postgres pipes, which set managed_table to false. This change is intentional to align with updated API expectations and ensure consistent handling of DB pipes. |
There was a problem hiding this comment.
@tpanetti could you please have a look? I have mixed feelings about this change - I'm not sure how false is possible for Postgres pipe remote state, as we default it to true.
There was a problem hiding this comment.
I think you may have this backwards. According to the validation, managedTable cannot be provided for database pipes Destination managedTable is required if source is not a database pipe and must not be provided for database pipes
|
@jkaflik we have a short |
|
@EvandroLG yes. Forgot to push. |
| "source_table": types.StringType, | ||
| "target_table": types.StringType, | ||
| "excluded_columns": types.ListType{ElemType: types.StringType}, | ||
| "use_custom_sorting_key": types.BoolType, |
There was a problem hiding this comment.
Nit: There's no validation that sorting_keys is provided when use_custom_sorting_key = true.
ilidemi
left a comment
ilidemi
left a comment
There was a problem hiding this comment.
Agreed with others' comments, looks good otherwise.
Also Postgres has tests, does BQ need them too?
Support for BigQuery snapshot ClickPipe creation, currently in private preview.
Comes with a self-contained example and a short, in-line documentation example.
Extra in this PR, I normalize how sample tfvars are organized for ClickPipes examples.