When a BigQuery DTS Transfer Run fails, this is not flagged in the Airflow task states in teradata_transfer_run_log_dag. The check_transfer_run_state task is marked as successful even if the state is failed.
Surface these more effectively, such as by failing the task/dag state. The dmt_logs.dmt_teradata_transfer_run_summary table does show the failed runs.
[2025-02-07, 20:39:02 UTC] {teradata_transfer_run_log_dag.py:537} INFO - TRANSFER_RUN_STATE: FAILED
[2025-02-07, 20:39:02 UTC] {teradata_transfer_run_log_dag.py:541} INFO - Skipping Validation
...
[2025-02-07, 20:39:02 UTC] {taskinstance.py:1206} INFO - Marking task as SUCCESS. ...
To reproduce: Provide invalid source DB connection details in the config.
Error loading Teradata Source Definition: Could not connect with provided parameters: [Teradata JDBC Driver] [TeraJDBC 20.00.00.34] [Error 1000] [SQLState 08S01] Login failure for Connection to 10.128.0.18 Fri Feb 07 20:38:39 UTC 2025 socket orig=10.128.0.18 cid=43b40233 sess=0 java.io.IOException: Connection to /10.128.0.18:1025 failed: Connection refused
When a BigQuery DTS Transfer Run fails, this is not flagged in the Airflow task states in
teradata_transfer_run_log_dag. Thecheck_transfer_run_statetask is marked as successful even if the state is failed.Surface these more effectively, such as by failing the task/dag state. The
dmt_logs.dmt_teradata_transfer_run_summarytable does show the failed runs.To reproduce: Provide invalid source DB connection details in the config.