Error Encountered During Composer Upgrade: BigQuery Job Fetching

[AaronTao]

We are upgrading Composer and Airflow from composer-1.19.10-airflow-2.2.5 to composer-2.8.2-airflow-2.7.3. Meanwhile, we use Terraform to manage the Composer environment creation. I copied one of the DAGs and kept the code snippet the same, but I encountered the following error:

“ Error on fetching BigQuery job.”

I am trying to determine whether this issue is due to my configuration or something related to the GCP platform itself.

I have included the code snippet and log for reference.

[NorieRam]

Hi @AaronTao,

Welcome to Google Cloud Community!

The "ERROR - Error on fetching BigQuery job" is possibly caused by authentication issues with the google-cloud-bigquery client library in your script, which is struggling to use the provided credentials correctly. There’s a good chance this is unrelated to the Composer upgrade or any changes to the BigQuery API.

You may check the following to determine what could be wrong in your situation:

• Service Account: Verify your Composer environment uses a service account with "BigQuery Job User" role and a valid, unexpired key file.
• Terraform: Ensure your Terraform configuration correctly sets up the service account, assigns roles, and provides the key file to Composer.
• Authentication Method: Confirm that your BigQuery.py script uses the right authentication method (likely Application Default Credentials (ADC)).

For the next steps, you may proceed to the following:

• Add Logging: Use logging in your BigQuery.py to pinpoint the issue.
• Terraform Validation: Re-apply your Terraform configuration to ensure the service account is correctly set up.
• BigQuery Authentication: You can refer to the documentation for guidance on authentication methods and error handling.

Reach out to Google Cloud Support for personalized help if you’ve followed these steps and still need assistance.

I hope the above information is helpful.

[AaronTao]

Hi Norie,

Many thanks for your reply. With the confirmation of our SRE colleague, the service account we used in the project comes with Bigquery administrator privilege.  And the authentication has been correctly configured with the Terraform and we can see it from the GCP composer website.

It would be great if you could provide more guidelines regarding this issue.

Thanks a lot!

Best regards!