Introduction to BigQuery pipelines

To request support or provide feedback for this feature, email bigquery-workflows-preview-feedback@google.com.

You can use BigQuery pipelines to automate and streamline your BigQuery data processes. With pipelines, you can schedule and execute code assets in sequence to improve efficiency and reduce manual effort.

Pipelines are powered by Dataform.

A pipeline consists of one or more of the following code assets:

You can use pipelines to schedule the execution of code assets. For example, you can schedule a SQL query to run daily and update a table with the most recent source data, which can then power a dashboard.

In a pipeline with multiple code assets, you define the execution sequence. For example, to train a machine learning model, you can create a pipeline in which a SQL query prepares data, and then a subsequent notebook trains the model using that data.

You can schedule a pipeline to automatically run at a specified time and frequency.

Limitations

Pipelines are subject to the following limitations:

  • You can't add an existing notebook or SQL query to a pipeline. To add a pipeline task, you need to create a new notebook or SQL query in the pipeline.
  • You can't grant access to a selected pipeline to other users.
  • Pipelines are available only in the Google Cloud console.
  • You can't change the region for storing a pipeline after it is created. For more information, see Set the default region for code assets.

Supported regions

All code assets are stored in your default region for code assets. Updating the default region changes the region for all code assets created after that point.

The following table lists the regions where pipelines are available:

Region description Region name Details
Africa
Johannesburg africa-south1
Americas
Columbus us-east5
Dallas us-south1 leaf icon Low CO2
Iowa us-central1 leaf icon Low CO2
Los Angeles us-west2
Las Vegas us-west4
Montréal northamerica-northeast1 leaf icon Low CO2
N. Virginia us-east4
Oregon us-west1 leaf icon Low CO2
São Paulo southamerica-east1 leaf icon Low CO2
South Carolina us-east1
Asia Pacific
Hong Kong asia-east2
Jakarta asia-southeast2
Mumbai asia-south1
Seoul asia-northeast3
Singapore asia-southeast1
Sydney australia-southeast1
Taiwan asia-east1
Tokyo asia-northeast1
Europe
Belgium europe-west1 leaf icon Low CO2
Frankfurt europe-west3 leaf icon Low CO2
London europe-west2 leaf icon Low CO2
Madrid europe-southwest1 leaf icon Low CO2
Netherlands europe-west4 leaf icon Low CO2
Turin europe-west12
Zürich europe-west6 leaf icon Low CO2
Middle East
Doha me-central1
Dammam me-central2

Quotas and limits

BigQuery pipelines are subject to Dataform quotas and limits.

Pricing

The execution of BigQuery pipeline tasks incurs compute and storage charges in BigQuery. For more information, see BigQuery pricing.

Pipelines containing notebooks incur Colab Enterprise runtime charges based on the default machine type. For pricing details, see Colab Enterprise pricing.

Each BigQuery pipeline run is logged using Cloud Logging. Logging is automatically enabled for BigQuery pipeline runs, which can incur Cloud Logging billing charges. For more information, see Cloud Logging pricing.

What's next