Orchestrate workflows

To request support or provide feedback for this feature, email [email protected].

This document describes how to orchestrate workflows, including how to schedule workflows and inspect scheduled workflow runs.

Workflows are powered by Dataform.

Each workflow schedule is run by a custom Dataform service account, which you select during schedule creation. For more information about the types of service accounts in Dataform, see About service accounts in Dataform.

Changes you make to a workflow are automatically saved, but are available only to you and to users granted the Dataform Admin role on the project. To update the schedule with a new version of the workflow, you need to deploy the workflow. Deploying updates the schedule to use your current version of the workflow. Schedules always run the latest deployed version.

Schedules of workflows that contain notebooks use a default runtime specification. During a scheduled run of a workflow containing notebooks, BigQuery writes notebook output to the Cloud Storage bucket selected during schedule creation.

Before you begin

Before you begin, create a workflow.

Enable workflow scheduling

To schedule workflows, you must grant the following roles to the service account which you plan to use for workflow schedules:

Service Account User (roles/iam.serviceAccountUser)

Follow Grant a single role on a service account to add your service account as a principal to itself. In other words, add the service account as a principal to the same service account. Then, grant the Service Account User role to this principal.

If your workflow contains SQL queries, you must grant the following roles to the service account which you plan to use for workflow schedules:

BigQuery Job User (roles/bigquery.jobUser)

Follow Grant a single role on a project to grant the BigQuery Job User role to your service account on projects from which your workflows read data.

BigQuery Data Viewer (roles/bigquery.dataViewer)

Follow Grant a single role on a project to grant the BigQuery Data Viewer role to your service account on projects from which your workflows read data.

BigQuery Data Editor (roles/bigquery.dataEditor)

Follow Grant a single role on a project to grant the BigQuery Data Editor role to your service account on projects to which your workflows write data.

If your workflow contains notebooks, you must grant the following roles to the service account which you plan to use for workflow schedules:

Notebook Executor User (roles/aiplatform.notebookExecutorUser)

Follow Grant a single role on a project to grant the Notebook Executor User role to your service account on the selected project.

Storage Admin (roles/storage.admin)

Follow Add a principal to a bucket-level policy to add your service account as a principal to the Cloud Storage bucket which you plan to use for storing output of notebooks executed in scheduled workflow runs, and grant the Storage Admin role to this principal.

Additionally, you must grant the following role to the default Dataform service account:

Service Account Token Creator(roles/iam.serviceAccountTokenCreator)

Follow Grant token creation access to a service account to add the default Dataform service account as a principal to your service account, and grant the Service Account Token Creator role to this principal.

To learn more about service accounts in Dataform, see About service accounts in Dataform.

Required roles

To get the permissions that you need to manage workflows, ask your administrator to grant you the following IAM roles:

• To delete workflows: Dataform Admin (roles/dataform.Admin) on the workflow
• To create, edit, run, and delete workflow schedules: Dataform Admin (roles/dataform.Admin) on the workflow
• To view and run workflows: Dataform Viewer (roles/dataform.Viewer) on the project
• To view workflow schedules: Dataform Editor (roles/dataform.Editor) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Dataform IAM, see Access control with IAM.

Create a workflow schedule

To schedule a workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click Schedule.

4. In the Schedule workflow pane, in the Schedule name field, enter a name for the schedule.

5. In the Service account field, select a service account.

6. If your workflow contains a notebook, in the Cloud Storage bucket field, click Browse and select or create a Cloud Storage bucket for storing the output of notebooks in your workflow.

   Your selected service account must be granted the Storage Admin IAM role on the selected bucket. For more information, see Enable workflow scheduling.

7. In the Schedule frequency section, do the following:

   1. In the Repeats drop-down, select the frequency of scheduled workflow runs.
   2. In the At time field, enter the time for scheduled workflow runs.
   3. In the Timezone drop-down, select the timezone for the schedule.

8. Click Create schedule.

When you create the schedule, the current version of the workflow is automatically deployed. To update the schedule with a new version of the workflow, deploy the workflow.

The latest deployed version of the workflow runs at the selected time and frequency.

Deploy a workflow

Deploying a workflow updates its schedule with the current version of the workflow. Schedules run the latest deployed version of the workflow.

To deploy a workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click Deploy.

The corresponding schedule is updated with the current version of the workflow. The latest deployed version of the workflow runs at the scheduled time.

Disable a schedule

To pause scheduled runs of a selected workflow without deleting the schedule, you can disable the schedule.

To disable a schedule for a selected workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click View schedule.

4. In the Schedule details table, in the Schedule state row, click the Schedule is enabled toggle.

Enable a schedule

To resume scheduled runs of a disabled workflow schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click View schedule.

4. In the Schedule details table, in the Schedule state row, click the Schedule is disabled toggle.

Manually run a deployed workflow

When you manually run a workflow deployed in a selected schedule, BigQuery executes the deployed workflow once, independently from the schedule.

To manually run a deployed workflow, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Click the name of the selected workflow schedule.

3. On the Schedule details page, click Run.

View all workflow schedules

To view all workflow schedules in your Google Cloud project, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Optional: To display additional columns with workflow schedule details, click view_column Column display options, and then select columns and click OK.

View workflow schedule details

To view details for a selected workflow schedule, follow these steps:

View past scheduled runs

To view past runs of a selected workflow schedule, follow these steps:

Edit a workflow schedule

To edit a workflow schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click View schedule, and then click Edit.

4. In the Schedule workflow dialog, edit the schedule, and then click Update schedule.

Delete a workflow schedule

To permanently delete a workflow schedule, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Do either of the following:

   • Click the name of the selected workflow schedule, and then on the Schedule details page, click Delete.

   • In the row that contains the selected workflow schedule, click more_vert View actions in the Actions column, and then click Delete.

3. In the dialog that appears, click Delete.

What's next

• Learn more about workflows in BigQuery.
• Learn how to create workflows.