The PRs which are not approved run subset of tests

[potiuk]

This PR is an implementation of optimisation - to only run
default values for build matrix in case PR does not have
"okay to test" label.

This "okay to test" label is set when the PR gets approved
but it was not approved before, also then a comment is generated
urging the committer to rebase the PR to run full set of tests.

Additionally a check is added (in-progress) that makes the PR
not yet ready to be merged. Only after re-running it it will
become truly ready to be merged.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

cc: @TobKed - > once we have the auto-labeling and re-triggering the workflow, this one implements only running one default combo of all tests.

[potiuk]

@dimberman @ashb @kaxil @turbaszek -> once we have the labeling in place, this one limits not only the actual tests run but also the number of images prepared. We only need and run one image (the default one) for such build, so the savings will be rather cool.

[potiuk]

cc: @TobKed -> testing without the selective checks and comments for now :).

[potiuk]

OK. We have it working with @TobKed. Please take a look. I'd love to merge it for tomorrow to see whether we will have some improvements even before moving to self-hosted runners.

It works as follows (and I will describe it in separate PR):

• by default when PR is not approved only "Default" variants of matrix are run as tests.

• when a committer approves the PR (generally when the state changes to "approved by all committers who reviewed) the PR review workflow is run and it

  • sets the label to "ok to test" and
  • adds the comment, that you should rebase or rerun the workflow to run all the test

Also a new check is added that will remain "in-progress" until you actually rebase the request or commiter re-runs the check:

I think at this point we might even decide to merge such request in case of simple changes that do not introduce too big of a risk. Maybe even we can change the comment to explicitly say so and leave the decision to committer.

• When someone rebases/reruns such PR with "okay to test" label, it will run with full suite of tests. This will continue with every rebase until some commiter "request changes" - in which case the label will be removed and the PR will return back to "selected tests only".

• It is a bit different for "doc-only" PRs that do not run any tests. When commiter approves the PR the PR gets "okay to merge" label and comment indicating that the PR can be merged:

And it can be merged immediately

[TobKed]

I released today v1.1 of TobKed/label-when-approved-action

[potiuk]

I also realised that I have to do an exception for image building in v1-10-stable/test branch. Pushing a fixup with both shortly.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

Shiny-new version! Here is the final description of how it works (note that we will not see that behavior here - only after we merge it)!

All the PRs before approval will run either "small" set of tests (default matrix values) or not at all (in case of doc-only changes).

Then interesting things will happen after PR gets approval from the committer:

1. The doc-only tests will get "okay to merge" label and this comment: "The PR is ready to be merged. No tests are needed!" : 

2. If the PR is not changing "Core" of airflow, the PR will get "okay to merge" label and this comment: "The PR should be OK to be merged with just subset of tests as it does not modify Core of Airflow. The committers might merge it or can add a label 'full tests needed' and re-run it to run all tests if they see it is needed!". 

3. If the PR is changing "Core" of airflow, the PR will get "full tests needed" label and this comment:  "This PR requires a full set of tests. Please rebase the PR on the latest master or ask a committer to re-run the tests". And the "okay to test" label should change to "full tests needed". We will add the "in-progress" check, to make the button 'gray' until it gets rebased and the full set of tests will run for it. The PRs with this label will run "full matrix" tests. 

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

cc: @TobKed -> merged