Run browser tests in parallel
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	awight
	Jun 28 2019, 8:58 PM

Description

(Apologies if this is a duplicate, I know there's been some discussion already but couldn't find it in Phabricator.)

Browser tests are slow, and there are a few opportunities for parallelization. The easiest step would be to run our QUnit and node-selenium tests in separate threads. After T199116 we could also run selenium tests for each repo in parallel, but this introduces the additional challenge of potential interaction between tests.

However, there are some questions to work through first:

Can our test webserver handle multiple clients? (See also T225218.)
Is it reasonable to run multiple chromedrivers on one machine? How much more memory will this require?
How many of our tests will become fragile, for example because they depend on a constant rather than randomly-generated title?

Details

Subject	Repo	Branch	Lines +/-
selenium: run tests concurrently	mediawiki/extensions/GrowthExperiments	master	+1 -0
selenium: run tests concurrently	mediawiki/extensions/Wikibase	master	+2 -0
selenium: run tests concurrently	mediawiki/extensions/Wikibase	master	+3 -3
selenium: run tests concurrently	mediawiki/extensions/ProofreadPage	master	+2 -1
selenium: run tests concurrently	mediawiki/extensions/PageTriage	master	+1 -0
[WIP] Run Selenium tests for each project in parallel	integration/quibble	master	+46 -72
selenium: Run tests in each suite concurrently	mediawiki/extensions/WikibaseLexeme	master	+2 -0
BrowserTests: Support splitting projects into two groups	integration/quibble	master	+17 -0
selenium: run tests concurrently	mediawiki/extensions/CheckUser	master	+96 -9
selenium: run tests concurrently	mediawiki/extensions/VisualEditor	master	+3 -1
selenium: run tests concurrently	mediawiki/extensions/Echo	master	+3 -1
selenium: run tests concurrently	mediawiki/extensions/CheckUser	master	+3 -1
selenium: Run test suites concurrently	mediawiki/extensions/GrowthExperiments	master	+1 -0
Release Quibble 1.3.0	integration/quibble	master	+38 -3
selenium: Run test suites concurrently	mediawiki/core	master	+1 -1
selenium: Run test suites concurrently	mediawiki/extensions/AbuseFilter	master	+2 -1
selenium: run tests concurrently	mediawiki/skins/MinervaNeue	master	+3 -1
selenium: run 4 tests in parallel	mediawiki/core	master	+2 -0
BrowserTests: Option to run npm install in parallel	integration/quibble	master	+55 -7
Prepare to run browser suites in parallel	integration/quibble	master	+145 -64

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T76586 K3. Run Flow tests with Parsoid on Jenkins
Resolved	Feature	kostajh	T218534 Run Parsoid service in quibble
Open		None	T225730 Reduce runtime of MW shared gate Jenkins jobs to 5 min
Open		None	T226869 Run browser tests in parallel
Declined		None	T291476 Migrate Wikibase selenium tests to Quibble+Apache, enable concurrency
Resolved		None	T285649 Switch all Quibble Selenium and api-testing jobs to use apache
Resolved		None	T276428 Introduce non-voting jobs with quibble+apache
Resolved		awight	T225218 Consider httpd for quibble instead of php built-in server
Resolved		kostajh	T282154 Wikibase: Special:EntityData compression encoding issues in local dev env
Open		None	T227352 Set up extension tests for Parsoid repo
Resolved		cscott	T248726 Run phan on code in extension/
Resolved		kostajh	T297480 Inject environment variable to indicate that Quibble is running with Apache
Open		None	T298514 "should mark the revert as a bot edit" test flaky with Apache + php-fpm
Resolved		kostajh	T299491 Switch QUnit tests to use Apache backend
Resolved		hashar	T299492 Quibble ci-fullrun jobs should use Apache backend
Open		None	T301100 Fix flaky ContributionsCount.js test
Resolved	PRODUCTION ERROR	hoo	T255566 [LEX] [TECH] Figure out why LexemeEditEntityTest::testGivenInvalidDataInClearRequest_errorIsReported is throwing an Exception/logging an error with AbuseFilter enabled
Resolved		Lucas_Werkmeister_WMDE	T298682 Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel
Resolved		None	T337862 Try to run browser tests of WMDE-TechWish projects in parallel
Resolved		Dreamy_Jazz	T340823 Fix CheckUser selenium tests so they can run concurrently
Resolved		Krinkle	T199393 Selenium tests sometimes fail due to deadlock in User::addToDatabase from api.php?action=createaccount
Resolved		hashar	T342088 Run Quibble Mariadb server with innodb_print_all_deadlocks
Resolved		None	T340943 TwoColConflict selenium tests run in parallel trigger db errors
Open		None	T344754 Browser tests video capture is shared between tests
Open		None	T345149 Special:RecentChanges Selenium test has a race condition when running concurrently

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 545661 abandoned by Awight:
Prepare to run browser suites in parallel

Reason:
Please see Ib2dc728980ce95 instead.

https://gerrit.wikimedia.org/r/545661

Jdforrester-WMF edited projects, added Quibble; removed Continuous-Integration-Infrastructure.Jun 25 2020, 5:14 PM

Daimona subscribed.Oct 7 2020, 11:25 AM

zeljkofilipin removed a project: User-zeljkofilipin.Dec 22 2020, 7:07 PM

Krinkle added a parent task: T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min.Apr 29 2021, 11:17 PM

Change 545653 abandoned by Thiemo Kreuz (WMDE):

[mediawiki/core@master] [DNM] Experiment with concurrent browser tests

Reason:

This is in conflict right now. It's a very trivial change that can be redone any time, if needed.

https://gerrit.wikimedia.org/r/545653

Change 545650 abandoned by Thiemo Kreuz (WMDE):

[mediawiki/extensions/Wikibase@master] [DNM] Experiment with concurrent browser tests

Reason:

This is in conflict right now. It's a very trivial change that can be redone any time, if needed.

https://gerrit.wikimedia.org/r/545650

awight added a subtask: T291476: Migrate Wikibase selenium tests to Quibble+Apache, enable concurrency.Sep 21 2021, 10:02 AM

kostajh subscribed.Nov 11 2021, 8:21 AM

Change 738061 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] Browser tests: run npm install in parallel

https://gerrit.wikimedia.org/r/738061

Change 738061 merged by jenkins-bot:

[integration/quibble@master] BrowserTests: Option to run npm install in parallel

https://gerrit.wikimedia.org/r/738061

Lens0021 subscribed.Jan 5 2022, 3:17 PM

Change 751761 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/core@master] selenium: run 4 tests in parallel

https://gerrit.wikimedia.org/r/751761

hashar added a subtask: T285649: Switch all Quibble Selenium and api-testing jobs to use apache.Jan 5 2022, 4:11 PM

Change 545650 restored by Hashar:

[mediawiki/extensions/Wikibase@master] [DNM] Experiment with concurrent browser tests

https://gerrit.wikimedia.org/r/545650

Change 751761 abandoned by Hashar:

[mediawiki/core@master] selenium: run 4 tests in parallel

Reason:

I will restore https://gerrit.wikimedia.org/r/c/mediawiki/core/+/545653

https://gerrit.wikimedia.org/r/751761

Change 545653 restored by Hashar:

[mediawiki/core@master] [DNM] Experiment with concurrent browser tests

https://gerrit.wikimedia.org/r/545653

Change 751767 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/skins/MinervaNeue@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/751767

As of this morning, CI exposes MediaWiki through Apache which allow concurrent requests (T285649) we should thus be able to get webdriver.io to run tests in parallel. I have restored and rebased some patches made by @awight and added one for MinervaNeue which has more than a few tests:

mediawiki/core	https://gerrit.wikimedia.org/r/545653
Wikibase client and repo	https://gerrit.wikimedia.org/r/545650
MinervaNeue	https://gerrit.wikimedia.org/r/751767

Change 747904 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] BrowserTests: Support splitting projects into two groups

https://gerrit.wikimedia.org/r/747904

zeljkofilipin added a project: User-zeljkofilipin.Jan 5 2022, 5:59 PM

Change 751767 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/751767

@kostajh If I understand correctly, this last patch would split at the job/quibble level, and thus not share a common db or web server. That's fine, I think, but that approach wouldn't have been blocked on using Apache I think. Can you confirm that?

Are we (also) considering letting wdio run chromium sessions concurrently? I imagine that may be tricky to some extent as tests could interfere with each other on the same server/db esp when involving a shared concept like RC feed, user prefs, or page title. But maybe we have alternate strategies for tests that rely on that, eg mandating new titles/users for everything always, and for RC feed perhaps check lower items and not just the last and hope there's not too many. This is somewhat different in direction compared to the direction of using more stable/deterministic titles in tests and using setup/tear down scripts, which is also in our backlog. That could potentially be addressed differently if we departed from testing an existing test/dev wiki and instead created an ad-hoc db+localsettings within the test runner, and/or formalise what we do with quibble and a lighter and more reusable way and discourage local bare running. Food for thought :-)

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.17; 2022-01-10).Jan 5 2022, 7:00 PM

In T226869#7599620, @Krinkle wrote:

@kostajh If I understand correctly, this last patch would split at the job/quibble level, and thus not share a common db or web server. That's fine, I think, but that approach wouldn't have been blocked on using Apache I think. Can you confirm that?

Yes, that's right – https://gerrit.wikimedia.org/r/747904 is not dependent on Apache. Maybe it should be tagged with a different task to clear up confusion. There's a few complementary approaches being proposed here:

within the existing Selenium job (e.g. https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/129061/console), run core + skin + extension selenium tests in parallel, using Quibble's ParallelCommand utility. If this "just worked", that would be nice, but I suspect it won't be easy for any number of reasons – there's a single chromedriver instance, tests messing up state from other tests, etc.
within the existing Selenium job, for each core/skin/extension test suite, parallelize the execution of tests in the suite (i.e. each it() statement). That will also surface some problems where test methods interfere with one anotehr when querying RC feed, prefs, page titles, etc as you noted. It also probably doesn't bring a big performance improvement, but it would be nice if it did :)
create two new Selenium jobs, group A and group B, where e.g. group A runs Abusefilter -> GrowthExperiments tests, and group B runs MobileFrontend -> Wikibase. That has the downside of requiring more VM resources to run an additional job, and also some work is done twice (Zuul clone, npm install), but has the benefit of cutting down the overall time spent on Selenium for any given patch by several minutes. In theory we could have a job that does the initial Zuul clone and npm install that group A and group B jobs then both make use of to save more time.

hashar mentioned this in T298682: Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel.Jan 6 2022, 11:19 AM

hashar added a subtask: T298682: Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel.

FYI:

Change by Awight merged:

[mediawiki/core] selenium: run 4 tests in parallel

https://gerrit.wikimedia.org/r/545653

Change 751940 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/AbuseFilter@master] selenium: Run tests in each suite concurrently

https://gerrit.wikimedia.org/r/751940

Change 751942 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/WikibaseLexeme@master] selenium: Run tests in each suite concurrently

https://gerrit.wikimedia.org/r/751942

Change 751957 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] selenium: Run test suites concurrently

https://gerrit.wikimedia.org/r/751957

Change 751959 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] selenium: Run test suites concurrently

https://gerrit.wikimedia.org/r/751959

Change 751961 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Run tests in parallel

https://gerrit.wikimedia.org/r/751961

Change 751940 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] selenium: Run test suites concurrently

https://gerrit.wikimedia.org/r/751940

kostajh mentioned this in T298735: Run api-testing tests in parallel.Jan 6 2022, 9:19 PM

Change 751959 abandoned by Kosta Harlan:

[mediawiki/core@master] selenium: Run test suites concurrently

Reason:

https://gerrit.wikimedia.org/r/751959

zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.Jan 7 2022, 1:29 PM

zeljkofilipin moved this task from Deep work 🌊 to Watching 📺 on the User-zeljkofilipin board.Jan 12 2022, 3:36 PM

Umherirrender mentioned this in T299077: CI job wmf-quibble-selenium-php72-docker get a deadlock on `change_tag_def` (NameTableStore).Jan 12 2022, 5:58 PM

Change 754524 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] Release Quibble 1.3.0

https://gerrit.wikimedia.org/r/754524

Change 754524 merged by jenkins-bot:

[integration/quibble@master] Release Quibble 1.3.0

https://gerrit.wikimedia.org/r/754524

Mentioned in SAL (#wikimedia-releng) [2022-01-17T16:16:21Z] <hashar> Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931

Lucas_Werkmeister_WMDE closed subtask T298682: Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel as Resolved.Jan 18 2022, 3:27 PM

thiemowmde added a project: WMDE-TechWish-Maintenance.Jan 31 2022, 10:08 AM

Change 768354 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run Selenium tests for each project in parallel

https://gerrit.wikimedia.org/r/768354

In T226869#7755609, @gerritbot wrote:

Change 768354 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run Selenium tests for each project in parallel

https://gerrit.wikimedia.org/r/768354

This is not a million miles off. Some observations:

The Apache config needs tuning, as the "Server unavailable" error shows up a couple of times:

Some tests fail on first attempt due to interference of other tests (or due to the 503s noted above), but then they succeed due to the automatic retries in wdio. Not ideal, but it mostly seems to work out.
Some tests need to be reworked slightly to be more robust. E.g. one of the Echo tests assumes the account isn't logged-in, but it is.
Some tests simulate blocking a user and then other tests which use the same user fail, because the user is blocked (https://integration.wikimedia.org/ci/job/integration-quibble-fullrun-extensions/75/artifact/log/Lexeme%253AForms-can-edit-statements-on-a-new-Form-2022-03-06T21-57-01-255Z.mp4)

@zeljkofilipin I was wondering if we could consider using a single wdio.conf.js in mediawiki core that each extension references. That way, we'd get WDIO's parallelization for free, as it already supports concurrent test running, if it knows about all the tests that need to be run.

Core's wdio.conf.js file would need to be adjusted to look in extensions/skins:

	specs: [
		'./tests/selenium/specs/**/*.js',
                './extensions/**/tests/selenium/specs/**/*.js'
	],

and each extension/skin would have to migrate its wdio.conf.js contents into before/beforeEach methods in specFiles, rather than in wdio.conf.js

Having a single configuration file for MW core + extensions/skins seems more akin to how we do things for PHPUnit.

thiemowmde moved this task from Incoming to Watching or blocked on the WMDE-TechWish-Maintenance board.May 5 2022, 11:13 AM

zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.May 6 2022, 12:34 PM

In T226869#7899039, @kostajh wrote:
@zeljkofilipin I was wondering if we could consider using a single wdio.conf.js in mediawiki core that each extension references. That way, we'd get WDIO's parallelization for free, as it already supports concurrent test running, if it knows about all the tests that need to be run.

Core's wdio.conf.js file would need to be adjusted to look in extensions/skins:
	specs: [
		'./tests/selenium/specs/**/*.js',
                './extensions/**/tests/selenium/specs/**/*.js'
	],
and each extension/skin would have to migrate its wdio.conf.js contents into before/beforeEach methods in specFiles, rather than in wdio.conf.js

Having a single configuration file for MW core + extensions/skins seems more akin to how we do things for PHPUnit.

Note that if we do this, some Quibble changes would be necessary (cc @hashar) because Quibble currently assumes a wdio specific config for each extension/skin that contains a list of tests to run for just that extension/skin. And that's how Quibble knows, for example, to first run tests for the extension/skin under test. Although, if we are running everything together in parallel, I guess it's not so important anymore to run the extension/skin browser tests ahead of the other ones.

Although PHPUnit integration tests and QUnit are centrally set in mediawiki/core, webdriver.io tests are split in each repositories. That is the model also used for to the linters (which are run via composer test and npm test) and let us use different versions of webdriver.io. It is too challenging if not impossible to force migrate all repositories at the same time.

From the list of attached patches there are a few repositories for which tests are broken when run concurrently (see patches associated to this task: bug:T226869 is:open. I imagine if we ran tests from different extensions in parallel we would encounter even more issues.

Lens0021 unsubscribed.May 12 2022, 4:03 PM

zeljkofilipin moved this task from Deep work 🌊 to Q2 👔 on the User-zeljkofilipin board.May 13 2022, 12:40 PM

In T226869#7899039, @kostajh wrote:

@zeljkofilipin I was wondering if we could consider using a single wdio.conf.js in mediawiki core that each extension references.

Sorry, looks like I didn't reply to your question. 🤦‍♂️

As far as I remember, that's how we started with the webdriver tests. All configuration and dependencies were in core. That simplified some things (like having to update configuration in just one place, core) but vastly complicated some other things (like making a breaking change in core causing a lot of tests in a lot of repositories to fail). I forgot all the details, but I think the consensus was that having to do more work (like updating configuration/dependencies in each repository) was much better long-term than having everything centralized in core.

@hashar and @Krinkle might remember more details.

zeljkofilipin moved this task from Q2 👔 to Watching 📺 on the User-zeljkofilipin board.Jun 24 2022, 12:49 PM

Change 751957 abandoned by Kosta Harlan:

[mediawiki/extensions/GrowthExperiments@master] selenium: Run test suites concurrently

Reason:

This is not going to be easy to implement, and not worthwhile at the moment given the relatively limited number of tests.

https://gerrit.wikimedia.org/r/751957

Michael mentioned this in T331732: [ES-M1] Investigate browser testing frameworks for Entity Schema Extension.Mar 17 2023, 1:08 PM

Change 934329 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/VisualEditor@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934329

Change 934331 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/Echo@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934331

Change 934333 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934333

Change 934333 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934333

Change 934331 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934331

Change 934329 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934329

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.16; 2023-07-04).Jun 30 2023, 1:01 PM

Change 935162 had a related patch set uploaded (by Dreamy Jazz; author: WMDE-Fisch):

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/935162

WMDE-Fisch closed subtask T337862: Try to run browser tests of WMDE-TechWish projects in parallel as Resolved.Jul 18 2023, 8:10 AM

Daimona mentioned this in T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min.Jul 18 2023, 1:00 PM

When crafting a small change to Selenium test https://gerrit.wikimedia.org/r/c/mediawiki/core/+/842391 the specs/recentchanges.js fail. It grabs the list of titles from recent change, pick the first title and fail cause another test created some other page so the expected title is no more the first in the entry. When looking at the captured output, I see the browser window creating pages with the title prefix BeforeEach-name-, however that does not come from spec/recentchanges.js but from specs/page.js!

In MediaWiki core I have set maxInstances: 4, what I suspect is that all four browser share the same X display and the video capture for a given test ends up having frame from another test running concurrently! I have no idea how to fix that though :)

Edit: I have filed the above as T344754: Browser tests video capture is shared between tests

hashar mentioned this in T344754: Browser tests video capture is shared between tests.Aug 22 2023, 6:05 PM

hashar added a subtask: T344754: Browser tests video capture is shared between tests.

hashar added a subtask: T345149: Special:RecentChanges Selenium test has a race condition when running concurrently.Aug 29 2023, 12:49 PM

Change 935162 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/935162

Dreamy_Jazz closed subtask T340823: Fix CheckUser selenium tests so they can run concurrently as Resolved.Aug 31 2023, 2:31 PM

Change 747904 abandoned by Hashar:

[integration/quibble@master] BrowserTests: Support splitting projects into two groups

Reason:

https://gerrit.wikimedia.org/r/747904