Page MenuHomePhabricator

WMF-JobQueue project description is out of date
Open, LowPublic

Description

WMF-JobQueue seems to have various amounts of out of date information...

The infrastructure used by Wikimedia Foundation for storage and execution of the MediaWiki job queue.

As of July 2018, the MediaWiki JobQueue infrastructure (at WMF) in a nutshell:

* Jobs are submitted from MediaWiki web servers to Kafka using EventBus.
* Jobs are scheduled using ChangeProp.
* Jobs are executed using rpc/RunSingleJob endpoint in wmf-config, on a dedicated "jobrunner" pool of MediaWiki app servers.


Workboard columns:

* **EventBus infra** - The modern infrastructure with jobs submitted from MediaWiki to Kafka using EventBus, and a standalone ChangeProp service that subscribes to Kafka which then spawns HTTP request to jobrunner servers to execute the jobs. The execution of jobs over HTTP uses the `rpc/RunSingleJob` endpoint.
** Git repository: [mediawiki/services/change-propagation/jobqueue-deploy](https://gerrit.wikimedia.org/g/mediawiki/services/change-propagation/jobqueue-deploy/+/master) ([GitHub mirror](https://github.com/wikimedia/mediawiki-services-change-propagation-jobqueue-deploy)).
** [Code Review](https://gerrit.wikimedia.org/r/#/projects/mediawiki/services/change-propagation/jobqueue-deploy,dashboards/default).
* **Legacy infra** - The older Redis-based infrastructure, including the Redis instances, and their PHP-based `jobchron` and `jobrunner` processes. This infrastructure is being decommissioned per T198220.
** [Code Review: mediawiki/services/jobrunner](https://gerrit.wikimedia.org/r/#/q/project:mediawiki/services/jobrunner).
** [Git repository](https://gerrit.wikimedia.org/g/mediawiki/services/jobrunner) ([GitHub mirror](https://github.com/wikimedia/mediawiki-services-jobrunner)).

For example, on https://github.com/wikimedia/mediawiki-services-change-propagation-jobqueue-deploy

This repository is no longer used, please refer to the Kubernetes documentation at https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments for instructions on how to deploy changeprop in production

Event Timeline

Reedy renamed this task from WMF-JobQueue description is out of date to WMF-JobQueue project description is out of date.Thu, Nov 21, 9:35 PM
Reedy triaged this task as Low priority.
Reedy added a project: MediaWiki-Platform-Team.