Just processed those edit requests.

For the record, I just found another set of templates we want to update on most of the affected files (Cc-by(-sa)-layout should bypass the SDC_statement_exist template) and filed edit requests for them.

@LucasWerkmeister how was the 10 year estimation calculated?

This type of Kafka Consumer lag for that job isn't unheard of. In fact, just recently we had way higher consumer lags for commons specifically.

Indeed, it looks like the refreshLinks_partitioner rule is easily keeping up with the "upstream" rate of new jobs [0] but the "real" refreshLinks rule on partition 3 (commons) has a rather deep backlog.

That’s great, thank you!

I have processed those editrequests, so at least when a page contains multiple license templates, it will only need to be reparsed once.

Currently, the number of links to Template:SDC statement has value (as counted by search – this lags somewhat behind the “real” number, as it depends on a further job, but it should be a decent approximation)

Per IRC discussion, marking as High priority. @AntiCompositeNumber reports that this results in category changes being slow to propagate (Category:Johann Baptist Hops not showing up in Category:Hops (surname) yet).

Monitoring for sustained latency impact on low-traffic jobs is now live.

Since the three job types critical to uploads have now been moved to dedicated consumers (T379035), the primary follow-up here is monitoring (T378609) to detect when these kinds of isolation failures occur so that we can reactively isolate the "antagonist" job.

Thanks for flagging, all. Yes, this looks like another isolation failure on the low-traffic consumer, and appears to have largely self-resolved as of ~ 14:50 UTC on the 10th. I'll follow up on T379462 for this particular instance, and aim to prioritize T379035 when I'm back this week.

Happened to me some months ago.

Possibly the cause of T379476?

Subscribing myself. I'm seeing this for a while, and yes, today some mw tags are taking forever to update.

That said, it does seem like the p99 for AssembleChunkUpload jobs has spiked to ~15 min for the last 2 hours (was fine before that point), so maybe that is just it. Sounds like a dedicated queue as Scott suggests would really help.

@MBH lets open a separate new task to investigate, as the cause could be something different than the job queue thing this task is about. If you want you could email the HAR file to me ( [email protected] ).

Now files are waiting ~10 minutes before publishing and doesn't published due to errors "Unknown server error" and "Incorrect CSRF token". Uploading (first UploadWizard step) was very slow too with the same behavior than in previous case: 3 files in queue and all other files waiting, after several minutes this 3 files uploaded and next 3 files in queue.

@Bawolff The problem described in T378276 is rised again. I have recorded a HAR file, please, give me an e-mail where should I send it. I will not clear any cookies because I don't know how to do it, I'll just seng you a raw file.

Thanks, @Bawolff - Yes, indeed, those both fan into the low-traffic consumer. While we don't really have a prioritization mechanism in this context that I'm aware of, it would probably be fairly straightforward to at least move them out of low-traffic to a dedicated consumer, as @Ladsgroup points out. I've opened T379035 to look into that.

It shouldn't be too hard to give it a dedicated lane with small concurrency

Just as an aside, I believe PublishStashedFile AssembleUploadChunks are considered low traffic job. Unlike normal jobs these are very latency sensitive, as they don't happen in the background, but the UI actually makes users wait well these jobs complete (See also T378276). It would be really great if somehow these jobs can be prioritized in a job queue backlog situation.

Thanks everyone!

Thanks @Ammarpad for the pointer to where this category came from, and thanks @cscott for providing some context. Indeed, this seems likely to be an interaction between the new category and the way flaggedrevs is used on dewiki.

MediaModeration jobs have gone back to normal processing.

I don't think they are related. That same tracking category appeared on every wiki, nothing explains why it would have cause a load spike for dewiki only. Unless perhaps flagged revs does something "funny" with categories that would cause much higher load?

In T378385#10269970, @Scott_French wrote:

Thanks, @kostajh.

FWIW, it looks like Kategorie:Wikipedia:Seite,_die_JsonConfig_verwendet is a brand new category created on the 24th, with a lot of associated pages.

FWIW, it looks like Kategorie:Wikipedia:Seite,_die_JsonConfig_verwendet is a brand new category created on the 24th, with a lot of associated pages.

In T378385#10269818, @Scott_French wrote:

Looking at the breakdown by wiki for flaggedrevs_CacheUpdate jobs among the last 1M entries in the executor log on mwlog1002:

$ tail -1000000 JobExecutor.log | grep flaggedrevs_CacheUpdate | cut -f 5 -d ' ' | sort | uniq -c | sort -nr
 261208 dewiki
   3026 arwiki
    111 ukwiki
 ...

So yeah, seems to effectively all be targeting dewiki. Example:

Looking at the breakdown by wiki for flaggedrevs_CacheUpdate jobs among the last 1M entries in the executor log on mwlog1002:

In T378385#10269442, @Dreamy_Jazz wrote:

In T378385#10269432, @Scott_French wrote:

Given the rate at which the backlog is draining, this should self-resolve in ~ 24h.

We can try to relax the concurrency limit for the low_traffic_jobs rule, but that might not make a significant difference in either clearing the backlog or improving queue times for other jobs handled by that rule.

I'm happy to give it a try, though.

I think it self-resolving in ~24 hrs would be okay. The concern was that this was a more long-term change to the wait times.