I was getting this just now (for watchlists and any search I did) for a couple of minutes, but it is working again now.

Request from [x] via cp1089 cp1089, Varnish XID 1067058835
Error: 503, Backend fetch failed at Fri, 03 May 2019 05:13:42 GMT

The errors are back - enwiki's main page just gave me "via cp1089 cp1089, Varnish XID 805012932 Error: 503, Backend fetch failed"

possible continuation of https://wikitech.wikimedia.org/wiki/Incident_documentation/20190503-varnish

@Marostegui did a restart on cp1081 at 2019-05-19T05:09 which helped. Most recently I'm getting the issue from cp1087.

I just came to report that this was happening again, eg: "via cp1087 cp1087, Varnish XID 381950846"

The problem is happening in Chinese Wikipedia. Mainly when saving pages, sometimes accessing pages. Some visual features (incl. the interface) sometimes cannot be rendered properly.
Update: From what I observe, the problem I face has relieved at the recent moment.

@jijiki did a restart on cp1087 at 2019-09-19T08:13 which should help for now.

The problem is happening in Wikidata

The problem is happening in Indonesian Wikipedia

@ReaperDawn @GerardM are you still getting 503s?

For posterity:

https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&from=1558225397000&to=1558268597000&var-datasource=eqiad%20prometheus%2Fops&var-cache_type=text&var-server=All&var-layer=backend

https://logstash.wikimedia.org/goto/94066e323078ed2e1a0ea6b24f801361

cp1081 was restarted at 05:09:48 UTC
cp1087 was restarted at 08:17:41 UTC

@jijiki No, it is alright now in id.wikipedia.

@CDanis just got another one trying to save on meta wiki:
... via cp1081 cp1081, Varnish XID 295561815 Error: 503, Backend fetch failed at Sat, 25 May 2019 08:08:27 GMT

Just got it again

via cp1075 cp1075, Varnish XID 1031998560
Error: 503, Backend fetch failed at Mon, 03 Jun 2019 08:11:59 GMT

The traffic team restarted two Varnish backends, the issue should be fixed now. Thanks a lot for the reports, please let us know if anything strange still happens :)

In T222418#5229410, @Ankit-Maity wrote:

Just a question: is this intermittent behaviour expected or is there something actually being fixed everytime we bring it up (no snark, genuine question)

Genuine anwser: So I wasn't involved on this, but to the best of my knowledge, these incidents above are not (very) related to each other, except on the symptoms that readers or editors suffer: an increase of 503 errors wich can be due to many different causes, normally due to a problem on the traffic/cache serving/connectivitiy/upper parts of the infrastructure (e.g. not the application).

Some can be due to maintenance or software/hardware problems, but some others can be due to external triggers. For example (not a real example), if one person, on purpose or by accident, generates an incredible amount of load (e.g. more than all other millions of users at the same time), some other users, on some locations, during some time can get instability. Work to minimize those is always ongoing, but new problems appear all the time that require new solutions and technologies, and sometimes it takes a long time to implement those. Allow me to be vague so to not tip people on how to abuse those weaknesses, but as soon as incidents are solved, analyzed and fixed/protections put in place we are pretty transparent and (if you are interested on the details), they are published at subpages at: https://wikitech.wikimedia.org/wiki/Incident_documentation . We DO understand those are indeed annoying and try to avoid problems at all cost.

Reporting these issues is actually helpful (as it is said above, not much the subsequent "+1"s) to understand the impact and be able to answer quickly, beyond the alerts and monitoring we already have in place. I would suggest, however, that probably a separate ticket each time would be preferred as they are normally issues not related to each other (503 errors). Software issues "XXX error" are sometimes related if they return the same errors, while infrastructure problems all tend to share a few error codes.

Hope that simplified explanation helps.

For extra context, 503 errors can also happen randomly, the current stats say that 99.999512% of requests are successful, and getting to 100% is almost impossible. That means that our of 200 000 requests, one may fail. The issue happens when the percentage is low (e.g. 99.8% availability and lower) and people suffer from that by having repeated errors on reading or editing.

There is a team (Site Reliability Engineering) entirely dedicated to keeping the "nines" high.