In T379044#10291795, @Joe wrote:

and something in eval.php tries to log the call. I guess it's something that changed in mwscript that makes a log call and then socket_sendto fails ofc.

Tue, Nov 5, 4:58 PM · SRE, MediaWiki-backport-deployments, Train Deployments, serviceops

Mon, Nov 4

thcipriani moved T378327: Modernize Jenkins Collapsible Console Sections plugin UI from INBOX to Doing 😎 on the Release-Engineering-Team board.

Mon, Nov 4, 11:42 PM · Release-Engineering-Team (Doing 😎), Upstream, Jenkins, Continuous-Integration-Infrastructure

thcipriani moved T378741: Don't redact stracktraces containing "eval()'d" in Phatality from Priority Backlog 📥 to Doing 😎 on the Release-Engineering-Team board.

Mon, Nov 4, 11:41 PM · Release-Engineering-Team (Doing 😎), Phatality

thcipriani lowered the priority of T378374: Audit space usage on the Catalyst k8s/k3s cluster from Medium to Low.

Mon, Nov 4, 5:49 PM · Catalyst (Camp Muir)

thcipriani moved T377463: Backfill BDD tests for the Catalyst API logs streaming usecase from In progress to Waiting for review on the Catalyst (Camp Muir) board.

Mon, Nov 4, 5:46 PM · Patch-For-Review, Catalyst (Camp Muir)

thcipriani changed the status of T374864: Catalyst API can deploy Wikifunctions, a subtask of T368002: [25Q2] Test Wikifunctions Services with 'Catalyst', from Open to Stalled.

Mon, Nov 4, 5:46 PM · Epic, Abstract Wikipedia team (25Q2 (Oct–Dec))

thcipriani changed the status of T374864: Catalyst API can deploy Wikifunctions from Open to Stalled.

Pending scaling of catalyst instance, then should be ready for deployment

Mon, Nov 4, 5:45 PM · Abstract Wikipedia team (25Q2 (Oct–Dec)), Catalyst (Camp Muir)

thcipriani added a subtask for T374864: Catalyst API can deploy Wikifunctions: T378749: scale k3s (vertically).

Mon, Nov 4, 5:43 PM · Abstract Wikipedia team (25Q2 (Oct–Dec)), Catalyst (Camp Muir)

thcipriani added a parent task for T378749: scale k3s (vertically): T374864: Catalyst API can deploy Wikifunctions.

Mon, Nov 4, 5:42 PM · Catalyst (Camp Muir)

thcipriani moved T378378: [Patchdemo] patchdemo checks that the mediawiki is running before giving user link to it from Backlog to Ready on the Catalyst (Camp Muir) board.

Mon, Nov 4, 5:40 PM · Catalyst (Camp Muir)

thcipriani assigned T378378: [Patchdemo] patchdemo checks that the mediawiki is running before giving user link to it to jeena.

Mon, Nov 4, 5:40 PM · Catalyst (Camp Muir)

thcipriani raised the priority of T377462: [Rescoping Needed] Add TTL (time to live) option to patchdemo which defaults to 30days from High to Needs Triage.

In T377462#10271067, @Tacsipacsi wrote:

Wouldn’t T372981: Auto-delete merged patches be a better solution than this date-based one? (And I still think that an opt-out mechanism should exist, we should trust users not to overuse it.)

Mon, Nov 4, 5:31 PM · Catalyst, Patch-For-Review

thcipriani merged T378748: audit memory pressure on k3s node into T379006: Gather logs for zombie process in pod.

Mon, Nov 4, 5:30 PM · Catalyst (Camp Muir)

thcipriani added a comment to T378748: audit memory pressure on k3s node.

Closed as dupe via discussion in Catalyst planning 2024-11-04

Mon, Nov 4, 5:30 PM · Catalyst

thcipriani merged task T378748: audit memory pressure on k3s node into T379006: Gather logs for zombie process in pod.

Mon, Nov 4, 5:29 PM · Catalyst

thcipriani changed the status of T378749: scale k3s (vertically) from Open to Stalled.

Mon, Nov 4, 5:28 PM · Catalyst (Camp Muir)

thcipriani updated the task description for T378749: scale k3s (vertically).

Mon, Nov 4, 5:27 PM · Catalyst (Camp Muir)

thcipriani moved T379006: Gather logs for zombie process in pod from Backlog to Ready on the Catalyst (Camp Muir) board.

Mon, Nov 4, 5:24 PM · Catalyst (Camp Muir)

thcipriani assigned T379006: Gather logs for zombie process in pod to EBomani.

Mon, Nov 4, 5:24 PM · Catalyst (Camp Muir)

thcipriani created T379006: Gather logs for zombie process in pod.

Mon, Nov 4, 5:20 PM · Catalyst (Camp Muir)

Thu, Oct 31

thcipriani reassigned T378204: Client error "TypeError: Cannot convert undefined or null to object" from dduvall to egardner.

Thu, Oct 31, 9:54 PM · MW-1.43-notes (1.43.0-wmf.28; 2024-10-22), MW-1.44-notes (1.44.0-wmf.1; 2024-10-29), Structured-Data-Backlog

thcipriani closed T375660: 1.44.0-wmf.1 deployment blockers as Resolved.

Thu, Oct 31, 9:26 PM · Release-Engineering-Team (Doing 😎), Release, Train Deployments

thcipriani moved T378736: RuntimeException: CAS update failed on user_touched. The version of the user to be saved is older than the current version. via InvalidateEmail from Untriaged to Oct 2024 on the Wikimedia-production-error board.

Thu, Oct 31, 3:58 PM · MW-1.44-notes (1.44.0-wmf.3; 2024-11-12), MediaWiki-User-management, Wikimedia-production-error

thcipriani created T378736: RuntimeException: CAS update failed on user_touched. The version of the user to be saved is older than the current version. via InvalidateEmail.

Thu, Oct 31, 3:58 PM · MW-1.44-notes (1.44.0-wmf.3; 2024-11-12), MediaWiki-User-management, Wikimedia-production-error

thcipriani added a comment to T351953: "RuntimeException: PCRE failure" displaying oldest revisions on eowiki.

In case it's helpful, here's a search (with too many filters) that will give you a list of pages that cause this: https://logstash.wikimedia.org/goto/246f0d53f76e1bba9a19d527448514dd

Thu, Oct 31, 3:51 PM · MediaWiki-Parser, User-brennen, Patch-For-Review, Essential-Work, Content-Transform-Team, Wikimedia-production-error, Wikimedia-database-issue (Bad data)

thcipriani moved T378731: PHP Notice: Trying to access array offset on value of type bool SearchVue from Untriaged to Oct 2024 on the Wikimedia-production-error board.

Thu, Oct 31, 3:40 PM · MW-1.44-notes (1.44.0-wmf.2; 2024-11-05), Unstewarded-production-error, SDAW-SearchVue, Structured-Data-Backlog, Wikimedia-production-error

thcipriani created T378731: PHP Notice: Trying to access array offset on value of type bool SearchVue.

Thu, Oct 31, 3:40 PM · MW-1.44-notes (1.44.0-wmf.2; 2024-11-05), Unstewarded-production-error, SDAW-SearchVue, Structured-Data-Backlog, Wikimedia-production-error

thcipriani moved T378679: Error: Object of class stdClass could not be converted to string from Untriaged to Oct 2024 on the Wikimedia-production-error board.

Thu, Oct 31, 3:32 PM · Essential-Work, Content-Transform-Team-WIP, Parsoid, Wikimedia-production-error

thcipriani added a comment to T378498: TypeError: Wikimedia\Minify\IndexMap::addEncodedMap(): Argument #2 ($generatedSize) must be of type Wikimedia\Minify\IndexMapOffset, null given, .

More info in case that's useful:

Thu, Oct 31, 3:31 PM · Wikimedia-production-error, MediaWiki-Platform-Team, MediaWiki-ResourceLoader

thcipriani moved T378727: PermissionsError: The action you have requested is limited to users in one of the groups: <groups> from Untriaged to Oct 2024 on the Wikimedia-production-error board.

Thu, Oct 31, 3:21 PM · MW-1.44-notes (1.44.0-wmf.2; 2024-11-05), MW-1.43-notes, MW-1.43-release, MediaWiki-Parser, Unstewarded-production-error, MediaWiki-Special-pages, Wikimedia-production-error

thcipriani created T378727: PermissionsError: The action you have requested is limited to users in one of the groups: <groups>.

Thu, Oct 31, 3:21 PM · MW-1.44-notes (1.44.0-wmf.2; 2024-11-05), MW-1.43-notes, MW-1.43-release, MediaWiki-Parser, Unstewarded-production-error, MediaWiki-Special-pages, Wikimedia-production-error

thcipriani closed T377117: Account recovery help needed for Developer account jgiannelos as Resolved.

In T377117#10262450, @Jgiannelos wrote:

Thanks @Aklapper leaving it open until gitlab 2fa is also reset.

Thu, Oct 31, 2:40 PM · GitLab (Auth & Access), Trust-and-Safety

thcipriani added a comment to T378531: PHP Notice: Array to string conversion.

In T378531#10279254, @Jdlrobson wrote:

@dduvall I'm a little concerned that we reverted a change based on an educated guess (T378531#10274398) and just assumed that was correct without further investigation and probing.

Thu, Oct 31, 12:06 AM · MW-1.44-notes (1.44.0-wmf.1; 2024-10-29), MinervaNeue, Wikimedia-production-error

Wed, Oct 30

thcipriani moved T375145: Draft and get approval for next hypothesis to follow WE6.2.1 from Priority Backlog 📥 to Doing 😎 on the Release-Engineering-Team board.

Wed, Oct 30, 4:05 PM · Release-Engineering-Team (Doing 😎), User-bd808

thcipriani moved T375660: 1.44.0-wmf.1 deployment blockers from Priority Backlog 📥 to Doing 😎 on the Release-Engineering-Team board.

Wed, Oct 30, 4:05 PM · Release-Engineering-Team (Doing 😎), Release, Train Deployments

thcipriani edited projects for T375782: scap web interface: Create SpiderPig web UI, added: Release-Engineering-Team (Doing 😎); removed Release-Engineering-Team (Priority Backlog 📥).

Wed, Oct 30, 4:04 PM · Release-Engineering-Team (Doing 😎), Design-System-Team (Design-System-Product-Roadmap), Codex, Scap, Epic

thcipriani edited Description on Release-Engineering-Team (Doing 😎).

Wed, Oct 30, 3:57 PM

thcipriani set the image for Release-Engineering-Team (Doing 😎) to F57659847: profile.

Wed, Oct 30, 3:56 PM

thcipriani created Release-Engineering-Team (Doing 😎).

Wed, Oct 30, 3:53 PM

thcipriani moved T329735: Run logstash canary checks via logs-api.svc from Onboarding 🚀 to Priority Backlog 📥 on the Release-Engineering-Team board.

Wed, Oct 30, 3:52 PM · Release-Engineering-Team (Priority Backlog 📥), Scap, Observability-Logging

thcipriani edited projects for T255197: scap's logstash_checker.py is blissfully unaware of any logstash indexing latency, added: Release-Engineering-Team (Seen); removed Release-Engineering-Team (Onboarding 🚀).

Wed, Oct 30, 3:52 PM · Release-Engineering-Team (Seen), User-brennen, Sustainability (Incident Followup), Scap

thcipriani edited projects for T269242: Change deploy-promote to check previous groups have been promoted beforehand, added: Scap, Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (Onboarding 🚀), MediaWiki-Release-Tools.

Wed, Oct 30, 3:52 PM · Release-Engineering-Team (Priority Backlog 📥), Scap

thcipriani added a project to T329735: Run logstash canary checks via logs-api.svc: Scap.

Wed, Oct 30, 3:50 PM · Release-Engineering-Team (Priority Backlog 📥), Scap, Observability-Logging

thcipriani moved T271107: Have PipelineBot link directly to build console output on test from Onboarding 🚀 to Priority Backlog 📥 on the Release-Engineering-Team board.

Wed, Oct 30, 3:50 PM · Release-Engineering-Team (Priority Backlog 📥), Continuous-Integration-Infrastructure, Release Pipeline

thcipriani closed T290612: GitLab project templates include issues - figure out if these can be customized or removed as Declined.

Meh, we have repos/releng/gitlab-settings:configure-projects which ensures we can keep issues disabled for projects. Let's not worry too much about this.

Wed, Oct 30, 3:50 PM · Release-Engineering-Team (Onboarding 🚀), GitLab (Administration, Settings & Policy), User-brennen

thcipriani moved T336179: scap deploy-promote uses latest train blocker, which is wrong if a train has been blocked over the weekend from Onboarding 🚀 to Priority Backlog 📥 on the Release-Engineering-Team board.

Wed, Oct 30, 3:46 PM · Release-Engineering-Team (Priority Backlog 📥), User-brennen, Scap

thcipriani moved T303828: Delete wmf branches from Gerrit repositories from Onboarding 🚀 to Priority Backlog 📥 on the Release-Engineering-Team board.

Wed, Oct 30, 3:46 PM · Release-Engineering-Team (Priority Backlog 📥)

Tue, Oct 29

thcipriani added a comment to T378082: Requesting access to 'deployment' for 'Joely Rooke WMDE'.

In T378082#10271688, @tappof wrote:

@thcipriani your approval is needed for the deployment group.
Thanks.

Tue, Oct 29, 4:49 PM · SRE, SRE-Access-Requests

thcipriani added a comment to T378429: Allow members of restricted to run maintenance scripts.

I'm listed as approver for restricted and deployment.

Tue, Oct 29, 4:21 PM · MW-on-K8s, serviceops

thcipriani updated the task description for T378428: Backport and config deploy: Speculative future backports.

Tue, Oct 29, 12:03 AM · Release-Engineering-Team, MediaWiki-backport-deployments

thcipriani created T378428: Backport and config deploy: Speculative future backports.

Tue, Oct 29, 12:02 AM · Release-Engineering-Team, MediaWiki-backport-deployments

Mon, Oct 28

thcipriani moved T378372: Add API better logging on error from Backlog to Ready on the Catalyst (Camp Muir) board.

Mon, Oct 28, 5:00 PM · Catalyst (Camp Muir)

thcipriani edited projects for T378372: Add API better logging on error, added: Catalyst (Camp Muir); removed Catalyst.

Mon, Oct 28, 5:00 PM · Catalyst (Camp Muir)

thcipriani assigned T378372: Add API better logging on error to jnuche.

Mon, Oct 28, 5:00 PM · Catalyst (Camp Muir)

thcipriani assigned T377463: Backfill BDD tests for the Catalyst API logs streaming usecase to jeena.

Mon, Oct 28, 4:57 PM · Patch-For-Review, Catalyst (Camp Muir)

thcipriani triaged T378373: Draft communications plan for beta launch as Medium priority.

Mon, Oct 28, 4:56 PM · Catalyst (Camp Muir)

thcipriani assigned T378373: Draft communications plan for beta launch to SDunlap.

Mon, Oct 28, 4:56 PM · Catalyst (Camp Muir)

thcipriani moved T377858: Ensure text is properly escaped/rendered for landing page from Backlog to Ready on the Catalyst (Camp Muir) board.

Mon, Oct 28, 4:50 PM · Patch-For-Review, Catalyst (Camp Muir)

thcipriani assigned T377858: Ensure text is properly escaped/rendered for landing page to jnuche.

Mon, Oct 28, 4:50 PM · Patch-For-Review, Catalyst (Camp Muir)

thcipriani edited projects for T377462: [Rescoping Needed] Add TTL (time to live) option to patchdemo which defaults to 30days, added: Catalyst; removed Catalyst (Camp Muir).

Mon, Oct 28, 4:50 PM · Catalyst, Patch-For-Review

thcipriani assigned T378374: Audit space usage on the Catalyst k8s/k3s cluster to jnuche.

Mon, Oct 28, 4:46 PM · Catalyst (Camp Muir)

thcipriani triaged T378374: Audit space usage on the Catalyst k8s/k3s cluster as Medium priority.

Mon, Oct 28, 4:46 PM · Catalyst (Camp Muir)

thcipriani moved T378374: Audit space usage on the Catalyst k8s/k3s cluster from Backlog to Ready on the Catalyst (Camp Muir) board.

Mon, Oct 28, 4:45 PM · Catalyst (Camp Muir)

thcipriani updated the task description for T378374: Audit space usage on the Catalyst k8s/k3s cluster.

Mon, Oct 28, 4:45 PM · Catalyst (Camp Muir)

thcipriani set the point value for T378374: Audit space usage on the Catalyst k8s/k3s cluster to 2.

Mon, Oct 28, 4:41 PM · Catalyst (Camp Muir)

thcipriani updated the task description for T378374: Audit space usage on the Catalyst k8s/k3s cluster.

Mon, Oct 28, 4:39 PM · Catalyst (Camp Muir)

thcipriani triaged T376272: Repository pool: fetch and gc remote changes periodically as Medium priority.

Mon, Oct 28, 4:31 PM · Catalyst (Camp Muir)

thcipriani moved T376272: Repository pool: fetch and gc remote changes periodically from Backlog to Ready on the Catalyst (Camp Muir) board.

Mon, Oct 28, 4:31 PM · Catalyst (Camp Muir)

thcipriani assigned T376272: Repository pool: fetch and gc remote changes periodically to EBomani.

Mon, Oct 28, 4:31 PM · Catalyst (Camp Muir)

thcipriani set the point value for T376272: Repository pool: fetch and gc remote changes periodically to 2.

Mon, Oct 28, 4:29 PM · Catalyst (Camp Muir)

thcipriani added a subtask for T378380: Improve Catalyst disk usage: T376272: Repository pool: fetch and gc remote changes periodically.

Mon, Oct 28, 4:25 PM · Epic, Catalyst

thcipriani added a parent task for T376272: Repository pool: fetch and gc remote changes periodically: T378380: Improve Catalyst disk usage.

Mon, Oct 28, 4:25 PM · Catalyst (Camp Muir)

thcipriani added a parent task for T378234: [Chore] increase k3s-data to 255GB and decrease patchdemo4-production-wikis to 30GB: T378380: Improve Catalyst disk usage.

Mon, Oct 28, 4:24 PM · Catalyst (Camp Muir)

thcipriani added a subtask for T378380: Improve Catalyst disk usage: T378234: [Chore] increase k3s-data to 255GB and decrease patchdemo4-production-wikis to 30GB.

Mon, Oct 28, 4:24 PM · Epic, Catalyst

thcipriani added a subtask for T378380: Improve Catalyst disk usage: T377462: [Rescoping Needed] Add TTL (time to live) option to patchdemo which defaults to 30days.

Mon, Oct 28, 4:24 PM · Epic, Catalyst

thcipriani added a parent task for T377462: [Rescoping Needed] Add TTL (time to live) option to patchdemo which defaults to 30days: T378380: Improve Catalyst disk usage.

Mon, Oct 28, 4:24 PM · Catalyst, Patch-For-Review

thcipriani added a subtask for T378380: Improve Catalyst disk usage: T378374: Audit space usage on the Catalyst k8s/k3s cluster.

Mon, Oct 28, 4:23 PM · Epic, Catalyst

thcipriani added a parent task for T378374: Audit space usage on the Catalyst k8s/k3s cluster: T378380: Improve Catalyst disk usage.

Mon, Oct 28, 4:23 PM · Catalyst (Camp Muir)

thcipriani created T378380: Improve Catalyst disk usage.

Mon, Oct 28, 4:23 PM · Epic, Catalyst

thcipriani added a comment to T374864: Catalyst API can deploy Wikifunctions.

@jnuche has this working locally now after making a few changes to the chart—there is no patch for the chart changes yet, coming soon. We are currently making a few needed followup changes to the catalyst API.

Mon, Oct 28, 4:10 PM · Abstract Wikipedia team (25Q2 (Oct–Dec)), Catalyst (Camp Muir)

Fri, Oct 25

thcipriani added a comment to T377117: Account recovery help needed for Developer account jgiannelos.

In T377117#10262034, @Jgiannelos wrote:

Hey @Aklapper unfortunately i couldn't reset it with only my old 2fa (it needs both OTPs). Can we arrange a video call to reset my 2fa? (cc'ing my manager @SLopes-WMF )

Fri, Oct 25, 9:33 PM · GitLab (Auth & Access), Trust-and-Safety

thcipriani added a comment to T375659: 1.43.0-wmf.28 deployment blockers.