This could a great test case for the new apt staging repo (to easily e.g. upgrade the Hadoop test cluster)! If you're interested, we can figure out the details next week.

In T375729#10326750, @brouberol wrote:

@MoritzMuehlenhoff I'm ready to create the airflow-platform-eng-ops LDAP group with the following members:

member: uid=cparle,ou=people,dc=wikimedia,dc=org
member: uid=mfossati,ou=people,dc=wikimedia,dc=org
member: uid=mlitn,ou=people,dc=wikimedia,dc=org
member: uid=fab,ou=people,dc=wikimedia,dc=org
member: uid=htriedman,ou=people,dc=wikimedia,dc=org
member: uid=bpirkle,ou=people,dc=wikimedia,dc=org
member: uid=cicalese,ou=people,dc=wikimedia,dc=org
member: uid=daniel,ou=people,dc=wikimedia,dc=org
member: uid=kevinbazira,ou=people,dc=wikimedia,dc=org
member: uid=gmodena,ou=people,dc=wikimedia,dc=org
member: uid=hokwelum,ou=people,dc=wikimedia,dc=org
member: uid=tchin,ou=people,dc=wikimedia,dc=org
member: uid=xcollazo,ou=people,dc=wikimedia,dc=org
member: uid=sg912,ou=people,dc=wikimedia,dc=org

as per https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/4b9d3601a07d29c4f7f60889ad30edf16e50b861/modules/admin/data/data.yaml#1050

In T350794#10323046, @Arnoldokoth wrote:

The blocker right now is figuring out firewall access to Puppet DB. @MoritzMuehlenhoff Would it be okay to open up the host to be accessible from any pod running on the k8s-aux cluster?

In T375729#10322821, @brouberol wrote:

@MoritzMuehlenhoff I'm ready to create the airflow-search-ops LDAP group with the following members:

member: uid=ebernhardson,ou=people,dc=wikimedia,dc=org
member: uid=dcausse,ou=people,dc=wikimedia,dc=org
member: uid=gehel,ou=people,dc=wikimedia,dc=org
member: uid=bearloga,ou=people,dc=wikimedia,dc=org
member: uid=tjones,ou=people,dc=wikimedia,dc=org
member: uid=pfischer,ou=people,dc=wikimedia,dc=org
member: uid=dr0ptp4kt,ou=people,dc=wikimedia,dc=org

as per https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/4b9d3601a07d29c4f7f60889ad30edf16e50b861/modules/admin/data/data.yaml#722

I had a closer look and I can confirm that the use of openssl is self-contained within the gitlab monorepo package:

Final status update: The VMs with the legacy setup have been removed and the obsolete Puppet code removed.

I created https://gerrit.wikimedia.org/r/c/operations/puppet/+/1090807 and also added the three new groups to
https://wikitech.wikimedia.org/w/index.php?title=SRE/LDAP/Groups&diff=prev&oldid=2243774 (which is our canonical list of NDA-sensitive groups)

BTW, there is also a much simpler option than writing LDIFs, running the following on ldap-maint1001 would have the same effect:

Thanks for the update, there's is no hurry, since we still have the old server(s), which ganeti2042 would eventually replace. I was just curious :-)

Will Supermicro send a replacement CPU for this server?

In T375729#10311884, @Stevemunene wrote:

Hi @MoritzMuehlenhoff ,
Looking to create the airflow-wmde-ops group as part of T378438, I have airflow-wmde-ops.ldif ready to deploy

This was a lingering issue caused by an interface name change caused by the update to bookworm, now resolved.

In T378954#10309891, @BTullis wrote:

Sqoop fails with:

/ws/output/sqoop/sqoop-1.4.6/build.xml:1094: Execute failed: java.io.IOException: Cannot run program "python2.7"

This is not unexpected, because there is no python 2.7 in bookworm at all.

Happened again on ganeti2031 today.

This behaviour hasn't changed compared to the legacy implementation: Every channel only gets created once there is an edit event for a given combination of language and wiki. Hence, #en.wikimedia will usually be instantly available after a restart of ircstream (the software powering irc.wikimedia.org), while less active wikis might take a little longer.

The replacement mainboard probably shipped a newer BIOS revision which now by default enables SGX. The state doesn't really affects us either way, so we can also simply close the task (and anyone who ever runs into it finds a reference).

We don't use or need SGX for virtualisation servers. It's a feature invented by Intel (AMD never adopted it, which is telling by itself) which provides an encrypted storage (in their terminology an "enclave") which is also inaccesible to the OS. In theory this would allow some interesting use cases, but in practice the predominant use case is DRM (4k UHD BluRays need it).

One more update: The upstream author (Faidon) of ircstream fixed the underlying bug in https://github.com/paravoid/ircstream/commit/7ef7acea12020189dd450c2de6a91d8baaa18942

In T365650#10297670, @Jclark-ctr wrote:

@MoritzMuehlenhoff these where not handed over to service owner while. Luca and dceng researched License /provisioning issue. The reprovisioned this was brought up in irc last week while luca was troubleshooting supermicro licenses and

The irc.wikimedia.org recently switched to ircstream, which has a different architecture, marking this task as declined.

All done!

I had been running into issues with moving VMs to ganeti1041 this morning (which is already added to the Ganeti cluster) and after debugging various OS-level aspects I finally realised that ganeti1041 also lost /dev/kvm? Was it also re-re-reprovisioned? It's not mentioned on this task at all.

In T365650#10287518, @elukey wrote:

Fixed 1044. For some reason IPv6 support was disabled, so our settings like IPv6AutoConfigEnabled: False led to a HTTP 400. I connected to the Web UI, turned IPv6 on and re-run provision, all good. I've also set the host's status to Active in Netbox.

@VRiley-WMF @Jclark-ctr I think that this batch of Supermicro nodes should be ok now, please recheck everything and lemme know if anything is missing :)

Although Moritz was saying that the Intel Linux driver development cycle leaves a lot to be desired, with frequent updates and breaks in backwards compatibility.

In T378358#10289446, @Jhancock.wm wrote:

removed CPU 2. gonna let it run for a little and see if it generates errors. then we'll at least know which one is the problem

This is just some log spam from an ongoing Ganeti installation.

And then set up some daily auto-deploy or similar

The data set is really small, I'd suggest to simply pull in the data with rsync during the container build/deploy.

The generation of the reports already lives outside of the miscweb hosts; it runs on puppetdb2003 and needs to continue to run there as it needs direct access to the puppet database. profile::microsites::os_reports basically just rsyncs these files from puppetdb2003 to the local vhost. If you want to move it to k8s, you can simply adapt the sync so that it deploys to Kubernetes instead.

In T378809#10284244, @cmooney wrote:

In T378809#10284231, @CDanis wrote:

I'm pretty confident this is the same as T348730, and I think it would be okay to return ganeti1025 to service and close this task as a dup

Ok yes from our discussion on irc that seems ok. In terms of service the node is part of the cluster, just the primary instances that were on it are moved. So I'm not sure we need to do anything in particular to bring it back in to service. I'll mention to Moritz in case he wants to do a manual rebalance.

@tappof: thanks for opening a task, but we usually deal with these via Phab tasks. These two both relate to hosts being setup, so some churn is to be expected. To reduce confusion I've just merged a patch to send these mails only to the SRE IF team alias: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1087133

I had a look at the IPMI logs and there are still two more of these errors logged after you reseated the memory on Friday, so it seems this wasn't the memory:

For transparency: The ssotest03 user is used by myself for tests and has been temporarily added to cn=logstash-access.

irc.wikimedia.org is powered by ircstream 1.0 with no known bugs, marking this as resolved. The old VMs will be removed in two weeks.