I see the dhcp6 packets from my test VM arriving into neutron:

We got DNS integration half working:

The VM did not get the IPv6 assigned in the interface via dhcpv6 :-(

neutron virtual router has the right IPv6 address:

however, instance creation itself failed:

Mentioned in SAL (#wikimedia-cloud) [2024-09-27T10:04:30Z] <arturo> [codfw1dev] enable IPv6 on the neutron virtual router T375847

new instance creation will allocate an IPv6 by default for a VM:

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/64

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/64

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63

In T374712#10163608, @cmooney wrote:

@arturo /64s for VM usage I guess can be allocated from 2a02:ec80:a100::/56. We still need to decide how to set the routing (and firealling on cloudgw) up, and make the neccecary adjustments on our edge (RIR DB entries, RPKI objects and route policies) to make it usable on the internet. But you can probably get going with seeing how Neutron can assign some of that to VMs and get it to the point of pinging between two VMs etc.

And we 've just seen this on parsoidtest1001 which is bullseye. Old host, scandium is on buster.

after a powercycle puppetserver1001 is responsive again

Mentioned in SAL (#wikimedia-operations) [2024-09-27T03:26:31Z] <vgutierrez> powercycle puppetserver1001 - T375839

In T375795#10181161, @Urbanecm_WMF wrote:

Interesting, good to know. This is fairly inconvenient though, especially since this is a permanent (301) redirect, meaning my browser is now confused even w/o WikimediaDebug. Can we at least use the 302 redirect in this case? Alternatively, getting a "not here" error could work as well.

In T375795#10181157, @bd808 wrote:

I don't know if there is a task for this yet, but it is known. [...]

I don't know if there is a task for this yet, but it is known. The bug here is that we changed WikimediaDebug to support Wikitech once Wikitech is in the k8s cluster as part of T371537: MVP: Privately serve wikitech via mwdebug1001 (https://gerrit.wikimedia.org/r/c/performance/WikimediaDebug/+/1070275), but we have not yet moved Wikitech to the k8s cluster. This should be magically resolved by T292707: ☂ Migrate Wikitech to Kubernetes.

Opened ticket CS1011077 for the above updated google doc draft.

Mentioned in SAL (#wikimedia-operations) [2024-09-26T18:23:51Z] <sukhe> repooling cp2037; downtimed removed for some time, looks good to repool: T375766

after troubleshooting this, we had to reboot E1 managment switch. This issue should be cleared up.

After troubleshooting the cables and seeing multiple issues with other servers. It was recommended to reboot the switch. Logged it and then proceeded to reboot. It looks like this has cleard up the issue. Closing this now.

However, I checked and the globalblocks table in the labswiki db is empty

Mentioned in SAL (#wikimedia-operations) [2024-09-26T17:03:29Z] <sukhe> removing downtime on cp2037 but still keeping it depooled: T375766

firmware updated and event log cleared.

Inbound shipment ticket 00980858 for UPS 1Z20506Y0100053206 (already delivered today and got the shipment notice last night).

This task is Resolved, as the view change has been applied to all wiki replicas hosts (an-redacteddb1001 and clouddb10[13-20]*).

Draft body of support request for magru temp investigation: https://docs.google.com/document/d/1T-XwSS_Rwfb9nfC1aHQW4AjptLjxiviyZfGFdFcowZY/edit?usp=sharing

Cookbook cookbooks.sre.wikireplicas.update-views started by fnegri executed with errors:

• an-redacteddb1001.eqiad.wmnet (FAIL)
  • Ran Puppet agent
  • The maintain-views run failed, see OUTPUT of 'maintain-views ...' above for details