Page MenuHomePhabricator
Create Task
Maniphest T282272

(Need By: TBD) rack/setup/install copernicium
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of copernicium

Hostname / Racking / Installation Details

Please note Rob filled this section out with assumptions based on the host it is replacing, see comment T279170#7069678. Moritz may update this before the host arrives as needed.

Hostnames: copernicium
Racking Proposal: 10G rack, no other restrictions (it is replacing sodium not running in redundancy with it.)
Networking/Subnet/VLAN/IP: single 10G production network connection to public vlan (mirror1001.wikimedia.org)
Partitioning/Raid: hw raid10 setup of all 4 disks, then hwraid-1dev.cfg partman recipe
OS Distro: Bullseye (please quickly ping Moritz before installing)

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

copernicium:

  • - receive in system on procurement task T279170 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, 1g and 10g network, raid controller)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/705008
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
copernicium should be bullseyeoperations/puppetproduction+1 -0
fixing copernicium entriesoperations/puppetproduction+2 -2
copernicium imaging detailsoperations/puppetproduction+11 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedMoritzMuehlenhoffT282272 (Need By: TBD) rack/setup/install copernicium

Event Timeline

RobH created this task.May 7 2021, 7:17 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
Maintenance_bot added a project: SRE.May 7 2021, 7:45 PM
MoritzMuehlenhoff renamed this task from (Need By: TBD) rack/setup/install mirror1001 to (Need By: TBD) rack/setup/install copernicium.May 11 2021, 7:22 AM
MoritzMuehlenhoff updated the task description. (Show Details)
RobH unsubscribed.May 11 2021, 6:34 PM
Jclark-ctr updated the task description. (Show Details)Jun 2 2021, 7:32 PM
Jclark-ctr added a comment.EditedJun 2 2021, 7:34 PM

copernicium B4 U42 Port23 Cableid#5368

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Jun 2 2021, 7:36 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr subscribed.
• Cmjohnson reassigned this task from • Cmjohnson to Jclark-ctr.Jun 10 2021, 4:59 PM
• Cmjohnson subscribed.

@Jclark-ctr I went to assign this to B4 port 23 but netbox has cloudcephosd1017 in that port. Could you please verify the correct port. Thanks

• Cmjohnson claimed this task.Jun 28 2021, 3:39 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson reassigned this task from • Cmjohnson to RobH.Jun 28 2021, 3:45 PM
• Cmjohnson added a subscriber: RobH.

the idrac has been updated and netbox. This should also fix the port spamming Arzhel. @RobH can you do the install?

gerritbot added a comment.Jul 16 2021, 6:24 PM

Change 705008 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] copernicium imaging details

https://gerrit.wikimedia.org/r/705008

gerritbot added a project: Patch-For-Review.Jul 16 2021, 6:24 PM
gerritbot added a comment.Jul 16 2021, 6:26 PM

Change 705008 merged by RobH:

[operations/puppet@production] copernicium imaging details

https://gerrit.wikimedia.org/r/705008

RobH removed a project: Patch-For-Review.Jul 16 2021, 6:29 PM
RobH updated the task description. (Show Details)
RobH removed subscribers: • Cmjohnson, Jclark-ctr.
gerritbot added a comment.Jul 16 2021, 6:41 PM

Change 705009 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] fixing copernicium entries

https://gerrit.wikimedia.org/r/705009

gerritbot added a project: Patch-For-Review.Jul 16 2021, 6:42 PM
gerritbot added a comment.Jul 16 2021, 6:47 PM

Change 705009 merged by RobH:

[operations/puppet@production] fixing copernicium entries

https://gerrit.wikimedia.org/r/705009

ops-monitoring-bot added a comment.Jul 16 2021, 6:53 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

copernicium.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202107161853_robh_10440_copernicium_wikimedia_org.log.

Maintenance_bot removed a project: Patch-For-Review.Jul 16 2021, 7:10 PM
ops-monitoring-bot added a comment.Jul 16 2021, 7:19 PM

Completed auto-reimage of hosts:

['copernicium.wikimedia.org']

and were ALL successful.

gerritbot added a comment.Jul 16 2021, 7:22 PM

Change 705013 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] copernicium should be bullseye

https://gerrit.wikimedia.org/r/705013

gerritbot added a project: Patch-For-Review.Jul 16 2021, 7:22 PM

Change 705013 merged by RobH:

[operations/puppet@production] copernicium should be bullseye

https://gerrit.wikimedia.org/r/705013

ops-monitoring-bot added a comment.Jul 16 2021, 7:36 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

copernicium.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202107161935_robh_18460_copernicium_wikimedia_org.log.

Maintenance_bot removed a project: Patch-For-Review.Jul 16 2021, 8:11 PM
RobH reassigned this task from RobH to MoritzMuehlenhoff.Jul 16 2021, 8:12 PM

So this has an initial puppet run failure for megacli and bullseye, which I chatted with Mortiz about and he is aware. This task is reassigned to him for fix next week. I put the host into maint mode in icinga until then as well, as it will need reimage after the megacli fix (or puppet run and associated post image reimage script steps.)

ops-monitoring-bot added a comment.Jul 16 2021, 9:55 PM

Completed auto-reimage of hosts:

['copernicium.wikimedia.org']

Of which those FAILED:

['copernicium.wikimedia.org']
Stashbot mentioned this in T275873: Prepare our base system layer for Debian 11/bullseye.Jul 19 2021, 9:52 AM

Mentioned in SAL (#wikimedia-operations) [2021-07-19T09:52:16Z] <moritzm> imported megacli for bullseye-wikimedia T282272 T275873

RobH unsubscribed.Jul 19 2021, 4:15 PM
MoritzMuehlenhoff closed this task as Resolved.Jul 20 2021, 10:55 AM

Puppet has been fixed and the host rebooted, closing the racking task. Further setup of the mirror will happen via T286898

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits