Rack A2's hosts alarm for PSU broken
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	Jan 3 2019, 3:14 PM

Description

Hi!

Today is PSU failure day, after kafka1013 I got the following broken PSUs:

an-worker1078

-------------------------------------------------------------------------------
Record:      2
Date/Time:   01/03/2019 14:40:12
Source:      system
Severity:    Critical
Description: The power input for power supply 1 is lost.
-------------------------------------------------------------------------------
Record:      3
Date/Time:   01/03/2019 14:40:13
Source:      system
Severity:    Critical
Description: Power supply redundancy is lost.
-------------------------------------------------------------------------------

an-worker1079

-------------------------------------------------------------------------------
Record:      2
Date/Time:   01/03/2019 14:39:02
Source:      system
Severity:    Critical
Description: Power supply redundancy is lost.
-------------------------------------------------------------------------------
Record:      3
Date/Time:   01/03/2019 14:39:02
Source:      system
Severity:    Critical
Description: The power input for power supply 2 is lost.
-------------------------------------------------------------------------------

Other hosts in A2 are reporting the same issue, so it is likely a rack level problem :)

Affected hosts:

Related Objects
Search...

Status	Assigned	Task
Resolved	• Cmjohnson	T212861 Rack A2's hosts alarm for PSU broken
Resolved	• jcrespo	T213108 db1082 power loss resulted on mysql crash
Resolved	• Cmjohnson	T213748 swap a2-eqiad PDU with on-site spare
Resolved	Marostegui	T213858 s3 master emergency failover (db1075)
Resolved	Johan	T213864 Emergency database primary master failover on s3 primary master

Event Timeline

elukey triaged this task as High priority.Jan 3 2019, 3:14 PM

elukey created this task.

Judging by icinga there's a few other hosts with PS alerts, all in A2. I suspect it has to do with one of the rack PDU themselves

cloudelastic1001
db1082
db1107
ms-be1044
ms-be1045

elukey renamed this task from PSU broken on two Analytics Hadoop Workers to Rack A2's hosts alarm for PSU broken.Jan 3 2019, 3:20 PM

elukey updated the task description. (Show Details)

elukey merged a task: T212844: kakfa1013 shows a failed PSU.

elukey added subscribers: • Cmjohnson, MoritzMuehlenhoff.

• Banyek merged a task: T212909: db1082 has CRITICAL status in power supply.Jan 4 2019, 3:20 PM

• Banyek merged a task: T212910: db1107 has CRITICAL status in power supply.

• Banyek added subscribers: • Banyek, Marostegui, DBA, Aklapper.

Marostegui updated the task description. (Show Details)Jan 4 2019, 3:22 PM

I replaced the fuse on the wrong side initially and caused an outage. I then replaced the fuses on the correct phase and the power was not restored, I tried replacing them both a second time and still nothing. I am out of spare fuses, I do have a spare PDU but we are wanting to replace the PDU and there is an order request for a new set.

Do we want to leave these servers with non-redundant power until we can replace the PDU with a new one that should be ordered soon?
Do you want me to replace the PDU with the spare but will ultimately have to change it in the near future?

Milimetric moved this task from Incoming to Radar on the Analytics board.Jan 7 2019, 4:59 PM

In T212861#4859499, @Cmjohnson wrote:

Do we want to leave these servers with non-redundant power until we can replace the PDU with a new one that should be ordered soon?

Do you want me to replace the PDU with the spare but will ultimately have to change it in the near future?

So the PDU order is on T210776, but it had a mistake. Dell listed out the SMART not SWITCHED PDUs and I did not catch it. I've sent back to Dell to get it corrected, and once we have the timeline I think we'll know if we want to go with 1 or 2.

Since the A2 PDU is a 24 port per tower at present, and we're having issues, I think a swap is in order, just the question of new or old spare will be answered later today when we have a timeline for the order on T210776.

I am creating a subtask to fix db1082, which may have to be reimaged because the power loss.

^CC @Marostegui so you know why db1082 + db1124 + labsdb replication (s5) are broken or stopped

fgiunchedi mentioned this in T209618: rack/setup/install ms-be10[44-50].eqiad.wmnet.Jan 8 2019, 2:09 PM

So I asked for an update for the quote on T210776 and nothing yet. Dell acknowledges they received and are working on it.

If we do not have a quote back today, I'd recommend swapping this PDU out for one of the spare 48 port PDUs Chris mentioned he has spare.

• ayounsi subscribed.Jan 8 2019, 5:12 PM

• jcrespo closed subtask T213108: db1082 power loss resulted on mysql crash as Resolved.Jan 9 2019, 4:24 PM

I rebuilt db1082- we are no blocker for any maintenance on those servers, but we would prefer to stop mysql if there is a chance for the server to lose power, while it does not cause any user-visible outage, as it is very time consuming for us to recover a pooled server, and takes very little time to depool it and stop it.

Peachey88 removed a subscriber: DBA.Jan 15 2019, 8:02 PM

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jan 16 2019, 2:52 PM

RobH closed subtask T213748: swap a2-eqiad PDU with on-site spare as Resolved.Jan 17 2019, 8:55 PM

RobH reopened subtask T213748: swap a2-eqiad PDU with on-site spare as Open.Jan 18 2019, 4:05 PM

Marostegui closed subtask T213748: swap a2-eqiad PDU with on-site spare as Resolved.Jan 26 2019, 3:07 AM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:58 PM

The fuse for the PDU on A2 has been replaced all power is restored.

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM