The current s5 master, db1070 is old and needs to be decommissioned.
Further, it has a disk on predictive failure and it is in row D, which needs to get some masters away from it, to avoid a high concentration of masters on that row.
We are going to switchover db1070 to db1100 on Tuesday Tuesday 15th October from 05:00 to 05:30
Read only is required, and the following wikis will not allow writes during this maintenance window:
cebwiki dewiki enwikivoyage mgwiktionary shwiki srwiki
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| wmnet: Update s5-master alias | operations/dns | master | +1 -1 | |
| mariadb: Promote db1100 to s5 master | operations/puppet | production | +4 -4 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | RLazarus | T243314 FY2020-2021 Q1 DC switchover and switchback | |||
| Resolved | RLazarus | T243316 FY2020-2021 Q1 eqiad -> codfw switchover | |||
| Resolved | Marostegui | T186188 Failover DB masters in row D | |||
| Resolved | Marostegui | T217396 Decommission db1061-db1073 | |||
| Resolved | Marostegui | T234300 Switchover s5 primary database master db1070 -> db1100 - 15th Oct 05:00 - 05:30 UTC | |||
| Resolved | Trizek-WMF | T234303 Community Relations support needed a several read-only window for s5 |
Window blocked on the deployments calendar: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1839624&oldid=1839611
Change 540762 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1100 to s5 master
Change 540763 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s5-master alias
Mentioned in SAL (#wikimedia-operations) [2019-10-14T09:48:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1130 into s5 api, db1100 will be removed later in preparation for tomorrow's failover T234300', diff saved to https://phabricator.wikimedia.org/P9325 and previous config saved to /var/cache/conftool/dbconfig/20191014-094809-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2019-10-14T10:07:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1100 with weight 0 in preparation for tomorrow's failover T234300', diff saved to https://phabricator.wikimedia.org/P9326 and previous config saved to /var/cache/conftool/dbconfig/20191014-100758-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2019-10-15T04:15:46Z] <marostegui> Start pre-switchover steps T234300
Change 540762 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1100 to s5 master
Mentioned in SAL (#wikimedia-operations) [2019-10-15T05:00:08Z] <marostegui> Starting s5 failover from db1070 to db1100 - T234300
Mentioned in SAL (#wikimedia-operations) [2019-10-15T05:00:16Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Set s5 as read-only for maintenance T234300', diff saved to https://phabricator.wikimedia.org/P9336 and previous config saved to /var/cache/conftool/dbconfig/20191015-050016-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2019-10-15T05:00:43Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Promote db1100 to s5 master and remove read-only from s5 T234300', diff saved to https://phabricator.wikimedia.org/P9337 and previous config saved to /var/cache/conftool/dbconfig/20191015-050042-marostegui.json
Change 540763 merged by Marostegui:
[operations/dns@master] wmnet: Update s5-master alias
This was done
read only start: 05:00:17
read only stop: 05:00:43
Total read only time: 26 seconds