Page MenuHomePhabricator
Create Task
Maniphest T158562

Manage apt sources via puppet
Closed, ResolvedPublic
Actions

Description

/etc/apt/sources.list is currently set up by d-i, but not managed via puppet. This leaves room to all kinds of inconsistencies, e.g. some hosts are using external mirrors (e.g. multatuli is using ftp.nl.debian.org instead of mirrors.wikimedia.org) and crucial apt sources can even be omitted (e.g. planet2001 currently has the security apt sources uncommented (which made me notice this and file this task)).

apt.wikimedia.org and backports are already added via /etc/apt/sources.list.d/wikimedia.list and /etc/apt/sources.list.d/debian-backports.list (and optionally also the experimental component).

I think that at least on Debian systems we should also integrate mirrors.wikimedia.org, security.debian.org and jessie|stretch-updates via sub files in /etc/apt/sources.list.d and simply stub /etc/apt/sources.list with a comment like "# managed via puppetised sub files in /etc/apt/sources/list.d".

Opinions?

Details

SubjectRepoBranchLines +/-
Remove option to manage sources.listoperations/puppetproduction+62 -69
autoinstall: Also use mirrors.wikimedia.org for public/esamsoperations/puppetproduction+0 -7
Enabled managed sources.list for all of productionoperations/puppetproduction+1 -6
Enable managed sources.list for codfwoperations/puppetproduction+1 -0
Enabled managed sources.list for esams/eqsinoperations/puppetproduction+2 -0
Enabled managed sources.list for ulsfooperations/puppetproduction+1 -0
Have the puppetised sources.list depend on the wikimedia repositoryoperations/puppetproduction+1 -0
d-i: add contrib component to d-i configurationoperations/puppetproduction+9 -0
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Feb 20 2017, 1:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2017, 1:30 PM
MoritzMuehlenhoff mentioned this in T158583: Restructure our internal repositories further.Feb 20 2017, 4:41 PM
fgiunchedi subscribed.Feb 21 2017, 8:34 AM

+1, sounds good to me!

faidon subscribed.Feb 21 2017, 11:39 AM

apt::repository has a comment_old option that comments out the line in sources.list, and Apt::Repository[wikimedia] sets that to true, so this should be the case already. If it's not, it's probably a bug and not intentional.

(for esams, note that using ftp.nl is probably a better idea than mirrors.wikimedia.org due to network proximity -- but it doesn't matter much anyway)

Ottomata assigned this task to MoritzMuehlenhoff.Mar 6 2017, 6:54 PM
Ottomata triaged this task as Low priority.
Ottomata subscribed.

+1 in general, but yeah, I think this should basically already be happening.

Perhaps manually installing our own sources.list via an ERb template would be better than commenting lines?

@MoritzMuehlenhoff, I'm just triaging, feel free to re-assign as needed.

MoritzMuehlenhoff added a comment.May 16 2018, 1:51 PM

Another case I found: ms-be2013-ms-be2021 were unable to install the systemd update that was released via stretch-updates and it turned that that stretch-updates was missing in /etc/apt/sources.list. Those were among the first stretch hosts, so this was probably a one-off issue in early installations only. I've fixed up the apt config on the affected systems.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:04 PM
jbond added a project: User-jbond.Nov 4 2019, 11:18 AM
jbond moved this task from Unsorted 💣 to Friday tasks on the User-jbond board.
gerritbot added a comment.Nov 4 2019, 11:32 AM

Change 548230 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] d-i: add contrib component to d-i configuration

https://gerrit.wikimedia.org/r/548230

gerritbot added a project: Patch-For-Review.Nov 4 2019, 11:32 AM
gerritbot added a comment.Nov 4 2019, 5:40 PM

Change 548230 abandoned by Jbond:
d-i: add contrib component to d-i configuration

Reason:
moritz beat me to it

https://gerrit.wikimedia.org/r/548230

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2019, 6:10 PM
Krenair subscribed.Nov 4 2019, 11:23 PM
Aklapper removed MoritzMuehlenhoff as the assignee of this task.Jun 19 2020, 4:28 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

MoritzMuehlenhoff renamed this task from Manage apt sources via puppet? to Manage apt sources via puppet.Sep 11 2020, 9:08 AM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff raised the priority of this task from Low to Medium.
MoritzMuehlenhoff mentioned this in T262647: Provide failover capacity for package installations from main mirror.
gerritbot added a comment.Sep 11 2020, 6:29 PM

Change 626693 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Manage /etc/apt/sources.list via Puppet (WIP)

https://gerrit.wikimedia.org/r/626693

gerritbot added a project: Patch-For-Review.Sep 11 2020, 6:29 PM
Stashbot added a comment.Sep 25 2020, 10:28 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-25T10:28:26Z] <moritzm> reimaging sretest1002 to validate puppetised sources.list with a new installation T158562

Stashbot added a comment.Sep 25 2020, 11:10 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-25T11:10:15Z] <moritzm> reimaging sretest1001 to validate puppetised sources.list with a new installation T158562

MoritzMuehlenhoff added a comment.Sep 25 2020, 12:44 PM

I did a test installation with the new setting as I had a hunch there would be issues in early install and turns out I was right: The installer writes out an initial /etc/apt/sources.list and the puppetised /etc/apt/sources.list does not contain apt.wikimedia.org (it's handled via /etc/apt/sources.list.d/wikimedia). Then there's some Puppet logic to comment out entries from /etc/apt/sources.list in the apt::repository define once the sub sources list is added and the current processing order installation ferm and librdkafka in the stock Debian versions instead of the customised ones from apt.wikimedia.org

But I think this can be fixed by making the puppetised sources.list depend on apt::repository{'wikimedia'}.

An alternative and more generic way would be to only apply the puppetised version once the installer is complete, the reimage script could write out a file like /var/lib/wmf-installation-complete (which could be useful for other purposes as well) and then the puppetised sources.list would depend on the presence of /var/lib/wmf-installation-complete. But I'm not too fond of that, as it creates special cases and from my PoV Puppet should behave identical independent of the current installation process.

gerritbot added a comment.Sep 25 2020, 2:01 PM

Change 630179 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Have the puppetised sources.list depend on the wikimedia repository

https://gerrit.wikimedia.org/r/630179

gerritbot added a comment.Sep 28 2020, 12:47 PM

Change 630179 merged by Muehlenhoff:
[operations/puppet@production] Have the puppetised sources.list depend on the wikimedia repository

https://gerrit.wikimedia.org/r/630179

Stashbot added a comment.Sep 28 2020, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-28T13:19:15Z] <moritzm> reimaging sretest1001 to validate puppetised sources.list with a new installation T158562

MoritzMuehlenhoff added a comment.Sep 28 2020, 2:24 PM
In T158562#6494038, @MoritzMuehlenhoff wrote:

I did a test installation with the new setting as I had a hunch there would be issues in early install and turns out I was right: The installer writes out an initial /etc/apt/sources.list and the puppetised /etc/apt/sources.list does not contain apt.wikimedia.org (it's handled via /etc/apt/sources.list.d/wikimedia). Then there's some Puppet logic to comment out entries from /etc/apt/sources.list in the apt::repository define once the sub sources list is added and the current processing order installation ferm and librdkafka in the stock Debian versions instead of the customised ones from apt.wikimedia.org

This is fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/630179, confirmed in a reimage of sretest1001.

gerritbot added a comment.Sep 29 2020, 11:04 AM

Change 630785 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enabled managed sources.list for ulsfo

https://gerrit.wikimedia.org/r/630785

gerritbot added a comment.Sep 29 2020, 2:22 PM

Change 630785 merged by Muehlenhoff:
[operations/puppet@production] Enabled managed sources.list for ulsfo

https://gerrit.wikimedia.org/r/630785

gerritbot added a comment.Sep 29 2020, 2:48 PM

Change 630876 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] autoinstall: Also use mirrors.wikimedia.org for publi/esams

https://gerrit.wikimedia.org/r/630876

gerritbot added a comment.Sep 29 2020, 2:56 PM

Change 630879 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enabled managed sources.list for esams/ulsfo

https://gerrit.wikimedia.org/r/630879

gerritbot added a comment.Sep 30 2020, 7:29 AM

Change 630879 merged by Muehlenhoff:
[operations/puppet@production] Enabled managed sources.list for esams/eqsin

https://gerrit.wikimedia.org/r/630879

gerritbot added a comment.Sep 30 2020, 7:53 AM

Change 631139 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed sources.list for codfw

https://gerrit.wikimedia.org/r/631139

gerritbot added a comment.Sep 30 2020, 9:18 AM

Change 631139 merged by Muehlenhoff:
[operations/puppet@production] Enable managed sources.list for codfw

https://gerrit.wikimedia.org/r/631139

gerritbot added a comment.Oct 1 2020, 8:33 AM

Change 631396 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enabled managed sources.list for all of production

https://gerrit.wikimedia.org/r/631396

gerritbot added a comment.Oct 1 2020, 10:15 AM

Change 631396 merged by Muehlenhoff:
[operations/puppet@production] Enabled managed sources.list for all of production

https://gerrit.wikimedia.org/r/631396

gerritbot added a comment.Oct 23 2020, 6:48 AM

Change 630876 merged by Muehlenhoff:
[operations/puppet@production] autoinstall: Also use mirrors.wikimedia.org for public/esams

https://gerrit.wikimedia.org/r/630876

MoritzMuehlenhoff closed this task as Resolved.Oct 23 2020, 7:27 AM

/etc/apt/sources.list is managed by Puppet since a few weeks in production, closing the task (for Cloud VPS it's being considered to also enabled it in a separate task).

gerritbot added a comment.Jun 5 2023, 9:50 AM

Change 927130 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove option to manage sources.list

https://gerrit.wikimedia.org/r/927130

gerritbot added a comment.Jun 6 2023, 8:00 AM

Change 927130 merged by Muehlenhoff:

[operations/puppet@production] Remove option to manage sources.list

https://gerrit.wikimedia.org/r/927130

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits