Showing content from https://phabricator.wikimedia.org/T282787 below:
⚓ T282787 Configure dns and puppet repositories for new drmrs datacenter
Event Timeline Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host lvs6001.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host lvs6001.drmrs.wmnet with OS buster completed:
- lvs6001 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111181017_mmandere_20302_lvs6001.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> staged
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host lvs6002.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host lvs6003.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host lvs6002.drmrs.wmnet with OS buster completed:
- lvs6002 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111181105_mmandere_29884_lvs6002.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> staged
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host lvs6003.drmrs.wmnet with OS buster completed:
- lvs6003 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111181127_mmandere_31664_lvs6003.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> staged
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6009.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6009.drmrs.wmnet with OS buster completed:
- cp6009 (WARN)
- Downtimed on Icinga
- Set pooled=inactive for the following services on confctl:
{"cp6009.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6009.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6009.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203151810_sukhe_1359834_cp6009.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6010.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6010.drmrs.wmnet with OS buster completed:
- cp6010 (WARN)
- Downtimed on Icinga
- Set pooled=inactive for the following services on confctl:
{"cp6010.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6010.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6010.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203152026_sukhe_1375326_cp6010.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6011.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6011.drmrs.wmnet with OS buster completed:
- cp6011 (WARN)
- Downtimed on Icinga
- Set pooled=inactive for the following services on confctl:
{"cp6011.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6011.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6011.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203160011_sukhe_1403513_cp6011.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6012.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6012.drmrs.wmnet with OS buster completed:
- cp6012 (WARN)
- Downtimed on Icinga
- Set pooled=inactive for the following services on confctl:
{"cp6012.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6012.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6012.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161108_sukhe_1482638_cp6012.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6013.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6013.drmrs.wmnet with OS buster completed:
- cp6013 (WARN)
- Downtimed on Icinga
- Set pooled=inactive for the following services on confctl:
{"cp6013.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6013.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6013.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161227_sukhe_1494418_cp6013.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6014.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6014.drmrs.wmnet with OS buster completed:
- cp6014 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"cp6014.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6014.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6014.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161357_sukhe_1509133_cp6014.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6015.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6015.drmrs.wmnet with OS buster completed:
- cp6015 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"cp6015.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6015.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6015.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161446_sukhe_1518579_cp6015.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp6016.drmrs.wmnet with OS buster
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp6016.drmrs.wmnet with OS buster completed:
- cp6016 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"cp6016.drmrs.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-be"}
{"cp6016.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=varnish-fe"}
{"cp6016.drmrs.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=drmrs,cluster=cache_text,service=ats-tls"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203161607_sukhe_1531550_cp6016.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-be' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=varnish-fe' set/pooled=yes
sudo confctl select 'dc=drmrs,cluster=cache_text,service=ats-tls' set/pooled=yes
- Updated Netbox data from PuppetDB
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3