From david at comtec.net.uk Tue Feb 5 12:37:56 2019 From: david at comtec.net.uk (David Croft) Date: Tue, 5 Feb 2019 13:37:56 +0100 Subject: [Comtec Announce] [Core Network] Service Alert 5th February 2019 Message-ID: This is an incident notification regarding an outage on our core network. Date: Tuesday, 5th February 2019 Start time: 11:00 GMT End time: 11:13 GMT Services affected: Internet connectivity (external) Report: During this window packet loss and high latency were experienced to multiple destinations by ISPs worldwide. The root cause was multiple device failures at Tier 1 network service provider CenturyLink. The impact of this outage was not limited to our network and ISPs worldwide were affected during reconvergence. Connectivity to IP Voice services and directly peered networks was not impacted. External notes appear below. Regards, David Croft 02/05/2019 11:49:06 GMT - The IP NOC has engaged an equipment vendor, Tier III Technical Support, and Operations Engineering to investigate multiple devices that went down. Stability is being validated after most of the devices recovered without any intervention. 02/05/2019 11:19:42 GMT - On February 05, 2018 at 11:00 GMT, CenturyLink identified a service impact in between in all markets network wide. The IP NOC is engaged and currently investigating the suspected equipment issue. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. -- David Croft Lead Engineer Comtec Enterprises Ltd Comtec House 46a Albert Road North Reigate Industrial Estate Reigate Surrey RH2 9EL Tel: 0845 899 1400 Fax: 0845 899 1401 www.comtec.com For urgent operational issues please always contact noc at comtec.com or 0845 899 1423 and not any named individual.