[Comtec Announce] [IP Voice Services/Hosted Contact Centre] Service Alert 18th March 2014 - Update 4

[David Croft]

This is an incident notification regarding IP Voice Services and
Hosted Contact Centre.

Date: Tuesday, 18th March 2014
Start time: 11:45 GMT
End time: 14:21 GMT

Services affected:

IP Voice Services
Hosted Contact Centre

Impact:

HIGH.

Report:

During this time there were issues with one-way audio on calls on both
the IPVS and HCC platforms. On-net calls were not affected.

A subset of calls to and from the PSTN via certain routes on the PSTN
Gateway experienced one way speech or no media between 11:45 and
14:12.

The issue with media failing to establish correctly on calls to/from
the PSTN via certain routes on the PSTN Gateway was caused by a change
to the PE routers by the 21CN Plan and Build team who had been asked
to do some preparatory work (to reserve ports for another service i.e.
not IP Exchange) ahead of a maintenance window.

This change was reverted at 14:12 and service was returned to normal.

The PSTN Gateway Team has the root cause under investigation to
understand why this preparatory work was designated as non-service
affecting.

Timeline:

11:45     Test calls failed on Manchester 04 SBC
11:50     1st customer report of OWT received
11:55     3rd line engaged
12:00     Test calls indicated more SBCs affected, vendor engaged
12:01     Initial service alert sent to customers
12:08     Technical bridge call set up
12:38     Service alert update sent to customers
12:55     21C network team join technical bridge
13:35     PE routers identified as causing the issue
13:40     Notification of incident received from the PSTN Gateway Team
14:05     Service alert update sent to customers
14:12     Service restored
14:53     Notification received from the PSTN Gateway Team that issue
was resolved and service had been restored to normal
15:07     Service alert all-clear sent to customers

Regards,

David Croft

--
David Croft
Lead Network Engineer

Comtec Enterprises Ltd
Comtec House
46a Albert Road North
Reigate Industrial Estate
Reigate
Surrey RH2 9EL

Tel: 0845 899 1400
Fax: 0845 899 1401
www.comtec.com

For urgent operational issues please always contact noc at comtec.com
or 0845 899 1423 and not any named individual.