[Comtec Announce] [IP Voice Services] Service Alert 30th June 2014 - Update 5

[David Croft]

This is an incident notification regarding IP Voice Services.

Date: Monday, 30th June 2014
Start time: 18:00 BST (UTC+0100)
End time: 18:49 BST (UTC+0100)

Services affected:

IP Voice Services routing to/from PSTN.

Report:

The engineering team was investigating an issue affecting all off-net
call routing. On-net calls remained unaffected.

Root Cause:

The carrier SBC cluster was unable to process calls due to the license expiring.

The engineering team should have received notification from the
carrier SBC network management application from 4 weeks prior to the
license expiry date. Engineering and TAC use this application daily
and in this instance the system did not provide the notification
banner as seen previously when licenses were due to expire.

The carrier SBC network management system should also have provided an
email alert which was not generated.

Symptoms:

Unable to route off-net calls.

Once off-net call routing was restored, subsequent issues of excessive
post dial delay and intermittent delays with media establishing on
answer were experienced.

Resolution:

The vendor issued and installed a perpetual license to provide a
permanent fix to prevent the license expiry and associated
notification failure being an issue in the future.

Timeline:

18:05 - Network Monitoring system alerted the Engineering team that
the carrier SBC cluster was not processing calls.

18:07 - Engineering identified that the SBCs had stopped processing
calls due to the license they were using expiring.

18:08 - The Engineering team contacted the license vendor to request
immediate replacements.

18:46 - Replacement license were provided and installed by the Vendor.

At this point off-net service was restored.

However excessive PDD (Post Dial Delay) and delays with media
establishing were still being observed.

The Engineering team worked in conjunction with the Vendor to resolve
the ongoing problems.

It was identified that the heartbeat between the two devices was not
establishing correctly causing the secondary to periodically try and
come into service.

19:47 - Both of the SBCs were shut down so that they could be
restarted cleanly in turn to resolve the heartbeat problems. At this
point off-net calls were unavailable.

19:51 - One of the SBCs was returned to service and it was confirmed
that call processing to and from off-net destinations was restored and
there were no longer any issues being experienced.

20:00 - The second SBC was restored to service and heartbeat
re-established cleanly with no affect on service.

20:25 - Service was rolled from one SBC to the other to confirm
stability. During this process there was no effect on call processing;
all existing calls were sustained and new calls established normally.

The Engineering Team and the Vendor closely monitored the service
until 22:00 to confirm all service was restored and no residual issues
remained.

We apologise for the inconvenience this has caused.

This is the final update.

Comtec NOC was tracking this issue under ticket [#OEE-382-69090].

Best regards,

David Croft

--
David Croft
Lead Network Engineer

Comtec Enterprises Ltd
Comtec House
46a Albert Road North
Reigate Industrial Estate
Reigate
Surrey RH2 9EL

Tel: 0845 899 1400
Fax: 0845 899 1401
www.comtec.com

For urgent operational issues please always contact noc at comtec.com
or 0845 899 1423 and not any named individual.