[Comtec Announce] RFO - IP Voice Outage 6th August 2010

[David Croft]

Please find below the analysis regarding Friday's IP Voice outage for
your information:

-----

Intermittent Session Border Controller Issue on 06/08/2010

Date of incident: Friday 6th August 2010
Time of incident: Intermittent between approximately 17:45 and 18:45
Time of resolution: Approximately 18:45

Synopsis

An extension side Session Border Controller climbed to an unusually
high CPU usage at approximately 17:45 on Friday 6th August 2010. This
caused end users that were communicating with this SBC to experience
an intermittent loss of registration with the HIPCOM platform and
consequently a temporary loss of new and established calls.
Engineering were alerted to the issue as the CPU usage was spiking.
Processes on the affected SBC cluster were restarted in order to
correct the problem and logs were compiled to further investigate the
root cause. The problem was fully resolved at approximately 18:45 and
has been monitored since. End users that were communicating with other
extension side SBC clusters were not affected.

Current Status

The affected SBCs continue to be monitored, the symptoms have not
occurred again and specific alerts/monitoring have been implemented
for the symptoms that resulted in the high CPU usage. Logs taken
during and after the problem continue to be analysed by Engineering
and our Vendor Partners in order to fully resolve the root cause of
the issue to ensure that the problem does not occur again. A further
update will be released as soon as possible in order to communicate
further actions as a result of the investigations. This is the first
instance in the current year where any outage has occurred and the
first at any point involving the SBC cluster that caused the issue.


Please accept our apologies for any inconvenience caused.


Regards,

BT IPVS Support


Best regards,

David Croft

--
David Croft
Lead Network Engineer

Comtec Enterprises Ltd
Comtec House
46a Albert Road North
Reigate Industrial Estate
Reigate
Surrey RH2 9EL

Tel: 0845 899 1400
Fax: 0845 899 1401
www.comtec.com

For urgent operational issues please always contact noc at comtec.com or
0845 899 1423 and not any named individual.