Intermittent Inbound/Outbound Call Disruption
Incident Report for SpecTronics UK
Postmortem

At 10:13 on Thursday 26/07/2018 our monitoring reported a sharp spike in the number of calls being dialled on our platform – the number of call attempts per second – with the rate doubling inside of 20 seconds.

This significantly increased the load on servers performing vital call setup tasks, resulting in these servers becoming too busy to accept new outbound call connections. At 10:20 widespread call failures were experienced.

Investigation began immediately and identified a set of large customers responsible for the increase in traffic. They were contacted immediately, and the traffic stopped, however by this point the load on the servers was already at a critical level.

Calls that failed were re-attempting to connect to the servers, thus maintaining a relatively high number of call attempts per second. Between 10:25 and 10:40 the load gradually decreased, allowing new calls to connect intermittently. At 11:00 the load had decreased sufficiently for the majority of calls to succeed.

The sheer volume of traffic seen in this incident flooded the system faster than our protection measures could respond and we have identified further bottlenecks in database capacity that meant that this incident caused a wide spread impact.

We are working internally to deploy our VoIP platform into multiple data-centres and are in customer testing for a new VoIP environment.

Posted 7 months ago. Jul 27, 2018 - 17:15 BST

Resolved
We have been monitoring our systems closely, and are confident that service has returned to normal.

Once again please accept our apologies for any inconvenience caused.
Posted 7 months ago. Jul 26, 2018 - 12:34 BST
Monitoring
Our monitoring shows that service is fully restored, and the system recovering. We will continue to monitor the situation closely.

We apologise for any inconvenience this may have caused. If you continue to experience any interruption to calls, please contact our support team on support@spectronics.co.uk or 03333 444 114.
Posted 7 months ago. Jul 26, 2018 - 11:12 BST
Identified
We have identified the cause of this incident, and our monitoring shows call services returning. Customers may experience a delay in calls connecting to our services for a short period of time.

Please accept our apologies for any inconvenience caused. This status will be updated with further information by 11:00.
Posted 7 months ago. Jul 26, 2018 - 10:46 BST
Investigating
We are currently investigating reports of intermittent inbound and outbound call disruption. This appears to be a fault on an upstream suppliers network, and we are working with our suppliers to resolve this as soon as possible.

This status will be updated as soon as any further information is available.

Please accept our apologies for the inconvenience caused.
Posted 7 months ago. Jul 26, 2018 - 10:35 BST
This incident affected: SpecTronics Support Team Systems (Support Team - Phone), Voice Services - Trunks (SIP Trunking, IAX Trunking), and Voice Services - Hosted VoIP (Hosted VoIP - Single User, Hosted VoIP - Multi User).