Inbound/Outbound Call Disruption
Incident Report for SpecTronics UK
Postmortem

At 16:30, SpecTronics' network supplier responded to alerts showing a sharp reduction in internet traffic across their primary internet transit link, which provides routing to the public internet. To resolve the underlying issue, they decided to manually disable this primary transit link and push traffic over other links to allow time for investigation. This was due to a line card equipment failure in their network.

After disabling the primary transit link, internet traffic should have been immediately routed through a secondary transit. This rerouting was successful and was in place less than a minute after the disabling of the primary transit. This led our network supplier to believe that the risk to service had been averted. However, SpecTronics services continued to be down. An urgent priority fault was raised to our network supplier to investigate the disruption, which found that the secondary transit route was not accepting SpecTronics traffic in-spite of previous confirmation that it was provisioned to do so. At 18:17 the primary (London) transit was re-enabled, restoring traffic. We have since re-confirmed that the secondary transit route is definitely viable and will conduct a full out of hours test of this in the near future.

The loss of the network affected our Support and Sales teams ability to access the internet, preventing them from making and receiving calls.

SpecTronics has extensive service disruption procedures and policies, however, we have not previously rehearsed a network failure that affects ourselves and customers simultaneously. This has now been addressed, with new policies being developed and changes to our internal systems underway. This event has highlighted our own reliance on our network, and the risk that this presents.

Moving forward, additional disaster recovery measures are being developed to ensure such an event would not cause the widespread disruption that occurred in this instance.

We appreciate the debilitating effect this type of incident has on our customers and we offer our sincere apologies for the disruption to your service. We are committed to improving our services and preventing such incidents from occurring again in future.

Posted 7 months ago. Jul 27, 2018 - 17:10 BST

Resolved
We have had no further reports of disruption to calls/internet and now believe this incident to be fully resolved.

Please accept our apologies for the inconvenience caused.
Posted 7 months ago. Jul 16, 2018 - 12:57 BST
Monitoring
We have identified that at approximately 16:30 there was a problem with the main internet provider (Level3) to our main provider (Wavenet) for our Telehouse servers in London. Our traffic re-routed to their backup provider, however, it seems that this connection does not fully support access to all of SpecTronics’ network and as a result we were only partially available.

In addition, we found that our VoIP platform could not then connect to much of the internet and it became significantly overloaded as it re-tried calls multiple times.

Wavenet have re-enabled their main london provider and this has brought all services fully online.

We can see calls being processed and the systems are recovering their full capacity.

We will post a further update to this incident & a postmortem early next week.
Posted 7 months ago. Jul 13, 2018 - 18:28 BST
Identified
We have restored partial service, but can see multiple problems remaining with phone call handling. We are continuing to work on restoring full service.
Posted 7 months ago. Jul 13, 2018 - 18:03 BST
Update
We are currently investigating a potential router failure in Telehouse, which is causing an interuption to service for customers of our voice and broadband services. We are in the process of removing it from service which we expect will take upto 30 mins. We will provide a further update at 18:00.
Posted 7 months ago. Jul 13, 2018 - 17:17 BST
Investigating
We are currently investigating reports of inbound and outbound call failures.

Please accept our apologies for any inconvenience this may cause.
Posted 7 months ago. Jul 13, 2018 - 16:46 BST
This incident affected: Voice Services - Hosted VoIP (Hosted VoIP - Control Panel, Hosted VoIP - Single User, Hosted VoIP - Multi User), SpecTronics Support Team Systems (Support Team - Phone), Voice Services - Trunks (SIP Trunking, IAX Trunking), and Broadband/Connectivity Services (Broadband - ADSL, Broadband - FTTC).