At 16:30, SpecTronics' network supplier responded to alerts showing a sharp reduction in internet traffic across their primary internet transit link, which provides routing to the public internet. To resolve the underlying issue, they decided to manually disable this primary transit link and push traffic over other links to allow time for investigation. This was due to a line card equipment failure in their network.
After disabling the primary transit link, internet traffic should have been immediately routed through a secondary transit. This rerouting was successful and was in place less than a minute after the disabling of the primary transit. This led our network supplier to believe that the risk to service had been averted. However, SpecTronics services continued to be down. An urgent priority fault was raised to our network supplier to investigate the disruption, which found that the secondary transit route was not accepting SpecTronics traffic in-spite of previous confirmation that it was provisioned to do so. At 18:17 the primary (London) transit was re-enabled, restoring traffic. We have since re-confirmed that the secondary transit route is definitely viable and will conduct a full out of hours test of this in the near future.
The loss of the network affected our Support and Sales teams ability to access the internet, preventing them from making and receiving calls.
SpecTronics has extensive service disruption procedures and policies, however, we have not previously rehearsed a network failure that affects ourselves and customers simultaneously. This has now been addressed, with new policies being developed and changes to our internal systems underway. This event has highlighted our own reliance on our network, and the risk that this presents.
Moving forward, additional disaster recovery measures are being developed to ensure such an event would not cause the widespread disruption that occurred in this instance.
We appreciate the debilitating effect this type of incident has on our customers and we offer our sincere apologies for the disruption to your service. We are committed to improving our services and preventing such incidents from occurring again in future.