Incident Details

Resolved [18/11/2025 10:22]

At 19:34 on 16/11/2025, a core switching platform at our Telehouse West (THW) PoP experienced an unexpected internal memory exhaustion event. This caused a forwarding component to shut down to protect the system. When this occurred, traffic had to be processed by the control plane rather than hardware, temporarily reducing forwarding capacity. As a result, leased line and broadband circuits terminating at the site saw sustained packet loss and session disconnections. We were able to intervene and resolve this and service gradually returned to normal over a 30 minute window with all circuits regaining sessions at THW and other PoPs.

What caused it?

The device entered a low memory protection state triggered by a sudden spike in memory utilisation. The exact trigger is under investigation, but there is no evidence of malicious activity or configuration error. We are working with the vendor to determine whether this was caused by an internal memory leak, a hardware limitation reached, or an unexpected routing event.

Next Steps

- Increased monitoring of memory utilisation
- Vendor TAC case opened for deeper analysis

Update [16/11/2025 20:17]

Circuits, sessions and traffic are now returning to normal levels following this unexpected outage. Our network team are continuing to work on the issue and will monitor closely. Further details will be shared in due course.

Investigating [16/11/2025 19:38]

We are investigating an issue impacting our Telehouse West POP which may impact some broadband connections and leased lines which utilise this POP. We are investigating and will share updates.