Outage Notification
Tuesday Oct 19, 2010
Our Network Operations Center (NOC) has detected an incident in progress. This incident has been escalated to the engineers and technicians who are responsible for resolving this issue. We appreciate your patience as we diligently work to restore your services and we will provide more information as it becomes available.
Incident Details:
Incident ID: 7092038
Incident Start Date & Time (24h CST/CDT): Tuesday Oct 19, 2010 00:42
Incident End Date & Time (24h CST/CDT): Tuesday Oct 19, 2010 20:00
Status: Resolved
Department: Networking
Incident Details: During a scheduled network maintenance on one of our border routers in Houston (ibr01.hstntx1), a software bug was triggered on a remote border router in Dallas (ibr03.dllstx3) that caused it to crash and repeatedly reboot. At 1:07AM CT, we isolated ibr03.dllstx3 which alleviated most customer issues. At this point in time, we have completed an emergency code upgrade to ibr03.dllstx3 to correct the software bug, and have confirmed functionality is restored to normal.
Impact Statement: Starting at 12:42AM CT, customers would have experienced intermittent periods of packet loss or increased latency via some of our Internet paths due to route convergence caused by the repeated router flaps. Though router flaps were resolved at 1:07AM, there was some longer-lived high utilization on one of the network paths between Houston and Dallas. Customers would have experienced some network delays on that path until 2:00AM CT when additional traffic was shed to alleviate congestion.
LABEL HOSTING
Sunday, October 24, 2010
Powered by WHMCompleteSolution