Web Forwarding

• 04:48 PM – Interface between ORD1 and ORD3 reports down in monitoring causing connectivity issues
• 05:00 PM – Network engineers are notified of potential issue
• 05:12 PM – NOC creates master ticket
• 05:25 PM – DC operations engaged
• 05:48 PM – Network engineers working with local DC operations team to begin initial triage
• 05:52 PM – Critical event created as a multi-customer impacting event
• 06:30 PM – Networking equipment evaluated as possible root cause and deemed operational
• 07:00 PM – Local loops are put in place to troubleshoot physical failures along the path
• 07:25 PM – Optics located and replaced on both A and Z device ends
• 08:05 PM – Ticket raised with DC operator to investigate patch panels
• 08:15 PM – Scope of customer impact updated and provided
• 08:32 PM – Began to initiate a work around solution to the faulty hardware
• 08:39 PM – DC operator engaged for assistance on potential hardware issue between patch panels
• 09:40 PM – Network engineers work with DC operator to perform further troubleshooting
• 10:18 PM – Faulty hardware identified and replaced

Reason For Outage

• Faulty hardware between edge router and patch panel removed available path between ORD1 and ORD3

Incident Resolution

• Faulty hardware was replaced