Spitfire Service Status
Spitfire
Your session has timed out please click the link originally provided to view this page
Job Details
Fault Description
Number
CLOSED - Reports: SIP Services - Loss of Service - P1 - more info >>> Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. Further information will appear at http://status.spitfire.co.uk. P1 = high priority
Created
Status
Cleared
View
Worksheet Ref
Status
View
MWS0692545
CLEARED
Worksheet Details
Fault Repair
Worksheet
INCIDENT REPORT PUBLISHED 12/07/2022 INCIDENT RAISED: 30/06/2022 13:54 INCIDENT CLEARED: By 01/07/2022 09:00 SERVICE: Some SIP Services PEAK IMPACT: Loss of Service CONTINUOUS/INTERMITTENT (AT PEAK): Intermittent PEAK PRIORITY: P1 (High Priority) DOMAIN: Internal COMMONALITY: Core network commonality REPORT: At 13:50 on 30/06/2022 one of our transit routers dropped its BGP sessions to all its neighbours. Due to the high number of prefixes being shared this caused a spike in CPU load across each of our transit routers while the BGP routing table of each router was updated with 900,000+ routes. This caused some connectivity issues to some customers, mainly confined to hosted PBX and cloud 3CX customers whose servers use that particular router as their default route. The issue lasted for around 5 to 10 minutes with registrations quickly recovering back to previous levels. Following the diagnosis our engineers disabled BGP on the affected transit router so it was no longer participating in BGP with the core network. On 03/07/22 at 21:00 an IOS upgrade was performed on Transit2 to resolve the issue and BGP was restored to selected neighbours. This will now be monitored for 2 to 3 weeks and if it's still performing as expected we will re-enable the full BGP peering. We apologise for the inconvenience caused. CORRECTIVE ACTIONS: As detailed above PERFORMANCE: We regret that, in line with our Service Level Agreement, no Service Credits are available UPDATE: 01/07/2022 09:00 STATUS: RESOLVED SERVICES: N/A IMPACT: N/A CONTINUOUS/INTERMITTENT: N/A PRIORITY: N/A DOMAIN: N/A COMMONALITY: N/A UPDATE: Spitfire engineers have confirmed resolution of this incident. Services will be monitored further by Spitfire engineers. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: N/A UPDATE: 01/07/2022 08:30 STATUS: Mitigated SERVICES: Some SIP Services IMPACT: Loss of Service CONTINUOUS/INTERMITTENT: Intermittent PRIORITY: P1 DOMAIN: Internal COMMONALITY: Core network commonality UPDATE: Spitfire engineers have mitigated the impact for the majority of affected customers. We continue to work towards full resolution. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: 01/07/2022 09:30 UPDATE: 01/07/2022 08:00 STATUS: Diagnosed SERVICES: Some SIP Services IMPACT: Loss of Service CONTINUOUS/INTERMITTENT: Intermittent PRIORITY: P1 DOMAIN: Internal COMMONALITY: Core network commonality UPDATE: Spitfire engineers have diagnosed an incident. Further work is underway to establish and implement any viable mitigatory measures and work towards resolution. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: 01/07/2022 09:00 UPDATE: 30/06/2022 18:50 STATUS: Diagnosed SERVICES: Some SIP Services IMPACT: Loss of Service CONTINUOUS/INTERMITTENT: Intermittent PRIORITY: P1 DOMAIN: Internal COMMONALITY: Core network commonality UPDATE: Spitfire engineers have diagnosed an incident. Further work is underway to establish and implement any viable mitigatory measures and work towards resolution. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: 01/07/2022 08:00 UPDATE: 30/06/2022 17:00 STATUS: Diagnosed SERVICES: Some SIP Services IMPACT: Loss of Service CONTINUOUS/INTERMITTENT: Intermittent PRIORITY: P1 DOMAIN: Internal COMMONALITY: Core network commonality UPDATE: Spitfire engineers have diagnosed an incident. Further work is underway to establish and implement any viable mitigatory measures and work towards resolution. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: 30/06/2022 18:00 UPDATE: 30/06/2022 16:00 STATUS: Diagnosed SERVICES: Some SIP Services IMPACT: Loss of Service CONTINUOUS/INTERMITTENT: Intermittent PRIORITY: P1 DOMAIN: Internal COMMONALITY: Core network commonality UPDATE: Spitfire engineers have diagnosed an incident. Further work is underway to establish and implement any viable mitigatory measures and work towards resolution. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: 30/06/2022 17:00 UPDATE: 30/06/2022 15:00 STATUS: Diagnosed SERVICES: Some SIP Services IMPACT: Loss of Service CONTINUOUS/INTERMITTENT: Intermittent PRIORITY: P1 DOMAIN: Internal COMMONALITY: Core network commonality UPDATE: Spitfire engineers have diagnosed an incident. Further work is underway to establish and implement any viable mitigatory measures and work towards resolution. Further information will appear at http://status.spitfire.co.uk. Customers experiencing issues should please raise a support case. Customers should speak to Spitfire Support (or their IT Support) if they require any assistance implementing workaround solutions purchased. NEXT UPDATE TARGET: 30/06/2022 16:00
Status