Microsoft Service Degradation

Resolved
Resolved

We will mark this issue as resolved. Microsoft is still in progress with the Post Incident Review (PIR)

Recovering

Microsoft update:

Summary of Impact: Between 07:05 UTC and 09:45 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in Public Azure regions, as well as other Microsoft services including M365 and PowerBI.

Preliminary Root Cause: We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, as well as connectivity between services in different regions, as well as ExpressRoute connections.

Mitigation: We identified a recent change to WAN as the underlying cause and have rolled back this change. Networking telemetry shows recovery from 09:00 UTC onwards across all regions and services, with the final networking equipment recovering at 09:35 UTC. Most impacted Microsoft services automatically recovered once network connectivity was restored, and we worked to recover the remaining impacted services.

Next Steps: We will follow up in 3 days with a preliminary Post Incident Report (PIR), which will cover the initial root cause and repair items. We'll follow that up 14 days later with a final PIR where we will share a deep dive into the incident.

You can stay informed about Azure service issues, maintenance events, or advisories by creating custom service health alerts (https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation) and you will be notified via your preferred communication channel(s).

Identified

Microsoft update : "Current Status:

We have identified a recent change to WAN as the underlying cause, and have taken steps to roll back this change. Our telemetry shows consistent signs of recovery from 09:00 UTC onwards across multiple regions and services, and we are continuing to actively monitor the situation.

With WAN networking now seeing recovery, we are working to ensure full recovery for impacted services.

The next update will be in 30 minutes or as soon as we have further information.

This message was last updated at 10:01 UTC on 25 January 2023"

Investigating

Microsoft: "Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in multiple regions, as well as other Microsoft services.

We are actively investigating and will share updates as soon as more is known.

This message was last updated at 08:53 UTC on 25 January 2023"

You can following the status of Microsoft here:

Began at:

Affected components
  • Support Services
  • Website
  • Environments
    • Microsoft Cloud
    • Boltrics Cloud
    • Boltrics (private) Cloud
  • External Boltrics
    • Azure
    • Other
  • Integrations
    • Datahub Servers (NL/BE/UK/AUS/US)
    • EDI
    • Customs
  • Other
    • Web portal
    • App platform
    • Telnet scanning
    • Boltrics API
    • Boltrics RDP01 server