Microsoft Cloud - Performance degradation and Unable to Login

Post incident Review Report Microsoft

Summary

On Wednesday, February 12, 2025, at 7:16 AM UTC, an escalation reported that users received an error when attempting to sign in to your environment. Investigation identified that a portion of the infrastructure supporting environment access was experiencing high resource utilization, resulting in impact. A failover to redundant infrastructure was performed, and following a review of service health diagnostics it was confirmed that environment access was restored on Wednesday, February 12, 2025, at 12:15 AM.

User Experience

Users were unable to access your environment, and may have received the error message “Something went wrong Please try again later. If you continue to see this error, provide the information below when requesting support.”

Incident Start Date and Time

Wednesday, February 12, 2025, 7:16 AM UTC

Date and Time Service was Restored

Wednesday, February 12, 2025, 12:15 AM UTC

Root Cause

We continue to investigate the root cause of the high resource utilization that prevented environment access on Wednesday, February 12, 2025. Additional details and results from this investigation will be provided within three weeks.

Microsoft update:

Final Status: We have completed both the failover to redundant infrastructure as well as scaling out available infrastructure. After a period of monitoring, we have confirmed that environment access and environment performance have both been restored.

A Post Incident Report will be published within five days.

Preliminary Root Cause: A portion of the backend infrastructure supporting environment access experienced high resource utilization.

Next Steps: We’re reviewing our code for improved performance and potential automated recovery options to reduce or avoid similar issues in the future.

All services should now be operational. We will share a Root Cause Analysis (RCA) as soon as Microsoft publishes it, which typically takes at least 24 hours.

If your production environment is still experiencing issues, please inform our support department. Kindly note that it may take up to another hour for performance to be fully restored.

We sincerely apologize for the inconvenience this has caused and appreciate your patience.

Update from Microsoft:

As our efforts to failover to redundant infrastructure, as well as scaling out the available infrastructure continue, users should be once again able to access their environment. However environment performance remains degraded while this process completes.

Update from Microsoft:

We are continuing our efforts to failover to redundant healthy infrastructure, as well as scaling out the available infrastructure.

It appears that some services are being restored. Please try to log in and check if your environment is working.

We will continue to monitor the situation closely and provide further updates as soon as we have more information. At this moment, there is no need to report whether your environment is working or not. We will keep monitoring all environments.

We have not yet received an update from Microsoft regarding this issue. The impact affects multiple clusters within NL. We will continue to stay in contact with Microsoft to obtain updates and will share any new information as soon as it becomes available.

Update from Microsoft:

After our investigation, we identified that a portion of the backend infrastructure supporting environment access was experiencing high resource utilization, leading to failures when logging in, or degraded performance within your environment. As an initial mitigation step, we are performing a failover to redundant healthy infrastructure as well as scaling out the available infrastructure to provide more available resources.

Update from Microsoft:

We are aware of an emerging issue where users may be unable to access your environment, or may be experience degraded performance once logged in. We are investigating the issue and will provide another update within the next 30 minutes.

We have received multiple reports that several environments are down or performing very slowly. The following environments are currently known to be affected:

Environments: PROD-005 | PROD-010 | PROD-020 | PROD-038 | PROD-042 | PROD-076 PROD-079 | PROD-084 | PROD-110 | PROD-112 | PROD-114 | PROD-116 PROD-117 | PROD-137 | PROD-151 | PROD-154 | PROD-175 | PROD-177 PROD-190 | PROD-203 | PROD-211 | PROD-228 | PROD-230 | PROD-231 PROD-242 | PROD-243 | PROD-246 | PROD-259 | PROD-277 | PROD-281 PROD-295 | PROD-296 | PROD-312 | PROD-333 | PROD-335

If your environment is also impacted, please submit a ticket through: Environment down

We appreciate your cooperation. Resolving this issue is our top priority, and we are working with Microsoft to find a solution as quickly as possible. Further updates will be provided as soon as they become available.

We have received multiple reports from users experiencing login issues with their environment. We are actively investigating this matter in collaboration with Microsoft and are working to resolve it as soon as possible.

If your environment is inaccessible and you are unable to create a ticket, please report the issue via the following link: environment down.

We will continue working on this issue until it is fully resolved and will provide further updates as soon as we have more information.

We are actively investigating this issue in collaboration with Microsoft and are working to resolve it as soon as possible.

We will provide further updates as soon as we have more information.

We have received confirmation from Microsoft that they are still investigating the issue. Their initial findings suggest a possible regression related to OData requests.

We are awaiting further updates from Microsoft and will continue to monitor the situation closely. Any new developments will be shared on this status page.

Thank you for your patience.

We are continuing to work closely with Microsoft to identify the root cause of the issue.

We understand the impact this is having on your operations and remain fully committed to resolving it as quickly as possible. Our team will actively monitor and investigate until a resolution is found.

Further updates will be provided on our status page as soon as new information becomes available.

We are currently experiencing an increasing number of reports from customers regarding performance degradation in Microsoft Dynamics 365 Business Central. Users may notice slow response times or delays in App Platform (Scanning) and in Web Client.

Current Actions:

We have reported these issues to Microsoft Support and are actively working with them to investigate the root cause. Our team is monitoring the situation closely and gathering data to assist in resolving the issue as quickly as possible. Next Steps: We will provide updates as soon as we receive more information from Microsoft or identify any relevant findings.

We appreciate your patience and will continue to keep you informed.