Investigating issues with Microsoft private cloud

Response from Microsoft:

Title: Environment inaccessible

User Impact:

Users may have been unable to access your environment.

Final Status:

We have completed the rollback and confirmed via diagnostic analysis that environment access has been restored.

Preliminary Root Cause: A recent deployment introduced a code misconfiguration on a portion of the infrastructure responsible for environment access.

Next Steps: We're reviewing our standard service update procedures to avoid similar impact in the future.

User Experience:

Users were unable to access their environment.

Incident Start Time: Wednesday, February 22, 2023, 4:20 AM UTC Incident End Time: Wednesday, February 22, 2023, 1:00 PM UTC

All environments are working again, waiting on an official response from Microsoft

Most environments are working again, waiting on an official response from Microsoft.

Update from Microsoft:

Title: Environment inaccessibleUser

Impact: Users may be unable to access your environment.

Current Status: After our investigation, we identified that users may be unable to access your environment due to a code misconfiguration introduced in a recent deployment. We are gathering and reviewing service diagnostics to determine our next troubleshooting steps.

User Experience: Users are unable to access their environment.

We are still in contact with Microsoft, issue not yet identified.

Some customers are experiencing problems with our service right now. We're so sorry about that. Please standby for further updates, and we'll be in touch as soon as we know more.

Impacted environments: 038|043|095|112|123|138|144|146|152|155|159|164|190|203|210|217|225|228|230|231|233|234|246|249|250|251|256|264

Situation: Environment down