Some customers currently having issues with printing (private cloud)

RCA Report from Microsoft: Printing Issue on Microsoft Private Cloud

Background

Our back-end team has identified the root cause of the printing issues that some of our customers have been experiencing. The issue is related to recent changes made to two components which altered the internal format of request page filters. This report provides an overview of what happened and outlines the steps we will take to prevent similar issues in the future.

Incident Details

On Friday 22/09/2023, our team deployed new changes to the two components in question. These changes were intended to update the internal format of request page filters. The deployment started on Saturday, but unfortunately, only one of the components was successfully upgraded on the ISV embed clusters. The deployment of the second component failed.

Under normal circumstances, when a deployment fails, we roll back to the last working version. In this case, however, the issue arose due to the fact that both components needed to be upgraded simultaneously. As we ended up with two different versions, one component was unable to decode the filters generated by the other component. As a result, request pages started to fail, leading to the printing issues that our customers experienced.

Root Cause

The root cause of the issue was twofold. Firstly, the failure to deploy the second component on the ISV embed clusters created a mismatch with the upgraded version of the first component. Secondly, the changes made to the internal format of request page filters required both components to be upgraded simultaneously. Without this synchronization, the decoding of filters generated by one component became incompatible with the other, causing the failure in request pages.

Corrective Measures

To ensure similar issues do not occur in the future, we will take the following corrective measures:

Deployment Synchronization: Going forward, we will ensure that both components are always deployed and rolled back in sync. If one component fails to deploy, we will roll back both components to the last working version. This will prevent any mismatch and compatibility issues between the components.

Addressing Deployment Failure: We will investigate the root cause of the deployment failure for the second component. By identifying and addressing the underlying reason for the failure, we can prevent similar failures from occurring in future deployments.

Conclusion

We apologize for the inconvenience caused by the printing issues experienced by some of our customers on the Private Cloud. We have identified the root cause of the issue and have taken steps to prevent similar incidents in the future.

Microsoft now resolved the incident. Thanks for your patience.

Microsoft fixed the core issue, and are waiting for things to recover.

Unfortunately, we cannot provide any additional information beyond what has already been shared.

Microsoft is actively investigating the issue and it remains their number 1 priority for their backend developers.

Affected Services: Printing on the Private Cloud

Collaboration with Microsoft: We continue to work closely with Microsoft to expedite the resolution of this issue. Since this issue is platform-related, our team is in constant communication with Microsoft's experts to ensure it is addressed promptly and efficiently.

We understand the frustration this may be causing some of you, and we sincerely apologize for any inconvenience you have experienced. Rest assured, we are doing everything in our power to resolve this matter as swiftly as possible.

We will keep you updated with any significant developments or changes in the situation.

We believe there may be an issue, we're looking into it.

The scope of the problem: All private cloud customers, so public cloud is not impacted.

Impacted customers: 038|043|095|123|152|155|159|246|249|251|256|264

Some people are experiencing problems with our service right now. We're so sorry about that. Please standby for further updates, and we'll be in touch as soon as we know more.

Issue: Customers are unable to print labels and documents and getting redirected to an error page 'something went wrong'.