We're currently having issues with scanning

RCA Report: Boltrics certificate failure

Incident Details

On Sept 28 2023 01:59 UTC, our Boltrics certificate expired and caused all SOAP and OData web service calls to fail. This affected our app platform/Telnet platform and EDI. We resolved the issue by manually updating the bindings and verifying the web service calls.

Root Cause

The root cause of this incident was a human error in the certificate replacement procedure that did not update all bindings correctly. The procedure was supposed to update the bindings for all TCP ports that were used by the web service calls, but it missed TCP ports 7147 and 7148. This caused the web service calls to use the old certificate, causing the failure of our external connections.

Corrective Measures

How are we making incidents like this less likely or less impactful?

Our ITOPS team is working on improvements to the current distribution of new Boltrics TLS Certificates across multiple environments. We will automate the certificate replacement procedure and include the appropriate tests to make sure that the certificate is rolled out correctly and bound to the correct TCP ports (Estimated completion: Q1 2024).

By implementing these corrective measures, we aim to enhance the reliability and security of our systems and prevent similar incidents in the future.

Alle services should be back to normal.

We apologize for any inconvenience this may have caused you. We solved the root cause and take steps to prevent similar incidents in the future.

We've confirmed there is a problem, we're working to resolve it.

This is because of an certificate that is not working anymore, we will solve this as soon as possible.

Impact:

• Boltrics Cloud, is restoring and should work now.
• These customers we will get online soon: 017|018|021|022|029|048|050|056|061|074|086|119|125|134|165|170|184|186|198|199|202|218|222|232|245|267|275

Some people are experiencing problems with our service right now. We're so sorry about that. Please standby for further updates, and we'll be in touch as soon as we know more.

Impact: Telnet scanning en app platform.