RCA Report: Boltrics certificate failure
Incident Details
On Sept 28 2023 01:59 UTC, our Boltrics certificate expired and caused all SOAP and OData web service calls to fail. This affected our app platform/Telnet platform and EDI. We resolved the issue by manually updating the bindings and verifying the web service calls.
Root Cause
The root cause of this incident was a human error in the certificate replacement procedure that did not update all bindings correctly. The procedure was supposed to update the bindings for all TCP ports that were used by the web service calls, but it missed TCP ports 7147 and 7148. This caused the web service calls to use the old certificate, causing the failure of our external connections.
Corrective Measures
How are we making incidents like this less likely or less impactful?
Our ITOPS team is working on improvements to the current distribution of new Boltrics TLS Certificates across multiple environments. We will automate the certificate replacement procedure and include the appropriate tests to make sure that the certificate is rolled out correctly and bound to the correct TCP ports (Estimated completion: Q1 2024).
By implementing these corrective measures, we aim to enhance the reliability and security of our systems and prevent similar incidents in the future.
Alle services should be back to normal.
We apologize for any inconvenience this may have caused you. We solved the root cause and take steps to prevent similar incidents in the future.
We've confirmed there is a problem, we're working to resolve it.
This is because of an certificate that is not working anymore, we will solve this as soon as possible.
Impact:
Some people are experiencing problems with our service right now. We're so sorry about that. Please standby for further updates, and we'll be in touch as soon as we know more.
Impact: Telnet scanning en app platform.
We’ll find your subscription and send you a link to login to manage your preferences.
We’ve found your existing subscription and have emailed you a secure link to manage your preferences.
We’ll use your email to save your preferences so you can update them later.
Subscribe to other services using the bell icon on the subscribe button on the status page.
You’ll no long receive any status updates from Boltrics status, are you sure?
{{ error }}
We’ll no longer send you any status updates about Boltrics status.