Interruption of service with SSO
View Event Details
On 13th April 2021, we suffered a fault with SSO (Azure) preventing some users being able to log in.
Between the hours of 0815-1030 BST on Tuesday 13th April 2021, users encountered errors when logging into Synthetix.cloud using Single Sign-On (SSO).
We have identified that this was due to a re-declaration of a variable within the core code of the API, this variable would only be utilized by SSO (Azure) users and for a majority of users would have remained dormant.
At 0930 Synthetix developers have been able identify the anomalous code and have since re-declared the variables with different keys to prevent overwriting of previously declared.
This was tested locally before being deployed to staging and tested. Once confirmed internally that the issue was no longer present the patch was then moved to production to confirm that live users were now able to login using SSO (Azure). The patch has now been deployed successfully.
• 08:15 BST – Issue identified by users
• 08:30 BST – Issue confirmed by support staff
• 08:35 BST – Incident opened
• 08:40 BST – Work began by developers to identify cause
• 09:30 BST – Problem code located and patched locally
• 09:40 BST – Patch deployed to development servers for testing
• 09:50 BST – Patch confirmed to be effective and migrate to staging
• 10:15 BST – Fix pushed to production servers
• 10:24 BST – Issue confirmed fixed
• 10:30 BST – Incident closed
This incident affected approximately 20% of users for 2 hours. Only users whom use SSO (Azure) rather than the traditional username and password method will have been affected.
Produced by: Luke Addington - Head of Software
We have resolved the issue affecting the SSO authentication.
We apologise for any inconvenience caused, and will be producing an RCA to follow within 24 hours.
We have deployed a fix for the SSO authentication.
We will confirm that this has resolved the issue shortly.