Rate limites

» View Event Details | Created Thu, 15 Apr 2021 12:11:00 +0000

Resolved

All services are now operational, we will continue to keep an eye on the service. We apologize for any loss of service you have have experienced during this short period .

Investigating

We are aware that some clients may be hitting rate limits when using our applications. We are looking into this and will update when we have any news.

Interruption of service with SSO

» View Event Details | Created Tue, 13 Apr 2021 08:24:00 +0000

Post-Mortem

Overview

On 13th April 2021, we suffered a fault with SSO (Azure) preventing some users being able to log in.

What Happened

Between the hours of 0815-1030 BST on Tuesday 13th April 2021, users encountered errors when logging into Synthetix.cloud using Single Sign-On (SSO). We have identified that this was due to a re-declaration of a variable within the core code of the API, this variable would only be utilized by SSO (Azure) users and for a majority of users would have remained dormant.

Resolution

At 0930 Synthetix developers have been able identify the anomalous code and have since re-declared the variables with different keys to prevent overwriting of previously declared. This was tested locally before being deployed to staging and tested. Once confirmed internally that the issue was no longer present the patch was then moved to production to confirm that live users were now able to login using SSO (Azure). The patch has now been deployed successfully.

Timeline

• 08:15 BST – Issue identified by users

• 08:30 BST – Issue confirmed by support staff

• 08:35 BST – Incident opened

• 08:40 BST – Work began by developers to identify cause

• 09:30 BST – Problem code located and patched locally

• 09:40 BST – Patch deployed to development servers for testing

• 09:50 BST – Patch confirmed to be effective and migrate to staging

• 10:15 BST – Fix pushed to production servers

• 10:24 BST – Issue confirmed fixed

• 10:30 BST – Incident closed

Impact

This incident affected approximately 20% of users for 2 hours. Only users whom use SSO (Azure) rather than the traditional username and password method will have been affected.

Produced by: Luke Addington - Head of Software

Resolved

We have resolved the issue affecting the SSO authentication.

We apologise for any inconvenience caused, and will be producing an RCA to follow within 24 hours.

Update

We have deployed a fix for the SSO authentication.

We will confirm that this has resolved the issue shortly.

MFA Emails

» View Event Details | Created Tue, 13 Apr 2021 07:00:00 +0000

Update

We have implemented a temporary fix to allow MFA to be sent until such time as we have found a more permanent solution. We will update when we have any more information.

Investigating

We are aware that some users are not receiving MFA codes via email when attempting to log in.

Developers are currently investigating and working on a fix for this, however we are unable to provide a timeframe; in the meantime please try clicking 'resend'.

Alternatively you can switch to using a mobile application such as Authy on your smartphone or PC:

iOS : https://apps.apple.com/us/app/twilio-authy/id494168017

Android : https://play.google.com/store/apps/details?id=com.authy.authy&hl=en

PC : https://authy.com/download/