All Systems are Online

MFA Emails

Update

We have implemented a temporary fix to allow MFA to be sent until such time as we have found a more permanent solution. We will update when we have any more information.


Posted: Fri, 16 Apr 2021 09:59:00 +0000

Investigating

We are aware that some users are not receiving MFA codes via email when attempting to log in.

Developers are currently investigating and working on a fix for this, however we are unable to provide a timeframe; in the meantime please try clicking 'resend'.

Alternatively you can switch to using a mobile application such as Authy on your smartphone or PC:

iOS : https://apps.apple.com/us/app/twilio-authy/id494168017

Android : https://play.google.com/store/apps/details?id=com.authy.authy&hl=en

PC : https://authy.com/download/


Posted: Tue, 13 Apr 2021 07:00:00 +0000

Overview

Legacy Products & Services
Good Service
Administration
Good Service
Synthetix Cloud App
Good Service
API - 2.0
Good Service
Website Integration (CDN)
Good Service
Editor
Good Service

Metrics

API 2.0 - Uptime

Average - %

API 2.0 - Response Time

Worldwide Average - ms

CDN - Response Time

Worldwide Average - ms

Recent Events

Last 7 Days

Saturday, April 17th 2021

There are no reported events.

Friday, April 16th 2021

Redacted messages

» View Event Details | Created Fri, 16 Apr 2021 11:10:00 +0000

Update

The redacted messages have now been restored.


Posted: Fri, 16 Apr 2021 16:33:00 +0000

Update

We are currently restoring the redacted messages and an update will go out as soon as the restore has been completed.


Posted: Fri, 16 Apr 2021 14:32:00 +0000

Update

We are still working on identifying the ongoing issue with redacted chat transcripts and preparing a data restore. Please standby for further updates.


Posted: Fri, 16 Apr 2021 12:33:00 +0000

Thursday, April 15th 2021

Rate limites

» View Event Details | Created Thu, 15 Apr 2021 12:11:00 +0000

Resolved

All services are now operational, we will continue to keep an eye on the service. We apologize for any loss of service you have have experienced during this short period .


Posted: Thu, 15 Apr 2021 12:22:00 +0000

Investigating

We are aware that some clients may be hitting rate limits when using our applications. We are looking into this and will update when we have any news.


Posted: Thu, 15 Apr 2021 12:00:00 +0000

Wednesday, April 14th 2021

There are no reported events.

Tuesday, April 13th 2021

Interruption of service with SSO

» View Event Details | Created Tue, 13 Apr 2021 08:24:00 +0000

Post-Mortem

Overview

On 13th April 2021, we suffered a fault with SSO (Azure) preventing some users being able to log in.

What Happened

Between the hours of 0815-1030 BST on Tuesday 13th April 2021, users encountered errors when logging into Synthetix.cloud using Single Sign-On (SSO). We have identified that this was due to a re-declaration of a variable within the core code of the API, this variable would only be utilized by SSO (Azure) users and for a majority of users would have remained dormant.

Resolution

At 0930 Synthetix developers have been able identify the anomalous code and have since re-declared the variables with different keys to prevent overwriting of previously declared. This was tested locally before being deployed to staging and tested. Once confirmed internally that the issue was no longer present the patch was then moved to production to confirm that live users were now able to login using SSO (Azure). The patch has now been deployed successfully.

Timeline

• 08:15 BST – Issue identified by users

• 08:30 BST – Issue confirmed by support staff

• 08:35 BST – Incident opened

• 08:40 BST – Work began by developers to identify cause

• 09:30 BST – Problem code located and patched locally

• 09:40 BST – Patch deployed to development servers for testing

• 09:50 BST – Patch confirmed to be effective and migrate to staging

• 10:15 BST – Fix pushed to production servers

• 10:24 BST – Issue confirmed fixed

• 10:30 BST – Incident closed

Impact

This incident affected approximately 20% of users for 2 hours. Only users whom use SSO (Azure) rather than the traditional username and password method will have been affected.

Produced by: Luke Addington - Head of Software


Posted: Tue, 13 Apr 2021 09:46:00 +0000

Resolved

We have resolved the issue affecting the SSO authentication.

We apologise for any inconvenience caused, and will be producing an RCA to follow within 24 hours.


Posted: Tue, 13 Apr 2021 09:26:00 +0000

Update

We have deployed a fix for the SSO authentication.

We will confirm that this has resolved the issue shortly.


Posted: Tue, 13 Apr 2021 09:19:00 +0000

MFA Emails

» View Event Details | Created Tue, 13 Apr 2021 07:00:00 +0000

Update

We have implemented a temporary fix to allow MFA to be sent until such time as we have found a more permanent solution. We will update when we have any more information.


Posted: Fri, 16 Apr 2021 09:59:00 +0000

Investigating

We are aware that some users are not receiving MFA codes via email when attempting to log in.

Developers are currently investigating and working on a fix for this, however we are unable to provide a timeframe; in the meantime please try clicking 'resend'.

Alternatively you can switch to using a mobile application such as Authy on your smartphone or PC:

iOS : https://apps.apple.com/us/app/twilio-authy/id494168017

Android : https://play.google.com/store/apps/details?id=com.authy.authy&hl=en

PC : https://authy.com/download/


Posted: Tue, 13 Apr 2021 07:00:00 +0000

Monday, April 12th 2021

There are no reported events.

Sunday, April 11th 2021

There are no reported events.

Twitter Updates

Subscribe to Updates