Resolved -
After closely monitoring our system for 42 hours without further issue, the incident impacting our Core API has been resolved and all affected service continue to operate normally.
Nov 6, 20:26 UTC
Update -
All core services are performing normally, and we are maintaining our standard monitoring posture overnight.
Nov 6, 01:32 UTC
Update -
The system remained stable overnight, and service has fully normalized.
We confirmed that impacted endpoints performed within normal operating parameters throughout the night's critical processing window. We are returning to our standard monitoring posture.
Thank you for your patience during this period of elevated observation.
Nov 5, 18:01 UTC
Update -
All core services are performing normally, and we are maintaining our elevated monitoring posture overnight. We will post another update if unusual activity is detected.
Nov 5, 02:27 UTC
Update -
All affected endpoints continue to operate normally. We are closely monitoring the system after the rollback of the fix that caused the temporary errors and latency.
Nov 5, 01:15 UTC
Monitoring -
The rollback for the issue impacting several endpoints has been deployed. We are now closely monitoring the system to confirm full recovery.
Nov 5, 00:09 UTC
Identified -
We are currently experiencing increased latency and error rates impacting certain endpoints due to an additional mitigation measure / optimization being rolled out as part of the incident.
We are actively rolling back the fix. We will provide an update within the next hour or as more information becomes available.
Nov 4, 23:49 UTC
Update -
Since our last update, the Core API remains stable and fully operational with no further errors or latency spikes for over 24 hours. The team has identified the underlying issue, and we have observed that our mitigation measures have addressed the intermittent latency. Given the stability over the past day, we anticipate resolving this incident tomorrow, but will continue active monitoring until then.
Nov 4, 18:02 UTC
Update -
We received a burst of traffic around 9:30pm PT that caused the issue to resume momentarily. We assigned more resources, see that the impact has subsided, and are actively monitoring.
Nov 3, 06:08 UTC
Monitoring -
No further errors or latency issues have occurred since last posting. We will continue to monitor the effectiveness of the deployed fix.
Nov 2, 11:13 UTC
Update -
We have implemented mitigation measures and have seen recovery in latency and error rates. Our engineers continue to monitor the situation and are actively working to identify the root cause.
Nov 2, 10:18 UTC
Update -
We are continuing to investigate the issue.
Nov 2, 09:15 UTC
Investigating -
We are currently investigating increased latency and error rates impacting our Core API. Users may experience intermittent errors when making API requests. Our engineers are actively working to identify the root cause and restore full functionality as quickly as possible. We will provide an update within the next hour as more information becomes available.
Nov 2, 08:15 UTC