This incident is resolved. As part of the manual recovery, some snapshots were deleted and may need to be re-run.
While the cluster is stable at this time, we remind all customers that our Elastic Block Storage service will be deprecated by June 1st, 2021. Until then we will continue to provide best-effort support for this service. https://metal.equinix.com/developers/docs/resilience-recovery/elastic-block-storage/
Our Customer Success team is available to consult with you regarding other options and support your move to a new solution where possible.
May 12, 22:28 UTC
Both nodes have been reloaded. Impact is resolved at this time.
May 12, 21:41 UTC
We have successfully reloaded one node in the cluster and restored IO to all volumes on the node. At this time, we are reloading the second node to fully restore connectivity to all volumes. There is a potential for some data loss, but it should be possible to remount drives at this time.
May 12, 21:23 UTC
We have confirmed that there are two failed drives across two nodes in the storage cluster. At this time, we are attempting to restore the cluster without these two drives, which will enable recovery to proceed.
May 12, 20:09 UTC
Our partners at Datera have confirmed all 3 nodes in the cluster are currently running in a normal state, and that the logs indicate there may have been a brief failure overnight. Troubleshooting continues, and we will recover the impacted volumes as soon as possible.
May 12, 13:54 UTC
The Elastic Block Storage service in AMS1 is currently suffering from a partial outage. We are working with our partners at Datera to resolve the issue, until then certain blocks may not be accessible by customers. Please stand by while we resolve the issue.
May 12, 10:53 UTC