Elastic Block Storage Partial Outage in AMS1
Incident Report for Equinix Metal
Resolved
This incident is resolved. As part of the manual recovery, some snapshots were deleted and may need to be re-run.

While the cluster is stable at this time, we remind all customers that our Elastic Block Storage service will be deprecated by June 1st, 2021. Until then we will continue to provide best-effort support for this service. https://metal.equinix.com/developers/docs/resilience-recovery/elastic-block-storage/

Our Customer Success team is available to consult with you regarding other options and support your move to a new solution where possible.
Posted May 12, 2021 - 22:28 UTC
Monitoring
Both nodes have been reloaded. Impact is resolved at this time.
Posted May 12, 2021 - 21:41 UTC
Update
We have successfully reloaded one node in the cluster and restored IO to all volumes on the node. At this time, we are reloading the second node to fully restore connectivity to all volumes. There is a potential for some data loss, but it should be possible to remount drives at this time.
Posted May 12, 2021 - 21:23 UTC
Update
We have confirmed that there are two failed drives across two nodes in the storage cluster. At this time, we are attempting to restore the cluster without these two drives, which will enable recovery to proceed.
Posted May 12, 2021 - 20:09 UTC
Update
Our partners at Datera have confirmed all 3 nodes in the cluster are currently running in a normal state, and that the logs indicate there may have been a brief failure overnight. Troubleshooting continues, and we will recover the impacted volumes as soon as possible.
Posted May 12, 2021 - 13:54 UTC
Identified
The Elastic Block Storage service in AMS1 is currently suffering from a partial outage. We are working with our partners at Datera to resolve the issue, until then certain blocks may not be accessible by customers. Please stand by while we resolve the issue.
Posted May 12, 2021 - 10:53 UTC