Error rates and latency have returned to normal levels. We are working with our provider to conduct a root cause analysis and prepare a postmortem report.
Posted Feb 28, 2017 - 21:58 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 28, 2017 - 21:10 UTC
Update
Our provider is reporting reduced errors rates and we are confirming a marked drop in latency and error rate. We are continuing to monitor as our infrastructure recovers.
Posted Feb 28, 2017 - 21:10 UTC
Update
Our service provider continues to work toward resolution based on their understanding of the root cause. Meanwhile, we are pushing as much traffic as possible through unimpacted regions, including our Oregon region and our Asia Pacific regions.
Posted Feb 28, 2017 - 19:45 UTC
Update
Our upstream provider has identified the problem and is working to recover.
Posted Feb 28, 2017 - 18:56 UTC
Identified
Our upstream provider has confirmed high error rates originating from their storage layer in Virginia. We are manually failing systems that depend on this service and falling back to standby resources in Oregon.
Posted Feb 28, 2017 - 18:38 UTC
Update
We are working with an upstream provider to investigate a major networking issue.
Posted Feb 28, 2017 - 18:02 UTC
Investigating
We're investigating increased API error rates and latency globally.