Intermittent Connectivity Issues With File Storage
Incident Report for Trackops
Resolved
This issue is now resolved.

We apologize for the inconvenience today's extended outage caused. We work hard to make sure these sorts of things don't happen, but in the rare instance that an issue is beyond our control (http://www.usatoday.com/story/tech/news/2017/02/28/amazons-cloud-service-goes-down-sites-scramble/98530914/), we're confident in weathering the storm with Amazon, the world's largest cloud provider.

We will attach a post-mortem once we have all the details from AWS surrounding the cause of this issue, so please subscribe to this status page for future updates and maintenance announcements.

Thanks again, and we appreciate and value your support!
Posted Feb 28, 2017 - 16:01 MST
Monitoring
AWS engineers have confirmed that file transfers are now functioning at full capacity (http://status.aws.amazon.com) for both uploading and downloading. We have confirmed that error rates have dropped off considerably. At this time, you should be able to upload files without issue. Tests appear to confirm that this issue should be resolved, but we will continue to monitor all systems for stability.

If you continue to have issues with file uploads or downloads, please don't hesitate to open a support ticket.

Thank you for your patience while we wrap this issue up, especially in such a critical time as the last day of the month.
Posted Feb 28, 2017 - 15:23 MST
Update
We are starting to see some successful activity with downloading/viewing files from the system. File uploads are still not functioning properly, and file downloads are still intermittent with some failures are still occurring. We are, however, seeing continued stabilization, which indicates the issue appears to be turning the corner towards resolution. We will continue to monitor progress and will post periodic updates until the issue is fully resolved. Thank you your continued patience and support during this issue!
Posted Feb 28, 2017 - 14:08 MST
Update
We're continuing to monitor AWS' progress to restore access to S3 file storage. AWS engineers have confirmed that they are actively repairing connectivity (https://status.aws.amazon.com), and we will continue to post periodic updates until this issue is resolved. Thanks again for your patience while we work through this problem.
Posted Feb 28, 2017 - 12:48 MST
Update
We continue to receive high error rates with file uploads/downloads and related media access. We're closely monitoring Amazon's progress on this issue and will continue to provide periodic updates until this issue is resolved.
Posted Feb 28, 2017 - 12:05 MST
Update
This issue continues to persist, although we're able to narrow the scope of the issue. At this time, it appears all features related to file uploads, downloads, photos, update attachments etc.. are affected. We are continuing to work with AWS to resolve this issue.
Posted Feb 28, 2017 - 11:12 MST
Identified
We have received confirmation from our cloud provider Amazon Web Services (AWS) that there is a widespread connectivity issue, affecting many systems across the internet. We are working with them to resolve this issue.
Posted Feb 28, 2017 - 10:59 MST
Investigating
We are investigating reports of connectivity issues related to our file storage system. We are looking into this.
Posted Feb 28, 2017 - 10:54 MST