Title: Amazon Web Services Outage 12/9/09 Document #: CPO120909
Advisory Class: Cloud Provider Outage Impact: Severe
Date Published: 12/09/09 Date of Last Update: 12/09/09
Outage Description
Apparent Network's Cloud Performance Center (
www.appparentnetworks.com/cpc) recently confirmed Amazon Web Services experienced connectivity loss which caused an Amazon Elastic Compute Cloud services outage in a single availability zone in their Northern Virginia Data Center. The outage occurred on December 9, 2009 beginning at approximately 3:34AM EST and lasted until 4:19AM EST or about 44 minutes. Amazon later confirmed the connectivity loss was due to a power failure.
During that time, access to systems in the Northern Virginia Data Center was unavailable. Businesses utilizing Amazon Elastic Compute Cloud services from this datacenter were affected.
Apparent Networks' Cloud Performance Center (CPC), a free service that offers performance data on leading cloud services providers such as Amazon, Google and GoGrid, detected the connectivity loss on 12/9/09. The Cloud Performance Center provides real-time and historical data to help inform IT teams of network performance issues that could impact effective delivery of cloud services. It provides network path performance metrics -including bandwidth, jitter and latency-between the cloud providers and major cities throughout North America.
The Cloud Performance Center utilizes Apparent Networks' PathView Cloud service to test the performance of cloud service providers. PathView Cloud software was configured to sample path performance to a series of pre-determined targets hosted at Amazon's eastern region data center every 120 seconds.
Testing Results
Apparent Networks' Cloud Performance Center (CPC), a free service that offers performance data on leading cloud services providers such as Amazon, Google and GoGrid, detected the connectivity loss on 12/9/09. The Cloud Performance Center provides real-time and historical data to help inform IT teams of network performance issues that could impact effective delivery of cloud services. It provides network path performance metrics -including bandwidth, jitter and latency-between the cloud providers and major cities throughout North America.
The Cloud Performance Center utilizes Apparent Networks' PathView Cloud service to test the performance of cloud service providers. PathView Cloud software was configured to sample path performance to a series of pre-determined targets hosted at Amazon's eastern region data center every 120 seconds.
PathView Cloud noted Amazon outage-related connectivity loss events as follows:
| PathView Cloud - Monitored From |
Connectivity Lost |
Connectivity Restored |
Outage Duration |
| Dallas, TX (Rackspace) |
3:39:50 AM |
4:24:32 AM |
0:44:42 |
| Newark, NJ(Hosting.com) |
3:34:43 AM |
4:19:27 AM |
0:44:44
|
| San Francisco, CA (GoGrid) |
3:35:46 AM |
4:20:30 AM |
0:44:44
|
| VA Data Center (Amazon) |
3:37:56 AM |
4:22:50 AM |
0:44:54 |
Click on any of the links below to access reports generated by PathView Cloud showing the outage reported from each of the data centers above.
Performance Report from Dallas Data Center
Performance Report from Newark Data Center
Performance Report from VA Data Center
Performance Report from San Francisco Data Center
Cloud Providers Tested
Amazon Web Services
Vendor Information, Solutions and Workarounds
Amazon Service Health Dashboard Records for this outage are below (
http://status.aws.amazon.com/).
1:08 AM PST We are investigating connectivity issues for instances in the US-EAST-1 region.
1:26 AM PST We are experiencing power issues for a subset of instances in a single availability zone in the US-EAST-1 region.
1:51 AM PST The underlying power issue has been addressed. Instances have begun to recover.
2:11 AM PST Most affected instances have regained power and are operating normally. We are working to recover a small number of instances that have not yet recovered.
3:33 AM PST We are continuing to work on recovering the remaining affected instances.
5:31 AM PST We are continuing to work on recovering the small number of remaining affected instances.
About PathView Cloud
PathView Cloud is a hosted network management tool that measures the performance of complete network paths from source to destination, including segments that pass through service providers' and carriers' networks. It enables IT teams and network managers to assess, troubleshoot and continuously monitor thousands of network paths simultaneously. A free version of the tool allowing users to monitor and test five network paths simultaneously is available at www.apparentnetworks.com.
About the Cloud Performance Center
The Cloud Performance Center (www.apparentnetworks.com/cpc) provides real-time, region-by-region service delivery performance scores for cloud providers. Visitors can view an interactive map of the North American market and select locations that are relevant to their businesses. Based on their selection, the CPC provides a detailed Cloud Provider Scorecard for the major cloud service providers in those geographies. Service providers are ranked on an aggregate or overall performance score, as well as on a lengthy list of specific performance metrics, such as jitter, available bandwidth and packet loss. Users can also customize the performance metrics to suit their particular application or service needs. Armed with this unbiased performance data, companies can select the best provider or set of providers in the locations that matter most to them.
About Apparent Networks
Apparent Networks is the only IT performance management provider that delivers the end-to-end service insight required for today's cloud applications. By experiencing network performance without affecting it, the company's patented path solutions (including PathView Cloud, PathView and AppCritical) assess network readiness, monitor service levels, and diagnose problems otherwise hidden from sight. Leading companies rely on Apparent Networks to assure application delivery and expand their service portfolios with confidence. For more information, visit www.apparentnetworks.com.
Disclaimer
The contents of this advisory are copyright (c) 2009 Apparent Networks Inc. and may be distributed freely as long as proper credit is given.