Title: Rackspace Cloud Outage 12/18/09 Document #: CPO121809
Advisory Class: Cloud Provider Outage Impact: Severe
Date Published: 12/21/09 Date of Last Update: 12/21/09
Outage Description
Apparent Network's Cloud Performance Center (www.apparentnetworks.com/cpc) recently confirmed Rackspace experienced connectivity loss which caused an outage in Rackspace Cloud services in their Dallas-Forth Worth data center. The outage occurred on December 18, 2009 beginning at approximately 4:37PM EST and lasted until 5:12PM EST or about 35 minutes. According to Rackspace's cloud system status page, the connectivity loss was due to a network issue in a peering location outside of their Dallas-Forth Worth data center.
During that time, access to systems in the Dallas-Forth Worth data center was unavailable. Businesses utilizing Rackspace Cloud services from this datacenter were affected.
Testing Results
Apparent Networks' Cloud Performance Center (CPC), a free service that offers performance data on leading cloud services providers such as Amazon, Google and GoGrid, detected the connectivity loss on 12/18/09. The Cloud Performance Center provides real-time and historical data to help inform IT teams of network performance issues that could impact effective delivery of cloud services. It provides network path performance metrics -including bandwidth, jitter and latency-between the cloud providers and major cities throughout North America.
The Cloud Performance Center utilizes Apparent Networks' PathView Cloud service to test the performance of cloud service providers. PathView Cloud software was configured to sample path performance to a series of pre-determined targets hosted at Rackspace Cloud's Dallas-Fort Worth every 120 seconds.
PathView Cloud noted Rackspace Cloud outage-related connectivity loss events as follows:
| PathView Cloud - Monitored From |
Connectivity Lost |
Connectivity Restored |
Outage Duration |
| Dallas, TX (Rackspace) |
4:38 PM EST |
5:12 PM EST |
0:34 |
| Newark, NJ(Hosting.com) |
4:37 PM EST |
5:12 PM EST |
0:35 |
| San Francisco, CA (GoGrid) |
4:38 PM EST |
5:13 PM EST |
0:35 |
| VA Data Center (Amazon) |
4:38 PM EST |
5:13 PM EST |
0:36 |
According to Rackspace's cloud system status page, the connectivity loss was due to a network issue in a peering location outside of their Dallas-Forth Worth data center.
Click on the link below to access a report generated by PathView Cloud showing the outage reported from one of the PathView Cloud datacenters.
/uploadDocs/RackSpace_Outage_121809.pdf
Also visit the Cloud Performance Center (www.apparentnetworks.com/cpc) for live data showing the Rackspace service interruption.
Cloud Provider Tested
Rackspace Cloud
Vendor Information, Solutions and Workarounds
Rackspace Cloud System Status page updates for this outage are below: (http://status.rackspacecloud.com/):
As of 3:45 PM CST, we are currently experiencing an issue within our Dallas-/ Fort Worth data center. We are investigating the issue and will post an update momentarily.
UPDATE: As of 3:55 PM CST, to clarify: This is a networking issue affecting Cloud Sites in our DFW data center.
UPDATE: As of 4:05 PM CST, networking engineers are quickly working to address this issue. We should have a resolution shortly.
UPDATE: As of 4:14 PM CST, another point of clarification: This is not a power issue in DFW, all power is confirmed up and has not been down. This is a networking issue.
Later replaced by:
Starting at 3:37 PM CST this afternoon, we experienced a network issue at a peering location outside of our data center facilities. The network has been back up since 4:12 PM CST.
About PathView Cloud
PathView Cloud is a hosted network management tool that measures the performance of complete network paths from source to destination, including segments that pass through service providers' and carriers' networks. It enables IT teams and network managers to assess, troubleshoot and continuously monitor thousands of network paths simultaneously. A free version of the tool allowing users to monitor and test five network paths simultaneously is available at www.apparentnetworks.com.
About the Cloud Performance Center
The Cloud Performance Center (www.apparentnetworks.com/cpc) provides real-time, region-by-region service delivery performance scores for cloud providers. Visitors can view an interactive map of the North American market and select locations that are relevant to their businesses. Based on their selection, the CPC provides a detailed Cloud Provider Scorecard for the major cloud service providers in those geographies. Service providers are ranked on an aggregate or overall performance score, as well as on a lengthy list of specific performance metrics, such as jitter, available bandwidth and packet loss. Users can also customize the performance metrics to suit their particular application or service needs. Armed with this unbiased performance data, companies can select the best provider or set of providers in the locations that matter most to them.
About Apparent Networks
Apparent Networks is the only IT performance management provider that delivers the end-to-end service insight required for today's cloud applications. By experiencing network performance without affecting it, the company's patented path solutions (including PathView Cloud, PathView and AppCritical) assess network readiness, monitor service levels, and diagnose problems otherwise hidden from sight. Leading companies rely on Apparent Networks to assure application delivery and expand their service portfolios with confidence. For more information, visit www.apparentnetworks.com.
Disclaimer
The contents of this advisory are copyright (c) 2009 Apparent Networks Inc. and may be distributed freely as long as proper credit is given.