Title: Hosting.com Datacenter Outage 6/1/10 Document #: CPO060110
Advisory Class: Cloud Provider Outage Impact: Severe
Date Published: 6/2/10 Date of Last Update: 6/2/10
Outage Description
Apparent Network's Cloud Performance Center (www.apparentnetworks.com/cpc) recently confirmed Hosting.com experienced connectivity loss which caused an outage in their Newark, New Jersey data center. The outage occurred on June 1, 2010 beginning at approximately 6:45PM and ending 8:29PM EDT. There were intermittent periods of connectivity with high data packet loss between those times, and the number of connectivity loss events and duration varied slightly by location. According to Hosting.com's Twitter feed (http://twitter.com/HDCOps), "One dedicated switch failed. It failed over to a second switch which crashed as well."
During that time, access to systems in the NJ data center was severely degraded and often unavailable. Businesses utilizing Hosting.com's services from this datacenter were affected.
Testing Results
Apparent Networks' Cloud Performance Center (CPC), a free service that offers performance data on leading cloud services providers such as Amazon, Google and GoGrid, detected the connectivity loss on 6/1/10. The Cloud Performance Center provides real-time and historical data to help inform IT teams of network performance issues that could impact effective delivery of cloud services. It provides network path performance metrics -including bandwidth, jitter and latency-between the cloud providers and major cities throughout North America.
The Cloud Performance Center utilizes Apparent Networks' PathView Cloud service to test the performance of cloud service providers. PathView Cloud software was configured to sample path performance to a series of pre-determined targets hosted at Hosting.com's New Jersey data center every 60 seconds.
PathView Cloud noted Hosting.com's outage-related connectivity loss events as follows:
|
PathView Cloud Monitoring Location |
Provider |
Initial Connectivity Lost Event |
Full Connectivity Restored |
# of Loss Events |
Total Connectivity Loss Time* |
Outage Duration** |
|
Atlanta, GA |
Peer 1 |
18:45 EDT |
20:29 EDT |
4 |
1:07 |
1:44 |
|
Austin, TX |
Hostway |
18:45 EDT |
20:29 EDT |
3 |
1:10 |
1:44 |
|
Dallas, TX |
Rackspace |
18:44 EDT |
20:27 EDT |
6 |
1:15 |
1:43 |
|
Miami, FL |
Peer 1 |
18:45 EDT |
20:29 EDT |
2 |
1:09 |
1:44 |
|
San Francisco, CA |
GoGrid |
18:45 EDT |
20:28 EDT |
4 |
1:11 |
1:43 |
|
San Jose, CA |
Verio |
18:46 EDT |
20:29 EDT |
7 |
1:16 |
1:43 |
|
Sterling / Dulles, VA |
Verio |
18:46 EDT |
20:29 EDT |
5 |
1:12 |
1:43 |
|
Virginia |
Amazon AWS |
18:46 EDT |
20:29 EDT |
6 |
1:13 |
1:43 |
|
Washington, DC |
Rackspace |
18:45 EDT |
20:29 EDT |
5 |
1:12 |
1:44 |
* Excludes intermittent periods when connectivity was partially restored
** Time from first connectivity loss until full connectivity was restored
According to Hosting.com's operations Twitter feed, their team acknowledged the connectivity loss due to Cisco 6509 switch failures in both a primary and backup switch. The cause of the failure was traced to a software bug in the switches.
Click on the link below to access a report generated by PathView Cloud showing the outage reported from one of the PathView Cloud datacenters.
/uploadDocs/6_1_10_Hosting_Outage_From_Dallas_PathView_Report.pdf
Also visit the Cloud Performance Center (www.apparentnetworks.com/cpc) for data showing the Hosting.com service interruption.
Cloud Providers Tested
Hosting.com
Vendor Information, Solutions and Workarounds
Vendor information about this outage can be found at http://twitter.com/HDCOps.
About PathView Cloud
PathView Cloud is a hosted network management tool that measures the performance of complete network paths from source to destination, including segments that pass through service providers' and carriers' networks. It enables IT teams and network managers to assess, troubleshoot and continuously monitor thousands of network paths simultaneously. A free version of the tool allowing users to monitor and test five network paths simultaneously is available at www.apparentnetworks.com.
About the Cloud Performance Center
The Cloud Performance Center (www.apparentnetworks.com/cpc) provides real-time, region-by-region service delivery performance scores for cloud providers. Visitors can view an interactive map of the North American market and select locations that are relevant to their businesses. Based on their selection, the CPC provides a detailed Cloud Provider Scorecard for the major cloud service providers in those geographies. Service providers are ranked on an aggregate or overall performance score, as well as on a lengthy list of specific performance metrics, such as jitter, available bandwidth and packet loss. Users can also customize the performance metrics to suit their particular application or service needs. Armed with this unbiased performance data, companies can select the best provider or set of providers in the locations that matter most to them.
About Apparent Networks
Apparent Networks is the only IT performance management provider that delivers the end-to-end service insight required for today's cloud applications. By experiencing network performance without affecting it, the company's patented path solutions (including PathView Cloud, PathView and AppCritical) assess network readiness, monitor service levels, and diagnose problems otherwise hidden from sight. Leading companies rely on Apparent Networks to assure application delivery and expand their service portfolios with confidence. For more information, visit www.apparentnetworks.com.
Disclaimer
The contents of this advisory are copyright (c) 2010 Apparent Networks Inc. and may be distributed freely as long as proper credit is given.