Title: Symantec Endpoint Protection 11v5 Document #: PD111809
Advisory Class: Performance Degradation Impact: Severe
Date Published: 11/18/09 Date of Last Update: 11/18/09
Performance Degradation Description
PathView Cloud network performance system (www.apparentnetworks.com) recently uncovered a parasitic network performance loss at the client side of the network path when the client was running Symantec Endpoint Protection ("SEP") 11v5 with Network Threat Protection enabled. SEP 11v5 combines end point security, messaging security and backup/recovery into a single client-installed agent.
The problem was uncovered when PathView Cloud's diagnostic tests detected that test traffic was being clamped at regular intervals on the host system. Upon query, the host system identified a list of drivers which were considered potential sources of the problem. SEP 11v5 was among the suspects and therefore another test was run with SEP disabled. PathView Cloud then detected no problem. Upon additional testing, PathView Cloud results showed that severe packet loss and throughput disruption occurred at clients running SEP 11v5 as well. To confirm, Apparent Networks engineers performed independent tests offline-the results and methodology of those tests are outlined in this alert.
While it's expected that any additional layers of added-value management (be it security, data protection, monitoring, etc) can often exact some level of performance impact, the performance impact of SEP can be up to 70% of the available bandwidth. This means that SEP effectively caps TCP throughput to approximately 250 Mbps on a 1 GB network.
Performance Results
In general, at low to mid network throughput levels (namely at or around 100 Mbits/sec), SEP 11v5 had negligible adverse affects on either TCP or UDP performance. However, as the GigE network was pushed to higher throughputs, SEP did begin to impact throughput performance on both TPC and UDP and introduce significant packet loss on UDP.
See table below:
|
Network Configuration and Test |
Symantec Endpoint Protection 11v5 Base Configuration (Malware Only) |
Symantec Endpoint Protection 11v5 with Network Threat Protection (NTP) Enabled |
|
|
Impact |
Impact |
|
1GBit TCP Throughput |
None |
Up to 70% |
|
1GBit UDP Throughput |
None |
Up to 30% |
|
1GBit UDP Packet Loss |
None |
Up to 45% |
|
100MBit TCP Throughput |
None |
0% |
|
100MBit UDP Throughput |
None |
0% |
|
100MBit UDP Packet Loss |
None |
0% |
Vendor Information, Solutions and Workarounds
No known workaround is available, except to disable the Network Threat Protection.
Test Description
To verify the problem, a simple Gigabit local area network was constructed consisting of three nodes connected via GigE switch. A Linux server, a WinXP Pro SP3 client running SEP 11v5, and an identical WinXP Pro SP3 client without SEP. In order to remove any possibility that the PathView Cloud measurement point was somehow affecting the results, it was turned off and an open source packet-flooding tool, iPerf (http://en.wikipedia.org/wiki/Iperf) was used to generate load and measure both TCP and UDP performance. See Appendix A for more details about testing configuration and methodology.
In particular, SEP effectively capped TCP throughput to approximately 250Mbits/sec on a 1GBit network. This was validated by establishing baseline performance without SEP installed and comparing the results with SEP installed in Network Threat Protection (NTP) enabled and disabled. With SEP NTP disabled, performance returned to near baseline level. When re-enabled, the performance impact returned. In addition, at higher throughput levels, UDP performance was equally impacted and significant packet loss (up to 45%) was introduced.
Click here for full testing results.
About PathView Cloud
PathView Cloud is a hosted network management tool that measures the performance of complete network paths from source to destination, including segments that pass through service providers' and carriers' networks. It enables IT teams and network managers to assess, troubleshoot and continuously monitor thousands of network paths simultaneously. A free version of the tool allowing users to monitor and test five network paths simultaneously is available at www.apparentnetworks.com.
About Apparent Networks
Apparent Networks is the only IT performance management provider that delivers the end-to-end service insight required for today's cloud applications. By experiencing network performance without affecting it, the company's patented path solutions (including PathView Cloud, PathView and AppCritical) assess network readiness, monitor service levels, and diagnose problems otherwise hidden from sight. Leading companies rely on Apparent Networks to assure application delivery and expand their service portfolios with confidence. For more information, visit www.apparentnetworks.com.
Disclaimer
The contents of this advisory are copyright (c) 2009 Apparent Networks Inc. and may be distributed freely as long as proper credit is given.