A Perfect Storm

Sam Wang, territory manager for Network Instruments, discusses how one clients’ use of retrospective network analysis captured and diagnosed the perfect network storm.

When tasked with providing examples of Observer in action along with the GigaStor, the recorder appliance that enables users to perform Retrospective Network Analysis (RNA), nothing beats a real-life experience.

Inevitably in the sales cycle, many of my potential clients want to “kick the tires” on a GigaStor before making a capital investment and to see a practical demonstration of its capabilities. It was during one of these sessions that I encountered the “Perfect Storm” i.e. the network behaved so badly that the client saw the value of the GigaStor almost immediately and thus inducted it as a mission critical tool to be used on a daily basis.

The GigaStor was installed on the client’s production network after-hours around 5pm or so. They experienced a major slowdown of their WAN shortly thereafter and I was called back the following morning to help sell the concept of RNA. By “rewinding” back to the last evening’s events, we were very quickly able to determine the following best practice issues and network errors:

1. SNMP traffic: Their network monitoring software was set to poll its SNMP devices in ten second intervals. Due to the heavy polling frequencies, the amount of sheer monitoring traffic was fairly large. We reduced the polling period to 10 minutes and saw a significant decrease in unnecessary traffic.
2. Patch Management: Although it was after business hours in Chicago where we were, the client has offices in the west coast. They had scheduled their Microsoft patch management to start at 5pm CST but had not remembered that their California offices were two hours behind. As a result, the WAN links were completely bogged down making the California offices completely unproductive.
3. Acceptable Use: We discovered several users absorbing more than their fair share of bandwidth on the Internet.
4. Voice over IP: The GigaStor was able to capture a communication error in their VoIP equipment during a call set-up. We shared it with the VoIP manufacturer and discovered it was a known bug that required a firmware upgrade.

Suffice to say, once we determined and resolved the noted problem areas, the client’s WAN resumed to operate normally. It is important to point out that this was a very typical network with very typical network problems. If acting individually, most of our anomalies would go unnoticed. However, by combining them together, they were able to form our “Perfect Storm”.

And yes, more importantly for me, I got the sale.

Network Resources
RNA White Paper from Network Instruments
GigaStor Product Page