My primary contact at the site phoned me to let me know that part of our process, which took 20 minutes last quarter, was still running after two hours. We immediately established a remote session to review the issue. In the past, we have experienced some issues with individual PCs processing at slower speeds due to poor network infrastructure, but more recently this firm invested in better network hardware to support their rapidly growing business.
We play a limited role for this client and focus on their quarterly reporting and billing systems. Though we are IT experts, it is not our responsibility to oversee and manage their IT infrastructure; however, at quarter end, a processing issue where systems are operating at a fraction of their normal speed rapidly becomes our problem.
I am very familiar with the bottlenecks that can slow Axys performance. The most critical of these is network speed. 100MB Ethernet (full-duplex) is an older standard, and we still find it in limited use at many offices. Gigabit Ethernet (full-duplex) is the current standard that should be in use by nearly all investment firms. Theoretically, gigabit is ten times faster, but you won’t see that in practice. You actually get six to seven times the performance of 100MB Ethernet with gigabit Ethernet over decent cabling.
My technical contact at the firm first assured me that all of the systems were connected to gigabit switches, and nothing had changed since last quarter. We discussed the wiring of the network in detail and I was eventually able to find out that they had added a new Dell switch in the server room, but assured again that it was a gigabit switch. I asked them to double-check the switch and let me know the model.
Though most of our own experiences purchasing equipment from Dell are good, Dell isn’t perfect. Perhaps the Dell sales rep didn’t know one gigabit switch from another. Our client thought they had purchased a managed gigabit switch where all ports were gigabit. They had, in fact, bought and installed a 100MB managed switch with two gigabit uplink ports. Further discussion revealed that the gigabit uplink ports were not being used either.
For those not familiar with network nomenclature, the primary switch to which all of your other switches, routers and servers are connected is considered your network backbone. It is a best practice to implement a backbone that has throughput greater than or equal to that of the devices connected to it.
When two network devices auto-negotiate to communicate with each other the maximum speed is usually the highest speed commonly supported by both devices. Other environment specific issues, such as the quality of cabling between two devices, can further degrade the speed at which two network devices communicate.
In this particular case, our client had unknowingly installed a switch that was forcing all of their servers with gigabit Ethernet to communicate with the rest of the network using 100MB Ethernet instead of gigabit. Users that were not connected directly to the backbone had a gigabit connection though another switch, and assumed that everything was fine.
The short-term fix for this client was to connect their file server hosting Axys to one gigabit uplink switch and use the other uplink to connect to their larger gigabit switches. They also called Dell and had them send the right switch overnight, which they installed today.
Having an up-to-date network diagram is a best practice. If you don’t have one, have your technical staff or IT provider create and maintain a network diagram documenting your systems, so you can proactively manage problems with network performance and reliability.
About the Author:
Kevin Shea is President of InfoSystems Integrated, Inc. (ISI); ISI provides a wide variety of outsourced IT solutions to investment advisors nationwide. For details, please visit isitc.com or contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@isitc.com.