Analyzing and Optimizing Performance

A bottleneck can be caused by a problem with a component or by the inherent limitations of the component. An example might be a network that has 10/100-Mbps hubs and switches and computers with 10/100-Mbps network cards, but only Category 3 twisted-pair in the cabling infrastructure. Since Category 3 twisted-pair cable does not support a high data transfer rate, the cable is the bottleneck that slows down the network.

It is doubtful that all network components can be fine-tuned so precisely that they all operate at exactly the same speed. Nonetheless, optimizing performance involves finding the bottlenecks and upgrading, reconfiguring, or replacing the components to bring their performance up to or above the level of the rest of the network components. This process almost inevitably creates a new bottleneck elsewhere, but with good planning the new bottleneck should be less restrictive than the old one.

To be able to diagnose which network server component is causing a system bottleneck, there must be performance-monitoring software tools specific to the network server operating system. Most network server operating systems have at least a basic performance-monitoring software package included with the operating system. Third-party vendors also have performance-monitoring software tools available for most network server operating systems.

Examples of performance-monitoring software tools for various network server operating systems include the following:

• Performance– For Microsoft Windows 2000 Server
• sar, iostat, vmstat, ps– For UNIX/Linux

The primary performance-monitoring tool for Microsoft’s Windows 2000 Server is just called Performance, which is shown in Figure. Performance can monitor nearly all hardware and software components on a Windows 2000 server. The monitoring can take the form of creating a real-time chart of resource utilization, or it can take the form of a log file for later analysis. The log file can be analyzed or exported in a format that can be used by other software, such as a standard statistics software package for analysis or Microsoft Excel to generate charts or statistics. Windows 2000 Server Tool (Performance) is a snap-in to the Microsoft Management Console. The NT 4.0 performance monitor appearance is slightly different, but the basic functionality is almost identical.

The various versions of the UNIX/Linux operating systems have command-line utilities that can be used to monitor performance of the UNIX/Linux network server. The primary tools aresar,vmstat,iostat, andps. The flags used by these commands can vary among the different versions of UNIX/Linux. Use the UNIX/Linuxmancommand to get specifics about the use of these commands. The information displayed by themancommand also tells how to interpret the output generated by the command.

The UNIX/Linux System Activity Reporter, orsarcommand, collects and reports system activity, which includes processor utilization, buffer utilization, and so forth for the UNIX/Linux network server.

The UNIX/Linux Virtual Memory Statistics, orvmstatcommand, provides information about the virtual memory system on the UNIX/Linux network server.

The UNIX/Linux Input/Output Statistics, oriostatcommand, provides information about disk subsystem input/output on the UNIX/Linux network server.

The UNIX/Linux Processes (ps) command lists all the processes that are running on the UNIX/Linux network server.

Processor Bottlenecks
Use the performance-monitoring tools available for the network server operating system to monitor the utilization of the processor(s) in the network server. The two counters to watch most closely are the percent processor utilization and the processor queue length. The percent processor utilization is the percentage of the time that the processor is busy. If the processor utilization percentage is constantly greater than 70%, the processor is the bottleneck in the network server. The processor queue length is the number of processes waiting for the processor. If the processor queue length is constantly greater than 2, the processor is the bottleneck in the network server. Alleviate the processor bottleneck by doing any of the following:

• Upgrade the existing processor to a faster processor.
• Add an additional processor to the network server.
• Move applications or services from this network server to another network server.

Memory Bottlenecks
On network servers with operating systems that do not support virtual memory, memory bottlenecks are rather easy to detect. Memory shortages manifest themselves on these systems as the inability to load additional software or the lack of “buffers”, which is the temporary storage between the system and its disk drives. The buffers are easily detected using standard server management software. The solution to a memory shortage is to add more memory.

On network servers with operating systems that do support virtual memory, several indicators or counters should be monitored. The first counter is the amount of available memory, or memory that is not being used, after all the network server software has been loaded. To totally prevent paging or swapping of memory pages to the disk, just have more actual or physical memory than is needed by all the software running on the network server.

The other counter that should be monitored to check for memory bottlenecks is the number of pages that are read from or written to the paging file on the disk in one second. The number of pages per second that is acceptable depends on the speed of the processor. Generally the number should be less than 2,000 per second. Heavy paging often manifests itself as high disk utilization, because the memory pages are written to the disk when the memory is needed by another program.

Another indicator of lack of memory is the number of page faults generated per second. A page fault is the condition encountered when there is an attempt to access a program page, the page is no longer in memory, but it must be retrieved from the disk. A page fault problem is indicated if the number of page faults per second is more than double the number of page faults recorded on the server baseline measurements taken under a normal workload.

To alleviate a memory bottleneck, add more memory to the network server.

Paging performance can often be improved by placing the paging file, or the swap file, on the fastest disk in the network server. Also, this disk must not contain the network server operating system or the data files used by the major application running on the network server.

Disk Subsystem Bottlenecks
The disk subsystem can cause system bottlenecks, especially when the network server has the role of a network server requiring a fast disk subsystem as in a database server.

The disk subsystem counters that should be monitored are disk subsystem utilization and disk queue length. A disk subsystem utilization percentage should not be constantly greater than 90%. This very high disk subsystem utilization rate indicates that the disk subsystem is the bottleneck.

The disk subsystem queue length indicates how many items are waiting to be written to the disk subsystem. A disk subsystem queue length that is constantly greater than two indicates that the disk sub-system is too slow. Several things might improve disk subsystem performance:

• Defragment the disk drive. Having fragmented files on the disk drive can drastically reduce disk performance.
• Move the network server operating system paging file to the fastest disk drive in the network server and off of the disk drive where the network server operating system is located.
• Move the application causing high disk utilization to another network server.
• Replace the disk controller with a disk controller that has a built-in memory cache.
• Replace the disk drives with disk drives with a higher rotational speed and, therefore, faster access time.
• Replace the entire disk subsystem, which includes the controller and disk drives, with a faster disk subsystem.

Network Subsystem Bottlenecks
Network subsystem bottlenecks are perhaps the most difficult to detect and solve. Typical performance-monitoring software can supply counts of packets sent and received and little else. To analyze network performance, a network analyzer for hardware or software is needed. A network analyzer can actually capture packets on the network for later analysis. The percent network utilization, a very valuable indicator of network performance, can be determined easily. A network analyzer can also determine the type of network packets (for instance, broadcasts, multicasts, or unicasts) and the protocols that are consuming the most network bandwidth.

A network utilization percentage that is constantly greater than 65% is generally considered to be high utilization and can constitute a bottleneck.

Solutions to a network subsystem bottleneck are generally expensive and include the following:

• Replace the network subsystem with a faster network sub-system. Replacing a 10 Mbps Ethernet network with a 100Mbps Fast Ethernet network will likely eliminate the network subsystem as a bottleneck.
• Replace Ethernet hubs with Ethernet switches. In an Ethernet hub, all the available network bandwidth is shared. Using an Ethernet switch instead of a hub provides each port a full amount of bandwidth, which means there is no sharing of the bandwidth.
• If the bottleneck appears to be caused by too many clients trying to get to the network server, consider using adapter teaming to increase the network bandwidth available to the network server. This requires multiple NICs to be installed in the network server and a Fast Ethernet switch that supports bandwidth aggregation to be installed.
• Eliminate unnecessary protocols from the network. When multiple protocols are installed on a network server, announcements sent from the network server are sent once for each installed protocol. This increases network traffic.

The following are a couple of well-known network monitoring and analysis tools:

• Microsoft Network Monitor
• Novell LANalyzer for Windows