SGI WebFORCE Server Tuning Guide

Table of Contents

• Scope of this Document
• Server Performance Factors
  • CPU
  • Memory
  • Disk
  • Network
• Understanding Memory Usage
  • Monitoring of Memory Usage
  • Netscape Server Process Basic Memory Requirements
  • DNS named Memory Requirements
  • Networking Kernel Memory Buffer Requirements
  • Using netstat to Determine IRIX Kernel Networking Memory Utilization
• Network (TCP/IP) Kernel Tuning Parameters
  • TCP/IP Buffer Space
  • Network Buffer Size
  • Closing of Idle Connections
  • Maximum Number of Pending Socket Connections
• Tuning for Many Server Processes
  • How Can I Avoid These Process Limits?
• IRIX Operating System Patches
• Modifying Kernel Parameters
• Further Reading

Scope of this Document

This document is intended as a general treatment of WebFORCE server tuning. It covers the most commonly encountered tuning questions concerning HTTP serving, and may not cover special cases (such as specialized applications working in conjunction with an HTTP server). This document is also used as an enhancement to the release notes for WebFORCE HTTP servers -- this way, one can always refer to this page for the latest tuning information.

Currently, WebFORCE servers ship with version 6.2 of the IRIX operating system and the Netscape Enterprise server or the Netscape FastTrack server. The NCSA server (also known as the "OutBox" server) is also shipped as part of the OS bundle on all SGI systems. (See a related document called Tuning Hints for the OutBox Server). This document covers issues for both the IRIX 6.2 and 5.3 operating systems and applies to most web-based services (but most importantly HTTP servers).

Server Performance Factors

For just about any server which provides a network-based service (such as an HTTP server), there are four basic components to the server which must be considered. The

• CPU
  Basic HTTP serving (for example, largely static text and images) is not very CPU-intensive. However, as dynamic elements are added to a Web site (such as APIs, parsed HTML, CGI scripts, etc.), CPU performance can dramatically affect server performance. Tuning for CPU bottlenecks either involves optimizing the application, or using faster (or more) CPUs. If it is suspected that there are application-specific problems or issues, one might consider par as an aid in characterizing or determining bottlenecks. Performance Co-Pilot can also be used to characterize performance issues.
  
  
• Memory
  Memory is vital to web server performance. Insufficient RAM as well as improper tuning can lead to memory swapping (virtual memory paging to disk), which drastically effects web server performance. Ideally, the web server should have enough RAM to handle all necessary applications, network buffers, etc., as well as caching of requested file system data. With many or most or all of the web pages, images, etc. requested by the server cached in the I/O buffer cache (part of main memory reserved by the OS), server performance is greatly improved since it can just get the data directly from memory rather than re-reading it from the hard disks which can introduce large latencies. Some tools to determine memory usage for tuning considerations are discussed below.
  
  
• Network
  Network performance is the most common bottleneck for basic web serving (once CPU and memory considerations have been dealt with). The bandwidth for a specific network interface is fixed. For example, the (theoretical) bandwidths for typical LAN networks range from 10 Mbps (for ethernet /10baseT) up to 100 Mbps (for FDDI or 100baseT). On the lower end of the performance spectrum, serial modems or ISDN connections have performance ranging from 9.6 Kbps to 128 Kbps -- an order of magnitude lower than LANs. Nonetheless, more efficient utilization of the bandwidth at hand can be obtained as a result of TCP/IP tuning, and is treated in some detail below.
  
  
• Disk
  For most situations, hard drive (disk) performance does not affect web server performance (the network is more commonly the bottleneck). However, for some I/O intensive applications, disk striping (spreading a file system across multiple disks and possibly multiple I/O controllers) is a commonly deployed tactic to improve disk I/O.
  

Understanding Memory Usage

As previously mentioned, and it cannot be stressed enough, memory swapping kills web server performance. Memory utilization is probably the single most tunable variable of the four performance factors just discussed. Taking this into consideration, the following paragraphs will outline a few considerations when monitoring/tuning memory use.

Monitoring of Memory Usage

Memory usage on a WebFORCE server can be monitored using any of the following tools:

• sar
• osview
• swap
• gmemusage (available on 6.2)
• ps
• top
• WebMeter or Performance Co-Pilot for Web

Network-specific memory usage can be monitored via netstat -m, as seen below.

Netscape Server Process Basic Memory Requirements

The Netscape Enterprise and FastTrack servers are configured in the following manner:

• Each server listens for requests on a specific port (like 80) for either all interfaces or for a specific IP address. (Note: the server may respond to one or more hostnames in different ways without the use of IP aliasing, making use of "Software Virtual Servers." This works without having to spawn multiple instances of the server, as was necessary for earlier versions of Netscape. See the Netscape Administration server guide for details.)
  
  
• Each server spawns a certain number of processes. These processes manage the load of the system, increasing or decreasing the number of child processes, or threads, as appropriate.
  
  
• Threads are spawned by the processes, and each shares memory with the parent process.

Thus, an active Netscape server with one server, one process, and 4 to 128 threads, will look like this in ps -efl:

The first process, PID 588, is the main server process. It spawned PID 592, which in turn spawned three children, bringing the total active processes to the minimum of 4. Note that the four processes share their space, so the total resident set size (RSS) for this server is 3402 4 KB pages, or approximately 13 MB, and a total of 25 MB has been reserved for all threads belonging to the one process. These are typical sizes for the Netscape 2.0 server.

The total number of active processes (other than the main server process, which usually has a PPID of 1) determine the number of simultaneous connections that can be serviced by your WebFORCE server. (Recall that simultaneous only means actively serving data, and does not represent the total number of users who are currently viewing your pages.)

When configuring the Netscape server, realize that each server process requires approximately 13 MB of memory, and may require as much as 25 MB. This includes space used by all of the threads under that process as well. (See Performance Tuning in the Netscape Admin server guide.) Often, just a single process is sufficient and other times multiple processes (say, 4 or 5) may improve performance. It is best to experiment a bit with this, but keep the memory requirements in mind, and avoid swapping.

DNS named Memory Requirements

named is the Internet domain name server. named is utilized by a Web server to map the originating IP address of each incoming HTTP requests to its fully qualified domain name (e.g. 192.26.51.11 to sgigate.sgi.com). named caches these mapping requests so it can handle the mappings in a timely fashion. Unfortunately, for a very high traffic Web server, named and its internal cache can grow quite large because requests will come literally from all over the world, yielding a high number of unique IP addresses and fully qualified domain names which must be cached.

Turning off domain name lookup for HTTP request logging would significantly reduce or eliminate named memory requirements. The following table highlights the memory requirements for named:

Of course, enabling DNS lookup can reduce the performance of the web server (in some cases significantly). Simply turning on logging (even without DNS lookup) can slow down the performance of the web server by as much as 15% (based on WebSTONE measurements).

Networking Kernel Memory Buffer Requirements

To service as many requests as possible, each process of the Netscape server attempts to process an incoming request and send out the response as fast as possible so it can handle the next incoming request. Unfortunately, the connections to the clients are not infinitely fast, especially if the client connection is over a 14.4 kbps modem. If each server process had to wait for the client to receive all of the data associated with the response, the server could not handle as many connections as it might if it could just hand off the data.

The solution to this problem is built in to the IRIX kernel: rather than having the process send data directly to the client, the data is instead passed into a network buffer. The IRIX kernel takes care of transferring the data from the network buffer to the client, and (at the same time) the server process can go on to handle the next request.

Control for both the size and number of network buffers available. Enough buffers are needed to be able to handle the large number of requests expected without forcing the server processes to wait for an available buffer (such delays can be seen via netstat -m, for example). When sizing these buffers, keep two things in mind:

1. If the server process wants to send something which is larger than the buffer, it pushes out a buffer-sized chunk first, waits until that has fully drained to the client, and then sends out the next chunk. The server process is incapable of handling the next request until it has sent the last chunk of data to the network buffer. Thus, the network buffers should be large enough so that the server processes don't have to spend all of their time breaking off chunks of data to send to the buffers.
   
   
2. At the same time, clients often drop connections leaving large amounts of data sitting in kernel buffers waiting to be drained to the client. This data never actually gets transmitted to the client so it is dead-weight in the kernel until the kernel does buffer cleanup. Thus, buffers should also not be too large.

Obviously, a compromise is necessary. For a typical web server, 95% or more of the content is less than 30 KB. We recommend reducing the maximum size for the TCP send buffers to 30 KB down from the 60 KB default setting. This is controlled by the TCP/IP kernel parameter, tcp_sendspace. (See TCP/IP Tuning below). Each connection is able to reserve and utilize an outgoing (send) buffer of up to 30 KB. There are two basic scenarios for how the Web server utilizes these buffers:

1. HTTP response <= 30 KB: server process sends data to the buffer, and is immediately available to service another request, letting the IRIX kernel drain the buffer to the client and close the connection.
   
   
2. HTTP response > 30 KB: server process sends data up to the buffer limit (30 KB) to the network buffer, waits for the buffer to drain to the client, then sends next batch of data up to the buffer limit. This continues until the last 30 KB or less of data is sent at which time the server process is able to service the next request, letting the IRIX kernel drain the buffer to the client and close the connection.

In one special case, it may be beneficial to decrease the buffer size further. For Ethernets, a situation can arise known as the Ethernet capture effect which can lead to excessive collisions. If excessive collisions are persistent on your Ethernet, try tuning down tcp_sendspace to 12 KB. Note that it may be required to increase the number of buffers (nm_clusters, see below) if this is done.

So, there is a need to have sufficient buffer space in the kernel to support not only the connections which are associated with Netscape server processes but there is also a need to have sufficient space for the buffers for the connections which are draining to the clients. This number is almost impossible to predict because it is based on factors such as:

• Number of outstanding requests
• Transfer throughput rate of Internet connection from server
• Transfer throughput rate of clients
• Size of files attempting to be transferred

All of these factors essentially determine the amount of kernel buffer storage. We want to have enough buffer space so as not to be the bottleneck resource of the web server. If insufficient buffer space were available, a server process would block its sending of data until buffer space became available. This insufficient buffer space situation is denoted as a "request for memory denied" or "request for memory delayed" in netstat -m (or Performance Co-Pilot).

A rough estimate of the kernel memory requirements is 2-4 times the number of server processes (total number of threads) configured for the server times the buffer size (tcp_sendspace) reserved per outgoing buffer (e.g., 30 KB). The 2-4 multiplier takes into consideration overhead of other supporting data structures. It is possible that a server would require more or less than this amount of memory dedicated to networking data structures. The IRIX kernel dynamically allocates chunks of network memory in 4 KB units called clusters. Allocation occurs up to a limit called nm_clusters (another kernel parameter.) See TCP/IP Tuning section below. If memory requests for are denied or delayed, it may be an indication that your Internet connection throughput is insufficient during peak load; the connections may be backing up and clients may be experiencing long wait times.

Using netstat to Determine IRIX Kernel Networking Memory Utilization

The command: