Multi-User Performance Testing in a Client/Server Environment

By Bruce Huang

'This article discusses an often overlooked but critical ingredient of developing a client/server environment--the need to conduct a multi-user performance test before putting the client/server system into production. The author cites cases in which failure to implement this test caused significant problems and expenses after rollout.'

Client/server computing growth continues to accelerate at an enormous rate, bringing with it complex technical challenges to the entire systems management process. Companies start client/server system development projects to provide better-quality, efficient, and user-friendly systems. However, once their systems roll out, many companies find that their technical support desks are swamped with performance problems and availability complaints. Why does this happen? What goes wrong?

A decade of experience with client/server performance issues has taught me that the common mistake is failing to conduct a multi-user performance test prior to the product rollout. Implementing a multi-user performance test is an essential step toward minimizing potential business risks, and it should be built into all client/server system development projects.

A multi-user performance test validates the entire client/server system under a stress environment in which multiple client workstations simultaneously send different requests to the server to simulate an actual production workload. This time-consuming test requires skills and resources, but it is essential to achieve the following key management objectives:
 * Satisfying end users
 * Improving overall user performance
 * Maximizing return on investment

Multi-User Performance Test Components
The three key components of a multi-user performance test are to validate:
 * Repetitive end-user scenario
 * Product integration
 * End user performance/capacity

Repetitive End-User Scenario Validation
With an automated, non-intrusive client/server test tool such as the IBM Solutions Evaluation Tool (SET), you can drive an end-user scenario repeatedly to test for the following potential problems:
 * Interrupt conflicts
 * Memory address conflicts
 * Memory leakage
 * Inefficient system resources allocation and de-allocation
 * Excessive dynamic resources allocation (common in graphical user interface [GUI] applications)

A recent engagement with a major life and casualty insurance company illustrates how serious these problems can be. The company was unable to duplicate a field-reported error until we tested the application in the manner just described. We found that, under certain circumstances, the system always hung at the 54th invocation. It was very expensive to correct this type of problem in a production environment. If the problem was discovered before the rollout, the correction cost could have been 80 percent less.

Product Integration Validation
A complex client/server system runs more than one application. For example, an office system for an insurance company might include an insurance claims application, a word processor, a spreadsheet, a calendar, and an electronic mail package. Will the insurance claims application cause any systems conflicts or capacity or performance problems, running concurrently on the same client workstation?

Only testing can tell prior to putting the system into production. In one case where the client skipped multi-user testing, we encountered numerous system and application traps while running multiple applications concurrently.

We have also found that the recommended network optimization parameter value for an application can be a bottleneck for other applications executing concurrently.

End-User Performance/Capacity Validation
Key factors dictating client/server system capacity are:
 * System and network resources availability
 * End-user perceived response time

In complex client/server environments, having a fast CPU with the maximum installable amount of system memory and disk storage does not always translate into the best system performance and capacity. Sometimes it can actually become a bottleneck, due to system overhead required for excessive system resources management. It is more efficient to determine the system requirements with real data prior to spending a fortune on unnecessary hardware. A multi-user performance test provides an in-depth view of the systems characteristics, and the data collected can be used to determine:
 * CPU, system memory, and disk storage requirements
 * Server placement and network topology requirements
 * Number of users per server, and therefore the number of servers required
 * An optimal network and system configuration for maximizing performance
 * End-user perceived response time and server transaction rate

Often we concentrate only on understanding the capacity of a server system, ignoring potential performance issues on client workstations. As the complexity of workstation programming environments grows, the possibility of finding performance bottlenecks on the client workstation increases. A multi-user performance test exercises the entire client/server system in an end-to-end environment, validating the server, the client workstation, network infrastructure, system software, and end-user applications.

The performance benefits gained from performance testing can be great. For instance, in an engagement with a major information processing company, we found a client workstation performance bottleneck after excessively allocating and de-allocating unnecessary system resources. Correcting this error with a change in the program implementation resulted in a 75 percent performance improvement in end-user response time.

Multi-User Performance Test Planning
Careful planning is the key to a successful multi-user performance test project. A comprehensive test plan is a document that describes the results of the planning. The three major components of a comprehensive test plan are:
 * 1) Test objective and criteria planning. The first step toward a successful performance test project is to clearly understand your:
 * Overall test environment
 * Test objectives and methodology
 * Production workload
 * Success criteria for test entry and exit
 * Project deliverables


 * 1) Test lab facility planning. Rapid changes in computer technology make it difficult for any test lab to have a comprehensive facility set up to respond to every test requirement. Therefore, it is essential to establish an alternate plan for acquiring all the necessary hardware, software, tools, and skills requirements.
 * 2) Test implementation planning. Performance testing is a time-consuming process. All specific work items, daily schedules, and individual responsibilities must be well defined prior to the test.

A Proven Methodology
Figure 1 shows a proven methodology for developing the comprehensive multi-user performance test plan just described. This methodology was developed by the Solutions Validation Lab, an IBM integrated services offering to assist customers in all phases of client/server performance test and analysis.



'''Figure 1. Methodology for Developing a Multi-User Performance Test Plan'''