| Item | Description | LoadRunner | OpenSTA |
| Protocols | The communication protocols that can be captured, manipulated and replayed by the tool. | Many supported. Protocols are charged per item. Has a multi-protocol recording feature. | HTTP 1.0 / 1.1 / HTTPS (SSL) only. |
| Playback functions | Replaying of the script and script debugging facilities. | Extended logging supports view of parameter values and Server messages. Also view and comparison with ‘recorded’ version of web page view and client response messages. Debugging facilities in script generator, step and breakpoints. | Similar playback facilities, but no integrated comparison function. The debugging functions are in the controller, including set break points and single stepping. |
| Scripting language | The medium used to represent the captured protocol data and manipulate the data for play-back. | Called TSL, it uses standard syntax for “C” & allows C function libraries to be added. Has extensive customised functions for the different protocols supported by the tool. | Called SCL, it uses a proprietary, “BASIC” like language that has special automation scriptingfacilities. Is limited in available functions, such as string manipulation and supports direct DOM addressing. |
| Extensibility | The ability to increase the functionality of the tool. | Additional TSL or “C” function libraries, limited to functional capabilities of the tool. | SCL script modules can be defined in â��Includeâ�� files. Open Source therefore new tool functionality can be added using C++. |
| Scripting Interface | The interfaces supplied by the tool application for the purpose of script editing. | Captures in several modes, high level context based and low level HTTP view. Has both a graphical tree structure & a script view. Script view has function sensitive entry. | Has low-level HTTP protocol view and provides graphical tree representation of the �Document Object Model� (DOM) structure. Viewable captured HTML rendering and addressable server-header table. Some language sensitive, syntax colour coding functionality. |
| Correlation | The task of substituting values in dynamic data to enable successful playback. | Automated correlation faculties. Including during recording, after recording and comparing recordings with playback results. Not available for all modes of capture. | Manual correlation using graphical interactive �DOM� structure. Facility to automatically generate script code to aid variable substitution. |
| Cookie Management | Detection, recording and playback of HTTP cookies. Both tools need additional code to manage JavaScript generated cookies. | HTTP header cookies are managed automatically and can be manipulated manually if required. | HTTP header cookies are managed automatically and can be manipulated manually if required. |
| Parameterisation | Automatically changing dynamic data values for more accurate emulation of real users. Often essential for session management. | Extensive facilities for data entry including wizard interface to DB interrogation. No standard function to lock data sources and maintain uniqueness of concurrently accessed data across distributed tests. | Extensive facilities for data entry including wizard interface to automatically generate test data. Standard functions for sequential, random and pseudo-random data-file access. Has standard common locking facilities for maintaining uniqueness of parameters for an individual load injector or across all injectors on a distributed test. |
| Controller | Application that manages and conducts a test. | Facilities for real-time monitoring. Automatic scenario generation. Individual control of vusers, scripts and groups of scripts. Scheduling, percentage runs & iterations. | Facilities for real-time monitoring. Simple drag & drop multi-scenario test configuration supporting modular scripting and addition of new scenarios/Vus at run-time. No automatic scenario generation. Allows test-wide and user-specific HTTP monitoring and debugging while multi-user load in progress. |
| Monitoring | Resource usage information is captured during execution. It can be shown during execution and used to build performance reports. | Many supported. On-line graphics during execution. Supplied with Apache, Netscape & IIS, other monitors are charged per item. Results used for reporting. New facility to allow remote users to monitor real-time results via a browser based interface. Note: To monitor through a firewall requires TCPIP access through a specific port. Future versions of LoadRunner should use HTTP messaging & avoid this issue. | Supports Integrated real-time, graph view of Windows NT/2000 Performance (Perfmon) & SNMP collectors. Various measurements of test progress including VU-specific, custom status and activity information. Web-relay allows monitors to run on remote machines beyond firewall. On-line graphics during execution, & monitored results used for reporting. |
| Distributed tests | The ability to distribute the generation of load across multiple load-injector machines. | Supports multiple load-injectors managed by a single controller. | Supports multiple load-injectors managed by a single controller. Uses TCP/IP if on the same network or the Web-relay feature uses HTTP to control load-injectors located within remote DMZs. |
| IP Spoofing | The ability to emulate the behaviour of different IP addresses accessing a system. Particularly useful with Load balancing systems. | Supports IP spoofing including automated router updates for IP forwarding. | No inbuilt features to enable IP spoofing. |
| WAN/LAN emulation | The ability to emulate the behaviour of different network infrastructures during a test. | New feature added to version 7.6. Allows the emulation of latency, packet loss, link faults & dynamic routing effects over the LAN used in a test. Requires a special license. | No inbuilt features to enable WAN/LAN emulation. |
| Caching | The ability to emulate the caching of pages as performed by a web browser. | Can control browser cache emulation during playback and control setting for each individual vuser. | No specific faculties, although can be emulated with simple script code. |
| User connection speed emulation | The ability to emulate the different network speeds that can be used by real users. | Can emulate different network speeds during playback | No inbuilt features to emulate user-connection speed emulation. |
| Distributed/Remote load generation | To generate large loads additional load generator machines are required. These machines must be centrally controlled. | Can control multiple load generators and collect results. | Can control multiple load generators and collect results. |
| | Can control load generators located at remote networks (through a firewall) if required using HTTP ports. | Can control load generators located at remote networks (through a firewall) if required using HTTP ports. |
| Reporting & analysis | The facilities to examine and investigate the results of a test including timers and monitored resources and display the results in graphical format. | Sophisticated & large range of charts and graphs with overlay facilities. -Automatic report generation into MS-Word. Analyser is a separate application that can be distributed to users. | Simple charts and graphs sufficient for analysing key load related statistics and resource usage monitors. Resource usage monitors supports graph overlays. Can be exported to Microsoft Excel. No license restrictions on OpenSTA distribution thus stats can be viewed by any user with access. -Free tools and excel macros available through public user-forums. |
| Scalability | The ability of the tool to generate numbers of virtual users and the corresponding resource usage. Actual resource use depends on the number, size and complexity of the scripts. | Resource limitations are number of threads and RAM. Approx 1 vuser per ½ Mb RAM for NT/W2K. Windows 95, 98 & Unix are less efficient. Max. approx 1,500 VUs per PC. | Main resource used is RAM. Tested for simple ASP pages, reaching up to 3,000 users on load generator of 1Gb RAM on a single P4 processor & W2K. Unconfirmed report of a limit of 1664 Vus per Win2K machine for complex scripts. Suspected thread limitation. No licensing limitations. |
| Initial Costs | Purchase costs of software and licenses, excluding upgrades or support. | From £16,000 for basic package with no virtual users. Additional charge for each protocol and monitored resource and for virtual users. | FREE� download from SourceForge via www.OpenSTA.org. Downloads available: Previous versions; Automatic installer or current source code (with simple build instructions for MS C++ Visual Studio 6). |
| Costs of Virtual users | Most commercial tools charge on the basis of the number of virtual user available. Extra hardware is an additional cost. | Prices vary but for the purpose of expectation: additional vusers cost from £10K for 25 VUs to £66K for 1000 VUs. Also temporary VU days at £3.50 each per day (min 1000). This is not a quoted price. | FREE. No licensing limitations. |
| Support & consultancy | The support services available for the tool and some costs. | By M.I., approx 1/5 of initial cost per annum. Includes upgrades. MI & their partner companies also provide consultancy (including etest associates). | Various independent resources. etest associates charged from £50 per incident for remote tech support. Consultancy rates available on request.Numerous online resources including web and e-mail forums. Upgrades are free (approx every 3-6 months) |
| Training | The training services available for the tool. | MI has a range of courses from around £400 per day per person. Many partners also provide training. | Specialist companies provide tailored training, prices vary. |
| System requirements | The operating systems required to host the tool. (Not the OS of the system under test). | MS windows 2000, NT4 (sp6a),XP-Pro (also 95 & 98 cannot run multi-threaded) - Load generators also support limited generator functionality on: Unix: HP, Solaris, Linux | MS windows 2000, NT4 (sp5+), XP-Pro. |
| | | |
| Hardware requirements. | The hardware requirements to host the tool. (Not the hardware of the system under test). | Min: Pentium 350 & 128M - Load generators: Pentium 1GHz & 1 MB per VU. | Min : Pentium 200 & 80MB RAM. Pref: Pentium 500MHz+ & 128MB+ RAM. |
| Access to source code | Availability of the source code of the tool itself. | Unavailable. | Open Source GNU public license - ‘C++’ Language. |
| A users opinion. | Opinion of Internet performance engineers who have experience with both tools. | Has a very user friendly user interface and fantastic monitoring & results analysis. Automatic correlation and improved script recording facilities can aid productivity. Very flexible scripting functions & great help documents. Complexity of options and layout of controller are drawbacks. The availability (at a cost) of numerous protocols for recording is a significant strength. | Easy to use interface and excellent scalability. Inbuilt results analysis is slightly limited in comparison to LoadRunner. The captured data is open and can also be exported to Excel in two clicks. Drag & drop scenario set-up and control is very intuitive and easy to interact with, simplifying creation of scenarios with modular scripts. Manual correlation can be a headache but is eased slightly by using third-party ‘Diff’ tools and the inbuilt GUI DOM addressing feature. A relative shortage of standard script language functions but enough to get most HTTP load testing jobs done. If not, with “Includes” support and the availability of the source code, it is very extensible |
第二篇来自 AdventNet 公司的一份评估报告，宣传自家产品的意味比较浓。| Features | Comparisons |
|---|
| QEngine | LoadRunner | OpenSTA | OpenLoad |
|---|
| Cost / License | Pricing starts at $599 | Pricing starts at $38,000 for 25 virtual users. ($22,000 for Controller and $16,000 for 25 users.) | Free | Express - $4,995
Advanced - $37,995
|
| Top |
| Platform Support |
| Platform Support | Windows, Linux | Windows | Windows & UNIX | Windows, Linux & Solaris |
| Top |
| PlayBack |
| Session Tracking (Cookies / URL redirection) | Yes | Yes(Additional code for JS) | Yes(Additional code for JS) | Yes |
| Page Validation (String/RegEx) | Yes | Yes | no integrated comparison function | Yes |
| IP Spoofing | Yes | Yes, with automated router updates for IP forwarding. | No inbuilt features | Yes |
| Server Monitoring (WMI/SNMP/RSTAT) | Yes | Yes | Yes | Yes |
| Simulate Multiple Browsers | Yes | Yes | Yes | Yes |
| Scalability | 1000VU on a P4, 512 MB RAM 1.6 Ghz CPU | Approx 1 VU / ½Mb RAM for NT/W2K. Windows 95, 98 & Unix are less efficient.
Max. approx 1,500 VUs / PC. | Simple pages - up to 3,000 VUs 1Gb RAM, P4, W2K.
Complex scripts - 1664 VUs W2K | large workloads- 1000 VUs |
| Distributed tests | Yes | Multiple load-injectors managed by a single controller. | Multiple load-injectors managed by a single controller.
TCP/IP - same network or Web-relay feature
HTTP - Remote DMZ | --- |
| Distributed/Remote load generation | Yes | Yes, can control multiple load generators and collect results. | Yes, can control multiple load generators and collect results. | --- |
| Dynamic Load (Virtual User Rampup) | Yes | Yes | Yes | Yes |
| Simulate Multiple Connection Speeds | Yes | Can emulate different network speeds during playback | No inbuilt features | Yes |
| Correlation (substituting values in dynamic data while playback) | Yes (DB, CSV) | Partially supported | Yes | Yes |
| Caching | Partially supported | Yes | Partially handled | --- |
| Reporting & analysis | Yes - Graphs and Charts | Yes - Graphs and Charts, MS-Word | Yes - Graphs and Charts, MS-Excel | Yes - Graphs and Charts, Text/Spreadsheet/PDF |
| Basic/Digest/NTLM Authentication Support | Yes | Yes | --- | Yes |
| Recording |
| Protocol | HTTP,HTTPS | Many. Protocols are charged per item. Has a multi-protocol recording feature. | HTTP, HTTPS (SSL) | HTTP, HTTPS (SSL) |
| Scripting language | Jython | TSL, uses standard syntax for "C" & allows C function libraries. Customised functions for different protocols | SCL, "BASIC" like language, has special automation scriptingfacilities. limited functions available | Script Free recording and playback |
| Scripting Interface | Yes | Yes | Yes | No |
| Parameterisation | Yes | Yes | Yes | Yes |
| Ajax request recording | Yes | ? | ? | ? |
