The SAPS Benchmark

Buying computer hardware is never easy. As discussed in the previous article, sizing a system correctly is the key. But once a sizing has been achieved, what should you do next? You might start looking at benchmark data to compare systems. When looking at and comparing benchmark data it is important to understand what the benchmark measures and how it is measured. For example, a CPU benchmark will give you an idea of the raw power of a systems processor or processors; it will not give you any idea of the memory performance or network capacity of a system. It is also the case that some benchmarks measure only one aspect of the performance of a specific component. This can be seen with the popular CPU benchmarks Whetstone and Dhrystone, these benchmarks test floating point and integer arithmetic performance respectively. Neither gives a complete account of the processor.

Another aspect of benchmarking is knowing how a benchmark result will translate to your workload. A CPU benchmark will benchmark a CPU, just as a memory benchmark will benchmark memory; both are clues to overall system performance but are not a complete answer. Due to this lack of holistic benchmarking, SAP has developed its own benchmark standard known as SAP Standard Application Benchmark. This benefits SAP customers in two ways. Firstly the benchmark tests the system as a whole and gives a standard result. In addition customers do not need to rely on proprietary benchmarks results given by different computer hardware manufacturers. All systems regardless of manufacturer, platform and OS are tested in the same way.

The Standard Application Benchmark is designed to test a SAP system doing what a SAP system does. The benchmark consists of user simulations on standard SAP components, a full list of the applications in the benchmark and the specific tests run for each application can be found here.

It is important to understand that SAP is a three tier solution. A SAP system comprises a presentation tier, an application tier and the database tier. The presentation tier will often be the SAP GUI, but is often SAP Portal or other web based front end. The application tier is the SAP application along with all its components. Finally the database tier is the Relational database management system (RDBMS), such as DB2 or Oracle.

The SAP Standard Application Benchmark does not test presentation performance. The benchmark measures the application performance, which relies on the performance of the database. Therefore when a system is submitted for certification and benchmarking, it is certified with a specific Operating System and RDBMS. However, a system can be submitted as a three tier solution (where the application and RDBMS operate on separate systems), only the application is installed on the certified system, but on the whole most systems are submitted as two tier solutions.

The result of the SAP Standard Application Benchmark is SAPS. Each system submitted for certification by SAP has its performance measured in SAPS. One SAP equals 20 fully processed line items per hour". For more information follow this link.

Now we know what a SAP is, the next question is "how is the SAPS figure obtained?"

When a system is submitted for certification simulated user session are launched. Each user’s session conducts tasks a real user would perform, such as log in, create a customer order, create an invoice etc. The simulated number of users is steadily increased until the dialogue response time drops to around 1 second. At this point, many thousands of simulated users could be accessing the system; in addition, the overall capacity of the system will be extremely high. CPU utilisation will normally be over 95%. This is point where the benchmark is taken. This high watermark of utilisation is where the number of fully processed line items per hour is calculated.

When a system is certified, the following information is recorded.

Date of Certification

Technology Partner - IBM, HP, Sun, Dell, etc

Number of Benchmark Users - The number of simulated users reached during the benchmark process.

Average Dialog Response Time (sec)

Dialog Steps Per Hour

SAPS - The final sizing figure.

Fully Processed Line Items Per Hour

Operating System Release - The OS variant and version used during testing.

RDBMS Release - The database product and version used during testing.

ERP Release - The ERP version used for the benchmark.

Server Segmentation - Marked as true if the benchmarking target is a virtual machine.

Central Server (*) Additional Specifications - Model name of the server hardware used.

Processors - The number of populated processor sockets in the target system.

Cores - Total number of cores.

Threads - Total number of threads.

Cpu Type - CPU manufacture and type. 

Cpu Speed - Speed of processor/s in Mhz.

Cache - Details of processor cache.

Central Server Memory (MB) - Amount of main memory installed in the server.

Certification Number - Official SAP certification number.

All the results of all benchmarks are publicly available from the SAP website.

Vendor bias

You might think that thanks to SAP's benchmarking that comparing the performance of different systems would be easy, the fact is it is not. The reason is that each two tier benchmarked solution has an Operating System and RDBMS chosen by the vendor. As many of the hardware vendors have hardware, Operating Systems and RDBMS that are in direct competition with each other, each vendor tends to submit systems running their own OS and DB or an OS and DB from a company that is not a competitor. For example many of SAP's IBM users with POWER systems run Oracle's RDBMS. IBM would of course prefer these users to use their own RDBMS, DB2; therefore out of 32 IBM POWER benchmarks only 3 configurations have been tested with Oracle, two in 2007 and 1 in 1997. The same is true of Sun Microsystems, who are now owned by Oracle. As Sun is a competitor of IBM out of the 57 Sun systems certified a system running DB2 has never been submitted, although DB2 is available for Solaris.

The following table shows some of the preferences that vendors have when submitting hardware.

This vendor bias means that although the SAP Standard Application Benchmark is transparent and open, the vendors control the result we see by their selective submission. I'm sure that many users and people in the IT industry would love to see the same IBM POWER system submitted with both Oracle and DB2 so we could see the performance difference, if any. In the same way, we may want to know how DB2 compares to MS SQL server on a HP x86 server.

We are unlikely to see these results but knowing how the benchmark works, what data is recorded and how the data can be obtained leads us firmly to the next article in this series. The next article will describe how to interpret the benchmark results, some surprising comparisons and how to make the most from your hardware investment.

Speak to one of our technical experts now, call us on 0115 951 9666 or email us at This email address is being protected from spambots. You need JavaScript enabled to view it.