[DPS Class] Benchmarks

A Benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it. The term 'benchmark' is also mostly utilized for the purposes of elaborately-designed benchmarking programs themselves.

Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems.

Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures.

Wikipedia: Benchmark

Supercomputing Benchmark

The number and type of benchmark suites used to study supercomputer performance has varied widely over the years. In early studies, an ad hoc collection of programs was typically used to measure the performance of a given system relative to a known performance benchmark. Eventually, this practice evolved into groups of programs explicitly designed as supercomputer benchmark suites.

The most widely used benchmarks for performance on supercomputing clusters are: the SPEChpc96 suite; the Livermore Loops; and for scientific machines, the Linpack Kernels.

Some general examples of individual computer benchmarks:

• Dhrystone - Integer benchmark for UNIX systems
• Whetstone - Floating point benchmark for minicomputers
• I/O benchmarks
• MIPS
• Synthetic benchmarks
• Kernel benchmarks
• SPECint / SPECfp
• Summarizing

Types of Benchmarks

1. Real program:
Word processing software tool software of CDA user's application software (i.e.: MIS)
2. Microbenchmark:
Designed to measure the performance of a very small and specific piece of code.
3. Kernel:
Contains key codes normally abstracted from actual program popular kernel: Livermore loop linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language) results are represented in MFLOPS
4. Component Benchmark/ micro-benchmark:
Programs designed to measure performance of a computer's basic components automatic detection of computer's hardware parameters like number of registers, cache size, memory latency
5. Synthetic Benchmark:
Procedure for programming synthetic benchmark: take statistics of all types of operations from many application programs get proportion of each operation write program based on the proportion above Types of Synthetic Benchmark Whetstone, Dhrystone. These were the first general purpose industry standard computer benchmarks. They do not necessarily obtain high scores on modern pipelined computers.
6. I/O benchmarks
7. Database benchmarks: to measure the throughput and response times of database management systems (DBMS')
8. Parallel benchmarks: used on machines with multiple cores, processors or systems consisting of multiplemachines

Linpack

This is the benchmark used to test the members of Top500 list.

The LINPACK Benchmark was introduced by Jack Dongarra. A detailed description as well as a list of performance results on a wide variety of machines is available in postscript form from netlib. Here you can download the latest version of the LINPACK Report: performance.ps. A parallel implementation of the Linpack benchmark and instructions on how to run it can be found at http://www.netlib.org/benchmark/hpl/.

The benchmark used in the LINPACK Benchmark is to solve a dense system of linear equations. TOP500 uses a version of the benchmark that allows the user to scale the size of the problem and to optimize the software in order to achieve the best performance for a given machine. This performance does not reflect the overall performance of a given system, as no single number ever can. It does, however, reflect the performance of a dedicated system for solving a dense system of linear equations. Since the problem is very regular, the performance achieved is quite high, and the performance numbers give a good correction of peak performance.

By measuring the actual performance for different problem sizes n, a user can get not only the maximal achieved performance Rmax for the problem size Nmax but also the problem size N1/2 where half of the performance Rmax is achieved. These numbers together with the theoretical peak performance Rpeak are the numbers given in the TOP500. In an attempt to obtain uniformity across all computers in performance reporting, the algorithm used in solving the system of equations in the benchmark procedure must conform to LU factorization with partial pivoting. In particular, the operation count for the algorithm must be 2/3 n^3 + O(n^2) double precision floating point operations. This excludes the use of a fast matrix multiply algorithm like "Strassen's Method" or algorithms which compute a solution in a precision lower than full precision (64 bit floating point arithmetic) and refine the solution using an iterative approach.

Top500: LinPack Benchmark

If you want to use Linpack in your system, follow these steps

1. Download the LinPack Benchmark: LinPack 10.3.4
2. Extract the content of the packet.
3. Open the path: ~/linpack_10.3.4/benchmarks/linpack/
4. Run the executable xlinpack_xeon64 or xlinpack_xeon32 according your architecture
5. Set these inputs:

   Input data or print help ? Type [data]/help : data
   Number of equations to solve (problem size): 5000
   Leading dimension of array: 10000
   Number of trials to run: 10
   Data alignment value (in Kbytes): 1000000
   

6. Press enter and wait to the finish

This is my output

There are also a package to run Linpack in HP Clusters, but I will review that when the cluster are ready :)

References

• Wikipedia: Benchmark
• Top500 LinPack Benchmark
• Supercomputing Benchmarking