/*******************************************

Copyright 2003, Aggelos Economopoulos, all rights reserved. *
Author: Aggelos Economopoulos <aoiko@cc.ece.ntua.gr>
Based on original work by: Con Kolivas <contest@kolivas.org> *
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version. *
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details. *
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. * *******************************************/

Contest v0.61

WHAT IS THIS?

This program is designed to test system responsiveness by running kernel compilation under a number of different load conditions. It is designed to compare different kernels, not different machines. It uses real workloads you'd expect to find for short periods in every day machines but sustains them for the duration of a kernel compile to increase the signal to noise ratio.

Some of the load conditions are applications originally sourced by B.Matthews' irman tool also licensed under GPL and modified to suit.

Null load - No load
Cache Run - A no load run directly after a previous run without memory

flushing
Process load - Fork and exec N processes, connected in a

        unidirectional ring by pipes.  Insert M<<N chunks of data into
        the ring and pass them around (nice IPC/switch test)
Memory load - Repeatedly reference 110% of RAM in a pattern
        designed to cause cache misses
IO Load - copies /dev/zero continually to a file the size of
        the physical memory.

IO Other - same as IO load at a different location Read Load - Reads a file the size of the physical memory List Load - Lists the entire file system (ls -lRa /) CTar Load - Repeatedly creates a tar of the kernel tree XTar Load - Repeatedly extracts a tar of the kernel tree Dbench Load - Runs dbench N (where N is 16*num_cpus) repeatedly

HOW DO I INSTALL IT?

Extract the tarball.
cd to the contest directory and type

make
make install

To override install dir, simply pass the INSTPATH variable

make INSTPATH=/path/to/install/dir install (installs in /usr/bin by default)

HOW DO I RUN IT?

Download a linux 2.4.19 kernel source tree, and extract it where you wish to be running your benchmarks. Copy the contest.config file in this tarball to that directory as .config. Download dbench and compile it (optional). Copy dbench somewhere into your PATH and copy the .txt files from the dbench source into your kernel test tree. Then:

make oldconfig
make dep

Reboot into the kernel you wish to test, disabling any apm in the bios. At lilo preferably boot into single user mode:

linux single vga=normal apm=off

Then cd to the kernel source directory and type (from the man page):

contest [-cdr] [-k name] [-t file] -o file [-n nrruns] [load...]

OPTIONS

These are the commonly used options:

     -b      Print a progress bar if information from previous runs is avail-
             able. Do not use this for remote logins / testing as it will
             taint the results.

     -k NAME
             Use the kernel name NAME in the report generated by contest
             (default is the internal kernel name from uname.)

     load
             The name of the load you wish to run - can specify multiple loads
             Available loads are: no_load, cacherun, pro-cess_load, ctar_load,
             xtar_load, io_load, read_load, list_load, mem_load and
             dbench_load (default all loads.)

     -n N    Number of times to run each load in contest to generate useful
             averages (default 3.)

     -r      Generate report on all the cumulative log files in the current
             directory.

     -t FILE
             Use the file FILE as the temporary file to use for io loads - can
             specify a full path such as /tmp/tmpfile (default ./dump.)
     
     -o FILE
             target file for io_other (required unless io_other is not specified).

OTHER OPTIONS

     You shouldn't need to touch these.
     -c      Assume a cold cache. This bypasses the memory flushing routines
             that occur between different loads in contest that are normally
             used to minimise the effects of caching of data from previous
             compiles. It is of use only to test contest functionality and
             will invalidate any testing you do.

     -d      Print debugging information if contest was compiled with debug-
             ging enabled (will flood the console).

     -p      Dont cleanup after each load is run - speeds up the running of
             contest but can use up massive amounts of disk space.

NOTES

contest must be run in the top directory of a 2.4.19 linux kernel tree, and the contest.config file from the contest source should be copied to that directory as .config.
contest assumes the existence of the common utilities, cc and dbench in your PATH
The version of cc used must not vary between benchmarks.
dbench_load requires both the existence of dbench in your PATH and the presence of the .txt files from the dbench source in your testbed tree.
contest should be run by itself in single user mode or as the sole init process to exclude the effects of any other loads on the system.
contest requires large amounts of spare disk space to perform dbench_load, the tar loads (ctar_load and xtar_load), read_load, io_load and io_other.
io_load requires as much spare disk space as the physical RAM the test machine has.
io_other needs the -o parameter to tell it where to do the other io load. Running the io load on a separate hard disk tests different aspects of the kernel.
cacherun is only meaningful if run in combination with no_load.
must be run with sufficient permissions to execute swapon/swapoff.
The contest name is a play on words due to the Author's name being Con.
Never do make dep again after running contest as it will change the results of subsequent runs.

     *Changing the vga from framebuffer mode to non-fb and vice versa changes
     results.

There are four numbers generated for each load; the time taken to compile the kernel, the average cpu percentage to do the compile, the amount of times the load performed its task and the cpu% the load used while running. The kernelname.log file contains more detailed information.

A report generated with -r gives the most useful information. It sorts the kernels, lays out the information and creates a new result called ratio which is simply the ratio of the time result compared to the noload result for the kernel.

FAQ

WHAT DO THE NUMBERS MEAN

The lower the time (and ratio) the better, and the higher the cpu percentage the better. The absolute number is not important, but the differences are. When the difference between noload and the loads is small it shows that on that kernel, the system is able to respond to requests for normal tasks (ie responsiveness). When the cpu percentage is high it shows that although the other loads are high, a cpu intensive application (kernel compile) can still use as much as it needs. Ideally the load should be able to perform some reasonable amount of work and the kernel compile take slightly longer. If the balance shifts substantially in one direction or the other it is indicating a large change in the fairness of scheduling (which is really what contest is testing.)

From version 0.50 information about how much work the load itself has done is also included. An internal number is generated by each work load to show how much work is done called Loads in the results. Also included is the average cpu% the load used while running (LCPU%). The best performance by a kernel will be some compromise in minimising the time the kernel takes to compile while not greatly decreasing the loads' work done - the kernel hackers are the best people to interpret this information.

Do NOT change the hardware you test it on, the kernel tree you use for the testing, gcc version or the hardware settings (eg with hdparm) as this will render the results useless. Testing it on different hardware will be interesting only to compare how the kernels compare on that piece of hardware. Comparing results from different hardware is meaningless.

WHY DO MY NUMBERS LOOK WORSE NOW THAT MY HARDWARE IS BETTER?

This is of concern to the kernel design! If faster hard disks mean they are so busy writing that other tasks wont get adequate attention then this needs to be addressed.

CAN I TEST THE RESPONSIVENESS WHILE WRITING TO ANOTHER HARD DISK?

You can specify where the file written by io_loads is located by specifying the io_other tempfile as a file on a different hard disk.

eg: if you have two hard disks mounted at / and /mnt/disk and have your test kernel tree in /usr/src/linux you can specify

contest -o /mnt/disk/dump

SMP?

The cpu% may read >100%; it should be divided by the number of cpus.

HOW MANY TIMES SHOULD I RUN CONTEST?

The more the merrier to get the best results. However, good results are obtained with the default 3 runs.

TODO

Lots I haven't thought of yet.

HOW DO I CONTACT YOU

Feel free to contact me with questions, suggestions, comments or patches email me at: contest at kolivas dot org Last update: Tue Feb 18 2003

Changelog: v0.60 to 0.61 Bugfixes. Simpler command line options. Implementation of io_other. Improved resolution of results. Logs are incompatible again with previous versions. Dbench corrected back to 16*num_cpus. New URL & contact details.

v0.51 to 0.60
Massive change with a complete rewrite of all elements in c by Aggelos Economopoulos (the unofficial maintainer of the actual program now.) Implementation of the internally used cacherun and dbench_loads. Progress bar, cold cache, debug and no cleanup options added. Change default to 3 runs of contest and all loads. Rewrite of process_load by Rene Herman. First version of manual page.

v0.50 to 0.51
Added list_load, read_load, xtar_load, ctar_load. Further cleanups and compiler issues.

v0.42 to 0.50
Included work done and cpu% by loads. io_load converted from using head to using dd (to work with high memory machines and to present greater io_load). Lots of internal changes to support the changes. Results while similar to 0.4x have different output so are not compatible with previous versions.

v0.41 to 0.42
Added averaging of results during parsing. Added precision option to ratio (to allow integer only bash to do all sorts of stuff). Older gcc fix (A. Morton)

v0.40 to 0.41
Major restructure of priming for each test gives much greater resolution in results. Removed priming compile and streamlined startup for faster runtimes.

v0.37 to 0.40
Modified process_load to be more of a process_load than a cpu load. Changes results dramatically hence major version upgrade (R. Herman). Better SMP support

v0.36 to 0.37
Dropped IO_halfmem (no useful info) and changed name of IO_fullmem to IO_load. Performed sync,swapoff,swapon b/w tests. Added ratio app to create ratios for results. Intrinsic results parser added. Fixed PATH in contest. Added number of runs option. (lots)

v0.35 to 0.36
Added command line arguments. Choose tests to perform at runtime. Optional install path (R. Maureira)

v0.34 to 0.35
Moved everything to /usr/bin. Extra kernel compile performed prior to each benchmark to ablate effects of prior benchmark. Load started sooner. mem_load modified to accept argument -p (percentage load) for future version.

v0.33 to 0.34
Better messages, more information, ability to specify io load file, removed chance of loads overlapping. Removed accidentally commented out lines.

v0.32 to 0.33
Improved messages, names corrected, noload script removed

v0.30 to 0.32
Signal trapping, kernel name specifying and code cleanup (lots of people)

v0.22 to 0.30
Code cleanup (R.v. Riel) (now -Wall compat.)

v0.21 to 0.22
mem_load corrected for low mem systems by (R.v. Riel)

v0.20 to 0.21
Minor change to meminfo for high mem systems (R. Hron) Change time to $time

v0.12 to 0.20
Changed IO load (idea by A. Morton)

v0.11 to 0.12
Modified mem_load to work with 2.5.x kernels (R.v. Riel)

v0.1 to 0.11
Internal nonsense

THANKS

Thanks Aggelos Economopoulos, Rik van Riel, Andrew Morton, Randy Hron, Rene Herman, Cliff White, Robinson Maureira, Paolo Ciarrocchi and more!

Written by Con Kolivas (official maintainer).