kcbench - Man Page

kcbench [options]

Kcbench tries to compile a Linux kernel really quickly which can be used to test a system's performance or stability.

Note: The number of compile jobs ('-j') that delivers the best result depends on the machine being benched. See the section "On the Default Number of Jobs" below for details.

To get comparable results from different machines you need to use the exact same operating system on all of them. There are multiple reasons for this recommendation, but one of the main reasons is: the Linux version this benchmark downloads and compiles depends on the operating system's default compiler.

If you choose to ignore this recommendation at least make sure to hard code the Linux version to compile ('-s 5.4'), as for example compiling 5.7 will take longer than 5.4 or 4.19 and thus lead to results one cannot compare. Also, make sure the compiler used on the systems you want to compare is from similar, as for example gcc10 will try harder to optimize the code than gcc8 or gcc9 and thus take more time for its work.

Kcbench is accompanied by kcbenchrate. Both are quite similar, but work slightly different:

• kcbench tries to build one kernel as fast as possible. This approach is called 'speed run' and let's make start multiple compilers jobs in parallel by using 'make -j #'. That way kcbench will use a lot of CPU cores most of the time, except during those few phases where the Linux kernel build process is singled threaded and thus utilizes just one CPU core. That for example is the case when vmlinux is linked.
• kcbenchrate tries to keep all CPU cores busy constantly by starting workers on all of them, which each builds one kernel with just one job ('make -j 1'). This approach is called 'rate run'. It takes a lot longer to generate a result than kcbench; it also needs a lot more storage space, but will utilize the machine and its processors better.

The optimal number of compile jobs (-j) to get the best result depends on the machine being benched. On most systems you will achieve the best result if the number of jobs matches the number of CPU cores. That for example is the case on this 4 core Intel processor without SMT:

[cttest@localhost ~]$ bash kcbench -s 5.3 -n 1
Processor:            Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz [4 CPUs]
Cpufreq; Memory:      Unknown; 15934 MByte RAM
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 4):         250.03 seconds / 14.40 kernels/hour
Run 2 (-j 6):         255.88 seconds / 14.07 kernels/hour

The run with 6 jobs was slower here. Trying a setting like that by default looks like a waste of time on this machine, but other machines deliver the best result when they are oversubscribed a little. That's for example the case on this 6 core/12 threads processor, which achieved its best result with 15 jobs:

[cttest@localhost ~]$ bash kcbench -s 5.3 -n 1
Processor:            Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz [12 CPUs]
Cpufreq; Memory:      Unknown; 15934 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 12):        92.55 seconds / 38.90 kernels/hour
Run 2 (-j 15):        91.91 seconds / 39.17 kernels/hour
Run 3 (-j 6):         113.66 seconds / 31.67 kernels/hour
Run 4 (-j 9):         101.32 seconds / 35.53 kernels/hour

You'll notice attempts that tried to utilize only the real cores (-j 6) and oversubscribe them a little (-j 9), which look liked a waste of time. But on some machines with SMT capable processors those will deliver the best results, like on this AMD Threadripper processor with 64 core/128 threads:

$ kcbench
[cttest@localhost ~]$ bash kcbench -s 5.3 -n 1
Processor:            AMD Ryzen Threadripper 3990X 64-Core Processor [128 CPUs]
Cpufreq; Memory:      Unknown; 15934 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 128):       26.16 seconds / 137.61 kernels/hour
Run 2 (-j 136):       26.19 seconds / 137.46 kernels/hour
Run 3 (-j 64):        21.45 seconds / 167.83 kernels/hour
Run 4 (-j 72):        22.68 seconds / 158.73 kernels/hour

This is even more visible when compiling an allmodconfig configuration:

[cttest@localhost ~]$ bash kcbench -s 5.3 -n 1 -m
Processor:            AMD Ryzen Threadripper 3990X 64-Core Processor [128 CPUs]
Cpufreq; Memory:      Unknown; 63736 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 128):       260.43 seconds / 13.82 kernels/hour
Run 2 (-j 136):       262.67 seconds / 13.71 kernels/hour
Run 3 (-j 64):        215.54 seconds / 16.70 kernels/hour
Run 4 (-j 72):        215.97 seconds / 16.67 kernels/hour

This can happen if the SMT implementation is bad or something else (memory, storage, ...) becomes a bottleneck. A few tests on above machine indicated the memory interface was the limiting factor. A AMD Epyc from the same processor generation did not show this effect and delivered its best results when the number of jobs matched the number of CPUs:

[cttest@localhost ~]$ bash kcbench -s 5.3 -n 1 -m
Processor:            AMD EPYC 7742 64-Core Processor [256 CPUs]
Cpufreq; Memory:      Unknown; 63736 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 256):       128.24 seconds / 28.07 kernels/hour
Run 2 (-j 268):       128.87 seconds / 27.94 kernels/hour
Run 3 (-j 128):       141.83 seconds / 25.38 kernels/hour
Run 4 (-j 140):       137.46 seconds / 26.19 kernels/hour

This table will tell you now many jobs kcbench will use by default:

 #                             Cores: Default # of jobs
 #                             1 CPU: 1 2
 #           2 CPUs (    no SMT    ): 2 3
 #           2 CPUs (2 threads/core): 2 3 1
 #           4 CPUs (    no SMT    ): 4 6
 #           4 CPUs (2 threads/core): 4 6 2
 #           6 CPUs (    no SMT    ): 6 9
 #           6 CPUs (2 threads/core): 6 9 3
 #           8 CPUs (    no SMT    ): 8 11
 #           8 CPUs (2 threads/core): 8 11 4 6
 #          12 CPUs (    no SMT    ): 12 16
 #          12 CPUs (2 threads/core): 12 16 6 9
 #          16 CPUs (    no SMT    ): 16 20
 #          16 CPUs (2 threads/core): 16 20 8 11
 #          20 CPUs (    no SMT    ): 20 25
 #          20 CPUs (2 threads/core): 20 25 10 14
 #          24 CPUs (    no SMT    ): 24 29
 #          24 CPUs (2 threads/core): 24 29 12 16
 #          28 CPUs (    no SMT    ): 28 34
 #          28 CPUs (2 threads/core): 28 34 14 18
 #          32 CPUs (    no SMT    ): 32 38
 #          32 CPUs (2 threads/core): 32 38 16 20
 #          32 CPUs (4 threads/core): 32 38 8 11
 #          48 CPUs (    no SMT    ): 48 55
 #          48 CPUs (2 threads/core): 48 55 24 29
 #          48 CPUs (4 threads/core): 48 55 12 16
 #          64 CPUs (    no SMT    ): 64 72
 #          64 CPUs (2 threads/core): 64 72 32 38
 #          64 CPUs (4 threads/core): 64 72 16 20
 #         128 CPUs (    no SMT    ): 128 140
 #         128 CPUs (2 threads/core): 128 140 64 72
 #         128 CPUs (4 threads/core): 128 140 32 38
 #         256 CPUs (    no SMT    ): 256 272
 #         256 CPUs (2 threads/core): 256 272 128 140
 #         256 CPUs (4 threads/core): 256 272 64 72

The compilation is unlikely to fail, as long as you are using a settled GCC version to natively compile the source of a current Linux kernel for popular architectures like ARM, ARM64/Aarch64, or x86_64. For other cases there is a bigger risk that compilation will fail due to factors outside of what kcbench/kcbenchrate control. They nevertheless try to catch a few common problems and warn, but they can not catch them all, as there are to many factors involved:

• Brand new compiler generations are sometimes stricter than their predecessors and thus might fail to compile even the latest Linux kernel version. You might need to use a pre-release version of the next Linux kernel release to make it work or simply need to wait until the compiler or kernel developers solve the problem.
• Distributions enable different compiler features that might have an impact on the kernel compilation. For example gcc9 was capable of compiling Linux 4.19 on many distributions, but started to fail on Ubuntu 19.10 due to a feature that got enabled in its GCC. Try compiling a newer Linux kernel version in this case.
• Cross compilation increases the risk of running into compile problems in general, as there are many compilers and architectures our there. That for example is why compiling the Linux kernel for an unpopular architecture is more likely to fail due to bugs in the compiler or the Linux kernel sources that nobody had noticed before when the compiler or kernel was released. This is even more likely to happen if you start kcbench/kcbenchrate with '-m/--allmodconfig' to build a more complex kernel.

Running benchmarks is very tricky. Here are a few of the aspects you should keep mind when doing so:

• Do not compare results from two different archs (like ARM64 and x86_64); kcbench/kcbenchrate compile different code in that case, as they will compile a native kernel on each of those archs. This can be avoided by cross compiling for a third arch that is not related to any of the archs compared (say RISC-V when comparing ARM64 and x86_64).
• Unless you want to bench compilers do not compare results from different compiler generations, as they will apply different optimizations techniques. For example to not compare results from GCC7 and GCC9, as the later optimizes harder and thus will take more time generating the code. That's also why the Linux version compiled by default depends on the machine's compiler: you sometimes can't compile older kernels with the latest compilers anyway, as new compiler generations often uncover bugs in the Linux kernel source that need get fixed for compiling to succeed. For example, when GCC10 was close to release it was incapable of compile the then latest Linux version 5.5 in an allmodconfig configuration due to a bug in the Linux kernel sources.
• Compiling a Linux kernel scales very well and thus can utilize processors quite well. But be aware that some parts of the Linux compile process will only use one thread (and thus one CPU core), for example when linking vmlinuz; the other cores will idle meanwhile. The effect on the result will grow with the number of CPU cores.

If you want to work against that consider using '-m' to build an allmodconfig configuration with modules; comping a newer, more complex Linux kernel version can also help. But the best way to avoid this effect is by running kcbenchrate.

• kcbench/kcbenchrate by default set CCACHE_DISABLE=1 when calling 'make' to avoid interference from ccache.

To let kcbench decide everything automatically simply run:

$ kcbench

On a four core processor without SMT kcbench by default will compile 2 kernels with 4 jobs and 2 with 6 jobs. You can specify a setting like this manually: .

: $ kcbench -s 5.4 --iterations 3 --jobs 2 --jobs 4

This will compile Linux 5.4 first 3 times with 2 jobs and then as often with 4 jobs.

By default, the lines you are looking for look like this:

Run 1 (-j 4): 230.30 sec / 15.63 kernels/hour [P:389%, 24 maj. pagefaults]

Here it took 230.30 seconds to compile the Linux kernel image. With a speed like this the machine can compile 15.63 kernels per hour (60*60/230.30). The results from this 4 core machine also show the CPU usage (P) was 389 percent; 24 major page faults occurred during this run – this number should be small, as processing them takes some time and thus slows down the build. This information is omitted, if less than 20 major page faults happen. For details how the CPU usage is calculated and major page faults are detected see the man page for GNU 'time', which kcbench/kcbenchrate rely on for their measurements.

When running with "-d|--detailedresults" you'll get more detailed result:

Run 1 (-j 4): 230.30 sec / 15.63 kernels/hour [P:389%]
Elapsed Time(E): 2:30.10 (150.10 seconds)
Kernel time (S): 36.38 seconds
User time (U): 259.51 seconds
CPU usage (P): 197%
Major page faults (F): 0
Minor page faults (R): 9441809
Context switches involuntarily (c): 69031
Context switches voluntarily (w): 46955

• some math to detect the fastest setting and do one more run with it before sanity checking the result and printing the best one, including standard deviation.

kcbenchrate(1), time(1)

Thorsten Leemhuis <linux [AT] leemhuis [DOT] info>

kcbenchrate(1).