Welcome to HBench-OS 1.0!
This file is a brief guide to the HBench-OS operating system
benchmarking suite, and contains both pointers to further information
about HBench-OS as well as important copyright and licensing
information. Please read it through completely before using the
If you have any questions or comments that are not addressed either in
this file or the accompanying documentation, please send them to the
author of HBench-OS, Aaron Brown (firstname.lastname@example.org), or the
current maintainer (reachable at email@example.com).
What is HBench-OS?
HBench-OS is a suite of portable benchmarks designed to measure the
performance of primitive functionality provided by an operating
system/hardware platform. HBench-OS was designed primarily as a
research tool for the systems research and OS development communities,
and thus its major design goal is to provide a flexible system for
reproducible, accurate, and statistically-sound performance analysis
and evaluation. These characteristics make it ideal for both
performance research as well as more traditional consumer-oriented
performance measurement and analysis.
HBench-OS's features include:
- a broad range of benchmarks, covering hardware capabilities,
low-level OS abstractions, and high-level OS primitives
- an accurate timing system that automatically measures and
removes its own overhead
- most benchmarks dynamically determine how long to run to overcome
low timing resolution
- all benchmark data is preserved for analysis
- support for user-defined statistical analysis policies,
including viewing raw data distributions
- basic automatic analysis of benchmark data
- support for obtaining cold-cache results (no loops) when
timing resolution is high enough
- a flexible control system that allows you to configure exactly
which benchmarks are executed (and their parameters) in a
- built-in support for multiple benchmarking runs, including
statistical processing across multiple runs
- a machine-independent framework that supports high-resolution
timing via hardware cycle counters
(currently supports Intel Pentium and later chips)
- a machine-independent framework to support hardware event
counters for detailed profiling of the benchmark tests
(currently supports Intel Pentium Pro under modified
HBench-OS has its roots in Larry McVoy's lmbench1.1 package, but has
diverged significantly from that original base. See the section below
entitled "HBench-OS and lmbench" for a discussion of how HBench-OS's
philosophy differs from that promulgated by lmbench, and how the
benchmark tests have been expanded and enhanced relative to those in
Building and Running HBench-OS
Full details on compiling, configuring, customizing, and running
HBench-OS can be found in the file "doc/using-hbench".
If you want to get HBench-OS up and running quickly with its default
configuration, see the instructions in the file "USING" in this
directory. We still recommend that you read the files in the doc
directory so that you understand the range of possible configuration
options that HBench-OS offers.
About the Benchmarks
Detailed information on each of the benchmarks, including what they
measure and what each parameter does, is available in the file
"doc/benchmark-descriptions". We recommend that you read this file
before customizing the selection of benchmarks to be included in a
run, and before attempting to analyze the data produced by a benchmark.
About the Results
The HBench-OS package includes several scripts to help reduce and
process the mass of data produced by the full set of benchmark
tests. These scripts, as well as the standard analysis policies used
by HBench-OS, are described in the file "doc/interpreting-results".
Database of Collected Results
We are building a publicly-accessible master database of HBench-OS
results, available at http://www.eecs.harvard.edu/vino/perf/hbench/.
We would appreciate any contributions for this database, especially if
they include results for machines and operating systems that are not
yet in the database. If you would be interested in contributing to
this database, please send any results that you acquire with HBench-OS
to firstname.lastname@example.org in the form of a tar file of the appropriate
HBench-OS and Larry McVoy's lmbench offer two different approaches to
operating system benchmarking. Although HBench-OS has its roots in
lmbench, it has diverged significantly in both philosophy and in the
benchmark code itself.
HBench-OS's philosophy toward measurement is to provide maximum
flexibility, detail and reproducibility for the researcher or
developer. Thus HBench-OS provides a configurable run system that
allows for easy selection of individual benchmarks; its benchmark
tests preserve all of the data they acquire, allowing the user to make
his or her own decisions about which data are important and which can
be reduced via statistics; and it provides a great deal of flexibility
in choosing benchmark parameters so that it is useful for both a broad
profile of a system as well as detailed study of a small subset of
system functionality. Additionally, HBench-OS's benchmarks have been
written to be self-consistent and reproducible, a necessary condition if
the benchmarks are to be used as the foundation of a sophisticated
performance analysis methodology. One example of such a methodology,
one that relies on the unique properties of HBench-OS, is described in
the SIGMETRICS '97 paper found in the doc subdirectory of this
distribution (doc/sigmetrics97-paper.ps.gz). This paper also details
some of the weaknesses in lmbench and the changes between lmbench and
HBench-OS has explicitly not been designed for blanket comparisons
of wildly-divergent hardware and OS platforms. You might notice that
it does not provide a script to compare selected results on all the
tested machines, as does lmbench. We feel that such comparisons are,
more often than not, misleading and prone to producing false
conclusions. HBench-OS is much more suited for use as a tool for
understanding system performance through detailed analysis, and its
design goals and structure represent this bias.
You might find that in many cases the HBench-OS benchmarks report the
same or similar values as do the lmbench benchmarks. This should not
be surprising; the lmbench benchmarks are still quite good, and the OS
and hardware functionality measured by both benchmark suites overlaps
considerably. In these cases, however, HBench-OS still offers what
lmbench does not: preservation of all data, user-definable data
reduction policies, simple automatic analysis, and the consistency and
structure that guarantee that the result is useful and correct.
Finally, the following is a list of the most significant changes that
distinguish HBench-OS from lmbench; more details are available in the
paper in doc/sigmetrics97-paper.ps.gz:
- Several bugfixes, including significant fixes to bw_file_rd
- Expanded repertoire of benchmarks, including L1/L2 cache
bandwidths, more system call measurements, and variable
transfer buffer sizes in bulk data transfer tests.
- Vastly more flexible data acquisition and analysis.
- All result data is preserved for off-line analysis via
built-in or user-defined data reduction policies.
- More rigorous statistical treatment of data.
- Benchmarks restructured for internal and inter-benchmark
- Dynamic benchmark sizing to overcome low timing resolution.
- Support for hardware cycle counters and event counters in a
machine-independent framework (although currently
implemented only for Intel Pentium and later architectures)
Copyright and Licensing Information
All files, source code, programs, scripts and documentation included
in the HBench-OS package are:
Copyright © 1997 The President and Fellows of Harvard College.
All rights reserved.
Copyright © 1997 Aaron B. Brown.
Portions copyright © 1994, 1995 Larry McVoy.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation (either version 2 of the License, or
(at your option) any later version), with the following additional
restriction, which overrides any conflicting restrictions in the
These unmodified benchmarks, and any results obtained through their
use, may not be described in any public forum or publication
under any name other than "HBench-OS", unless an exemption is
explicitly granted by either Aaron Brown or Margo Seltzer.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program, in the file COPYING in this distribution;
if not, write to the Free Software Foundation, Inc., 675 Mass Ave,
Cambridge, MA 02139, USA.
This work is derived from, but can no longer be called, lmbench.
We also request that if you have any fixes or enhancements to the benchmarks
that you mail them to the current maintainer of HBench-OS at
or to the author at email@example.com.