Welcome to HBench-OS 1.0!

This file is a brief guide to the HBench-OS operating system benchmarking suite, and contains both pointers to further information about HBench-OS as well as important copyright and licensing information. Please read it through completely before using the HBench-OS benchmarks.

If you have any questions or comments that are not addressed either in this file or the accompanying documentation, please send them to the author of HBench-OS, Aaron Brown (abrown@eecs.harvard.edu), or the current maintainer (reachable at hbench@eecs.harvard.edu).

What is HBench-OS?

HBench-OS is a suite of portable benchmarks designed to measure the performance of primitive functionality provided by an operating system/hardware platform. HBench-OS was designed primarily as a research tool for the systems research and OS development communities, and thus its major design goal is to provide a flexible system for reproducible, accurate, and statistically-sound performance analysis and evaluation. These characteristics make it ideal for both performance research as well as more traditional consumer-oriented performance measurement and analysis.

HBench-OS's features include:

HBench-OS has its roots in Larry McVoy's lmbench1.1 package, but has diverged significantly from that original base. See the section below entitled "HBench-OS and lmbench" for a discussion of how HBench-OS's philosophy differs from that promulgated by lmbench, and how the benchmark tests have been expanded and enhanced relative to those in lmbench1.1.

Building and Running HBench-OS

Full details on compiling, configuring, customizing, and running HBench-OS can be found in the file "doc/using-hbench".

If you want to get HBench-OS up and running quickly with its default configuration, see the instructions in the file "USING" in this directory. We still recommend that you read the files in the doc directory so that you understand the range of possible configuration options that HBench-OS offers.

About the Benchmarks

Detailed information on each of the benchmarks, including what they measure and what each parameter does, is available in the file "doc/benchmark-descriptions". We recommend that you read this file before customizing the selection of benchmarks to be included in a run, and before attempting to analyze the data produced by a benchmark.

About the Results

The HBench-OS package includes several scripts to help reduce and process the mass of data produced by the full set of benchmark tests. These scripts, as well as the standard analysis policies used by HBench-OS, are described in the file "doc/interpreting-results".

Database of Collected Results

We are building a publicly-accessible master database of HBench-OS results, available at http://www.eecs.harvard.edu/vino/perf/hbench/. We would appreciate any contributions for this database, especially if they include results for machines and operating systems that are not yet in the database. If you would be interested in contributing to this database, please send any results that you acquire with HBench-OS to hbench@eecs.harvard.edu in the form of a tar file of the appropriate results directory.

HBench-OS and lmbench

HBench-OS and Larry McVoy's lmbench offer two different approaches to operating system benchmarking. Although HBench-OS has its roots in lmbench, it has diverged significantly in both philosophy and in the benchmark code itself.

HBench-OS's philosophy toward measurement is to provide maximum flexibility, detail and reproducibility for the researcher or developer. Thus HBench-OS provides a configurable run system that allows for easy selection of individual benchmarks; its benchmark tests preserve all of the data they acquire, allowing the user to make his or her own decisions about which data are important and which can be reduced via statistics; and it provides a great deal of flexibility in choosing benchmark parameters so that it is useful for both a broad profile of a system as well as detailed study of a small subset of system functionality. Additionally, HBench-OS's benchmarks have been written to be self-consistent and reproducible, a necessary condition if the benchmarks are to be used as the foundation of a sophisticated performance analysis methodology. One example of such a methodology, one that relies on the unique properties of HBench-OS, is described in the SIGMETRICS '97 paper found in the doc subdirectory of this distribution (doc/sigmetrics97-paper.ps.gz). This paper also details some of the weaknesses in lmbench and the changes between lmbench and HBench-OS.

HBench-OS has explicitly not been designed for blanket comparisons of wildly-divergent hardware and OS platforms. You might notice that it does not provide a script to compare selected results on all the tested machines, as does lmbench. We feel that such comparisons are, more often than not, misleading and prone to producing false conclusions. HBench-OS is much more suited for use as a tool for understanding system performance through detailed analysis, and its design goals and structure represent this bias.

You might find that in many cases the HBench-OS benchmarks report the same or similar values as do the lmbench benchmarks. This should not be surprising; the lmbench benchmarks are still quite good, and the OS and hardware functionality measured by both benchmark suites overlaps considerably. In these cases, however, HBench-OS still offers what lmbench does not: preservation of all data, user-definable data reduction policies, simple automatic analysis, and the consistency and structure that guarantee that the result is useful and correct.

Finally, the following is a list of the most significant changes that distinguish HBench-OS from lmbench; more details are available in the paper in doc/sigmetrics97-paper.ps.gz:

Copyright and Licensing Information

All files, source code, programs, scripts and documentation included in the HBench-OS package are:
Copyright © 1997 The President and Fellows of Harvard College. All rights reserved.
Copyright © 1997 Aaron B. Brown.
Portions copyright © 1994, 1995 Larry McVoy.

This work is derived from, but can no longer be called, lmbench.

We also request that if you have any fixes or enhancements to the benchmarks that you mail them to the current maintainer of HBench-OS at hbench@eecs.harvard.edu, or to the author at abrown@eecs.harvard.edu.