In his paper, John Chapin of MIT observed that current schedulers have two major drawbacks in the desktop computing environment. First, they optimize for throughput, rather than for the performance of the tasks that are most critical to the user. Second, although they do a good job of prioritizing processor allocation, when memory is tight the performance of high-priority applications often suffers. To address these shortcomings, John proposed the use of a new performance metric, system memory cycles per instruction (system MCPI) to evaluate the success of operating system improvements.
Brian Noble from CMU outlined some of the reasons that it is difficult to perform rigorous, reproducible experiments on wireless networking systems. He also described the use of trace-modulation to emulate the performance characteristics of a target network. Trace-modulation allows both synthetically generated and empirically collected traces to be combined into a "hybrid" trace in which the simulator selects a particular trace based on data from a "location simulator." The resulting system provides flexibility in the types of environment to be simulated, and realistic simulation.
The first speaker was Margo Seltzer of Harvard University. Her paper was titled "Self-Monitoring and Self-Adapting Operating Systems." This paper focused on performance analysis and optimization tools that are being built into the VINO operating system, although Margo argued that the same principles could be applied to other systems.
Margo started by pointing out that although we all agree that system performance is important, there is a dearth of people who are good at tuning systems. This problem will only be exacerbated by extensible operating systems, which offer the promise of allowing each application developer to fine-tune system behavior to optimize the performance of their own application. This flexibility improve the performance of any application nobody can figure out how to use it.
To help get a handle on this problems, Margo proposed the use of self-monitoring systems. In other words, systems that collect statistics and traces of their own performance. In VINO, each module collects a variety of statistics about its performance. These statistics are collected into a database of performance activity, which is analyzed off-line to determine the system's normal performance and resource usage patterns. From this profile, thresholds for "normal behavior" are be derived. At run time, system performance is monitored, and when performance falls outside of these thresholds, the operating system will attempt to dynamically change its behavior to improve performance.
The most challenging part of this process is, of course, the devising the heuristics that the system might use to adapt its behavior to changes in the user's workload. Margo proposed several techniques that might be useful, but admitted that many of the heuristics are "pie in the sky" at this point.
[Could extend this to include a description of in situ simulation and Margo's proposed heuristics for dealing with paging.]
[The same system interfaces where traces and logs are collected can also be extended to feed the traces (module inputs) to an in situ simulator of an alternate policy. The outputs of this simulator can then be compared against the outputs of the real module.]
Discussion:
The next talk was by John Chapin, recently of Stanford, and now on the faculty at MIT. His paper was titled, "A Fresh Look at Memory Hierarchy Management." John started his talk with a disclaimer that so far this work only represents a hypothesis; he has not implementation or data to support it.
[Include description of John's opening graph?]
John started by describing how the usage of computer system has changed since the 60s and 70s. Historically, computer systems were large time sharing systems that ran many concurrent jobs for a variety of users. Given the demands of this environment, the focus was on optimizing for throughput. In the current desktop computing environment, however, the performance needs are different.
John claimed that there are only two classes of jobs that are run on most systems today. There are "primary jobs," the tasks that you bought the system to run, and there are "secondary jobs," which are anything else that you do to take advantage of the machine's extra capacity.
John argued that current operating systems do a lousy job of prioritizing in this setting. For example, if you try running a simulator as a background job on a conventional UNIX workstation, it frequently has a noticeable impact on interactive foreground jobs. [Dave Presotto (Lucent Bell Labs) questioned the need to run things like simulations on desktop machines, arguing that there are usually other nodes on the network that are better designed for running large CPU- and memory-intensive tasks.]
John pointed out that the UNIX scheduler, a hold-over from time-sharing days, is not well suited to the desktop computer environment that he described. There are two problems. First, although the "nice" mechanism has twenty different settings, in actual use, it is very rare for anyone to use anything other than the highest and lowest settings. The second problem is that nice priorities only affect CPU scheduling, so the scheduling is not effective when the system is bottlenecked on resources other than the CPU.
John proposed a simpler scheduling scheme. Give the user a dipole switch, specifying whether or not a job is important. This raises the question of how the system should respond to this switch. What should it do to prioritize system resources?
John's idea is to use a new metrics, "system memory cycles per instruction" (or MCPI) to evaluation. This measurement extends the conventional notion of MCPI to include the overhead incurred by all of the levels of the memory hierarchy, including disk access time during I/O system calls and during page faults.
To use system MCPI, you would start by measuring the system MCPI of you primary applications when they run by themselves on the machine. Since you brought the machine to run these applications, the expectation is that you will get a relatively good system MCPI. Then, while the system is running a multiprogrammed workload, it would monitor the system MCPI of the primary application(s), and attempt to improve the resource allocation policies for those applications if their system MCPI diverges too far from that observed when they ran by themselves.
Discussion:
The final presentation of the Performance Dynamics session was by Brian Noble of Carnegie Mellon University. Brian's paper was titled "The Role of Trace Modulation in Building Mobile Computing Systems." Brian's research attacks the problem of evaluating the performance of wireless communication systems. These systems are typically difficult to conduct experiments and the nature of the medium frequently results in irreproducible measurements.
There are a variety of complications that arise when trying to measure the performance of a wireless communication system. Performance can vary with the location of mobile system and with the amounts of interference in its environment. The wireless media itself is difficult, if not impossible, to isolate, as other devices, including those not under the control of the researchers (e.g., in nearby labs, buildings, etc.) can generate interferences. Furthermore, outdoor experiments can be complicated by the weather. All of these complexities are multiplicative, making it nearly impossible to conduct reproducible experiments. As a result, it is extremely difficult to explore new networking technologies and topologies.
Brian's solution is to build a realistic simulation environment on top of a real network, such as ethernet. He modifies the network layer of the operating system's protocol stack to drop or delay packets in response to an input trace. This allows the wireless network environment to be simulated in a reproducible manner. Once this simulation environment has been constructed, a researcher can run whatever benchmarks he/she is interested in.
Brian described three different types of traces that can be used to drive the system. Synthetic traces are idealized functions that are typically generated by simple programs. Empirical traces are based on measured performance characteristics of a single machine traversing a path through a wireless networking environment. Hybrid traces allow the researcher to combine multiple traces by selecting between them based on a location simulator. Thus, hybrid traces allow a researcher to collect empirical traces from a variety of locations and conditions and then combine them based on a hypothetical traversal through the environment from which the traces were collected.
There were a few caveats that Brian issued about this system. First, a single trace captures only a single point is a very large space of possible performance profiles, so while it is possible to realistically simulate a wireless environment, it is impossible to exhaustively simulate all of the possible conditions that might be encountered in that environment. Brian also pointed out that all of the components of this simulation environment are live, other than the network and because the low level is still a simulation, it is possible to miss some subtle phenomena.
In conclusion, Brian reported that the system is almost entirely real, and works well. They have done work to compare their simulated environment to real wireless networks and have found that their simulation is very accurate. They have an upcoming SIGCOMM paper on this system.
Discussion: