A user-level, user-extensible network interface architecture

Matt Welsh
David Oppenheimer

26 August 1998

Abstract. U-Net/SLE (Safe Language Extensions) is a network interface architecture supporting both direct user-level access and virtualization (as in the case of U-Net and U-Net/MM) as wellas user extensibility through a virtual machine implemented on the NI. The virtual machine is intended to be a subset of the Java VM, thus providing ease-of-use, powerful programming abstractions, and buzzword-compliance.
Papers. U-Net/SLE: A Java-based User-Customizable Virtual Network Interface. M. Welsh, D. Oppenheimer, and D. Culler. To be presented at the Java for High-Performance Network Computing workshop at EuroPar '98, Southampton, England, September 4, 1998.

User Customization of Virtual Network Interfaces with U-Net/SLE. D. Oppenheimer and M. Welsh. UC Berkeley Tech Report CSD-98-995, December, 1997. Other formats available here.

Poster. The slides from a poster providing an overview of U-Net/SLE, presented for UC Berkeley CS262, December 12, 1997, are available here on-line. Note that these results are very preliminary and further work must be done before the system is released.
News. 20 December 1997: We have completed a prototype implementation of U-Net/SLE using the Myricom Myrinet network interface and the Solaris 2.x operating system, although the software itself should be portable across U-Net implementations. We have implemented a subset of the Java Virtual Machine (which we call Java Implementation for Vertical Extensions, or JIVE), and are now able to run Java applets on the Myrinet LanAI processor and allow these applets to interact with the operation of U-Net.

JIVE is a simplified Java VM which does not implement many of the features supported by Java, such as multiple classes, inheritance, and virtual methods. Instead all applet code (for a particular U-Net endpoint) must be contained within a single class, UnetSLEApplet. The idea here is that we wish to exploit the type- and computation-safety of the Java language without having to deal with the overhead of the many object-oriented features Java provides. In fact, we expect that the kinds of processing that JIVE applets will want to accomplish will have little use for much of Java's high-level functionality. JIVE is to Java what C is to C++.

Technically there is nothing limiting JIVE's implementation other than design time and the memory size of the Myrinet board (we are using 256K-populated SBus boards). We hope that JIVE can at least establish a baseline performance/functionality datapoint against which future systems can be compared.

Design overview. One drawback of user-level network interfaces is that all protocol processing must be executed at the user application level, meaning that protocol processing time is framed by the host operating system scheduler. Similarly, for incoming messages, an interrupt and a context switch is necessary before any processing can occur. Several projects have demonstrated that pushing protocol processing into the network interface has real performance advantages, however, these systems limit the flexibility of user applications to circumvent the NI-level processing. Additionally in all cases these mechanisms are implemented in an ad hoc manner and are not portable across different NIC architectures.

One potential solution is to enable user applications to specify and download their own protocol-specific code into the network interface. This approach raises several issues: safety, resource management, and performance, and flexibility.

To address the safety issue we propose that a safe language (such as Java or Modula-3) be used as the user extension language; projects such as SPIN and Exokernel have shown that these languages are applicable to enabling user-level extensions within a protected environment. Resource management is a more difficult issue to solve and involves preventing one application's extensions from consuming too many resources (such as CPU time or memory) on the (usually limited) NI. Performance considerations are important as programmable NICs often employ slow CPUs (37 MHz in the case of the Myrinet LanAI); one potential solution is to employ just-in-time compilation of the safe language code. Flexibility issues include defining the extent of the user-level extensions and how they may interact both with the NI and the user application.

The goal of U-Net/SLE (Safe Language Extensions) to extend the U-Net user-level network interface with a safe language virtual machine (for example, a subset of the Java VM) enabling applications to download their own protocol-specific code into the NI. Note that this architecture is independent of NIC hardware; the SLVM may be implemented either on the CPU of a programmable network interface or on the host within a protected co-routine. This design maps closely to the design of U-Net on both programmable and non-programmable NICs.

With the U-Net/SLE architecture, applications may either implement protocol code directly (as in the case with the current U-Net design) or provide NI-level extensions, or both. U-Net/SLE also allows NI-level protocol code to be portable across a range of NIC architectures, rather than implementing the code ad-hoc for a particular hardware configuration. Furthermore, U-Net/SLE allows the network interface to be adapted to unforseen protocol and application needs; rather than recoding the NI-specific protocol stack when application usage changes, applications can instead tailor their protocol code to their own needs. Obviously, it is up to applications to maintain the correct balance between application- level and NI-level processing code; future work will hopefully demonstrate where the boundary should be drawn in general.

Analysis of the U-Net/SLE design will involve a cross-comparison of various network protocols implemented at various levels. For instance, we hope to compare AM-II as implemented over U-Net, as implemented directly on the Myrinet LanAI, and as implemented using U-Net/SLE. Even if the performance of the U-Net/SLE implementation lags somewhat behind that of the ad hoc Myrinet implementation, we believe that the flexibility and extensibility of U-Net/SLE will outweigh these concerns.

Other applications of U-Net/SLE involve continuous media transfer protocols (which often have very different flow-control and reliability needs than traditional messaging layers); multicast and group-synchronization protocols, such as barriers (for which small ACKs or tokens need not require application intervention); packet filtering for TCP and UDP; as well as various coscheduling algorithms for clusters of workstations. Our intent is to develop a general-purpose, user-extensible network interface architecture which can be readily applied to all of these applications.

References. U-Net, a user-level network interface architecture. Home page at Cornell.

SPINE, a Safe Programmable and Integrated Networking Environment. SOSP 1997 Work-in-progress page.

Mark Fiuczynski's home page and SPINE pages.

Matt Welsh's research page, including information on U-Net/MM and U-Net/Myrinet.

matt welsh home page