U-Net Installation Notes


Last modified 21-Jan-1998 by M. Welsh

These are the installation notes for U-Net version 2.1, 13-January-1998. Please see the U-Net documentation tree and release notes first.

Overview

Installation of U-Net is not automated, but it's meant to be quite easy. There are two steps: Compiling the code, and installing the various pieces. Compilation is covered in the next section, and installation for each implementation is covered later in this document.

The first step is to unpack the U-Net release distribution; create a directory unet-2 and unpack the gzipped tarfile there. The subdirectories are described in the Release Notes.

Compiling the Software

Before you compile the U-Net release, please check the Makefiles in each directory relevant for your release. All of the paths and CFLAGS should be correctly set; however, you may need to make small changes (e.g., if you are using a compiler other than gcc). Note that the SBA-200 device driver for Solaris is compiled using the SparcWorks cc compiler, and for SunOS is compiled using cc as shipped with the operating system. All other code is compiled using gcc.

Note that all Makefiles assume use of GNU make. If you don't have (or want to use) GNU make, you'll need to modify the Makefiles appropriately.

If you are using a FORE 200-series ATM board, please edit the file utils/uload-fore200/Makefile and modify the TOOLIBPATH variable for your system. You need to have toolib.a as compiled from the GNU i960 tools distribution (see the release notes for details). The subdirectory utils/uload-fore200/i960-binaries contains a binary of this library which may work for you; however, you need to modify the TOOLIBPATH in this Makefile anyway.

Other than this we hope that no other Makefile changes are necessary. Please let us know if this isn't so.

To compile the release, issue the command

make ARCH=architecture DEV=device
from the top-level unet-2 directory, where architecture is one of and device is one of This does not imply that all combinations of the above are supported.

For example, to compile for the PCA-200 under Linux, issue

make ARCH=linux DEV=pca200
This should compile all of the components for a particular operating system. You can also issue this command in the various subdirectories by hand, although no checks are made that ARCH and DEV are set in that case. Be careful!

To remove all binary files for a given architecture/device, use

make ARCH=architecture DEV=device clean

Once compiled, the various binary files will be left in the subdirectories

bin-device-architecture
of each subdirectory. You will need to follow the installation instructions below to make use of the binaries.

Installing the Kernel Driver

Notes on installing each U-Net driver are included below.

Tulip device, Linux

First, patch your kernel with the bigphysarea patch in install/linux. This adds several files to /usr/src/linux/init to allocate physical memory at boot time (the amount is controlled in init/unet.h) which can be used for U-Net endpoint mappings. Patches are available for 1.3.71 and 2.0.30 kernels.

If you are using Linux 2.0.30 or later, you also need to patch the kernel with install/linux/linux-2.0.30.tulip.patch which adds two entries to the list of symbols exported by the kernel for kernel modules. You can apply this patch before or after the bigphysarea patch.

Install the new kernel and reboot. Be sure to use the LILO boot option

  bigphysarea=
to specify the number of pages which should be pinned down for U-Net buffers systemwide. For example,
  bigphysarea=256
should suffice for most systems.

The Tulip device driver is located in dev-tulip-linux/tulip.o. It is a stand-alone kernel module. Simply load it with

insmod tulip.o
Once installed the device should print messages such as the following:
tulip: Found device index 0, bus 0, function 160.
tulip: Revision 0x12, using IRQ 9.
tulip: CBIO: 0xfc00, CBMA: 0xf0cffc00
tulip: Setting latency timer from 64 to 100 clocks.
tulip: MAC address 00:00:c0:17:bc:c7
tulip0: Rx ring at 0x00a34034, Tx ring at 0x00a34434
tulip: 1 devices found.
If you see error messages which you don't understand, please
contact us.

In order to use the device you need to create device nodes in /dev. The major number of the U-Net device is defined in devtulip.h. If this major number conflicts with your system, change it and recompile the driver.

The device numbering scheme works as follows: The minor number of the device corresponds to a single endpoint. devtulip.h contains a definition for TULIP_MAX_ENDPTS, the maximum number of endpoints supported by each physical Tulip device. If TULIP_MAX_ENDPTS is 10, minor numbers 0 through 9 correspond to the first Tulip device in the system, 10 through 19 to the second, and so forth.

You should create at least one minor device node in order to use the system; you may wish to create one for each potential endpoint. For each, execute the command

mknod /dev/tulipN c major N
Where major is the major number (say, 33) and N is the minor number as described above.

You should also chmod 777 the device entries, so that non-root users can use them.

Now you're ready to test it out; see the section Testing below.

PCA-200 device, Linux

First, patch your kernel with the bigphysarea patch in install/linux. This adds several files to /usr/src/linux/init to allocate physical memory at boot time (the amount is controlled in init/unet.h) which can be used for U-Net endpoint mappings. Patches are available for 1.3.71 and 2.0.30 kernels. Install the new kernel and reboot. Be sure to use the LILO boot option

  bigphysarea=
to specify the number of pages which should be pinned down for U-Net buffers systemwide. For example,
  bigphysarea=256
should suffice for most systems.

The PCA-200 device driver is located in dev-pca200-linux/pca200.o. It is a stand-alone kernel module. Simply load it with

insmod pca200.o
Once installed the device should print messages such as the following:
pca200: Found device index 0, bus 0, function 144.
pca200: Revision 0x0, using IRQ 11.
pca200: Board memory at 0xf0d00000 mapped at 0x0181c000.
pca200: Setting latency timer from 0 to 100 clocks.
pca200: MAC address prom of device 0 reads 00:20:48:17:03:27

In order to use the device you need to create device nodes in /dev. The major number of the U-Net device is defined in devpca200.h. If this major number conflicts with your system, change it and recompile the driver.

You should create at least one minor device node in order to use the system; you may wish to create one for each potential endpoint. For each, execute the command

mknod /dev/pcaN c major N
Where major is the major number (say, 33) and N is the minor number as described above.

You should also chmod 777 the device entries, so that non-root users can use them.

The next step is to load the i960 firmware onto the board. This can be done at any time after the PCA-200 device driver is loaded. See Loading the Firmware, below.

Myrinet device, Linux

First, patch your kernel with the bigphysarea patch in install/linux. This adds several files to /usr/src/linux/init to allocate physical memory at boot time (the amount is controlled in init/unet.h) which can be used for U-Net endpoint mappings. Patches are available for 1.3.71 and 2.0.30 kernels. Install the new kernel and reboot. Be sure to use the LILO boot option

  bigphysarea=
to specify the number of pages which should be pinned down for U-Net buffers systemwide. For example,
  bigphysarea=256
should suffice for most systems.

The Myrinet device driver is located in dev-myrinet-linux/unet-myrinet.o. It is a stand-alone kernel module. Simply load it with

insmod unet-myrinet.o
Once installed the device should print messages such as the following:
unet-myrinet: Loading devmyrinet-linux.c (Oct 30 1997 09:17:29)
unet-myrinet: EEPROM SRAM size reads 262144.
unet-myrinet: Found device index 0, bus 0, function 96.
unet-myrinet: Revision 0x1, using IRQ 9.
unet-myrinet: regbase 0xfdc00000, kernel_regbase 0x4e37000
unet-myrinet: membase 0xfdc00000, kernel_membase 0x4e37000
unet-myrinet: 1 devices found.

In order to use the device you need to create device nodes in /dev. The major number of the U-Net device is defined in devmyrinet.h. If this major number conflicts with your system, change it and recompile the driver.

You should create at least one minor device node in order to use the system; you may wish to create one for each potential endpoint. For each, execute the command

mknod /dev/myriN c major N
Where major is the major number (say, 46) and N is the minor number as described above.

You should also chmod 777 the device entries, so that non-root users can use them.

The next step is to load the LanAI firmware ("LCP") onto the board. This can be done at any time after the Myrinet device driver is loaded. See Loading the Firmware, below.

SBA-200 device, SunOS 4.1.x

The SBA-200 device driver is a standalone kernel module, located in dev-sba200-sunos/sba200.o. Simply load it with a command such as

modload -o /tmp/sba200.o sba200.o
Once installed the device should print messages such as the following:
sba200 U-Net driver loaded.
FORE,sba-2000 at  SBus slot 0 0x800000 and  SBus
 slot 0 0x1000000 pri 3 (sbus level 2)

In order to use the device you need to create device nodes in /dev. The major number of the U-Net device is defined in devsba200-sunos.h. If this major number conflicts with your system, change it and recompile the driver.

You should create at least one minor device node in order to use the system; you may wish to create one for each potential endpoint. For each, execute the command

mknod /dev/sbaN c major N
Where major is the major number (say, 33) and N is the minor number as described above.

You should also chmod 777 the device entries, so that non-root users can use them.

The next step is to load the i960 firmware onto the board. This can be done at any time after the SBA-200 device driver is loaded. See Loading the Firmware, below.

Note that use of the U-Net driver for SunOS is probably incompatible with use of the FORE Systems SBA-200 kernel driver. For best results use a SunOS kernel without the FORE Systems driver installed to prevent conflicts.

SBA-200 device, Solaris 2.x

The SBA_200 device driver is a standalone kernel module, located in dev-sba200-solaris/sba200. Use of the U-Net SBA-200 driver is probably incompatible with use of the original FORE SBA-200 driver (as they attach to the same device properties). You should uninstall your original FORE driver first, saving copies of the various files modified below. To install the driver,

  1. Copy the files sba200 and sba200.conf to a directory on your kernel driver path (probably /usr/kernel/drv).
  2. Add the following lines to /etc/devlink.tab:
    # U-Net/SBA-200 driver
    type=ddi_pseudo;name=FORE,sba-200       sba\N0
           
    Be sure to delete any lines associated with the FORE SBA-200 driver.
  3. Copy /etc/system to /etc/system.nounet and add the following line to /etc/system.nounet:
    exclude: drv/sba200
    If installation or use of the U-Net driver causes problems, you can always boot using the etc/system.nounet file to prevent the driver from being loaded at boot time.
  4. Execute the command:
    add_drv -i 'FORE,sba-200' -m '* 0666 root bin' sba200

This will load the device driver into the system, and automatically create /dev/sbaNNN entries for each endpoint, with the numbering system described above.

The next step is to load the i960 firmware onto the board. This can be done at any time after the SBA-200 device driver is loaded. See Loading the Firmware, below.

Note that use of the U-Net driver for SunOS is probably incompatible with use of the FORE Systems SBA-200 kernel driver. For best results unload the FORE Systems driver before loading the U-Net driver, and vice versa.

Myrinet device, Solaris 2.x

The Myrinet device driver is a standalone kernel module, located in dev-myrinet-solaris/unet-myrinet. Use of the U-Net Myrinet driver is incompatible with any other Myrinet driver or code, such as Myricom's original driver, Fast Messages, and so on. You should uninstall any other Myrinet driver before attempting to install U-Net. In most cases it is sufficient to simply unload the driver with a command such as:

rem_drv myrinet
rather than doing a complete uninstall of the software.

To install the driver,

  1. Copy the file unet-myrinet to a directory on your kernel driver path (probably /usr/kernel/drv).
  2. Add the following lines to /etc/devlink.tab:
    # U-Net/Myrinet driver
    type=ddi_pseudo;name=MYRICOM,mlanai	myri\N0
    	
  3. Copy /etc/system to /etc/system.nounet and add the following line to /etc/system.nounet:
    exclude: drv/unet-myrinet
    If installation or use of the U-Net driver causes problems, you can always boot using the etc/system.nounet file to prevent the driver from being loaded at boot time.
  4. Execute the command:
    add_drv -i 'MYRICOM,mlanai' -m '* 0666 root bin' unet-myrinet

This will load the device driver into the system, and automatically create /dev/myriNNN entries for each endpoint, with the numbering system described above.

The next is to load the LanAI firmware ("LCP") onto the board. This can be done at any time after the Myrinet device driver is loaded. See Loading the Firmware, below.

Loading the Firmware - PCA200 and SBA-200

For U-Net implementations on FORE Systems ATM boards, the i960 firmware must be loaded after installing the appropriate kernel driver. The uload program included in the U-Net distribution is used.

There are different firmware binaries for each FORE architecture. They are:

  • fore-200/bin-sun4/sba200.unet2 for SBA-200 boards
  • fore-200/bin-pca200/pca200.unet for PCA-200 boards

The uload binary is located in utils/uload-fore200/bin-arch where arch is one of pca200, sba200-sunos, or sba200-solaris.

To download the i960 firmware using uload, do the following:

  1. Set the environment variable G960BIN to the directory where the objcopy binary (from the GNU i960 tools) is located. Precompiled binaries (which may not work for you) are found in utils/uload-fore200/i960-binaries/bin-arch.
  2. Run the command
    uload -r -a -s device firmware
    Where device is an entry in /dev corresponding to the appropriate physical board, and firmware is the filename of the appropriate firmware binary (see above).

    You should see messages such as the following:

           Downloading /tmp/comma00531
           Sending /tmp/comma00531, 311 blocks:
           Sector 311 38k
           
  3. Once the firmware has been downloaded, run the ureset command, found in utils/bin-arch/ureset:
    ureset device
    This will reset the U-Net device driver and the firmware code. You should see messages from the kernel such as:
           pca0: U-Net reset
           pca0: U-Net firmware magic is 0xf00d.
           pca0: U-Net firmware version is 0x5.
           
    If the firmware magic does not match, then the firmware was not loaded correctly. If the version number does not match, the kernel driver and firmware version do not correspond.
  4. If instead you see that the reset timed out, then the firmware is not running on the i960. You can run uload without the -a and -s switches, which will allow you to enter commands to the i960 monitor running on the board. For example, uload -r device (without a firmware argument) produces the following:
    Mon960
     
    Mon960 monitor for the Intel i960 CA (step 04)
    Version 1.0        HPA-200 V1.1 (12), Fore Systems, Inc. Apr  4 1994
    Copyright 1992, Intel Corporation
    =>
           
    In this way you can verify that uload is at least talking to the i960 monitor correctly.

At this point the i960 firmware should be running and you're ready to test the system out.

Loading the Firmware - Myrinet boards

For U-Net implementations on Myrinet LanAI3.x and LanAI4.x boards, the LanAI firmware (LCP, or LanAI Control Program) must be loaded after installing the appropriate kernel driver. The uload program included in the U-Net distribution is used.

There are different firmware binaries for each Myrinet board architecture. They are:

  • firmware/lanai4.x/bin-sbus/lcp for SBus boards
  • firmware/lanai4.x/bin-pci/lcp for PCI boards

The uload binary is located in utils/uload-lanai/bin-arch where arch is one of myrinet-linux, myrinet-solaris, or myrinet-bsdi.

Use of uload depends on a configuration file which informs the LCP of the Myrinet network topology. This is necessary so that the LCP knows the route from the local node to all other nodes on the network. The format of the file is

 [switchnum | nodename] [switchnum | nodename] ...
 [switchnum | nodename] [switchnum | nodename] ...
where switchnum is a decimal number identifying a switch, and nodename is an alphanumeric hostname. The first column of each line specifies the switch number for that line; the remaining columns each correspond to one port on that switch. The first port entry in a line corresponds to port 0 of that switch, the second port entry to port 1, and so on.

For example, a simple four-port switch with four hosts connected would use the configuration file:

0 host1 host2 host3 host4
while two four-port switches connected together might use:
0 host1 1     host2 host3
1 host4 host5 0     host6
Here, we see that port 1 of switch 0 connects to port 2 of switch 1 (where ports are numbered 0-3).

Needless to say the format of the configuration file must correspond to the topology of your network. If a port is left unconnected you can use a 'dummy' placeholder hostname for that port, but all hostnames must be unique.

The format of the configuration file is identical to that used by Myrinet Fast Messages from the University of Illinois. (Thanks to that group for allowing me to steal their route-generation code!)

Before running uload you must define the environment variable UNET_HOSTS which assigns a numeric ordering to hostnames in the network. For example,

export UNET_HOSTS="host1 host2 host3 host4"
will make host1 host number 0, host2 host number 1, and so forth. U-Net applications use host numbers rather than host names to identify machines on the network. The hostname-to-hostnumber mapping is defined systemwide by the setting of UNET_HOSTS when uload is run. Note that different hosts may have different name-to-number mappings (that is different values of UNET_HOSTS may be used). For simplicity it's probably best to use the same value of UNET_HOSTS on all network nodes, and to advertise the value of the variable so that user programs can determine the name-to-number mapping.

To download the LanAI firmware using uload, do the following:

  1. Create a configuration file specifying the topology of your network.
  2. Set the environment variable UNET_HOSTS as explained above.
  3. Run the command
    uload device lcpfile configfile
    Where device is an entry in /dev corresponding to the appropriate physical board, lcpfile is the filename of the appropriate firmware binary (see above), and configfile is the name of the configuration file.

Testing U-Net

The test directory contains three programs which test the basic U-Net functionality and the libunet library:

  • pingpong: Simple ping-pong test between two machines.
  • loopback: Loopback send-receive test.
  • window: Simple bandwidth test.
Note that loopback will not work for the Tulip device unless the driver is recompiled to use full-duplex mode. Full-duplex operation is not recommended for normal use (as this disables collision detection).

pingpong

The syntax for pingpong is

pingpong [send | recv] device addr num size
Where device is the U-Net device for the desired endpoint (e.g., /dev/sba0), addr is the address of the machine to send to or receive from, num is the number of pingpongs, and size is the size of each pingpong.

The addr argument is a string corresponding to the address of some other machine communicating via U-Net. It is converted by routines in libunet to the appropriate format for U-Net channel addresses (see U-Net Device Reference). This argument should be a string in one of three formats:

  • An integer corresponding to the VCI of the desired station. U-Net does not perform signalling tasks; it is assumed that the given VCI will simply be used by the U-Net device to send and receive messages from some other station. VCIs should be allocated by some other signalling mechanism (or hardwired on the switch).
  • An Ethernet MAC address of the form
    xx:xx:xx:xx:xx:xx.yy
    Where xx are hexadecimal digits corresponding to the Ethernet MAC address of the destination controller, and yy is the hex value of a single-byte ``port'' address on the receiving end. Port addresses are used to allow multiple endpoints/channels to use one physical device; the sending and receiving ends of a U-Net connection over Fast Ethernet should agree on the port address(es) used. Note that only one endpoint on a single physical network device can listen to a particular port.
  • A Myrinet host number and a port number in the form
    nnn/ppp
    Where nnn is the decimal hostnumber of the destination machine (defined by the setting of UNET_HOSTS when uload was run), and ppp is a decimal ``port'' address (in the range 0-255). Port addresses are used to allow multiple endpoints/channels to use one physical device; the sending and receiving ends of a U-Net connection over Myrinet should agree on the port address(es) used. Note that only one endpoint on a single physical network device can listen to a particular port.

The size argument is limited by the maximum size of a PDU supported by the U-Net device; see Device Reference for details.

For example, to pingpong 40-byte messages between two systems (one at VCI 100, the other at VCI 200), you would execute the commands:

Receiver: pingpong recv /dev/sba0 200 1000 40
Sender: pingpong send /dev/sba0 100 1000 40
The ``receiver'' should be started before the ``sender''. In pingpong, both stations send and receive packets, but the receiver receives first, while the sender sends first.

On the sending end you should see something like:

Activated channel 0.
Sending 1000 40-byte messages.
Sent 1000 40-byte messages in 0.107453 seconds.
107.452989 usec/message or 2.840086 Mbit/sec
Deactivating channel...
Destroying endpoint...
Closing and exiting...
This time corresponds to the round-trip ping-pong time, so divide by two for the one-way latency per message.

loopback

The syntax for loopback is

loopback device addr num size
With arguments are as described above. For loopback on ATM, you should have the VCI addr configured at the switch for send/receive, or use a loopback cable (connect send/receive ends of the fibre to both ports on the card), in which case the VCI is arbitrary. For loopback on Fast Ethernet, the Tulip driver must be compiled for full duplex operation and a loopback cable used (hack up a UTP5 and twist the Rx and Tx lines together). For loopback on Myrinet, everything will work fine if the Myrinet board is connected to a switch; if you wish to use a loopback cable you must recompile the LanAI firmware with MYRINET_LOOPBACK_CABLE defined in lcp.c.

You should see something like:

Got channel 0.
Sending 10000 40-byte messages.
Sent 10000 40-byte messages in 0.441592 seconds.
44.159210 usec/message or 6.910807 Mbit/sec
Deactivating channel...
Destroying endpoint...
Closing and exiting...

window

The syntax for window is

window [send | recv] device addr num size window-size
With arguments as described above. The sender sends window-size messages of size bytes each to the receiver, which sends a single (small) acknowdledgement after receiving window-size messages. This is repeated num times.

Note that for large values of window-size messages may be silently dropped, either in your switch or by the U-Net implementation (if not enough free buffers or receive FIFO entries are available, for example). There is some practical limit to the arguments to window which should be determined empirically; no intelligent flow-control is being used by this benchmark application!

On the sending end you should see something like:

Sending 3000 1000-byte messages, window size 30.
Sent 3000 1000-byte messages (window size 30) in 0.346486 seconds.
115.495324 usec/message or 69.266873 Mbit/sec
Deactivating channel...
Destroying endpoint...
Closing and exiting...


U-Net should now be working for you; you can modify your system boot scripts to automatically load the drivers and firmware, if you wish. Please see the Device Reference for information on programming the U-Net device.


Return to U-Net Documentation.


M. Welsh, mdw at cs dot berkeley dot edu