Note: we are currently experiencing severe network problems on our 256 Kb school Internet connection. Please be understanding! I am still looking for a site willing to mirror the WebLouvre exhibit (30 Mb in all), preferably in the USA. -WebLouvre[31].
Table: Summary of Internet growth. Notice that while traffic over the
Internet only doubled last year, the percentage of that traffic for
which the World Wide Web is responsible grew by a factor of 8.
Traffic on the Internet has been growing at an exponential rate for several years, but so far network technology has kept pace. There have been few growing pains, and bandwidth today is still plentiful despite annual predictions that the Internet is on the verge of collapse. We fear, however, that Internet growth is accelerating, and we believe that more efficient caching techniques are required to conserve Internet bandwidth and to allow Internet information servers to keep up with demand.
Table 
summarizes the Internet's growth over
the past few years, and also charts the growth of a relatively young
Internet service, the World Wide Web [1]. Practically
non-existent in 1991, the Web has grown in only 3 years to account for 16%
of the total Internet traffic. The Web is popular, not only because of
the ease with which it allows information to be published, but also for the
ease with which it allows information to be browsed.
Using Web client software, a click of the mouse can move megabytes of data across countless network hops. This ease of use has proved quite popular, and millions of new users are signing up to access the Web through online services.
The only problem with the Web is that it is inherently inefficient, because it operates as a cache-less distributed system. When two clients retrieve a document from the same server, the document is transmitted twice, regardless of the two clients' proximity. This places more load on the server and uses more network capacity than might be strictly necessary.
Some Web browsers have addressed this problem by adding local client caches. These caches enable a client to avoid transferring the same document twice. Another approach to reducing server load and network bandwidth usage is the deployment of Web proxies [29][25] that prevent two clients in the same local area from transferring the same document twice.
The problem with both of these solutions is that they are myopic. A client cache does nothing to reduce traffic to a neighboring computer, and a local proxy does not help any other local proxies. Furthermore, these caches are typically limited in size: 5Mb for a client cache, and 100Mb-1Gb for a proxy cache are typical values. Disks might be inexpensive, but disk space will always be a limited resource and local caches will only be able to cache the most popular items, resulting in many copies of the most popular items but no copies of only slightly less popular items.
A better solution allows clients to share cache space cooperatively. The degree to which a file is replicated should be proportional to that file's global popularity, and objects should be cached where clients can reach them most efficiently. A system can achieve both goals by relegating caching decisions to the server. A server has the ability to monitor the request stream and decide both when and where objects should be cached. In making these decisions, the server can take advantage of its understanding of the network topology and the file's access history to maximize bandwidth savings and reduce load.