An Infrastructure for Connecting Sensor Networks and Applications
 | The Harvard Hourglass project is building a scalable, robust data collection system to support geographically diverse sensor network applications. Hourglass is an Internet-based infrastructure for connecting a wide range of sensors, services, and applications in a robust fashion. In Hourglass, streams of data elements are routed to one or more applications. These data elements are generated from sensors inside of sensor networks whose internals can be entirely hidden from participants in the Hourglass system. The Hourglass infrastructure consists of an overlay network of well-connected dedicated machines that provides service registration, discovery, and routing of data streams from sensors to client applications. In addition, Hourglass supports a set of in-network services such as filtering, aggregation, compression, and buffering stream data between source and destination. Hourglass also allows third party services to be deployed and used in the network. Overview | Publications | Members | Related Projects |Internal |
Overview
Sensor networks are becoming a pervasive part of our environment, and promise to deliver real-time data streams for applications such as environmental monitoring, structural engineering, and health care. Existing sensor networks are single purpose in that they communicate with a limited number of external systems through proprietary interfaces.
To address these issues, we are developing a scalable, robust data collection system, called Hourglass. The Hourglass infrastructure is a collection of Internet-connected systems that provides mechanisms for discovery of sensor network data and routes data streams from providers to requesters in a fault tolerant, delay-sensitive fashion. Sensor networks and end-user applications interface with Hourglass to publish locally-generated data streams or request streams of interest. The systems making up the Hourglass "core" are well-connected, well-provisioned servers maintained by organizations providing the Hourglass service.
Apart from providing a robust stream dissemination infrastructure, Hourglass supports a range of in-network services to facilitate efficient discovery, processing, and delivery of sensor network data. These services include filtering, compression, aggregation, and storage of event streams within the Hourglass. Hourglass dynamically adapts to changing network conditions and node failures by allocating in-network services to nodes to meet performance and reliability targets. For example, to reduce bandwidth requirements, a filtering service can be instantiated near an event source to filter out non-critical data.
The Hourglass architecture leverages recent research in overlay networks and peer-to-peer architectures for constructing self-organizing, robust services from a collection of hosts distributed across the Internet. Our approach differs from previous work in that it focuses on real-time event delivery for sensor network applications, the incorporation of mobile hosts and intermittent connectivity to clients, and dynamic in-network processing.
To facilitate the implementation of large-scale stream-based applications, such as Hourglass, we propose a novel underlying network infrastructure called a Stream-Based Overlay Network (SBON). SBONs are intended to support a new class of Internet-based applications that pull data from one or more streaming sources on the Internet and process the data in the network as it is delivered to potentially multiple end-user applications. By abstracting away the details of data path optimization, service naming and service discovery, SBONs will greatly simplify the development of Internet-based stream processing systems.
More details can be found on the SBON.
Publications
- Cobra: Content-based Filtering and Aggregation of Blogs and RSS Feeds
In Proceedings of the 4th USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI 2007), Cambridge, MA, April 2007.
Ian Rose, Rohan Murty, Peter Pietzuch, Jonathan Ledlie, Mema Roussopoulos, and Matt Welsh.
Available as: PDF - Network-Aware Operator Placement for Stream-Processing Systems
Proceedings of the 22nd International Conference on Data Engineering (ICDE'06), Atlanta, GA, April 2006
Peter Pietzuch, Jonathan Ledlie, Jeffrey Shneidman, Mema Roussopoulos, Matt Welsh, and Margo Seltzer
Available as: PDF - Supporting Network Coordinates on PlanetLab
Proceedings of the Second Workshop on Real, Large Distributed Systems (WORLDS'05), San Francisco, CA, December 2005
Peter Pietzuch, Jonathan Ledlie, and Margo Seltzer
Available as: PDF - A Cost-Space Approach to Distributed Query Optimization in Stream Based Overlays
Proceedings of the 1st IEEE International Workshop on Networking Meets Databases (NetDB'05), Tokyo, Japan, April 2005
Jeff Shneidman, Peter Pietzuch, Matt Welsh, Margo Seltzer, and Mema Roussopoulos
Available as: PDF - Evaluating DHT-Based Service Placement for Stream-Based Overlays
Proceedings of the 4th International Workshop on Peer-to-Peer Systems (IPTPS'05), Ithaca, New York, February 2005
Peter Pietzuch, Jeff Shneidman, Jonathan Ledlie, Matt Welsh, Margo Seltzer, and Mema Roussopoulos
Available as: PDF - Path Optimization in Stream-Based Overlay Networks
Harvard Technical Report TR-26-04
Peter Pietzuch, Jeffrey Shneidman, Mema Roussopoulos, Margo Seltzer, Matt Welsh
Available as: PDF - Hourglass: A Stream-Based Overlay Network for Sensor Applications
Harvard Industrial Partnership (HIP'04)
Peter Pietzuch, Jeff Shneidman, Jonathan Ledlie, Matt Welsh, Margo Seltzer, Mema Roussopoulos
Available as: PDF (poster) - Hourglass: An Infrastructure for Connecting Sensor Networks and Applications
Harvard Technical Report TR-21-04
Jeffrey Shneidman, Peter Pietzuch, Jonathan Ledlie, Mema Roussopoulos, Margo Seltzer, Matt Welsh
Available as: PDF - Open Problems in Data Collection Networks
SIGOPS European Workshop, Leuven, Belgium, September 2004
Jonathan Ledlie, Jeffrey Shneidman, Matt Welsh, Mema Roussopoulos, Margo Seltzer
Available as: PDF - Hourglass Data Collection Network
Harvard Industrial Liason Program (HIP'02)
Jeffrey Shneidman, Bryan Choi
Available as: PDF (poster) - Collecting Data for One Hundred Years
Harvard Technical Report: Fall 2002 Work in Progress Description
Jeffrey Shneidman, Bryan Choi, Margo Seltzer
Available as: PDF
Current Members
- Jonathan Ledlie
- Rohan Murty
- Peter Pietzuch
- Jeff Shneidman
Related Projects
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. 0330244. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.