|
To use their pool of resources efficiently, distributed
stream-processing systems push query operators to nodes
within the network. Currently, these operators, ranging
from simple filters to custom business logic, are placed manually
at intermediate nodes along the transmission path to meet
application-specific performance goals. Determining placement
locations is challenging because network and node conditions change
over time and because streams may interact with each other, opening
venues for reuse and repositioning of operators.
This paper describes a stream-based overlay network (SBON), a layer
between a stream-processing system and the physical network that
manages operator placement for stream-processing systems. Our design
is based on a cost space, an abstract representation of the network
and on-going streams, which permits decentralized, large-scale
multi-query optimization decisions. We present an evaluation of the
SBON approach through simulation, experiments on PlanetLab, and an
integration with Borealis, an existing stream-processing engine.
Our results show that an SBON consistently improves network
utilization, provides low stream latency, and enables dynamic
optimization at low engineering cost.
|
