Comprehending or summarizing large multi-dimensional datasets is a challenging task, often requiring extensive exploration by a user. While many tools exist to help a user produce a visualization of some aspect of a dataset, very few tools exist that help a user choose which aspects to visualize next. We present a framework for interactive data summarization in which the user is guided towards content that deviates from the currently observed visualizations. The advantage is to help users produce better summaries because they spend their time evaluating data that is of interest and are less likely to miss important exceptions to their observations. We demonstrate our techniques in a sample application for summarizing correlations between fields in a large datasets.