research

I work on problems in computer vision and computational photography. I am interested in exploring statistical properties of natural images and scenes, and developing practical inference algorithms that exploit their structure for different applications. This page contains brief descriptions of select research projects, including links to associated publications.

	Photography with Color-Coded Apertures We propose modifying the aperture of a conventional color camera so that the effective aperture size for one color channel is smaller than that for the other two. This produces an image where different color channels have different depths-of-field, and from this we can computationally recover scene depth, reconstruct an all-focus image and achieve synthetic re-focusing, all from a single shot. These capabilities are enabled by a spatio-spectral image model that encodes the statistical relationship between gradient profiles across color channels. This approach substantially improves depth accuracy over alternative single-shot coded-aperture designs, and since it avoids introducing additional spatial distortions and is light efficient, it allows high-quality deblurring and lower exposure times. We demonstrate these benefits with comparisons on synthetic data, as well as results on images captured with a prototype lens. Work with: Todd Zickler. Publications: ECCV'12. Links: Project Page.
	Statistics of Natural Hyperspectral Images Hyperspectral images provide higher spectral resolution than typical RGB images by including per-pixel irradiance measurements in a number of narrow bands of wavelength in the visible spectrum. The additional spectral resolution may be useful for many visual tasks, including segmentation, recognition, and relighting. Vision systems that seek to capture and exploit hyperspectral data should benefit from statistical models of natural hyperspectral images, but at present, relatively little is known about their structure. Using a new collection of fifty hyperspectral images of indoor and outdoor scenes, we derive an optimized "spatio-spectral basis" for representing hyperspectral image patches, and explore statistical models for the coefficients in this basis. Work with: Todd Zickler. Publications: CVPR'11. Links: Project Page (with database).
	Spatially Varying Blur Blur in an image, be it motion or defocus, is often spatially varying. We seek to estimate the parameters of this kind of blur— to be able to "de-blur" the image; and also because blur encodes useful information about the scene (depth, what's moving, etc.). We develop a model for what local sharp image regions look like and use that to reason about blur locally. We then combine this "blur cue" with color information to develop a method to segment out moving objects from a scene and estimate the direction and "speed" of motion. The method performs encouragingly on real world images— including those taken from consumer-level and cell-phone cameras. Work with: Todd Zickler, Bill Freeman. Publications: CVPR'10. Links: Project Page (with source code).
	Camera Pipelines As computer vision applications begin to get deployed "in the wild"— i.e., on images taken from the internet and personal photo collections— researchers have to deal with data from uncalibrated consumer-level cameras that are designed to produce images that are visually "pleasing", not photometrically accurate. We collect a database of calibrated images from various cameras to gauge the complexity and variability of mappings from the spectral distribution of incoming light to rendered pixel values. We verify the linearity of camera sensors, explore the variability in color spaces and the post-capture non-linear processing in each camera's pipeline. Work with: Daniel Scharstein, Todd Zickler. Publications: BMVC'09. Links: Project Page (with database).
	Color Constancy with Spatio-Spectral Statistics Often for recognition and other tasks, we want "color constancy"— a color representation of materials that doesn't depend on the color of the illuminant. We show that the strong correlations that exist between a pixel and its neighbors encode useful information about the illuminant color, and develop an algorithm to leverage this information efficiently for color constancy. This algorithm is able to incorporate knowledge about illuminant statistics when available, and performs well when evaluated on a database of natural images. Work with: Keigo Hirakawa, Todd Zickler. Publications: IEEE Trans. PAMI '12, CVPR'08. Links: Project Page (with source code).

You can download my PhD. dissertation here and defense slides here.