Birefractive stereo imaging is developed to capture color and depth information simultaneously from a single-shot image

The double refraction phenomenon was creatively innovated by a KAIST research team for stereo imaging. This breakthrough imaging technology enables us to capture color and depth information from a single shot image with double refraction. Instead of requiring multiple stereo cameras, just a single image is sufficient to capture depth information. This technology is expected to revolutionize various 3D imaging applications.

Article  |  Fall 2016

Depth estimation from images is very useful for many applications besides computer graphics, such as robotics, autonomous vehicles, augmented reality, and robotics. Common binocular/multiview stereo imaging techniques require at least two cameras to estimate disparity, resulting in additional cost and a large form factor. Many alternative approaches requiring a single camera have been proposed; however, they all present several shortcomings or require additional hardware, hindering their applicability in everyday scenarios.

A KAIST research team called the visual computing laboratory (http://vclab.kaist.ac.kr) led by Professor Min H. Kim developed a novel stereo imaging technology [1] (shown in Figure 1) as a joint project with the University of Zaragoza. The team extended their prior works [2] and [3] to develop a novel birefractive depth acquisition method, which allows for single-shot depth imaging by simply placing a birefringent material in front of the lens. While most transmissive materials present a single refractive index per wavelength, birefringent crystals like calcite posses two, resulting in a double refraction effect (Figure 2). The team developed an imaging model that leverages this phenomenon and the information contained in the ordinary and the extraordinary refracted rays, providing an effective formulation of the geometric relationship between scene depth and double refraction. They demonstrated several 3D stereo imaging applications including refocusing, anaglyph generation, automatic object segmentation, and depth-aware image editing.

As shown in Figure 1, this is easily achieved by simply placing a birefringent material in front of the lens of any conventional camera. It is therefore cheap, has a minimal impact on the form factor of the setup, does not sacrifice spatial resolution, introduces no visible image degradation (as the results show), and is readily useful for personal photography, allowing users to capture images without the burden of complicated setups or complex additional hardware. Figure 3 presents a captured image using the camera with calcite and an estimated depth map.

 
Figure 4 shows a refocusing application in computational photography. They propose a birefractive stereo image formation model to estimate a depth map from a double-refraction image. The proposed image formation can build a set of deconvolution kernels for each depth; an example is shown in the inset, consisting of the two-point spread functions for the o-ray and the e-ray. By applying different blur kernels across the estimated depths, a user can refocus the captured photograph.

This research falls in the realm of computational photography, where the team places an optical element to record the double refraction information arriving at the sensor; this double refraction is later decoded to extract additional information (depth in this case). Within this core research field, single-shot depth acquisition is an open problem worth investigating; although it is true that multiple-shot algorithms have the advantage of additional captured information, they are still mostly impractical in personal photography, as most people carry and use a single camera. This team has developed a novel birefractive stereo imaging model that they believe is a valuable contribution as it offers a compact and cost-effective alternative, does not require placing optical elements such as a coded aperture inside the lens, and can be used with mobile phones or, in the near future, in setups where the form factor imposes severe constraints, such as endoscopic imaging. This work [1] is to appear in the top computer graphics journal, ACM Transactions on Graphics (TOG) and will be presented at ACM SIGGRAPH Asia 2016.

[1] Birefractive Stereo Imaging for Single-Shot Depth Acquisition,
Seung-Hwan Baek, Diego Gutierrez, Min H. Kim
ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2016)
http://vclab.kaist.ac.kr/siggraphasia2016p1/