Replicate physical objects for AR/VR using a smartphone

Prof. Min H. Kim’s team of the KAIST School of Computing has developed a new method that replicates physical objects for augmented and virtual reality (AR/VR) using just a single smartphone, without any need for additional, and oftentimes expensive, supporting hardware.

Article  |  Fall 2019

To faithfully reproduce a real-world object in the VR/AR environment, we need to replicate the 3D geometry and appearance of the object. Traditionally, this has been either done manually by 3D artists, which is a labor-intensive task, or by using specialized, expensive hardware. Those setups might include a 3D laser scanner or multiple cameras, or a lighting dome with more than a hundred light sources. In contrast, this new technique only needs a single camera with a built-in flash, to produce high-quality outputs.

“Many traditional methods using a single camera can capture only the 3D geometry of objects, but not the complex reflectance of real-world objects, given by the SVBRDF,” notes Kim. The acronym SVBRDF, which stands for spatially-varying bidirectional reflectance distribution functions, is key in obtaining an object’s real-world shape and appearance. “Using only 3D geometry cannot reproduce the realistic appearance of the object in the AR/VR environment. Our technique can capture high-quality 3D geometry as well as its material appearance so that the objects can be realistically rendered in any virtual environment. It is also straightforward, cheap, and efficient, and reproduces realistic 3D objects by just taking photos from a single camera with a built-in flash.”

 

The research team demonstrated their framework in a series of examples in their paper, “Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography.” The novel algorithm, which does not require any input geometry of the target object, successfully captured the geometry and appearance of 3D objects with basic, flash photography and reproduced consistent results. Examples that were showcased in the work included a diverse set of objects that spanned a wide range of geometries and materials, including metal, wood, plastic, ceramic, resin, and paper, and comprised complex shapes like a finely detailed mini-statute of Nefertiti.

Prof. Kim and his collaborators, Diego Gutierrez, Professor of computer science at Universidad de Zaragoza in Spain, and KAIST PhD students Giljoo Nam and Joo Ho Lee, published the new study in ACM Transactions on Graphics (TOG) and presented it at ACM SIGGRAPH Asia 2018 in Tokyo. In future work, they plan to further simplify the capture process or to extend the method to larger scenes.