Research led by University of California San Diego’s Manmohan Chandraker to improve the way intelligent machines “reason” about the physical, three-dimensional world has received the CAREER Award from the National Science Foundation. The work establishes connections between computer vision, machine learning, computer graphics and perception.
Chandraker’s research has important applications for machines and systems that must navigate or interact with the real world, such as light field cameras, augmented reality devices, self-driving cars, or underwater robots used for exploration. Other applications include motion capture of humans and animals, gesture control and medical imaging.
Such intelligent, vision-based systems must be able to reason about 3D geometric and semantic properties while accounting for complex factors of variation such as shape, material, illumination, participating media and visibility. Chandraker’s research is poised to enable better semantic 3D scene understanding in turbulent water or foggy weather, for example, or could make virtual objects appear more realistic when rendered in real scenes.
“Recent advances in deep learning have led to impressive gains in several areas,” explains Chandraker, who is an Assistant Professor of Computer Science and Engineering and an affiliate of the UC San Diego Jacobs School of Engineering. “But generalization of their purely data-driven approach to handle such complexities is expensive. Our research seeks to handle the diversity of real-world images by incorporating physical models of image formation within deep learning frameworks.”
“For deep learning,” Chandraker continues, “we want to leverage insights from geometric and physical modeling to enable greater accuracy, generalization or ease of training. For 3D reasoning, we want to obtain high-quality shape and semantics in the face of real-world challenges due to material, illumination or medium.”
To conduct their research, Chandraker and his team plan to use physically-motivated models of image formation, such as simulation of training data, designing novel architectures, or using better optimization methods. The research will also create a cross-disciplinary education program in vision, graphics, learning and perception, with educational modules that provide insights into novel technologies such as self-driving cars or augmented reality.
The Faculty Early Career Development (CAREER) Program offers NSF's most prestigious awards in support of early-career faculty who have the potential to serve as academic role models in research and education and to lead advances in the mission of their department or organization.
“Humans are intuitively able to reason about shape and semantics, even in very complex situations,” notes Chandraker. “Our work seeks to enable computers to do the same. I think the combination of domain knowledge with data-driven priors is a promising way to achieve that goal.”