Deep Non-Line-of-Sight Reconstruction

In proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
 

Abstract

The recent years have seen a surge of interest in methods for imaging beyond the direct line of sight. The most prominent techniques rely on time-resolved optical impulse responses, obtained by illuminating a diffuse wall with an ultrashort light pulse and observing multi-bounce indirect reflections with an ultrafast time-resolved imager. Reconstruction of geometry from such data, however, is a complex non-linear inverse problem that comes with substantial computational demands. In this paper, we employ convolutional feed-forward networks for solving the reconstruction problem efficiently while maintaining good reconstruction quality. Specifically, we devise a tailored autoencoder architecture, trained end-to-end, that maps transient images directly to a depth map representation. Training is done using an efficient transient renderer for diffuse three-bounce indirect light transport that enables the quick generation of large amounts of training data for the network. We examine the performance of our method on a variety of synthetic and experimental datasets and its dependency on the choice of training data and augmentation strategies, as well as architectural features. We demonstrate that our feed-forward network, even though it is trained solely on synthetic data, generalizes to measured data from SPAD sensors and is able to obtain results that are competitive with model-based reconstruction methods.

Additional Material

Bibtex

@INPROCEEDINGS{GrauCVPR2020,
     author = {Grau, Javier and Hullin, Matthias B. and Wand, Michael and Iseringhausen, Julian},
      title = {Deep Non-Line-of-Sight Reconstruction},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
       year = {2020},
      month = jun,
   abstract = {The recent years have seen a surge of interest in methods for imaging beyond the direct line of
               sight. The most prominent techniques rely on time-resolved optical impulse responses, obtained by
               illuminating a diffuse wall with an ultrashort light pulse and observing multi-bounce indirect
               reflections with an ultrafast time-resolved imager. Reconstruction of geometry from such data,
               however, is a complex non-linear inverse problem that comes with substantial computational demands.
               In this paper, we employ convolutional feed-forward networks for solving the reconstruction problem
               efficiently while maintaining good reconstruction quality. Specifically, we devise a tailored
               autoencoder architecture, trained end-to-end, that maps transient images directly to a depth map
               representation. Training is done using an efficient transient renderer for diffuse three-bounce
               indirect light transport that enables the quick generation of large amounts of training data for the
               network. We examine the performance of our method on a variety of synthetic and experimental
               datasets and its dependency on the choice of training data and augmentation strategies, as well as
               architectural features. We demonstrate that our feed-forward network, even though it is trained
               solely on synthetic data, generalizes to measured data from SPAD sensors and is able to obtain
               results that are competitive with model-based reconstruction methods.}
}