Crowdsampling the Plenoptic Function
ECCV 2020 Oral
Overview Video
Abstract
Many popular tourist landmarks are captured in a multitude of online, public photos. These photos represent a sparse and unstructured sampling of the plenoptic function for a particular scene. In this paper, we present a new approach to novel view synthesis under time-varying illumination from such data. We present a new approach for reconstructing light fields under the varying viewing conditions (e.g., different illuminations) present in Internet photo collections. Our approach builds on the recent multi-plane image (MPI) format for representing local light fields under fixed viewing conditions. We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function, that allows for real-time synthesis of photorealistic views that are continuous in both space and across changes in lighting. Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time. We show how to learn a model of these effects in an unsupervised way from an unstructured collection of photos without temporal registration, demonstrating significant improvements over recent work in neural rendering.
Results
Below you will find links to videos of our results, as well as other methods for comparison.
Click "play all" button in each part for better visualization.
Space-Time Photos
A gallery showing results of space-time interpolation
Diverse Results
A gallery showing novel view synthesis of landmarks under diverse illumination
Comparison with NRW
A gallery of results comparing with Neural Rerendering in the Wild (NRW) [Meshry et al 2019]
Comparison with MUNIT
A gallery of results comparing with Multimodal Unsupervised Image-to-image Translation (MUNIT) [Huang et al 2018]
Failure Cases
Examples of failure cases
Paper
Click to view the paper.
Click to view the supp document.
Code
Code and dataset.
Acknowledgements
We thank Kai Zhang, Jin Sun, and Qianqian Wang for helpful discussions. This research was supported in part by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program.