NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng | ECCV 2020 Oral
In a Nutshell 🥜
This paper1 represents a novel new method of using neural networks as an implicit representation of a scene. Specifically, the network ‘overfits’ on a given set of images from different views so that it can synthesise new renderings on angles of the scene that had never exist before.
The input of the network comprises a set of 3-dimensional coordinates indicating location and a two-dimensional viewing direction, and the output is the corresponding density and color of the location viewing from that angle. Since density is invariant to the viewing direction, the network first predicts the density and a set of feature vectors via the location only, then uses the feature vector with the combination of viewing direction to predict the color.
One observation is that if the network directly operates on the 5-dimensional input, it performs poorly when the object has high-frequency variation of color and geometry. Thus, following Rahaman et al.2, they map the inputs to a higher dimensional space using high-frequency functions to fit data with high-frequency variations.
The results on both real and synthetic images from NeRF significantly outperforms pervious papers.
Some Thoughts ðŸ’
This method is particularly creative in that the network itself could even be smaller than a single high quality image for input.
The trade-off is that one scene takes around 1-2 two days of fitting to achieve convergence.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020, August). Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision (pp. 405-421). Springer, Cham.
Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., ... & Courville, A. (2019, May). On the spectral bias of neural networks. In International Conference on Machine Learning (pp. 5301-5310). PMLR.