NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

“NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, by Ben Mildenhall, Pratul P. Srinivasan and Matthew Tancik of UC Berkeley (equal contribution), Jonathan T. Barron of Google Research, Ravi Ramamoorthi of UC San Diego and Ren Ng of UC Berkeley, was posted to arXiv on 19 March 2020 and presented at the European Conference on Computer Vision (ECCV) in August 2020, where it received a Best Paper Honorable Mention. It fitted a multilayer perceptron mapping a 3D position and viewing direction to density and colour to one scene’s posed photographs, through classical volume rendering. Both arXiv versions add appendices, three in v1 and four in v2 (3 August 2020, with the ECCV main text); appendix material is labelled by version. The model and its family are covered in neural radiance fields.

Problem

The paper addressed novel view synthesis from a set of photographs. Its related work (§2) describes two lines of prior work, each falling short.

The first line encoded shapes in MLPs mapping a 3D point to a signed distance or an occupancy, the neural implicit representations of DeepSDF and Occupancy Networks, which needed ground-truth 3D geometry such as ShapeNet models. Niemeyer et al. and Sitzmann et al.’s Scene Representation Networks (SRN) learned such shapes from 2D images through differentiable rendering of a surface, but the authors judged them limited to simple shapes, with oversmoothed renderings.

The second line predicted classical geometry and appearance. Light field interpolation needed dense views. Meshes could be optimized through differentiable rasterizers or path tracers, but the authors found such optimization often difficult, likely because of local minima or poor conditioning, and dependent on a template mesh of fixed topology. Volumetric methods coloured voxel grids directly, trained networks on many scenes to predict sampled volumes, such as the multiplane images of Local Light Field Fusion (LLFF, Mildenhall et al., 2019), or paired CNNs with per-scene grids, as Neural Volumes (Lombardi et al., 2019) did. Volumes rendered complex shapes with fewer artifacts than meshes, but, the authors argued, discrete sampling limits their resolution, with time and storage growing to match. A continuous volume stored in network weights was their answer to both lines.

Contribution

The introduction (§1) lists three technical contributions:

• A representation: continuous scenes with complex geometry and materials as 5D neural radiance fields, parameterized as basic MLPs.
• A differentiable rendering procedure based on classical volume rendering, used to optimize the representation from ordinary RGB images, including a hierarchical sampling strategy that allocates the network’s capacity to space with visible content.
• A positional encoding that maps each input coordinate into a higher-dimensional space, which the authors found necessary to represent high-frequency content.

They claimed to outperform both per-scene neural representations and networks trained to predict sampled volumes, and, as far as they knew, to present the first continuous neural scene representation able to render high-resolution photorealistic novel views of real objects and scenes from RGB images captured in natural settings (§1).

Method

Sections 3–5 define the scene function $F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (\mathbf{c}, \sigma)$, with density depending on position only, render pixels by the quadrature $\hat C = \sum_i T_i \alpha_i \mathbf{c}_i$, and train a coarse and a fine network; neural radiance fields and volume rendering give the details.

The paper motivates the encoding with Rahaman et al.’s work on the spectral bias of deep networks (§5.1) and distinguishes it from the transformer’s positional encoding, which supplies token order rather than frequency content. The skip connection that concatenates the encoded position to the fifth layer’s activation, the output activations and the encoding sizes of 60 and 24 are printed only in the arXiv appendix (Appendix A, Fig. 7). There the versions differ: v1’s Fig. 7 has four 128-channel colour layers, v2’s one, as in §3, and a comment in the official code says the experiments used one. The derivation of the normalized device coordinates used for forward-facing scenes is in v2 only (Appendix C).

Results

Datasets and baselines

The experiments (§6.1) use three datasets, each with a separate network per scene:

• Diffuse Synthetic 360°: the four Lambertian objects of DeepVoxels, rendered at 512 × 512, with 479 input and 1,000 test views from the upper hemisphere.
• Realistic Synthetic 360°: eight path-traced objects of the authors’ own with complex geometry and non-Lambertian materials, six viewed from the upper hemisphere and two from the full sphere, with 100 input and 200 test views at 800 × 800. Later papers call it the Blender dataset.
• Real Forward-Facing: eight handheld cellphone captures, five from the LLFF paper and three new, with 20 to 62 images each at 1008 × 756 and one eighth held out for testing.

The baselines (§6.2) were SRN; Neural Volumes (NV), which predicts a $128^3$ RGBα voxel grid and needs a bounded volume in front of a separately captured background; and LLFF, which trains one 3D CNN on a large dataset and blends multiplane images predicted for nearby views. SRN and NV were fitted per scene like NeRF. Appendix B (arXiv only) adds that the baselines ran from their authors’ code, LLFF as a pretrained model, and that SRN’s memory needs limited its metrics to 512 × 512 on the synthetic data and 504 × 376 on the real scenes, lower than the others’. (That appendix gives the others’ real-scene resolution as 1008 × 752, against 1008 × 756 in §6.1.)

Table 1

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓
Diffuse Synthetic 360°	SRN	33.20	0.963	0.073
	NV	29.62	0.929	0.099
	LLFF	34.38	0.985	0.048
	NeRF	40.15	0.991	0.023
Realistic Synthetic 360°	SRN	22.26	0.846	0.170
	NV	26.05	0.893	0.160
	LLFF	24.88	0.911	0.114
	NeRF	31.01	0.947	0.081
Real Forward-Facing	SRN	22.84	0.668	0.378
	LLFF	24.13	0.798	0.212
	NeRF	26.50	0.811	0.250

NeRF had the best value in every column but one: on the real scenes, LLFF’s LPIPS was better (NV needs a bounded volume and could not be run there). The authors pointed to their supplementary video, where they judged NeRF more consistent across views and freer of artifacts (Table 1 caption, §6.3). They attributed the baselines’ failures to specific causes: SRN keeps only one depth and colour per ray, NV’s $128^3$ grid limits detail, and LLFF’s own sampling guideline asks for at most 64 pixels of disparity between input views, while the synthetic datasets have up to 400–500. The appendix breaks the results down by scene (v2, Appendix D, Tables 3–6; v1’s Tables 3–5 print different SSIM values). In v2, LLFF’s SSIM is higher on Leaves, Flower, Horns and the DeepVoxels Chair, although the text calls the breakdown consistent with NeRF outperforming all baselines and names only LLFF’s LPIPS as an exception.

Ablation

Table 2 removes components one at a time on Realistic Synthetic 360°, averaged over the eight scenes:

Row	What changes from the complete model	PSNR	SSIM	LPIPS
1	No positional encoding, view dependence or hierarchical sampling (256 coarse samples)	26.67	0.906	0.136
2	No positional encoding	28.77	0.924	0.108
3	No view dependence (position only)	27.66	0.925	0.117
4	No hierarchical sampling (256 coarse samples)	30.06	0.938	0.109
5	25 input images instead of 100	27.78	0.925	0.107
6	50 input images	29.79	0.940	0.096
7	5 frequencies instead of 10	30.59	0.944	0.088
8	15 frequencies	30.81	0.946	0.096
9	Complete model	31.01	0.947	0.081

Positional encoding and view dependence gave the largest gains, then hierarchical sampling (§6.4). With 25 images, NeRF still exceeded all three baselines trained on 100 on every metric (row 5 against Table 1). Raising $L$ from 10 to 15 did not help, which they attributed to $2^L$ already exceeding the highest frequency in the sampled images, roughly 1024.

Time and storage

§6.3 frames the trade-off as time against space. Every per-scene method took at least 12 hours, NeRF typically 1–2 days on one V100 (§5.3), while LLFF processed a small input set in under 10 minutes. LLFF, however, stored a voxel grid for every input image, over 15 GB for one Realistic Synthetic scene, against 5 MB of network weights for NeRF, which the authors call a 3000× compression and less than the input images of a single scene in any of their datasets. Rendering was not fast: about 30 seconds per frame on a V100 for 150–200 million network queries per image (arXiv Appendix A, both versions).

Checking the storage figure

The paper gives no parameter count. The script below counts the weights of the architecture in arXiv v2’s Fig. 7, with 60- and 24-dimensional encodings, and of the official code, whose encoder also passes the raw coordinates through (63 and 27 inputs, include_input in run_nerf_helpers.py).

def nerf_params(dim_x, dim_d, width=256, depth=8, skip_after=5):
    """Weights and biases of one NeRF MLP as in arXiv v2, Fig. 7."""
    n, n_in = 0, dim_x
    for layer in range(1, depth + 1):
        n += n_in * width + width
        n_in = width + (dim_x if layer == skip_after else 0)   # skip: concatenate gamma(x)
    n += width * 1 + 1                                         # density sigma
    n += width * 256 + 256                                     # 256-d feature vector
    n += (256 + dim_d) * 128 + 128                             # 128-channel colour layer
    n += 128 * 3 + 3                                           # RGB
    return n

L_x, L_d = 10, 4
for label, dx, dd in [("paper, Fig. 7: gamma only", 3 * 2 * L_x, 3 * 2 * L_d),
                      ("official code: raw input + gamma", 3 + 3 * 2 * L_x, 3 + 3 * 2 * L_d)]:
    one = nerf_params(dx, dd)
    print(f"{label:34s} inputs {dx}/{dd}: {one:,} per MLP, {2 * one:,} coarse + fine, "
          f"{2 * one * 4 / 1e6:.2f} MB as float32")

print(f"LLFF over 15 GB / NeRF 5 MB = {15e9 / 5e6:,.0f}x")
for name, n_img, w, h in [("Realistic Synthetic 360, 100 inputs", 100, 800, 800),
                          ("DeepVoxels, 479 inputs", 479, 512, 512),
                          ("Real Forward-Facing, smallest scene (20 images)", 20, 1008, 756)]:
    print(f"{name:48s} {n_img * w * h * 3 / 1e6:6.1f} MB as 8-bit RGB")

paper, Fig. 7: gamma only          inputs 60/24: 593,924 per MLP, 1,187,848 coarse + fine, 4.75 MB as float32
official code: raw input + gamma   inputs 63/27: 595,844 per MLP, 1,191,688 coarse + fine, 4.77 MB as float32
LLFF over 15 GB / NeRF 5 MB = 3,000x
Realistic Synthetic 360, 100 inputs               192.0 MB as 8-bit RGB
DeepVoxels, 479 inputs                            376.7 MB as 8-bit RGB
Real Forward-Facing, smallest scene (20 images)    45.7 MB as 8-bit RGB

Two networks of about 594,000 single-precision parameters take 4.75 MB, consistent with 5 MB for the coarse and fine networks together (our arithmetic; the paper does not say what it counts). The official code’s count, 1,191,688, matches the 1,191K that Mip-NeRF’s authors list for the TensorFlow NeRF (Barron et al., 2021, Table 2), which suggests that they counted the released implementation, not Fig. 7 (our inference). Even the smallest real scene’s images take about 46 MB uncompressed; the paper does not say whether it compared compressed files.

Impact

ECCV 2020 gave the paper a Best Paper Honorable Mention, listed on the conference’s awards page as “ECCV’20 Best Paper Honorable Mention” together with “Towards Streaming Perception”; the Best Paper Award went to RAFT (RAFT paper). The European Computer Vision Association’s award list records the same honorable mention.

In January 2022, Communications of the ACM published a shortened version in its Research Highlights section (65(1), 99–106), which refers to the ECCV paper for the hierarchical sampling. Frank Dellaert’s accompanying Technical Perspective described neural volume rendering as having exploded onto the scene in 2020, triggered by NeRF, and attributed the paper’s impact to its simplicity and to the many opportunities it left for improvement.

Follow-up work took them up in families that neural radiance fields surveys, from anti-aliasing and unbounded scenes to fast grid-based training, photo collections and dynamic scenes. In 2023, 3D Gaussian splatting kept NeRF’s image formation and compositing but replaced the network with explicit Gaussians rasterized in real time (its paper).

Limitations

The paper says little about its own failure cases. Its conclusion (§7) names two open problems, and its results and setup show two more restrictions:

• Time. Beyond the 1–2 days of fitting, efficient optimization and rendering remain open despite hierarchical sampling (§7).
• Interpretability. Unlike voxel grids and meshes, a scene stored in network weights gives no clear way to reason about expected rendering quality and failure modes (§7).
• Perceptual quality on real scenes. LLFF had the better LPIPS on Real Forward-Facing, which the authors acknowledge (Table 1).
• Scope. The method needs known poses, intrinsics and scene bounds, from COLMAP for real data (§5.3), and the real scenes are forward-facing, handled in normalized device coordinates with the far bound at infinity (arXiv v2, Appendix C).

Later work identified further limits, which neural radiance fields discusses: Dellaert’s perspective listed slow training and rendering, static scenes only, baked-in lighting and no generalization across scenes, and Mip-NeRF’s authors traced blurring and aliasing at mixed image resolutions to the single ray per pixel (Barron et al., 2021).

What Came After

The paper closed with the hope of a graphics pipeline built from radiance fields optimized from photographs of real objects and scenes (§7). Much of the work it prompted instead addressed the cost it had stated, which grid-based fields and then 3D Gaussian splatting cut from days to minutes of training and from seconds per frame to real-time rendering, and the assumptions its experiments left in place: bounded or forward-facing captures, static content and fixed lighting. Its evaluation protocol lasted too: the Realistic Synthetic 360° scenes, scored by PSNR, SSIM and LPIPS, became a common comparison, and Mip-NeRF’s Table 2, for example, keeps NeRF’s 31.01 PSNR as a reference row (Barron et al., 2021).