Neural Radiance Fields

A neural radiance field (NeRF) represents one scene as a multilayer perceptron that maps a 3D position and a viewing direction to a volume density and an emitted colour, and renders images from it by volume rendering along camera rays. Because the renderer is differentiable, the network can be fitted to a set of photographs with known camera poses, and then render the scene from poses that no photograph used. Mildenhall et al. introduced it at ECCV 2020. It is a coordinate network, one of the neural implicit representations, and a central model of neural rendering. This article covers the model and its family; the 2020 paper has its own article.

Problem Addressed

The task is novel view synthesis: given photographs of a static scene from known viewpoints, render it from new ones. Before NeRF, learned methods either predicted discrete volumes, such as multiplane images, with networks trained on many scenes, or fitted per-scene neural representations that rendered blurry or simplified geometry. The authors argue that voxel approaches scale poorly because finer images need finer sampling of 3D space, with time and storage to match, and that MLP scene representations such as Scene Representation Networks had been limited to simple shapes with oversmoothed renderings (Mildenhall et al., 2020, §2).

NeRF stores the scene as a continuous function in the weights of one network, a few megabytes (3D representations compares the storage), and fits it to the photographs of that scene alone.

Architecture

Inputs and outputs

The scene function is

$$F_\Theta : (\mathbf{x}, \mathbf{d}) \mapsto (\mathbf{c}, \sigma),$$

with $\mathbf{x} = (x, y, z)$ a point, $\mathbf{d}$ the ray’s viewing direction as a 3D unit vector (the paper’s 5D input counts it as two angles), $\sigma \ge 0$ the volume density and $\mathbf{c}$ an RGB colour. The density depends on $\mathbf{x}$ only and the colour on both; the authors restrict it this way to keep the representation consistent across views (§3). Colour that changes with $\mathbf{d}$ is how a radiance field reproduces specular highlights and other non-Lambertian appearance.

Positional encoding

Each coordinate passes through $\gamma(p) = \big(\sin(2^0 \pi p), \cos(2^0 \pi p), \dots, \sin(2^{L-1} \pi p), \cos(2^{L-1} \pi p)\big)$ (Eq. 4), with $L = 10$ for the position, normalized to $[-1, 1]$, and $L = 4$ for the direction. It counters spectral bias, the slow fitting of high frequencies from raw coordinates (positional encoding).

The MLP

$\gamma(\mathbf{x})$ goes through 8 fully connected ReLU layers of 256 channels, which output $\sigma$ and a 256-dimensional feature vector. The feature is concatenated with the encoded direction and passed through one more ReLU layer of 128 channels to the RGB colour (ECCV version, §3). The arXiv version’s appendix figure adds the details (arXiv v2, Appendix A, Fig. 7): a skip connection, following DeepSDF, concatenates $\gamma(\mathbf{x})$ to the fifth layer’s activation; $\sigma$ is rectified by a ReLU; the colour has a sigmoid; and the encodings have 60 and 24 dimensions ($3 \times 2 \times 10$ and $3 \times 2 \times 4$). The ECCV text does not mention the skip connection.

The late entry of the direction is the architectural prior: the geometry must explain every view, while the colour branch is too small to fake geometry for each viewpoint (our reading).

Rendering and hierarchical sampling

A pixel’s colour is the compositing sum $\hat C = \sum_i T_i \alpha_i \mathbf{c}_i$ of volume rendering, over stratified samples (Eq. 2) between near and far bounds $t_n$ and $t_f$. NeRF optimizes two networks of this form: a coarse one at $N_c = 64$ samples, whose normalized weights place $N_f = 128$ more by inverse transform sampling, and a fine one evaluated at all $N_c + N_f$, which gives the final colour (§5.2–5.3). The volume rendering article details both steps.

Training

Each scene is optimized separately from its RGB images, their camera poses and intrinsics, and scene bounds: ground truth for synthetic data, and COLMAP estimates for real data (§5.3). Each iteration samples a batch of rays from all pixels of all training images and minimizes the squared error of both renderings:

$$\mathcal{L} = \sum_{\mathbf{r} \in \mathcal{R}} \Big[ \big\lVert \hat C_c(\mathbf{r}) - C(\mathbf{r}) \big\rVert_2^2 + \big\lVert \hat C_f(\mathbf{r}) - C(\mathbf{r}) \big\rVert_2^2 \Big],$$

where $C(\mathbf{r})$ is the observed pixel colour (Eq. 6). The coarse loss is kept so that the coarse weights stay useful for placing the fine samples. The batch has 4096 rays; Adam’s learning rate decays exponentially from $5 \times 10^{-4}$ to $5 \times 10^{-5}$; and a scene typically takes 100,000–300,000 iterations, about 1–2 days on one NVIDIA V100 (§5.3).

The arXiv appendix adds the scene setup (arXiv v2, Appendix A). Synthetic objects are scaled into a cube of side 2 and queried only inside it. Real forward-facing captures, whose content can lie anywhere up to infinity, are mapped to normalized device coordinates, which bound the depth range by using disparity instead of depth. For real scenes, Gaussian noise of unit variance is added to the density before the ReLU during training, which the authors found slightly improves novel views.

Camera poses in practice

For their own scenes, the official repository points users to COLMAP through the LLFF code (NeRF README). NeRF’s code stores camera-to-world matrices with OpenGL camera axes, $x$ right, $y$ up and the camera looking along $-z$, whereas COLMAP and OpenCV store world-to-camera poses with $y$ down and $z$ forward; camera pose gives the conversion. Since the photometric loss is the only supervision, pose errors become blur: NeRF-W’s authors note sensitivity to calibration errors (Martin-Brualla et al., 2021, §5).

Inference

Rendering a new view casts one ray per pixel of the virtual camera. At test time NeRF evaluates 64 coarse and $64 + 128 = 192$ fine samples, 256 queries per ray; for 640,000 rays per synthetic image and 762,000 per real image, that is 150–200 million network queries and about 30 seconds per frame on a V100 (arXiv v2, Appendix A).

Extracting geometry

A radiance field holds density, not a surface. Two read-outs are common:

• Depth maps. The compositing weights give the expected termination depth $\hat D = \sum_i w_i t_i / \sum_i w_i$ of each ray (volume rendering); NeRF-W and BARF show such depth maps (Martin-Brualla et al., 2021, Fig. 6; Lin et al., 2021, Fig. 6).
• Meshes. Evaluate $\sigma$ on a grid and run marching cubes at a chosen density level; the official repository includes a notebook that does this (NeRF README). Rendered depth maps can also be fused into a mesh with TSDF fusion.

Both are approximations. The expected depth is a mean stopping distance, behind the front of a soft density; and nothing in training ties a density level to the surface. NeuS (Wang et al., 2021) extracted NeRF meshes at a density threshold of 25 and found them noisy, which the authors attribute to the lack of constraint on the density’s geometry; on DTU with object masks, their Table 1 reports a mean Chamfer distance of 1.54 for NeRF against 0.77 for NeuS, which models a signed distance function instead.

Performance

The authors evaluated NeRF on four Lambertian objects from DeepVoxels, eight path-traced objects of their own (Realistic Synthetic 360°, which Barron et al. (2021) call the Blender dataset) and eight real forward-facing cellphone captures (§6.1); the neural rendering hub describes the datasets and metrics. In their Table 1, NeRF reached PSNR 31.01, SSIM 0.947 and LPIPS 0.081 on Realistic Synthetic 360°, against 24.88, 0.911 and 0.114 for LLFF, and PSNR 26.50 on the real scenes against 24.13 for LLFF, whose LPIPS was better (0.212 against 0.250).

The ablation on Realistic Synthetic 360° (Table 2) shows which parts matter: without positional encoding PSNR fell to 28.77, without view dependence to 27.66, and without hierarchical sampling (256 coarse samples) to 30.06. Five frequencies gave 30.59 and fifteen 30.81.

Strengths

• Compact and continuous: megabytes of weights, queried at any resolution.
• View-dependent appearance and soft, thin or semi-transparent structure, which mesh pipelines handle poorly.
• Simple supervision: posed RGB images, no depth, masks or 3D data.
• Differentiable end to end, so poses, appearance codes or deformations can be optimized with the scene, as the variants below do.

Limitations

• Per-scene optimization. Nothing is learned across scenes; each capture is a new fit (§5.3).
• Slow training and rendering: days to fit, tens of seconds per frame.
• Static scenes and fixed lighting. The model assumes that a point seen from the same direction has the same colour in every image; moving objects and changing illumination violate it (Martin-Brualla et al., 2021, §4).
• Needs accurate poses from structure from motion, as above.
• Aliasing and scale. One infinitesimally thin ray per pixel blurs or aliases when training or test images see the scene at different resolutions (Barron et al., 2021).
• Bounded or forward-facing scenes only. NeRF uses Euclidean coordinates for objects and normalized device coordinates for forward-facing captures, neither of which suits scenes unbounded in every direction (Barron et al., 2022).
• Geometry is a by-product, as shown above.

Historical Importance

NeRF showed that a plain MLP with an input encoding, fitted through classical volume rendering, could render real scenes photorealistically from tens of photographs. Its formulation, posed images in and a field of density and colour out, became the common frame of later work; most variants below keep the rendering and change the representation, the sampling or the assumptions about the input.

Variants

Mip-NeRF (ICCV 2021): anti-aliasing

Barron et al. (2021) cast a cone through each pixel instead of a ray, with its radius at the image plane set to the pixel width times $2/\sqrt{12}$, and split it into conical frustums. Each frustum is approximated by a Gaussian with mean $\boldsymbol{\mu}$ and covariance $\Sigma$, and featurized by the integrated positional encoding, the expected value of the positional encoding under that Gaussian (Eqs. 13–14):

$$\gamma(\boldsymbol{\mu}, \Sigma) = \begin{bmatrix} \sin(\boldsymbol{\mu}_\gamma) \circ \exp\!\big(-\tfrac{1}{2} \operatorname{diag}(\Sigma_\gamma)\big) \\ \cos(\boldsymbol{\mu}_\gamma) \circ \exp\!\big(-\tfrac{1}{2} \operatorname{diag}(\Sigma_\gamma)\big) \end{bmatrix},$$

where $\boldsymbol{\mu}_\gamma$ and $\Sigma_\gamma$ are the mean and covariance lifted into the encoding’s frequency basis. Frequencies whose period is smaller than the frustum are attenuated towards zero, so the encoding tells the network how large a region it is asked about. Because scale is now an input, one network serves both sampling stages, which halves the parameter count (612K against 1,191K in their tables). On the Blender dataset (their Table 2), mip-NeRF reached PSNR 33.09, against 31.01 for the original TensorFlow NeRF and 31.74 for a JAX reimplementation; on a multiscale version with images downsampled by 2, 4 and 8 (Table 1), the reimplemented NeRF fell to 22.53 at 1/8 resolution while mip-NeRF reached 35.60. They trained for 1 million iterations at batch size 4096, which they call NeRF’s procedure; the NeRF paper reports 100,000–300,000.

Mip-NeRF 360 (CVPR 2022): unbounded scenes

For scenes where the camera turns 360° and content lies at any distance, Barron et al. (2022) make three changes. A contraction maps space into a ball of radius 2 (Eq. 10),

$$\operatorname{contract}(\mathbf{x}) = \begin{cases} \mathbf{x}, & \lVert \mathbf{x} \rVert \le 1, \\ \Big(2 - \dfrac{1}{\lVert \mathbf{x} \rVert}\Big) \dfrac{\mathbf{x}}{\lVert \mathbf{x} \rVert}, & \lVert \mathbf{x} \rVert > 1, \end{cases}$$

and is applied to the frustum Gaussians through its Jacobian, like an extended Kalman filter, with ray samples spaced linearly in disparity. A small proposal MLP predicts only density; its weights, supervised by a loss that penalizes them for not bounding the main MLP’s weights, choose where the large NeRF MLP is evaluated. A distortion loss on the normalized ray distances $s$ and weights $w$ (Eq. 15),

$$\mathcal{L}_{\text{dist}}(\mathbf{s}, \mathbf{w}) = \sum_{i,j} w_i w_j \left| \frac{s_i + s_{i+1}}{2} - \frac{s_j + s_{j+1}}{2} \right| + \frac{1}{3} \sum_i w_i^2 (s_{i+1} - s_i),$$

concentrates each ray’s weight into a compact interval, against floaters and “background collapse”. The proposal MLP has 4 layers of 256 units and the NeRF MLP 8 layers of 1024; training ran 250,000 iterations at batch size $2^{14}$. On their new dataset of 9 scenes (5 outdoor, 4 indoor) with COLMAP poses, Table 1 reports PSNR 27.69, SSIM 0.792 and LPIPS 0.237, against 24.04, 0.616 and 0.441 for mip-NeRF, a 57% reduction in mean squared error, with 6.89 hours of training on a 32-core TPU v2.

Instant-NGP (ACM TOG 2022): the multiresolution hash encoding

Müller et al. (2022) move most of the capacity from the network into trainable feature vectors. There are $L$ levels of grids whose resolutions grow geometrically from $N_{\min}$ to $N_{\max}$ (Eqs. 2–3):

$$N_l = \big\lfloor N_{\min} \, b^{\,l} \big\rfloor, \qquad b = \exp\!\left(\frac{\ln N_{\max} - \ln N_{\min}}{L - 1}\right).$$

At each level, the $2^d$ corners of the voxel containing $\mathbf{x}$ index a table of at most $T$ feature vectors of dimension $F$: one-to-one while the dense grid fits, $(N_l + 1)^d \le T$, and otherwise through the spatial hash (Eq. 4)

$$h(\mathbf{x}) = \Big( \bigoplus_{i=1}^{d} x_i \pi_i \Big) \bmod T,$$

with $\oplus$ the bitwise XOR, $\pi_1 = 1$, $\pi_2 = 2\,654\,435\,761$ and $\pi_3 = 805\,459\,861$. The corner features are $d$-linearly interpolated, and the levels are concatenated, with auxiliary inputs, into the input of a small MLP. Collisions are not resolved explicitly; the authors argue that colliding gradients are dominated by the points that matter, such as visible surfaces, and that the coarse, collision-free levels and the MLP disambiguate the rest. Their Table 1 gives $L = 16$, $T = 2^{14}$ to $2^{24}$, $F = 2$, $N_{\min} = 16$ and $N_{\max} = 512$ to $524{,}288$; only $T$ and $N_{\max}$ are tuned per task.

For NeRF they use a density MLP and a colour MLP of 64 neurons, the direction encoded by spherical harmonics up to degree 4, an occupancy grid that skips empty space, and fully fused CUDA kernels. On the eight Blender scenes (their Table 2, arXiv v2), mean PSNR was 29.26 after 5 seconds of training, 31.41 after 15 seconds, 32.64 after 1 minute and 33.18 after 5 minutes, against 31.01 for NeRF and 33.09 for mip-NeRF after hours. Replacing the hash encoding by the frequency encoding with a NeRF-sized MLP in the same implementation reached 30.06 after 5 minutes; the authors attribute a 20–60× improvement to the hash encoding and the smaller MLP.

NeRF in the Wild (CVPR 2021)

For internet photos of landmarks, Martin-Brualla et al. (2021) relax the consistency assumption. Each image gets an appearance embedding, optimized with the network in the manner of Generative Latent Optimization and fed only to the colour branch, so lighting and post-processing can vary while geometry stays shared (Eq. 7). A transient head adds a per-image density and colour, with a second embedding, plus an uncertainty $\beta$ that is composited like colour; the loss is a Gaussian negative log-likelihood with an L1 penalty on transient density (Eq. 13), and only the static part is rendered at test time. Test appearance embeddings are fitted on the left half of each test image and metrics computed on the right half. On six Phototourism scenes, with COLMAP poses and 300,000 steps on 8 V100 GPUs (about 2 days), NeRF-W reached PSNR 29.08 on Brandenburg Gate, against 21.05 for NeRF (their Table 1).

Grid-based fields: Plenoxels and TensoRF

Plenoxels (Fridovich-Keil et al., 2022) drop the network: a sparse voxel grid stores a density and degree-2 spherical harmonic coefficients (27 per voxel), interpolated trilinearly and optimized with a total-variation regularizer and coarse-to-fine pruning. On the eight synthetic scenes, their Table 2 reports PSNR 31.71 after 11 minutes on one Titan RTX GPU, against 31.85 for JAXNeRF after 1.45 days. TensoRF (Chen et al., 2022) factorizes the 4D feature grid into low-rank components, either CP (vectors) or a vector-matrix product; in their Table 1, the VM-192 model reached PSNR 33.14 on Synthetic-NeRF in 17.4 minutes with 71.8 MB, and the CP-384 model 31.56 in 25.2 minutes with 3.9 MB, on one V100.

Unknown poses: BARF (ICCV 2021)

BARF (Lin et al., 2021) optimizes camera poses jointly with the field, parameterized in the Lie algebra $\mathfrak{se}(3)$, and observes that applying the full positional encoding from the start harms registration. It weights the $k$-th frequency band by $w_k(\alpha)$, rising from 0 to 1 as a schedule $\alpha$ grows, so that alignment starts from a smooth field and detail is added later, as in classical coarse-to-fine image alignment. On the forward-facing LLFF scenes with every camera initialized to the identity, their Table 3 reports a mean rotation difference of 0.573° from the SfM poses and PSNR 23.97, against 22.56 for NeRF trained with the SfM poses; with the full encoding, the rotation difference was 84.5°.

Dynamic scenes: D-NeRF (CVPR 2021)

D-NeRF (Pumarola et al., 2021) adds time: a deformation network $\Psi_t(\mathbf{x}, t) \mapsto \Delta\mathbf{x}$, fixed to zero at $t = 0$, maps each point into a canonical configuration, and a canonical network $\Psi_x(\mathbf{x} + \Delta\mathbf{x}, \mathbf{d}) \mapsto (\mathbf{c}, \sigma)$ renders it, trained from a monocular camera with one image per time instant.

Zip-NeRF (ICCV 2023)

Zip-NeRF (Barron et al., 2023) combines mip-NeRF 360 with Instant-NGP’s grids, using multisampling within each frustum and downweighting of fine grid levels by the sample’s scale to anti-alias the grid features. On the mip-NeRF 360 dataset, their Table 2 reports PSNR 28.54 against 27.57 for mip-NeRF 360, with 0.89 hours of training on 8 V100 GPUs against 21.69 hours for mip-NeRF 360 on the same hardware, 24× faster.

3D Gaussian splatting

3D Gaussian splatting (Kerbl et al., 2023), the subject of its own article, keeps the radiance field’s image formation and front-to-back compositing but replaces the network and the ray marching with explicit anisotropic Gaussians, initialized from structure-from-motion points and rasterized in real time.

Implementations

The original TensorFlow code is the bmild/nerf repository (NeRF README). Nerfstudio is an open-source framework for NeRF-style methods; as of October 2026 its latest release is v1.1.5 (November 2024). Its documentation has method pages for NeRF, Mip-NeRF, Instant-NGP, TensoRF, Zip-NeRF and Gaussian splatting (Splatfacto), among others, processes custom images or video with COLMAP through ns-process-data, and exports meshes by TSDF fusion or Poisson surface reconstruction. Its default model, nerfacto, is described as a combination of published techniques rather than a published method: pose refinement, per-image appearance embeddings, proposal sampling, scene contraction and a hash encoding.

A tiny NeRF in PyTorch

The code builds NeRF’s pipeline at toy scale on a scene whose answers are known exactly: a striped red ball and a blue ball with a broad highlight that moves with the viewing direction, both of density 40 with a soft edge. Sixteen training cameras and four held-out ones look at the origin from distance 2.5, with camera-to-world poses in NeRF’s OpenGL convention; the “photographs” are 32 × 32 renderings of the exact field with 512 samples per ray, on a white background. The four blocks run in sequence as one script (PyTorch 2.14.1 on CPU, about 44 seconds in total); use_deterministic_algorithms makes the hash-table gradients of the third block, which are accumulated by index, reproducible.

import math, torch
import torch.nn as nn

torch.manual_seed(0)
torch.set_num_threads(4)
torch.use_deterministic_algorithms(True)            # the hash-table backward accumulates by index
H = W = 32                                       # image size in pixels
F = 0.5 * W / math.tan(math.radians(20))         # focal length for a 40 degree field of view
NEAR, FAR = 1.5, 3.5                             # ray bounds t_n, t_f

def look_at_gl(center):
    """Camera-to-world rotation in NeRF's OpenGL convention: x right, y up, looking along -z."""
    back = center / center.norm()                                   # camera +z points away from the scene
    right = torch.linalg.cross(torch.tensor([0.0, 0.0, 1.0]), back)
    right = right / right.norm()
    return torch.stack([right, torch.linalg.cross(back, right), back], 1), center

def get_rays(R_wc, center):
    v, u = torch.meshgrid(torch.arange(H) + 0.5, torch.arange(W) + 0.5, indexing="ij")
    d_cam = torch.stack([(u - W / 2) / F, -(v - H / 2) / F, -torch.ones_like(u)], -1)   # y up, -z forward
    d = d_cam.reshape(-1, 3) @ R_wc.T                                # rotate into the world frame
    return center.expand(H * W, 3), d / d.norm(dim=1, keepdim=True)

def cameras(azimuths, elev, dist=2.5):
    e = math.radians(elev)
    return [get_rays(*look_at_gl(dist * torch.tensor([math.cos(e) * math.cos(math.radians(a)),
            math.cos(e) * math.sin(math.radians(a)), math.sin(e)]))) for a in azimuths]

train_cams = cameras(range(0, 360, 30), 20) + cameras(range(15, 360, 90), 50)   # 16 training views
test_cams = cameras(range(45, 360, 90), 35)                                     # 4 held-out views

# Analytic scene: a striped red ball and a glossy blue ball whose highlight depends on the view direction d.
C1, R1 = torch.tensor([-0.35, 0.0, 0.0]), 0.45
C2, R2 = torch.tensor([0.40, 0.15, 0.10]), 0.30
LIGHT = torch.tensor([0.3, 0.5, 0.8]) / math.sqrt(0.98)

def scene(x, d):
    """Exact density and colour at points x seen along directions d, both of shape (..., 3)."""
    r1, r2 = (x - C1).norm(dim=-1), (x - C2).norm(dim=-1)
    s1 = 40 * torch.sigmoid((R1 - r1) / 0.01)                       # density 40 inside, soft 0.01 edge
    s2 = 40 * torch.sigmoid((R2 - r2) / 0.01)
    stripes = 0.5 + 0.5 * torch.sin(2 * math.pi * x[..., 2] / 0.15)  # stripes of period 0.15 on ball 1
    c1 = torch.stack([0.9 + 0 * stripes, 0.2 + 0.6 * stripes, 0.1 + 0 * stripes], -1)
    n = (x - C2) / r2.unsqueeze(-1).clamp(min=1e-6)
    refl = 2 * (n @ LIGHT).unsqueeze(-1) * n - LIGHT                 # light direction mirrored about the normal
    spec = (refl * -d).sum(-1).clamp(min=0)                          # broad highlight, Phong exponent 1
    c2 = torch.tensor([0.1, 0.25, 0.8]) + 0.7 * spec.unsqueeze(-1)
    sigma = s1 + s2
    c = (s1[..., None] * c1 + s2[..., None] * c2) / sigma.clamp(min=1e-8)[..., None]
    return sigma, c.clamp(0, 1)

def render(field, o, d, n, stratified=False):
    """Compositing quadrature of the volume rendering article, over a white background."""
    u = torch.rand(len(o), n) if stratified else torch.full((len(o), n), 0.5)
    t = torch.linspace(NEAR, FAR, n + 1)[:-1] + u * (FAR - NEAR) / n         # NeRF Eq. 2, or bin centres
    delta = torch.diff(torch.cat([t, torch.full((len(o), 1), FAR)], 1), dim=1)
    x = o[:, None] + t[..., None] * d[:, None]
    sigma, c = field(x, d[:, None].expand_as(x))
    alpha = 1 - torch.exp(-sigma * delta)
    T = torch.exp(-torch.cumsum(torch.cat([torch.zeros(len(o), 1), sigma * delta], 1), 1))
    w = T[:, :-1] * alpha                                             # weights w_i = T_i alpha_i
    return (w[..., None] * c).sum(1) + T[:, -1:], w, t

def first_hit(o, d):
    """Exact distance to the first ball along each ray (inf if none) and which ball it is (0 or 1)."""
    ts = []
    for c, r in ((C1, R1), (C2, R2)):
        b = ((o - c) * d).sum(1)
        disc = b ** 2 - ((o - c) ** 2).sum(1) + r ** 2
        ts.append(torch.where(disc > 0, -b - disc.clamp(min=0).sqrt(), torch.tensor(float("inf"))))
    t = torch.stack(ts, 1)
    return t.min(1).values, t.argmin(1)

with torch.no_grad():                                                 # "photographs": 512 samples per ray
    train_rgb = [render(scene, o, d, 512)[0] for o, d in train_cams]
    test_rgb = torch.cat([render(scene, o, d, 512)[0] for o, d in test_cams])
O, D = torch.cat([o for o, _ in train_cams]), torch.cat([d for _, d in train_cams])
RGB = torch.cat(train_rgb)
hit_t, hit_ball = map(torch.cat, zip(*[first_hit(o, d) for o, d in test_cams]))
striped = (hit_t < float("inf")) & (hit_ball == 0)                   # held-out pixels showing each ball
glossy = (hit_t < float("inf")) & (hit_ball == 1)
print(f"{len(train_cams)} training views, {len(test_cams)} held-out views, {H}x{W} pixels, "
      f"{len(O)} training rays; {(hit_t < float('inf')).float().mean():.0%} of held-out pixels see a ball, "
      f"{striped.float().mean():.0%} the striped one")

16 training views, 4 held-out views, 32x32 pixels, 16384 training rays; 29% of held-out pixels see a ball, 20% the striped one

The model keeps NeRF’s layout at small scale: four ReLU layers of 64 units, density from $\mathbf{x}$ alone, and the encoded direction ($L = 2$) entering a small colour branch. Three simplifications are ours: one network with 32 stratified samples per ray instead of a coarse and a fine pass; a softplus on the density, as in Mip-NeRF 360, instead of NeRF’s ReLU; and the raw coordinate kept beside its encoding, so that $L = 0$ means raw input. Each step draws 512 random rays from all training pixels and minimizes the photometric mean squared error with Adam. Every model is a single run with seed 0. The table reports held-out PSNR over whole images and, at 1,000 iterations, over the pixels of each ball.

def encode(p, L):
    """NeRF's positional encoding (Eq. 4) plus the raw coordinate, so that L = 0 means raw input."""
    out = [p]
    for k in range(L):
        out += [torch.sin(2 ** k * math.pi * p), torch.cos(2 ** k * math.pi * p)]
    return torch.cat(out, -1)

class TinyNeRF(nn.Module):
    """NeRF's layout at small scale: density from x alone, colour from a feature and the encoded d."""
    def __init__(self, L_x, L_d=2, use_dir=True, width=64, depth=4):
        super().__init__()
        self.L_x, self.L_d, self.use_dir = L_x, L_d, use_dir
        layers, n_in = [], 3 * (1 + 2 * L_x)
        for _ in range(depth):
            layers += [nn.Linear(n_in, width), nn.ReLU()]
            n_in = width
        self.trunk = nn.Sequential(*layers)
        self.sigma, self.feature = nn.Linear(width, 1), nn.Linear(width, width)
        n_dir = 3 * (1 + 2 * L_d) if use_dir else 0
        self.color = nn.Sequential(nn.Linear(width + n_dir, width // 2), nn.ReLU(),
                                   nn.Linear(width // 2, 3), nn.Sigmoid())
    def features(self, x):
        return self.trunk(encode(x, self.L_x))
    def forward(self, x, d):
        h = self.features(x)
        sigma = nn.functional.softplus(self.sigma(h)).squeeze(-1)    # non-negative density
        f = self.feature(h)
        if self.use_dir:
            f = torch.cat([f, encode(d, self.L_d)], -1)              # the view direction enters late
        return sigma, self.color(f)

psnr = lambda a, b: (-10 * torch.log10(((a - b) ** 2).mean())).item()

def train(model, iters=1000, lr=5e-3, eps=1e-8, batch=512, n=32, report=(100, 250, 500, 1000)):
    opt = torch.optim.Adam(model.parameters(), lr=lr, eps=eps)
    log = []
    for it in range(1, iters + 1):
        idx = torch.randint(0, len(O), (batch,))                      # random rays from all training pixels
        loss = ((render(model, O[idx], D[idx], n, stratified=True)[0] - RGB[idx]) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
        if it in report:                                              # held-out PSNR, 64 samples per ray
            with torch.no_grad():
                pred = torch.cat([render(model, o, d, 64)[0] for o, d in test_cams])
            log.append(psnr(pred, test_rgb))
    return log + [psnr(pred[m], test_rgb[m]) for m in (striped, glossy)]   # per-ball PSNR at the end

print("held-out PSNR (dB) after         100    250    500   1000 iterations;  striped, glossy ball")
models = {}
for name, L in [("MLP, L = 0", 0), ("MLP, L = 6", 6)]:
    torch.manual_seed(0)
    models[name] = TinyNeRF(L_x=L)
    log = train(models[name])
    print(f"  {name:22s}" + "".join(f"{p:7.2f}" for p in log[:4]) + f"{log[4]:11.2f}{log[5]:8.2f}")

held-out PSNR (dB) after         100    250    500   1000 iterations;  striped, glossy ball
  MLP, L = 0              15.23  20.73  23.89  25.70      20.99   23.28
  MLP, L = 6              18.16  23.68  25.17  27.45      30.05   18.37

With raw coordinates the MLP renders the striped ball at 20.99 dB after 1,000 iterations; with $L = 6$ it reaches 30.05 dB, the spectral-bias effect that NeRF’s encoding addresses, and the whole-image PSNR rises from 25.70 to 27.45 dB. The encoded MLP is worse on the glossy ball (18.37 against 23.28 dB). We have not diagnosed this; one possible reading is that the high-frequency input lets the network paint the moving highlight onto the surface as fixed texture, which is wrong from new viewpoints.

The third block replaces the deep MLP by a two-level hash encoding: a $16^3$ grid, which fits densely in a table of $T = 2^{14}$ entries since $17^3 = 4913$, and a $64^3$ grid hashed into another such table with Instant-NGP’s primes, $F = 2$ features per entry, followed by one hidden layer. It uses Instant-NGP’s learning rate $10^{-2}$ and $\epsilon = 10^{-15}$, and is trained with and without the view direction.

PRIMES = (1, 2654435761, 805459861)                    # Instant-NGP's pi_1, pi_2, pi_3

class HashGrid(nn.Module):
    """Multiresolution hash encoding (Mueller et al., 2022, Eqs. 2-4) with two levels."""
    def __init__(self, resolutions=(16, 64), T=2 ** 14, F=2, bound=1.5):
        super().__init__()
        self.res, self.T, self.bound = resolutions, T, bound
        self.tables = nn.ParameterList(nn.Parameter(torch.empty(T, F).uniform_(-1e-4, 1e-4)) for _ in resolutions)
        self.corners = torch.tensor([[i, j, k] for i in (0, 1) for j in (0, 1) for k in (0, 1)])
    def index(self, ijk, N):
        if (N + 1) ** 3 <= self.T:                     # coarse level: dense grid, one entry per vertex
            return ijk[..., 0] + (N + 1) * (ijk[..., 1] + (N + 1) * ijk[..., 2])
        h = (ijk[..., 0] * PRIMES[0]) ^ (ijk[..., 1] * PRIMES[1]) ^ (ijk[..., 2] * PRIMES[2])
        return h % self.T                              # fine level: spatial hash, collisions unresolved
    def forward(self, x):
        u = ((x / self.bound + 1) / 2).clamp(0, 1)    # scene box -> unit cube
        out = []
        for N, table in zip(self.res, self.tables):
            p = u * N
            p0 = p.floor().long().clamp(max=N - 1)
            w = p - p0                                 # position inside the voxel
            ijk = p0[..., None, :] + self.corners      # the 8 voxel corners
            wc = torch.where(self.corners.bool(), w[..., None, :], 1 - w[..., None, :]).prod(-1)
            out.append((table[self.index(ijk, N)] * wc[..., None]).sum(-2))   # trilinear interpolation
        return torch.cat(out, -1)                      # concatenate the levels

class HashNeRF(TinyNeRF):
    """Same heads as TinyNeRF; the encoding and a one-layer trunk replace the deep MLP."""
    def __init__(self, use_dir=True):
        super().__init__(L_x=0, use_dir=use_dir, depth=1)
        self.grid = HashGrid()
        self.trunk = nn.Sequential(nn.Linear(4, 64), nn.ReLU())   # 2 levels x F = 2 features
    def features(self, x):
        return self.trunk(self.grid(x))

for name, use_dir in [("hash grid, no d", False), ("hash grid, with d", True)]:
    torch.manual_seed(0)
    models[name] = HashNeRF(use_dir)
    log = train(models[name], lr=1e-2, eps=1e-15)                  # Instant-NGP's learning rate and epsilon
    print(f"  {name:22s}" + "".join(f"{p:7.2f}" for p in log[:4]) + f"{log[4]:11.2f}{log[5]:8.2f}")
for name in ("MLP, L = 6", "hash grid, with d"):
    print(f"  {name}: {sum(p.numel() for p in models[name].parameters())} parameters")

  hash grid, no d         21.00  27.33  30.50  31.74      30.53   25.52
  hash grid, with d       21.93  27.23  30.04  33.39      31.69   28.38
  MLP, L = 6: 21924 parameters
  hash grid, with d: 72740 parameters

At equal iterations the hash grid is ahead of the MLP at every checkpoint and ends at 33.39 dB against 27.45, but with 72,740 parameters against 21,924, so this compares convergence per iteration, not equal capacity, and says nothing about wall-clock speed, which Instant-NGP also owes to fused CUDA kernels and empty-space skipping. Without the view direction the hash model loses 1.65 dB over whole images and 2.86 dB on the glossy ball, whose highlight depends on $\mathbf{d}$, against 1.16 dB on the striped ball, whose colour does not.

The last block reads geometry out of the fitted fields: the expected depth from the weights, against the exact distance to the first ball along each held-out ray, and the grid cells whose density exceeds a threshold, against the true balls.

# Geometry from the fitted fields: expected depth from the weights, and density thresholds on a grid.
g = torch.linspace(-1, 1, 64)
X = torch.stack(torch.meshgrid(g, g, g, indexing="ij"), -1).reshape(-1, 3)
inside = ((X - C1).norm(dim=1) < R1) | ((X - C2).norm(dim=1) < R2)
hit = hit_t < float("inf")
with torch.no_grad():
    w, t = map(torch.cat, zip(*[render(scene, o, d, 256)[1:] for o, d in test_cams]))
    e = (w * t).sum(1)[hit] / w.sum(1)[hit] - hit_t[hit]
    print(f"  exact scene: depth error on {hit.sum()} ball pixels, median {e.median():+.3f}")
    for name in ("MLP, L = 6", "hash grid, with d"):
        m = models[name]
        w, t = map(torch.cat, zip(*[render(m, o, d, 256)[1:] for o, d in test_cams]))
        depth = (w * t).sum(1) / w.sum(1).clamp(min=1e-8)             # expected depth (volume rendering article)
        e = depth[hit] - hit_t[hit]
        sigma = m(X, torch.zeros_like(X))[0]
        ious = [((sigma > s) & inside).sum().item() / ((sigma > s) | inside).sum().item() for s in (1, 3, 10, 30)]
        print(f"  {name}: depth error on {len(e)} ball pixels, median {e.median():+.3f}; "
              f"median density inside the balls {sigma[inside].median():.1f} (true 40)")
        print(f"     IoU of the grid cells with sigma > 1 / 3 / 10 / 30 against the true balls: "
              + " / ".join(f"{v:.2f}" for v in ious))

  exact scene: depth error on 1181 ball pixels, median +0.016
  MLP, L = 6: depth error on 1181 ball pixels, median +0.067; median density inside the balls 9.2 (true 40)
     IoU of the grid cells with sigma > 1 / 3 / 10 / 30 against the true balls: 0.72 / 0.75 / 0.37 / 0.00
  hash grid, with d: depth error on 1181 ball pixels, median +0.055; median density inside the balls 11.1 (true 40)
     IoU of the grid cells with sigma > 1 / 3 / 10 / 30 against the true balls: 0.75 / 0.79 / 0.61 / 0.00

The exact field’s expected depth lies 0.016 behind the true surface, the offset of a dense but soft medium (volume rendering). The fitted fields lie 0.067 and 0.055 behind it, and their median density inside the balls is about a quarter of the true value (9.2 and 11.1 against 40). Our reading is that the images cannot tell such densities apart: with 32 training samples per ray the step is 0.0625, a density of 10 already gives each step an opacity of 0.46, and a few steps make a ball opaque; and since the expected depth in a thick medium lies about $1/\sigma$ behind its front, a lower density puts the estimated surface deeper. Thresholds behave the same way. Cells with $\sigma > 1$ or $\sigma > 3$ match the balls with IoU 0.72–0.79; at $\sigma > 10$ the IoU falls to 0.37 and 0.61; and at $\sigma > 30$, still below the true density, it is zero. A level that suits one fit, such as the 25 that NeuS used for NeRF, need not suit another (our reasoning).