Neural Rendering

Neural rendering is image synthesis in which some part of the rendering pipeline is learned from images. A classical renderer turns a hand-built scene description into pixels by simulating light; a neural renderer learns the scene description, the rendering step, or both, from photographs of real scenes. The term covers networks that turn a rough rasterized image into a photograph, scene representations such as radiance fields and 3D Gaussians fitted to posed photographs, and generative models that produce 3D-consistent images of things never captured. The renderer for density fields is covered in volume rendering; neural implicit representations, neural radiance fields (NeRF), 3D Gaussian splatting and novel view synthesis have their own articles.

Definition

Tewari et al. (2020), in what they describe as the first survey of the field, note that neural rendering had no clear definition yet and define it as deep image or video generation that allows explicit or implicit control over scene properties such as illumination, camera parameters, pose, geometry, appearance and semantic structure. Their emphasis is on methods that combine learned components with physical knowledge from computer graphics, such as models of projection or a differentiable renderer inside the network.

The sequel survey (Tewari et al., 2022, Fig. 2) points out that the term had come to name two distinct ideas:

• 2D neural rendering, also called neural re-rendering or deferred neural rendering: a network takes a 2D input, such as a semantic label map or an image rasterized from proxy geometry, and is trained to produce the final image. The network learns to render.
• 3D neural rendering: a network or other learned structure represents the scene in 3D, and a fixed, analytically defined renderer turns it into images. The learned part does not render; it represents, and the image follows from the physics of image formation.

A working definition that covers both, in our words: neural rendering produces images of a scene, under controllable viewpoint, lighting or content, with a pipeline in which the scene representation, the rendering function, or both are learned from images.

Classical rendering

In graphics, rendering turns a scene definition, with cameras, lights, geometry and materials, into a simulated camera image. Tewari et al. (2020, §4.1.3–4.1.4) summarize the two standard approaches. Rasterization transforms geometry into the image, primitive by primitive; hardware-accelerated rendering typically relies on it. Ray tracing casts rays backward from the pixels into the scene and follows reflections and refractions recursively. Both approximate the rendering equation, which Tewari et al. attribute to Kajiya (1986): the radiance leaving a surface point is its emission plus the integral over incoming directions of incident radiance, weighted by the surface’s reflectance and the cosine of the incidence angle. Since the incident light at one point is the outgoing light of another, the equation has no closed-form solution for nontrivial scenes, and renderers approximate it by Monte Carlo sampling or by truncating the number of bounces.

Classical rendering is exact up to its models, but the models must be supplied. Building geometry, materials and lighting for a real scene is slow manual work or error-prone 3D reconstruction, and Tewari et al. (2020) describe estimating these parameters from images, known as inverse rendering, as closely related to neural rendering but limited by the predefined physical models.

Image-based rendering

Image-based rendering skips the scene model and makes new views by resampling captured images. Levoy and Hanrahan (1996) treated the input images as 2D slices of a 4D light field, the radiance along every ray through free space in a static scene with fixed illumination, and generated new views by extracting and resampling slices, without depth or feature matching. The Lumigraph of Gortler et al. (1996) samples the same 4D function, a subset of the plenoptic function, and adds approximate geometry to improve reconstruction when views are sparse. Later image-based methods warp captured views into the target view with a proxy geometry and blend them, and the result depends on the quality of that geometry, with seams or ghosting in complex scenes (Tewari et al., 2020, §1 and §4.1.5). Neural rendering inherits the image-based goal of photorealism from captured data, and learned components entered by replacing parts of it, such as the blending.

Intuition

Picture 100 photographs of a statue, with camera poses from structure from motion. A classical pipeline reconstructs a mesh and a texture and rasterizes them, so every reconstruction error shows in the image. A 2D neural renderer keeps the imperfect mesh but trains a network to turn its rasterized features into the photographs, learning to hide the errors. A 3D neural renderer starts from a generic scene representation, renders it from the 100 poses with a differentiable renderer, and adjusts it until the renderings match the photographs; it can then render poses that no photograph used.

In all three cases the photographs are the supervision. What differs is where the learned parameters live and how much of the image formation is fixed physics.

Formal Definition

Write $\mathbf{c}$ for the control parameters (camera, lighting, pose) and $I \in \mathbb{R}^{H \times W \times 3}$ for the image. Tewari et al. (2022, §1) frame neural rendering as learning a mapping

$$I = M(\mathbf{c}).$$

The families differ in how $M$ is built. A 2D neural renderer is a network $G_\phi$ applied to a guide image $\mathbf{g}(\mathbf{c})$ produced by a conventional, possibly non-differentiable, graphics pipeline:

$$I = G_\phi\big(\mathbf{g}(\mathbf{c})\big).$$

A 3D neural renderer is a fixed renderer $\mathcal{R}$ applied to a learned scene representation $S_\theta$, viewed by a camera with intrinsics $K_k$ and world-to-camera pose $[R_k \mid \mathbf{t}_k]$, so that $\mathbf{X}_c = R_k\mathbf{X}_w + \mathbf{t}_k$ (camera parameters). Given photographs $I_1, \dots, I_N$ with known cameras, per-scene fitting solves

$$\theta^\star = \arg\min_\theta \sum_{k=1}^{N} \ell\Big(\mathcal{R}\big(S_\theta;\, K_k, R_k, \mathbf{t}_k\big),\; I_k\Big),$$

with $\ell$ a per-pixel $L^1$ or $L^2$ loss, sometimes with perceptual or adversarial terms (Tewari et al., 2022, §3.3). A feed-forward model replaces the optimization with a network that predicts the representation from a few images, $S = E_\psi(I_1, \dots, I_n)$, with $\psi$ trained across many scenes. A generative model draws the representation from a distribution, $S = G_\psi(\mathbf{z})$, and is trained so that its renderings look like real images.

Gradient-based fitting needs, for every pixel $p$, the derivative of the rendered value with respect to the scene parameters:

$$\frac{\partial \ell}{\partial \theta} = \sum_{k} \sum_{p} \frac{\partial \ell}{\partial I_k(p)} \, \frac{\partial I_k(p)}{\partial \theta}.$$

Tewari et al. (2022, §3) stress that both the representation and the renderer must be differentiable to learn from real images; the term $\partial I(p) / \partial \theta$ is what a differentiable renderer provides. For radiance fields, $\mathcal{R}$ is the front-to-back compositing $\hat C = \sum_i T_i \alpha_i \mathbf{c}_i$ given in 3D representations and derived, with its gradients, in volume rendering.

A Taxonomy

Where the network sits

Image-space networks. The network works on 2D images. When the conditioning input is a label map or a rendered image of depth, normals or a diffuse shading pass, the problem becomes image-to-image translation, which Tewari et al. (2020, §5.2) note is a well-studied setting for conditional generative networks. Deferred neural rendering (Thies et al., 2019) is the clearest example of a hybrid: each object carries a neural texture, a learned feature map stored on its mesh like an ordinary texture but with 16 features per texel instead of a color in the authors’ setup. A standard rasterizer samples the neural texture into a screen-space feature map, and a U-Net-like network turns that map into the photograph; texture and network are trained end to end. Because the features are attached to 3D geometry, re-renderings stay consistent as the camera moves, and the network can correct for noisy or incomplete reconstructed geometry. The cost of working in 2D is that consistency across views is learned, not guaranteed.

Learned 3D scene representations with a fixed renderer. Here the learned part is a 3D scene, and the renderer is a differentiable version of a graphics algorithm. Neural Volumes (Lombardi et al., 2019) decodes a voxel grid of opacity and color with an encoder–decoder network and renders it by differentiable ray marching, trained on multi-view video of dynamic objects. Scene Representation Networks (Sitzmann et al., 2019) map world coordinates to features with a network and find surfaces with a learned, differentiable ray marcher, trained from posed images without depth. NeRF (Mildenhall et al., 2020) maps position and viewing direction to density and color and renders by volume rendering. 3D Gaussian splatting (Kerbl et al., 2023) drops the network for explicit anisotropic Gaussians that are projected and composited front to back. Tewari et al. (2022, §1–2) credit this family’s quality to its inductive bias: an intermediate 3D representation, rendered by physics, is 3D-consistent by design.

Generative 3D-aware models. A generative model produces new scenes rather than fitting a captured one, with a 3D representation and renderer inside. EG3D (Chan et al., 2022) is a GAN whose generator, built on StyleGAN2, outputs a hybrid tri-plane representation that a lightweight decoder, a neural volume renderer and a super-resolution module turn into images; it trains on collections of single-view photographs, such as the FFHQ faces and AFHQ cats, without 3D scans or multi-view supervision. DreamFusion (Poole et al., 2023) needs no 3D or multi-view data at all: it optimizes a randomly initialized NeRF, built on Mip-NeRF 360, so that its renderings from random viewpoints score well under a frozen, pretrained text-to-image diffusion model, Imagen. Its score distillation sampling loss noises a rendering, takes the difference between the noise the diffusion model predicts and the noise actually added as the update direction, omitting the expensive and poorly conditioned Jacobian of the diffusion network, and backpropagates it through the renderer into the NeRF.

Explicit, implicit and hybrid representations

3D representations defines explicit forms, which list geometry, and implicit ones, which store a function whose level set is the surface. Neural rendering blurs this vocabulary. Tewari et al. (2022, §3.1.3) point out that a NeRF-style network outputs density and color, so it parameterizes geometry explicitly as a field, not implicitly as a level set, although the literature commonly calls such networks “implicit” because the scene is stored in network weights. In practice the useful distinction is where the parameters live:

• Pure networks. A multilayer perceptron maps coordinates to values, as in SRN and NeRF; memory is small and fixed, but every sample along every ray needs a full network evaluation. Such networks need encodings of their input coordinates to fit fine detail (positional encoding).
• Grids. Voxels store values or features that are interpolated, as in Neural Volumes; lookups are fast, but dense grids grow with the cube of the resolution.
• Hybrids. Features stored in a data structure, decoded by a small network. Instant-NGP (Müller et al., 2022) stores trainable feature vectors in a multiresolution hash table, interpolates them at each query point, and passes them to a small MLP; collisions in the hash table are left for the network to resolve. The authors report training high-quality models in a matter of seconds. EG3D’s tri-planes, three axis-aligned feature planes, are another hybrid.
• Primitives. Points or Gaussians, each with position, shape and appearance, rendered by splatting or rasterization; 3D Gaussian splatting is the leading example, and neural point renderers attach learned features to points and decode them with a 2D network (Tewari et al., 2022, §3.2.1).

Per-scene optimization versus feed-forward models

Tewari et al. (2020, §5.5) already separated instance-specific methods, retrained for every object or person, from general methods trained once for a whole class, and noted that the specific ones gave higher quality at the price of long per-instance training. NeRF and 3D Gaussian splatting are per-scene: the representation is fitted to one capture from many views. Generalizable models predict a representation directly. pixelNeRF (Yu et al., 2021) conditions a NeRF on convolutional features of the input images, is trained across many scenes to learn a prior, and renders novel views from as few as one input image without test-time optimization; NeRF, given only three views, performs poorly. LRM (Hong et al., 2024) scaled the idea: a transformer of about 500 million parameters, trained on around one million objects, predicts a tri-plane NeRF from a single image in about 5 seconds, according to the authors. The feed-forward geometry models described in 3D reconstruction, such as DUSt3R and VGGT, are the reconstruction-side counterparts.

Differentiable Rendering

Fitting a scene to photographs means following the gradient of an image loss back to geometry, and the standard rasterizer does not provide it. Coverage and visibility are discrete decisions: a pixel is inside a triangle or not, and one surface is in front of another or not. Liu et al. (2019) identify this discrete sampling step as what prevents gradients from flowing from pixels to mesh vertices. Moving a vertex by less than the distance to the next pixel center changes no pixel; swapping the depth order of two surfaces changes the image abruptly. The image is a piecewise-constant function of the geometry, so its derivative is zero almost everywhere and undefined at the jumps.

Two remedies exist. Some differentiable rasterizers keep the hard forward pass and approximate the gradient; Liu et al. (2019) contrast their method with OpenDR and the Neural Mesh Renderer, which only approximate the rendering gradient in the backward pass. Soft Rasterizer (Liu et al., 2019) instead makes the forward pass itself smooth. Each triangle’s influence on a pixel is a probability $D_j = \operatorname{sigmoid}(\delta\, d^2 / \sigma)$, where $d$ is the distance from the pixel to the triangle’s edges, $\delta = \pm 1$ marks inside or outside, and $\sigma$ sets the blur (their Eq. 1). Colors are then aggregated with softmax weights over normalized inverse depth with temperature $\gamma$ (Eqs. 2–3); as $\gamma \to 0$ this reduces to z-buffering. Gradients reach occluded and distant triangles, which a z-buffer hides.

Volume rendering avoids the problem by construction. A ray does not hit a hard surface; it has some probability of interacting with the medium at every point, and the rendered color is a smooth function of density and color, which Tewari et al. (2022, §3.2.2) name as the reason it works well for learning pipelines that depend on well-behaved gradients. In our reading, 3D Gaussian splatting gets similar smoothness from its semi-transparent, smoothly decaying primitives. The price, in our reasoning and visible in the center-pixel values of the example below, is blur: a soft renderer’s image differs from a sharp one, so sharp surfaces must come from the data or from annealing the softness.

Example

The script renders a tiny synthetic scene: two discs of radius 12 pixels in a 48 × 48 image, a bright disc A and a dark disc B at depth 2.0, overlapping in the middle. A hard rasterizer covers a pixel when its center is inside a disc and keeps the nearer disc. A soft rasterizer in the style of Soft Rasterizer uses sigmoid coverage (Eq. 1 of Liu et al., 2019), a softmax over normalized inverse depth for the color (their Eqs. 2–3, without the background term), and their probabilistic silhouette, one minus the product of $(1 - D_j)$ (Eq. 4), to blend with the black background; combining the silhouette with the color this way is our simplification, not their formulation. Each renderer’s loss is the mean squared error to its own image of the true scene, with disc A at $x_A = 18$, $z_A = 1.9$, in front of B. Derivatives are central finite differences with step $10^{-3}$.

import numpy as np

H = W = 48
v, u = np.mgrid[0:H, 0:W] + 0.5                     # pixel centres
Z_NEAR, Z_FAR = 1.0, 3.0

def discs(xa, za):
    """Two discs (centre x, centre y, radius, depth, grey level). Disc A moves."""
    return [(xa, 24.0, 12.0, za, 0.9),               # A: bright, depth za
            (30.0, 24.0, 12.0, 2.0, 0.3)]            # B: dark, depth 2.0

def render_hard(scene):
    """Rasterization: a pixel is covered if its centre lies inside; nearest depth wins."""
    img, zbuf = np.zeros((H, W)), np.full((H, W), np.inf)
    for cx, cy, r, z, c in scene:
        win = (np.hypot(u - cx, v - cy) < r) & (z < zbuf)
        img[win], zbuf[win] = c, z
    return img

def render_soft(scene, sigma=2.0, gamma=0.02):
    """Soft rasterizer after Liu et al. (2019): sigmoid coverage per disc (their Eq. 1),
    a softmax over normalized inverse depth for colour (Eq. 3), and the probabilistic
    silhouette 1 - prod(1 - D) (Eq. 4) to blend with the black background."""
    D, Zi, C = [], [], []
    for cx, cy, r, z, c in scene:
        dist = np.hypot(u - cx, v - cy) - r          # < 0 inside the disc
        D.append(1 / (1 + np.exp(np.sign(dist) * dist**2 / sigma)))   # sigmoid(delta d^2 / sigma)
        Zi.append((1 / z - 1 / Z_FAR) / (1 / Z_NEAR - 1 / Z_FAR))      # larger = nearer
        C.append(c)
    D, Zi, C = np.array(D), np.array(Zi)[:, None, None], np.array(C)[:, None, None]
    w = D * np.exp((Zi - Zi.max()) / gamma)          # softmax weights over depth
    colour = (w * C).sum(0) / np.maximum(w.sum(0), 1e-300)
    return (1 - np.prod(1 - D, axis=0)) * colour     # background is 0

TRUE = (18.0, 1.9)                                  # A in front of B

def loss(render, xa, za):
    """Mean squared error to the same renderer's image of the true scene."""
    return np.mean((render(discs(xa, za)) - render(discs(*TRUE))) ** 2)

def fd(render, xa, za, which, h=1e-3):
    """Central finite difference of the loss w.r.t. x_A ('x') or z_A ('z')."""
    if which == "x":
        return (loss(render, xa + h, za) - loss(render, xa - h, za)) / (2 * h)
    return (loss(render, xa, za + h) - loss(render, xa, za - h)) / (2 * h)

hard, soft = render_hard, render_soft
print("1. Depth of disc A (disc B at z = 2.0), x_A = 18")
print("   z_A   hard loss  dL/dz    soft loss  dL/dz    centre pixel hard/soft")
for za in [2.20, 2.12, 2.04, 1.96, 1.88, 1.80]:
    print(f"  {za:.2f}  {loss(hard, 18, za):9.5f} {fd(hard, 18, za, 'z'):7.3f}"
          f"  {loss(soft, 18, za):9.5f} {fd(soft, 18, za, 'z'):7.3f}"
          f"     {hard(discs(18, za))[24, 24]:.3f} / {soft(discs(18, za))[24, 24]:.3f}")

print("2. Sub-pixel shift of disc A, z_A = 1.9")
print("   x_A    hard loss  dL/dx     soft loss   dL/dx")
for xa in [17.80, 17.85, 17.90, 17.95, 18.00, 18.05]:
    print(f"  {xa:.2f}  {loss(hard, xa, 1.9):9.6f} {fd(hard, xa, 1.9, 'x'):7.4f}"
          f"  {loss(soft, xa, 1.9):10.7f} {fd(soft, xa, 1.9, 'x'):8.5f}")

print("3. Gradient descent on z_A from 2.2 (step 0.5, 40 iterations)")
for name, render in [("hard", hard), ("soft", soft)]:
    za = 2.2
    for _ in range(40):
        za -= 0.5 * fd(render, 18, za, "z")
    print(f"  {name}: z_A = {za:.3f}, hard-image loss = {loss(hard, 18, za):.5f}")

Output:

1. Depth of disc A (disc B at z = 2.0), x_A = 18
   z_A   hard loss  dL/dz    soft loss  dL/dz    centre pixel hard/soft
  2.20    0.02688   0.000    0.03039   0.058     0.300 / 0.319
  2.12    0.02688   0.000    0.02370   0.115     0.300 / 0.364
  2.04    0.02688   0.000    0.01202   0.162     0.300 / 0.494
  1.96    0.00000   0.000    0.00186   0.071     0.900 / 0.710
  1.88    0.00000   0.000    0.00012  -0.011     0.900 / 0.850
  1.80    0.00000   0.000    0.00181  -0.026     0.900 / 0.891
2. Sub-pixel shift of disc A, z_A = 1.9
   x_A    hard loss  dL/dx     soft loss   dL/dx
  17.80   0.003828  0.0000   0.0000880 -0.00088
  17.85   0.002813  0.0000   0.0000496 -0.00066
  17.90   0.002109  0.0000   0.0000221 -0.00044
  17.95   0.001406  0.0000   0.0000055 -0.00022
  18.00   0.000000  0.0000   0.0000000  0.00000
  18.05   0.000625  0.0000   0.0000055  0.00022
3. Gradient descent on z_A from 2.2 (step 0.5, 40 iterations)
  hard: z_A = 2.200, hard-image loss = 0.02688
  soft: z_A = 1.900, hard-image loss = 0.00000

In the hard renderer, the center pixel of the overlap is either B’s 0.300 or A’s 0.900, and the loss is either 0.02688 or zero: it jumps once, where $z_A$ passes 2.0, and its finite-difference derivative is zero at every sampled depth. The soft renderer’s center pixel slides from 0.319 to 0.891 as A moves forward, and its loss falls smoothly to a minimum near the true depth, with a derivative that points towards it from both sides. The sub-pixel sweep shows the same for position: the hard loss changes in steps as pixel centers cross the disc boundary, but a shift of $10^{-3}$ pixels changes no pixel, so the measured derivative is again zero; the soft loss is a smooth bowl with zero slope at $x_A = 18$. Gradient descent makes the consequence concrete: started from $z_A = 2.2$, with A hidden behind B, the hard renderer never moves, while the soft renderer brings A to $z_A = 1.900$, where the hard rendering also matches exactly.

Historical Development

Year	Milestone	What it introduced
1996	Light Field Rendering (Levoy & Hanrahan); the Lumigraph (Gortler et al.)	New views by resampling a captured 4D ray function
2019	Deferred Neural Rendering (Thies et al., ACM TOG 38(4))	Learned neural textures on a mesh, decoded by a 2D network
2019	Neural Volumes (Lombardi et al., ACM TOG 38(4))	Voxel opacity and color from an encoder–decoder, differentiable ray marching
2019	Scene Representation Networks (Sitzmann et al., NeurIPS)	Continuous coordinate network with a learned differentiable ray marcher
2019	Soft Rasterizer (Liu et al., ICCV)	Smooth forward rasterization with probabilistic coverage and depth
2020	NeRF (Mildenhall et al., ECCV); first survey (Tewari et al., Computer Graphics Forum)	Coordinate MLP for density and color, volume rendering from posed images
2021	pixelNeRF (Yu et al., CVPR)	NeRF predicted feed-forward from one or a few images
2022	Instant-NGP (Müller et al., ACM TOG 41(4)); EG3D (Chan et al., CVPR); second survey	Hash-grid features with a small MLP; tri-plane 3D-aware GAN
2023	DreamFusion (Poole et al., ICLR); 3D Gaussian Splatting (Kerbl et al., ACM TOG 42(4))	Score distillation from a 2D diffusion model; real-time explicit Gaussian primitives
2024	LRM (Hong et al., ICLR)	Large transformer predicting a tri-plane NeRF from one image

Tewari et al. (2020) name the Generative Query Network (Eslami et al., 2018) as one of the first publications to use the term neural rendering. The sequel survey credits NeRF’s quality and simplicity with an “explosion” of follow-up work (Tewari et al., 2022, §1). Instant-NGP and 3D Gaussian splatting then attacked its main practical cost, the time to train and render.

Evaluation

View synthesis is evaluated on held-out views: some photographs of each scene are withheld from fitting, the method renders those camera poses, and the renderings are compared with the withheld photographs. Three image metrics dominate, and NeRF reported all three (Mildenhall et al., 2020, Table 1):

• PSNR, peak signal-to-noise ratio, $10 \log_{10}(L^2 / \mathrm{MSE})$ in decibels for images with peak value $L$, is the mean squared error on a logarithmic scale, as in image denoising; higher is better.
• SSIM, the structural similarity index of Wang et al. (2004), compares the local means, contrasts and correlations of the two images, computed in their paper with an 11 × 11 Gaussian window of standard deviation 1.5 pixels and averaged over the image; higher is better, with 1 for identical images.
• LPIPS (Zhang et al., 2018) compares deep network features of the two images, unit-normalized along the channel dimension and scaled per channel by linear weights calibrated on human two-alternative forced-choice similarity judgments (their §3, the “lin” variant); lower is better. Image-to-image translation discusses why per-pixel metrics favor blurry outputs and perceptual ones do not.

The metrics can disagree. In NeRF’s Table 1, on the real forward-facing scenes, NeRF has the higher PSNR (26.50 against 24.13) and SSIM (0.811 against 0.798), while LLFF has the better LPIPS (0.212 against 0.250); the authors point to their video for multi-view consistency, which no single-image metric measures.

The common datasets:

Dataset	Content	Protocol as printed
NeRF Realistic Synthetic 360° (Mildenhall et al., 2020)	8 path-traced objects with complex geometry and non-Lambertian materials	100 training and 200 test views per object at 800 × 800
Real Forward-Facing (Mildenhall et al., 2020)	8 handheld cellphone captures, 5 from the LLFF paper (Mildenhall et al., 2019) and 3 new	20 to 62 images per scene, 1/8 held out, 1008 × 756
Mip-NeRF 360 (Barron et al., 2022)	9 unbounded scenes, 5 outdoor and 4 indoor, each a central object with detailed background	Poses from COLMAP
Tanks and Temples (Knapitsch et al., 2017)	Large indoor and outdoor scenes captured on video, built as a reconstruction benchmark	3DGS used two of its scenes for view synthesis

Later papers call the first set the “Blender” dataset; Kerbl et al. (2023), for example, followed Mip-NeRF 360’s methodology of holding out every 8th photo for testing. Tanks and Temples was designed to score reconstructed geometry against laser scans (3D reconstruction); its use for view synthesis scores images only.

Common Misconceptions

• “Neural rendering means NeRF.” NeRF is one 3D method; the 2020 survey mainly covered 2D networks that learn to render (Tewari et al., 2022, Fig. 2), and generative 3D-aware models are a further family.
• “A neural renderer simulates physics.” Only the 3D family uses a physically motivated renderer, and even there the image formation is usually simplified to emission and absorption, without scattering (Tewari et al., 2022, §3.2.2). A 2D network imitates its training images and is consistent across views only as far as it has learned to be.
• “Implicit means stored in a network.” In the signed distance literature, implicit means a level set; NeRF’s density field defines geometry explicitly as a field, even though it is stored in weights (Tewari et al., 2022, §3.1.3).
• “Good held-out PSNR means good geometry.” The metrics score images. A radiance field can match the test views with density that does not concentrate on a surface, which is why extracting surfaces from these models is a research topic of its own (3D reconstruction); this follows from the metrics’ definition.

Where It Is Used

The 2020 survey lists novel view synthesis, semantic photo synthesis and manipulation, facial and body reenactment, relighting, free-viewpoint video, and photo-realistic avatars for virtual and augmented reality telepresence (Tewari et al., 2020, abstract and §6). The 2022 survey adds view synthesis of dynamic scenes, generalization across object classes, scene editing and composition, and relighting and material editing (Tewari et al., 2022, §4), and names the real-time SLAM system iMAP as a rare robotics use (§5).

• View synthesis of captured scenes, the task of the batch’s novel view synthesis article, with cameras from structure from motion.
• 3D content creation, from text with score distillation (Poole et al., 2023) or from a single image with feed-forward models (Hong et al., 2024); the first relies on a pretrained diffusion model, the second on large multi-view training sets.
• Faces and avatars, where deferred neural rendering edited facial expressions in video (Thies et al., 2019) and 3D-aware GANs generate consistent views of faces (Chan et al., 2022).

Open Problems

Tewari et al. (2022, §5) listed the first four of these as open challenges at the time of their survey:

• Integration with graphics tools. Decades of graphics software assume meshes and point clouds; neural representations are hard to edit, inspect and interpret, so editing tools have to be built anew.
• Scale. Most methods handle objects or bounded scenes; large, partially observed environments need new storage and retrieval structures, as voxel hashing provided for TSDF volumes.
• Generalization. Feed-forward models depend on large multi-view training sets with wide baselines, and few-view reconstructions are blurry where views are missing.
• Quality and speed. High-frequency detail and view-dependent appearance remain hard, and training is slow compared with rendering.
• Misuse. Realistic synthetic images of people raise the forgery and detection questions that the 2020 survey discusses in its section on social implications (Tewari et al., 2020, §8).