Novel View Synthesis

Novel view synthesis renders a scene from a viewpoint at which no photograph was taken: given images of the scene, usually with their camera poses and intrinsics, it predicts the image a camera at a new pose would record. Most neural rendering methods are built and benchmarked for this task, but it is older than them; image-based rendering in graphics posed it decades earlier.

Problem Definition

Given $N$ images $I_1, \dots, I_N$ of a scene, with intrinsics $K_k$ and world-to-camera poses $[R_k \mid \mathbf{t}_k]$ such that $\mathbf{X}_c = R_k \mathbf{X}_w + \mathbf{t}_k$ (camera parameters), and a target camera $(K_\ast, R_\ast, \mathbf{t}_\ast)$, predict the target image

$$\hat I_\ast = F\big(\{(I_k, K_k, R_k, \mathbf{t}_k)\}_{k=1}^{N};\; K_\ast, R_\ast, \mathbf{t}_\ast\big).$$

$F$ may be a fitted scene representation and renderer, a network, or a conditional sampler, which returns a plausible target rather than the true one, since the inputs do not determine content they never saw.

“Poses given” usually means “poses estimated”. Real captures are posed by structure from motion, usually COLMAP, whose cameras are treated as ground truth, as for NeRF’s real scenes and the Mip-NeRF 360 dataset (Mildenhall et al., 2020; Barron et al., 2022). RealEstate10K used ORB-SLAM2 and bundle adjustment, its authors having found COLMAP slow and failure-prone on video (Zhou et al., 2018, §4.2–4.3). Estimated cameras carry SfM’s errors, an arbitrary scale and the tool’s axis convention (camera pose).

Input and Output

• Input: one to hundreds of RGB images with intrinsics and poses; sometimes depth maps, foreground masks or scene bounds.
• Output: an RGB image at the target pose, sometimes a depth map, often a scene representation that renders any target.

Variants

• Interpolation versus extrapolation. A target among the input cameras sees mostly surfaces the inputs saw; a target outside them sees surfaces no input saw. Zhou et al. (2018) targeted extrapolation beyond the baseline of a narrow stereo pair.
• Dense versus sparse views. Light fields need dense sampling, per-scene radiance fields tens to hundreds of views, few-shot methods a handful; single-image synthesis, the extreme case, is closer to generation than to reconstruction.
• Static versus dynamic scenes. Most methods assume a static scene under fixed lighting; neural radiance fields covers dynamic and in-the-wild variants.
• Bounded objects, forward-facing scenes and unbounded 360° scenes, which need different scene parameterizations (Barron et al., 2022).

Challenges

• Pose errors. With photometric supervision only, a misplaced camera blurs the fitted scene; neural radiance fields covers joint pose optimization.
• Sparse views and overfitting. Few inputs are explained by many scenes; the CO3D authors saw more expressive models overfit, with a large gap between training and test PSNR (Reizenstein et al., 2021, §5.2).
• Disocclusions, surfaces hidden in every input, which are left as holes or filled from a prior.
• View-dependent effects. A texture-mapped mesh renders only Lambertian materials, and the apparent depth of a specularity changes with viewpoint (Mildenhall et al., 2019, §2.2).
• Real-time rendering at high quality; 3D Gaussian splatting reports at least 30 frames per second at 1080p (Kerbl et al., 2023, abstract).

Approaches

Image-based rendering

Image-based rendering warps, resamples and blends captured images, and needs either very dense views or proxy geometry whose errors become artifacts (Wang et al., 2021, §1). Beyond the dense light fields and the Lumigraph of the neural rendering hub, Mildenhall et al. (2019, §2.2) split methods by their geometry. Global methods reproject nearby views through one scene mesh and blend them, as in view-dependent texture mapping and the Unstructured Lumigraph’s per-pixel weights. Local methods reproject the nearest inputs through per-image depth maps, among them the view interpolation of Chen and Williams (1993), which Wang et al. (2021, §1) call pioneering. A pixel $\mathbf{p}$ with depth $Z$ moves into the target as

$$\mathbf{X}_w = R_k^\top \big(Z\, K_k^{-1} \mathbf{p} - \mathbf{t}_k\big), \qquad \mathbf{p}_\ast \simeq K_\ast \big(R_\ast \mathbf{X}_w + \mathbf{t}_\ast\big),$$

with a z-buffer keeping the nearest of the pixels that land together. Depth comes from stereo or multi-view stereo, and the family fails where it is wrong, at object boundaries, as the example below shows.

Layered representations: multiplane images

A multiplane image (MPI) is a stack of fronto-parallel RGBA planes at fixed depths in a reference camera. Zhou et al. (2018) predict one from a stereo pair with a convolutional network, using 32 planes spaced evenly in disparity between 1 m and 100 m (§3.5), and render a target by warping each plane with a homography and alpha-compositing back to front (§3.3). Trained on video frames, including targets beyond the input pair, the network learns to fill disocclusions. Local Light Field Fusion (LLFF; Mildenhall et al., 2019) predicts an MPI per input view of a handheld capture and blends adjacent MPIs’ renderings in real time. Its guideline, that the nearest object move at most a set number of pixels between views, needs up to about 4000 times fewer views than Nyquist-rate light field sampling (Fig. 1). MPIs are composited like radiance fields (volume rendering) but serve only a limited range of viewpoints.

Per-scene optimized radiance fields and Gaussians

Neural radiance fields (Mildenhall et al., 2020) fit a continuous density and color field to one scene’s posed images, removing the fixed planes and the limited range of viewpoints at the cost of one to two days of fitting per scene (Mildenhall et al., 2020, §5.3). 3D Gaussian splatting (Kerbl et al., 2023) keeps the image formation with explicit primitives, initialized from SfM points and rasterized in real time.

Generalizable feed-forward models

Feed-forward models are trained on many scenes and applied to a new one without optimization. pixelNeRF (Yu et al., 2021) conditions a radiance field on image features sampled at each 3D point’s projection into the inputs. IBRNet (Wang et al., 2021) is closer to image-based rendering: it reads colors and features from 10 nearby source views at render time, predicts densities along each ray with a “ray transformer”, and volume-renders. In their Table 1, on the real forward-facing scenes, pretrained IBRNet reached PSNR 25.13 against 24.13 for LLFF, and after per-scene fine-tuning 26.73 against 26.50 for NeRF. Large reconstruction models such as LRM (Hong et al., 2024) predict a radiance field from one image.

Generative view synthesis

When most of the target is unseen, synthesis becomes generation: Zero-1-to-3 (Liu et al., 2023) fine-tunes Stable Diffusion, a latent diffusion model, to produce an object’s image under a given relative rotation and translation, from one input image. It is trained on renderings of Objaverse, 800K+ 3D models with 12 random views each (§3.4). Different samples for the same target view differ in geometry and appearance (their Fig. 8), as an ill-posed problem requires (generative model).

Geometry-free transformers

The Scene Representation Transformer (Sajjadi et al., 2022) encodes posed or unposed images into a set of latent vectors with a transformer, and renders each target ray by attending into them, without any explicit 3D representation or volume rendering. It targets few inputs and interactive rates; in our reading, such models must learn 3D consistency instead of inheriting it from a renderer.

Why the field moved

In our reading, each family relaxed a limit of the last: warping needs exact depth, MPIs cover few viewpoints, per-scene fields need dense views, and learned priors trade fidelity for working from few views.

Example: Depth Warping Against a Cross-Fade

The script builds the local-geometry baseline on a known scene: two textured cards, at depths 2.5 and 4, before a textured wall at depth 8, ray-cast with $\mathbf{X}_c = R\mathbf{X}_w + \mathbf{t}$ by two input cameras 0.3 units apart. The novel camera sits on their line at $s$ times the baseline, interpolating for $0 < s < 1$. Each input’s depth is forward-warped into the target with a z-buffer, and each covered target pixel back-projected with it to read the input’s color bilinearly. Where the two warped depths agree within 5%, colors are blended with inverse-distance weights; otherwise the nearer surface wins. The cross-fade blends the input images with the same weights and no geometry.

import numpy as np
from scipy.ndimage import maximum_filter, minimum_filter

H, W, f = 96, 128, 110.0
K = np.array([[f, 0, W / 2], [0, f, H / 2], [0, 0, 1.0]])
v, u = np.mgrid[0:H, 0:W] + 0.5                       # pixel centres
pix = np.stack([u, v, np.ones_like(u)], -1)            # homogeneous pixels

# Scene: textured fronto-parallel rectangles in world coordinates (Z = depth plane).
PLANES = [  # (Z, x0, x1, y0, y1, texture frequency, colour phase)
    (2.5, -0.55, 0.05, -0.30, 0.35, 9.0, 0.0),        # near card
    (4.0, -0.20, 1.10, -0.60, 0.20, 6.0, 2.0),        # middle card
    (8.0, -1e9, 1e9, -1e9, 1e9, 3.0, 4.0)]            # background wall

def texture(X, Y, freq, phase):
    """Smooth RGB pattern on a plane, a function of world X, Y."""
    a = np.sin(freq * X + phase) * np.cos(0.7 * freq * Y)
    b = np.sin(1.3 * freq * (X + Y) + 2 * phase)
    return 0.5 + 0.4 * np.stack([a, b, a * b], -1)

def camera(cx, R=np.eye(3)):
    """World-to-camera pose with X_c = R X_w + t for a centre at (cx, 0, 0)."""
    C = np.array([cx, 0.0, 0.0])
    return R, -R @ C

def render(R, t):
    """Ray-cast the planes: returns the RGB image and the depth map (Z_c)."""
    C = -R.T @ t
    d = (np.linalg.inv(K) @ pix[..., None])[..., 0] @ R    # world ray directions R^T K^-1 p
    img, Z = np.zeros((H, W, 3)), np.full((H, W), np.inf)
    for Zp, x0, x1, y0, y1, fr, ph in PLANES:
        s = (Zp - C[2]) / d[..., 2]                     # ray parameter at the plane
        X, Y = C[0] + s * d[..., 0], C[1] + s * d[..., 1]
        zc = s * (d @ R.T)[..., 2]                      # camera-frame depth
        hit = (X >= x0) & (X <= x1) & (Y >= y0) & (Y <= y1) & (s > 0) & (zc < Z)
        img[hit], Z[hit] = texture(X[hit], Y[hit], fr, ph), zc[hit]
    return img, Z

def bilinear(img, x, y):
    """Sample img at continuous pixel coordinates (pixel centres at +0.5)."""
    x, y = np.clip(x - 0.5, 0, W - 1.001), np.clip(y - 0.5, 0, H - 1.001)
    x0, y0 = x.astype(int), y.astype(int)
    ax, ay = (x - x0)[..., None], (y - y0)[..., None]
    return ((1 - ay) * ((1 - ax) * img[y0, x0] + ax * img[y0, x0 + 1])
            + ay * ((1 - ax) * img[y0 + 1, x0] + ax * img[y0 + 1, x0 + 1]))

def warp(img, Z, src, dst):
    """Forward-warp the source depth into the target with a z-buffer, then fetch
    colour from the source by back-projecting each target pixel with that depth."""
    (Rs, ts), (Rd, td) = src, dst
    Kinv = np.linalg.inv(K)
    Xw = (Z[..., None] * (pix @ Kinv.T) - ts) @ Rs      # X_w = R_s^T (X_c - t_s)
    Xd = Xw @ Rd.T + td                                 # X_d = R_d X_w + t_d
    p = Xd @ K.T
    ui = np.floor(p[..., 0] / p[..., 2]).astype(int)
    vi = np.floor(p[..., 1] / p[..., 2]).astype(int)
    ok = (ui >= 0) & (ui < W) & (vi >= 0) & (vi < H) & (Xd[..., 2] > 0)
    zbuf = np.full((H, W), np.inf)
    np.minimum.at(zbuf, (vi[ok], ui[ok]), Xd[..., 2][ok])   # z-buffer: nearest wins
    hit = np.isfinite(zbuf)
    Xw_t = (zbuf[hit][:, None] * (pix[hit] @ Kinv.T) - td) @ Rd   # back-project target
    q = (Xw_t @ Rs.T + ts) @ K.T                        # ... and project into the source
    out = np.zeros((H, W, 3))
    out[hit] = bilinear(img, q[:, 0] / q[:, 2], q[:, 1] / q[:, 2])
    return out, zbuf

def synthesize(views, dst, cx):
    """Warp both inputs, keep the nearer surface, blend where depths agree."""
    (I0, Z0, c0, x0), (I1, Z1, c1, x1) = views
    A, za = warp(I0, Z0, c0, dst)
    B, zb = warp(I1, Z1, c1, dst)
    w0 = 1 / (abs(cx - x0) + 1e-6); w1 = 1 / (abs(cx - x1) + 1e-6)
    w0, w1 = w0 / (w0 + w1), w1 / (w0 + w1)            # inverse-distance weights
    with np.errstate(invalid='ignore'):
        same = np.abs(za - zb) < 0.05 * np.minimum(za, zb)
    out = np.where((za < zb)[..., None], A, B)          # nearer surface wins
    out = np.where(same[..., None], w0 * A + w1 * B, out)
    hole = np.isinf(za) & np.isinf(zb)
    return out, hole, (w0, w1)

def in_frame(Zt, dst, src):
    """Does each target pixel's true surface point project inside the source image?"""
    (Rd, td), (Rs, ts) = dst, src
    Xw = (Zt[..., None] * (pix @ np.linalg.inv(K).T) - td) @ Rd
    q = (Xw @ Rs.T + ts) @ K.T
    x, y = q[..., 0] / q[..., 2], q[..., 1] / q[..., 2]
    return (x >= 0) & (x < W) & (y >= 0) & (y < H)

def psnr(a, b, mask):
    return 10 * np.log10(1.0 / np.mean((a[mask] - b[mask]) ** 2))

BASE = 0.3
cams = [camera(0.0), camera(BASE)]
inputs = [render(*c) for c in cams]
views = [(inputs[0][0], inputs[0][1], cams[0], 0.0), (inputs[1][0], inputs[1][1], cams[1], BASE)]

print("s = novel camera centre / baseline (inputs at s = 0 and s = 1)")
print("    s   warp PSNR  bad px (at edges)  holes (disoccluded)  cross-fade PSNR")
for s in [0.25, 0.5, 0.75, 1.25, 1.5, 2.0, -0.5, -1.0]:
    dst = camera(s * BASE)
    gt, Zgt = render(*dst)
    out, hole, (w0, w1) = synthesize(views, dst, s * BASE)
    fade = w0 * inputs[0][0] + w1 * inputs[1][0]        # no geometry: blend pixels
    bad = (np.abs(out - gt).max(-1) > 0.05) & ~hole     # pixels off by > 0.05
    edge = maximum_filter(Zgt, 3) - minimum_filter(Zgt, 3) > 0.5   # next to a depth jump
    seen = in_frame(Zgt, dst, cams[0]) | in_frame(Zgt, dst, cams[1])
    print(f"{s:5.2f}   {psnr(out, gt, ~hole):6.2f} dB  {bad.sum():5d} ({(bad & edge).sum():2d})       {100 * hole.mean():5.2f}%"
          f" ({100 * (hole & seen).mean():4.2f}%)     {psnr(fade, gt, ~hole):6.2f} dB")

print("Depth noise at s = 0.5 (Gaussian, relative to depth), seeds 0-9")
dst = camera(0.5 * BASE); gt, _ = render(*dst)
for rel in [0.0, 0.005, 0.01, 0.02, 0.05]:
    vals = []
    for seed in range(10):
        rng = np.random.default_rng(seed)
        noisy = [(I, Z * (1 + rel * rng.standard_normal(Z.shape)), c, x) for I, Z, c, x in views]
        out, hole, _ = synthesize(noisy, dst, 0.5 * BASE)
        vals.append((psnr(out, gt, ~hole), 100 * hole.mean()))
    p, h = np.array(vals).T
    print(f"  sigma = {100 * rel:3.1f}%: PSNR {p.min():.2f}-{p.max():.2f} dB, holes {h.min():.2f}-{h.max():.2f}%")

Output:

s = novel camera centre / baseline (inputs at s = 0 and s = 1)
    s   warp PSNR  bad px (at edges)  holes (disoccluded)  cross-fade PSNR
 0.25    35.04 dB     55 (55)        0.00% (0.00%)      22.77 dB
 0.50    40.24 dB     59 (59)        0.00% (0.00%)      20.82 dB
 0.75    49.97 dB     23 (23)        0.00% (0.00%)      23.21 dB
 1.25    36.20 dB     68 (68)        1.27% (0.49%)      19.34 dB
 1.50    45.50 dB     46 (46)        2.77% (1.20%)      15.00 dB
 2.00    34.93 dB     71 (71)        5.31% (2.18%)      11.50 dB
-0.50    43.27 dB     33 (33)        2.73% (1.17%)      15.14 dB
-1.00    40.64 dB     47 (47)        5.31% (2.18%)      11.47 dB
Depth noise at s = 0.5 (Gaussian, relative to depth), seeds 0-9
  sigma = 0.0%: PSNR 40.24-40.24 dB, holes 0.00-0.00%
  sigma = 0.5%: PSNR 39.45-40.25 dB, holes 0.00-0.00%
  sigma = 1.0%: PSNR 38.25-40.17 dB, holes 0.00-0.04%
  sigma = 2.0%: PSNR 35.78-37.45 dB, holes 0.08-0.20%
  sigma = 5.0%: PSNR 33.85-35.31 dB, holes 0.23-0.31%

Interpolating with exact depth, warping reaches 35–50 dB on covered pixels and the cross-fade, which superimposes offset copies of each card (ghosting), 21–23 dB. At every position, the warp’s error sits in 23 to 71 of 12,288 pixels, all next to a depth jump, where rounding a warped point to a pixel puts it on the wrong side of an occlusion boundary. These few pixels dominate the mean squared error, so PSNR jumps between positions without a trend.

Holes appear only when extrapolating: 1.27% of the target at $s = 1.25$, 5.31% at $s = 2$ or $s = -1$, of which 2.18 percentage points are disocclusions, surfaces inside an input’s view but hidden in both, and the rest outside both images. PSNR on the covered pixels stays high, so the score depends on whether holes count. The cross-fade falls to 11.5 dB.

Over seeds 0–9, 0.5% relative depth noise costs at most 0.8 dB, 2% costs 2.8–4.5 dB and opens cracks in 0.08–0.20% of the image, and 5% costs 4.9–6.4 dB: the dependence on geometry that learned methods relax.

Datasets and Benchmarks

Dataset	Content	Protocol as printed
NeRF Realistic Synthetic 360° (“Blender”)	8 path-traced objects	100 training and 200 test views per object, 800 × 800 (Mildenhall et al., 2020)
Real Forward-Facing (LLFF scenes)	8 handheld cellphone scenes, 5 from LLFF	20 to 62 images per scene at 1008 × 756, 1/8 held out for testing (Mildenhall et al., 2020)
Mip-NeRF 360	9 unbounded scenes, 5 outdoor and 4 indoor	COLMAP poses (Barron et al., 2022); every 8th photo held out for testing, which Kerbl et al. (2023) describe as Mip-NeRF 360’s methodology
Tanks and Temples	Large scenes captured as 8-megapixel video, built for reconstruction (Knapitsch et al., 2017)	Kerbl et al. (2023) used two scenes, Truck and Train, with the same every-8th split
DTU, sparse-view protocol	124 robot-arm scenes, 49 or 64 views each at 1600 × 1200, 80 used in the paper’s evaluation (Aanæs et al., 2016, and its official page)	pixelNeRF: 88 training and 15 test scenes without overlap, images downsampled to 400 × 300, fixed informative input views (Yu et al., 2021, §5.3)
RealEstate10K	Camera trajectories for about 80,000 clips from about 10,000 YouTube videos, about 10 million frames	Video URLs, timestamps, intrinsics and poses only; about 90% of clips for training, 10% for testing (official page, as of October 2026)
CO3D	1.5 million frames from nearly 19,000 videos circling objects of 50 MS-COCO categories, COLMAP cameras and point clouds	Per-category 9:1 video split; frames split 8:2 into known and unseen; 1, 3, 5, 7 or 9 source views per test sample (Reizenstein et al., 2021, §5)

pixelNeRF’s authors replaced the MVSNet split of DTU because it shares scenes between training and testing. The paper’s RealEstate10K had about 7,000 clips and 750,000 frames, with 1,329 test sequences (Zhou et al., 2018, §4–5).

Evaluation Metrics

Withheld views are rendered at their poses and compared with the photographs by PSNR, SSIM and LPIPS, defined in the neural rendering hub. Their limits:

• Disagreement with perception. In the two-alternative tests of Zhang et al. (2018, §4.1), $\ell_2$ distance agreed with human choices 63.2% of the time and SSIM 63.1%, against 67.0–68.9% for features of classification-trained networks and 73.9% for human consistency; SSIM was not designed for geometric distortion, such as misalignment from pose error. Blur changes $\ell_2$ little and perception a lot (their §1), so PSNR favors blurry averages of uncertain content. On CO3D, IDR won on LPIPS with less detailed renderings but lost on PSNR (Reizenstein et al., 2021, §5.2).
• Masks and distance. CO3D scores PSNR on foreground pixels only, and also by bins of distance from the source views (§5).
• No ground truth for invented content. When several targets are plausible, a pixel metric punishes all but the photograph. Zero-1-to-3 still reports PSNR, SSIM and LPIPS against the true views, with the Fréchet Inception Distance (FID), which compares image distributions, not pairs; on Google Scanned Objects it reached PSNR 18.378, LPIPS 0.088 and FID 0.027 (Liu et al., 2023, Table 1), an FID far below usual values, printed without its protocol. In our reading, FID scores realism, not agreement with the scene, and none of these metrics checks that generated views are consistent with one 3D scene.

Applications

• Virtual and augmented reality and virtual exploration of captured places, which motivated stereo magnification (Zhou et al., 2018), LLFF (Mildenhall et al., 2019) and SRT, shown on Street View imagery (Sajjadi et al., 2022).
• 3D content from a single photo, with Zero-1-to-3 and LRM, related to 3D reconstruction.

Open Problems

In our assessment, as of October 2026, the challenges above remain open: methods need posed images, sparse-view priors invent unscorable content, dynamic scenes and changing light lag static captures, and the standard metrics ignore multi-view consistency.