3D Generation

3D generation is the task of producing new three-dimensional objects or scenes, as representations that can be rendered from any viewpoint, rather than single images. The input may be nothing but random noise, as in a 3D-aware generative adversarial network that produces new faces or cats; a text prompt (text-to-3D); or one photograph of an object (image-to-3D, also called single-image 3D generation or zero-shot 3D reconstruction). The output is a 3D representation, such as a neural radiance field, a tri-plane or a textured polygon mesh, whose renderings must look right and agree with each other from every side. The methods that dominate the field do not learn from 3D data alone, which is scarce, but borrow the knowledge of 2D image and video generators through a differentiable renderer, the subject of Neural Rendering. The main families are 3D-aware GANs such as EG3D (Chan et al., 2022); optimization against a pretrained 2D diffusion model by score distillation sampling, which Magic3D’s authors (Lin et al., 2023) credit to DreamFusion (Poole et al., 2023) and refine; diffusion models conditioned on the camera, from Zero-1-to-3 (Liu et al., 2023) to the video-based SV3D (Voleti et al., 2024); and feed-forward models that predict a 3D representation in one pass. The DreamFusion paper is an ICLR paper that was not read for this article; it is described here only through the readable papers that build on it.

Problem Definition

A 3D generator outputs parameters $\theta$ of a scene representation together with a renderer $g$, so that $x = g(\theta, \pi)$ is the image seen from camera $\pi$. The requirement that separates the task from image generation is that one $\theta$ serves every camera: the renderings must be multi-view consistent, because they are projections of a single object. What must be generated depends on the variant:

• Unconditional or class-level generation: sample $\theta$ so that renderings from the cameras of the training photographs follow the distribution of those photographs, as a 3D-aware GAN does for faces.
• Text-to-3D: find $\theta$ such that renderings from all viewpoints are probable images of the prompt under a text-to-image model.
• Image-to-3D: find $\theta$ whose rendering from the input camera reproduces the given photograph and whose other views are plausible completions of the unseen sides.

The task is ill-posed in every variant. Many objects match a prompt, and the back of an object in a single photograph is invisible; Zero-1-to-3’s samples for the same target view differ in geometry and appearance (Liu et al., 2023, Fig. 8). A generator is judged by whether its outputs are plausible, consistent and faithful to the condition, not by agreement with one true answer; in this sense the task is the generative counterpart of novel view synthesis and of 3D reconstruction.

Input and Output

Variant	Input	Output (in the papers here)	Training data
3D-aware GAN (EG3D)	Latent code and camera pose	Tri-plane features decoded by volume rendering, plus image-space super-resolution	Single-view photographs with estimated camera poses
Text-to-3D by score distillation (DreamFusion, Magic3D)	Text prompt	NeRF (DreamFusion); hash-grid neural field, then textured mesh (Magic3D)	None for the 3D model; a pretrained text-to-image model
View-conditioned diffusion (Zero-1-to-3)	One image and a relative camera rotation and translation	A novel view; a 3D model by score distillation from it	Renderings of a large set of 3D models
Multi-view or video diffusion (SV3D)	One image and a camera orbit	An orbit of 21 views; a mesh fitted to them	Renderings of a large set of 3D models
Feed-forward reconstruction	One image or a few generated views	Point cloud, neural field or mesh, in one network pass	Large 3D or multi-view sets

The camera conventions matter in every row: EG3D associates each training photograph with intrinsics and extrinsics from off-the-shelf pose detectors (§4), Zero-1-to-3 conditions on the relative rotation and translation between input and target views (§3), and SV3D parameterizes a view by elevation and azimuth on an orbit around an object at the origin (§3).

Variants

• Text-to-3D creates an asset from a description. Magic3D, for instance, generates models for the 397 prompts published on DreamFusion’s website (§5).
• Image-to-3D lifts one photograph to a full object. Zero-1-to-3 and SV3D both frame it as reconstruction from a single image (Zero-1-to-3 abstract; SV3D §1), although most of the object is generated.
• Unconditional 3D-aware synthesis learns a distribution of objects of one category from photographs, as EG3D does with FFHQ faces and AFHQ cats (§5).
• Editing and personalization. Magic3D edits a coarse model by fine-tuning it with a modified prompt, and generates subjects from a few photographs by first fine-tuning its diffusion priors with DreamBooth (§6), the personalization method for text-to-image models.
• Scenes rather than objects. Most methods target single objects on a plain background; Zero-1-to-3 names generalization to scenes with complex backgrounds as an open challenge (§5.1).

Challenges

3D data are scarce. Magic3D’s authors attribute the slow progress of 3D generation, compared with image generation, largely to the lack of diverse large-scale 3D datasets: 3D content is much less accessible on the Internet than images and video (§1). Earlier 3D generators were mostly trained per category (§1). The authors therefore ask whether 3D generation can be achieved by leveraging text-to-image models, whose training sets comprise billions of image–text samples (§1).

2D models do not know where the camera is. A text-to-image model learns the distribution of photographs, not of views of an object. Zero-1-to-3’s authors name two problems with extracting 3D knowledge from such models: their representations do not explicitly encode correspondences between viewpoints, and they inherit the viewpoint biases of the Internet; samples of “a chair” from DALL·E 2 and Stable Diffusion v2 mostly show forward-facing chairs in canonical poses (§3, Fig. 2).

Multi-view consistency. SV3D’s authors call multi-view consistency the most critical requirement for novel view synthesis and 3D generation, and identify its lack in the underlying generative models as a key issue of generation-based reconstruction (§1, §2.1). Score-distillation methods, in their account, are prone to the Janus problem and to over-saturated textures (§2.2). The Janus problem is an object that shows its front, a face for instance, on several sides; SV3D cites it to DreamFusion and Latent-NeRF, a paper not read here.

Cost and resolution. DreamFusion supervises its NeRF with $64 \times 64$ renderings and, according to Magic3D’s authors, takes 1.5 hours per prompt on average on TPUv4 (§1); its global MLP limited it to an effective batch size of 8 (Magic3D supplement, B). 3D-aware GANs must render tens of millions of images during training, which is why EG3D’s authors call explicit voxel grids and neural implicit representations too memory-inefficient or slow for a high-resolution 3D GAN (§1).

No ground truth. A generated object has no reference to compare with, except in image-to-3D benchmarks built from scanned or synthetic objects, so evaluation mixes image metrics, geometry metrics against held-out 3D models, and human preference (see Evaluation Metrics).

Approaches

3D-aware GANs

The first route keeps the GAN recipe and puts a 3D representation and a renderer inside the generator, so that images are rendered rather than painted. The discriminator sees only single photographs; consistency comes from the generator’s 3D-structure-aware inductive bias, as EG3D’s authors put it (§1–2). EG3D (Chan et al., CVPR 2022), which the Neural Rendering article summarizes, is the representative design:

• Tri-planes. A StyleGAN2 backbone outputs a $256 \times 256 \times 96$ feature image, split into three 32-channel axis-aligned planes. A 3D point is projected onto each plane, the three bilinearly interpolated features are summed, and an MLP with one hidden layer of 64 units decodes density and a 32-channel feature, with no positional encoding or view direction (§3, §4.1). For a side length of $N$ features, tri-planes scale with $O(N^2)$ rather than the $O(N^3)$ of dense voxels (§3).
• Rendering and super-resolution. Volume rendering produces a $128^2$ feature image with 96 samples per ray, and two StyleGAN2-modulated blocks upsample it to $256^2$ or $512^2$ (§4.1–4.2).
• Dual discrimination. Image-space upsampling can break view consistency, so the first three channels of the raw rendering are upsampled and concatenated with the final image, and the discriminator judges both together; real images are paired with blurred copies of themselves (§4.3).
• Pose conditioning. The discriminator receives the camera matrices, and the generator’s mapping network receives them too, so that attributes correlated with pose in FFHQ (smiling, for example) can be modelled; at inference the generator is conditioned on a fixed pose, and during training the conditioning pose is swapped for a random one half the time to avoid degenerate “billboards” facing the camera (§4.3–4.4).

EG3D renders in real time at $512^2$ on one RTX 3090 (Table 3), inherits StyleGAN’s latent space for style mixing, and can reconstruct a face’s 3D shape from one photograph by GAN inversion with pivotal tuning (§5.3). Its limits are those of the family: a known distribution of camera poses (§6) and, as Magic3D’s authors put it of 3D-aware GANs, a single object category per model, such as cars, chairs or human faces, which lacks scalability and creative control (§2).

Score distillation from 2D diffusion models

DreamFusion removed 3D training data altogether. In Magic3D’s summary (§3), it combines a scene model $x = g(\theta)$, a volumetric renderer of a coordinate MLP, with a frozen text-to-image diffusion model whose noise prediction $\hat\epsilon_\phi(x_t; y, t)$ supplies a gradient that pushes every rendering toward high-density regions of the prompt-conditioned image distribution. The gradient is called score distillation sampling (SDS):

$$\nabla_\theta \mathcal{L}_{\text{SDS}}\big(\phi, g(\theta)\big) = \mathbb{E}_{t, \epsilon}\Big[ w(t)\, \big(\hat\epsilon_\phi(x_t; y, t) - \epsilon\big)\, \frac{\partial x}{\partial \theta} \Big]$$

(Magic3D Eq. 1), where $x_t = \alpha_t x + \sigma_t \epsilon$ is the rendering noised to level $t$ and $w(t)$ a weighting. The Jacobian of the diffusion network does not appear: the residual between predicted and added noise is used directly as the gradient on the image and backpropagated through the renderer, as the Neural Rendering article describes. Sauer et al. (2024, §2) note that SDS is also known as Score Jacobian Chaining, the framework Zero-1-to-3 builds on. DreamFusion used a variant of Mip-NeRF 360 with an explicit shading model and Imagen as the prior (Magic3D §3), random camera and light augmentations, an initial spatial density bias toward the centre, an opacity regularization (Magic3D supplement, B) and an MLP environment map for the background (Magic3D §4.3). Magic3D’s authors also report that DreamFusion relies on large classifier-free guidance weights for quality (§3), replacing $\hat\epsilon_\phi$ by its guided combination.

What SDS optimizes. With an exact prior the gradient has a closed form. Write $p_t$ for the noisy image distribution at level $t$, so that $\hat\epsilon = -\sigma_t \nabla \log p_t(x_t)$ for the exact model. It follows that

$$\mathbb{E}_\epsilon\big[\hat\epsilon(\alpha_t x + \sigma_t \epsilon) - \epsilon\big] = -\frac{\sigma_t}{\alpha_t}\, \nabla_x\, \mathbb{E}_\epsilon\big[\log p_t(\alpha_t x + \sigma_t \epsilon)\big] = \frac{\sigma_t}{\alpha_t}\, \nabla_x\, D_{\text{KL}}\big(\mathcal{N}(\alpha_t x, \sigma_t^2 I)\, \Vert\, p_t\big),$$

because $\mathbb{E}[\epsilon] = 0$ and the entropy of the Gaussian does not depend on $x$. SDS is therefore gradient descent, through the renderer, on a weighted sum over noise levels of the divergence from a narrow Gaussian around each rendering to the noised prior. This derivation is ours. Sauer et al. refer to an SDS objective proposed by DreamFusion, whose derivation was not read here, and state that ADD’s distillation loss, with a squared $L_2$ distance and a specific weighting, becomes equivalent to it, as shown in their supplementary material, which was not read (§3.3; see Diffusion Distillation). Two consequences follow. The divergence in this direction is small when the rendering sits in any high-density region, so SDS seeks a mode rather than a sample; and the $-\epsilon$ term, which has mean zero, does not change the objective but does change each step’s noise. With guidance $s$ (here $s = 1$ means none, the convention of the classifier-free guidance article’s $s = 1 + w$), the exact guided noise prediction is that of $p_t(x \mid y)^s\, p_t(x)^{1 - s}$, so large $s$ seeks renderings that are probable under the prompt and improbable without it.

Magic3D (Lin et al., CVPR 2023) kept SDS and changed the scene models and priors, in two stages (§4):

1. Coarse: a neural field on Instant NGP’s multiresolution hash grid, with single-layer MLPs of 32 hidden units predicting albedo, density and normals, octree-based empty-space skipping and density-based pruning, optimized with eDiff-I’s base model at $64 \times 64$ (§4.1–4.2; supplement, B). SDS samples $t \sim U(0, 1)$ with $w(t) = 1$ (supplement, B).
2. Fine: a textured mesh extracted with DMTet, a deformable tetrahedral grid whose vertices carry signed distances and offsets, initialized from the coarse density field and rendered by a differentiable rasterizer at $512 \times 512$, so that the latent diffusion model Stable Diffusion can serve as the prior (§4.2–4.3). The gradient then passes through the encoder as well (Eq. 2), and SDS samples $t \sim U(0.02, 0.5)$ with $w(t) = \sigma_t^2$; the authors found $t_{\max}$ between 0.5 and 0.7 to work well (supplement, B).

The coarse stage took about 15 minutes and the fine stage 25 minutes on 8 A100 GPUs, against DreamFusion’s reported 1.5 hours (§1, §5). Their ablations favored the two-stage design: meshes optimized from scratch with the high-resolution prior failed to give high-quality results, and a single-stage neural field rendered at low resolution and upsampled for the latent prior gave worse shapes (§5, Fig. 4). Their supplement adds a style-guided variant with two guidance weights, for text and for joint text-and-image conditioning; weights of 0 and 100 are equivalent to a single guidance weight, and combinations around 50 and 50 worked best (supplement, D and Fig. 2).

Score distillation is not confined to 3D. Sauer et al. (§2) list its uses in text-to-3D video synthesis and image editing, and its descendant as a loss for training one-step image generators is covered in Diffusion Distillation.

View-conditioned diffusion

In our reading, a prior that knows the camera removes the cause of the Janus problem at its source, as the example below shows on a toy prior. Zero-1-to-3 (Liu et al., ICCV 2023) fine-tunes Stable Diffusion to generate the view of an object under a relative rotation and translation $(R, T)$ from one input image (§3). The condition enters two ways: a CLIP embedding of the input image concatenated with $(R, T)$ (“posed CLIP”) through cross-attention, and the input image concatenated channel-wise with the image being denoised; conditions are dropped at random during training so that classifier-free guidance can be applied (§3.2). Training pairs come from Objaverse, 800K+ 3D models by 100K+ artists, each rendered from 12 random viewpoints (§3.4). For 3D reconstruction, the model replaces the text-to-image prior in Score Jacobian Chaining, with an MSE loss on the input view, a depth smoothness loss and a near-view consistency loss, on a voxel radiance field from which a mesh is extracted by marching cubes (§3.3, §4.1). They keep a technique of SJC, inspired by DreamFusion, of setting the guidance value significantly higher than usual, which they say decreases the diversity of each sample but improves the fidelity of the reconstruction (§3.3). The Novel View Synthesis article places Zero-1-to-3 among view synthesis methods.

Multi-view and video diffusion

Models that generate one novel view at a time, such as Zero-1-to-3 and its successors, are not designed to be multi-view consistent inherently, as SV3D’s authors put it; they group the responses (§2.1):

• MVDream, SyncDreamer, HexGen3D and Zero123++ generate several specific views at once, with interaction between views through cross-attention, but not arbitrary viewpoints;
• Consistent-1-to-3 and SPAD use epipolar geometry;
• SyncDreamer and ConsistNet put a 3D representation inside the latent diffusion model;
• Vivid-1-to-3, SVD-MV and IM-3D use video diffusion models, the line SV3D follows.

SV3D (Voleti et al., ECCV 2024) fine-tunes Stable Video Diffusion to generate an orbital video of $K = 21$ views around an object at $576 \times 576$, conditioned on one image, with the elevation and azimuth of each frame embedded and added to the noise-level embedding (§3, Fig. 2). Static orbits circle the object at the input’s elevation; dynamic orbits jitter the azimuths and vary the elevation with sinusoids, and the pose-conditioned model trained progressively on static then dynamic orbits, SV3Dp, was the best performing (§3, §3.1, Tables 1–4). At inference, guidance rises linearly from 1 at the front view to 2.5 at the back and falls again, a triangle schedule that avoided the over-sharpened last frames of the linear schedule inherited from Stable Video Diffusion (§3, Fig. 4). Training took about six days on 32 80 GB A100 GPUs (§3).

For 3D, SV3D treats the generated orbit as photographs, a pipeline its authors compare with photogrammetry in which capture is replaced by view synthesis (§1). A coarse Instant-NGP NeRF is fitted to the generated views with photometric losses (MSE, mask and LPIPS), then refined as a DMTet mesh; a model of 24 spherical Gaussians for white illumination is fitted jointly to reduce lighting baked into the texture (§4). SDS with SV3D as the prior is applied only through a soft mask on surfaces unlikely to be visible in the reference orbit, because naive SDS made training unstable and gave over-saturated or blurry textures (§4.1, Fig. 9).

Feed-forward generation and reconstruction

Optimization per object takes minutes to hours. Feed-forward models predict the representation in one pass instead, trained on large 3D or multi-view collections. SV3D’s 3D comparisons include Point-E and Shap-E, which they found often produce incomplete shapes, One-2-3-45++, which trains an additional 3D network on a 2D generator’s outputs, and DreamGaussian (§2.1, §4.2). Zero-1-to-3 describes Point-E as a diffusion model over colored point clouds trained on OpenAI’s internal 3D dataset, with an impressive zero-shot ability limited by its 4,096 points (§4.2, §4.5). Large reconstruction models such as LRM (Hong et al., 2024) belong to this family; the Neural Rendering article describes LRM.

Output representations

Representation	Where it appears here	Strength for generation
Neural radiance field (MLP)	DreamFusion’s Mip-NeRF 360 variant	Handles topological change smoothly (Magic3D §4.2); slow
Hash-grid neural field	Magic3D coarse stage, SV3D coarse stage	Fast; high-frequency detail at low cost
Tri-plane	EG3D	$O(N^2)$ memory; planes generated by a 2D CNN
Voxel radiance field	Zero-1-to-3’s reconstruction	Simple; memory grows as $N^3$
Textured mesh (DMTet)	Magic3D fine stage, SV3D	Fast high-resolution rasterization; usable in graphics software
Point cloud	Point-E	Direct output of a feed-forward model; sparse

The common pattern, in our reading, is coarse-to-fine: a volumetric representation for the first stage, where shapes must form and change topology, and a mesh for the last, where resolution and export matter (Magic3D and SV3D both follow it). In our reading, the explicit Gaussian primitives of 3D Gaussian splatting can fill the same role, since they are rendered differentiably and fast; none of the papers read here uses them, although SV3D compares with DreamGaussian, whose title, as SV3D’s reference list gives it, names generative Gaussian splatting. The 3D Representations article compares the forms and conversions between them.

Example: Score Distillation with an Exact View Prior

The example runs SDS on a $16^3$ voxel object with an exact Gaussian mixture standing in for the pretrained image model, so that every score is exact and only the SDS objective, the view conditioning of the prior and the guidance scale are under study. The prior is a mixture over $16 \times 16$ grey renderings of two known objects (class 0, an ellipsoid “figure” with eyes and a mouth in front, an ear spot on its right side and a stripe on its left; class 1, a “box” with a window, two dots and a band), each in a full-contrast and a faint variant, from four views, with pixel noise of standard deviation 0.1. Front views get weight 0.55 and each other view 0.15, mimicking the viewpoint bias Zero-1-to-3 shows in text-to-image models. The prompt is class 0. A view-agnostic prior conditions on the class only, as a text prompt does; a view-conditioned prior conditions on class and view, in the spirit of Zero-1-to-3 and SV3D. The unconditional prior, used for guidance, is the whole mixture.

Conventions: the volume is indexed vol[y, x, z] with $y$ down the image, $x$ to the right and $z$ toward the camera; rendering is orthographic emission–absorption along $-z$ with voxel size 1 and a background of $-1$, with density $\operatorname{softplus}(\rho)$ and albedo $\tanh(a)$. View $v$ turns the object by $v$ quarter turns about the vertical axis (torch.rot90 from $+x$ toward $+z$), so view 1 shows the object’s $+x$ side (“right”) and view 2 its back. The diffusion uses the linear schedule of DDPM, $\alpha_t = \sqrt{\bar\alpha_t}$, $\sigma_t = \sqrt{1 - \bar\alpha_t}$. SDS uses $w(t) = 1$, as Magic3D’s first stage does, one noise level $t$ per view drawn uniformly from 20 to 980, all four views each step, Adam with learning rate 0.03 for 1,500 steps, from a density blob (an initial density bias, as DreamFusion and Magic3D use) with random albedo. A 2D run applies the same SDS to four free images started from the same renderings, which shows what the objective itself gives without any 3D coupling. The script prints the data’s properties, checks the SDS identity above, and then, over seeds 0–4, the number of views whose rendering is nearest the class-0 image of its own view (of 20), the number of non-front views nearest a front image (Janus, of 15), the number of seeds whose front view is nearest the full-contrast variant (of 5), the mean absolute pixel value (saturation) and the RMS distance to the nearest valid class-0 object’s four renderings. It needs PyTorch (output from PyTorch 2.14.1 and NumPy 2.5.4, one thread, float64) and ran in about a minute on a laptop CPU; two runs gave identical output. Two printed checks were settled with the editor after pilot runs and before the first run. The planned form of check 2 compared the whole SDS direction, including the subtracted $\epsilon$, with the KL gradient, which a finite sample of noises cannot match exactly; the check therefore compares the average predicted noise, with its tolerance fixed in advance, and prints the size of the mean of $\epsilon$ separately. The data check prints the two back-view variant pairs on their own line, because the plan had wrongly assumed them identical. Nothing was changed after the first run.

import numpy as np
import torch

torch.set_num_threads(1)
dt = torch.float64
N, BG = 16, -1.0
c = torch.arange(N, dtype=dt) + 0.5 - N / 2
Y, X, Z = torch.meshgrid(c, c, c, indexing="ij")          # vol[y, x, z]: y down, x right, z toward camera
VIEWS = ["front", "right", "back", "left"]

def render(sigma, alb, v):
    """Orthographic emission-absorption render along -z after v quarter turns about the y axis."""
    s = torch.rot90(sigma, v, dims=(1, 2)).flip(2)        # ray order: nearest voxel (z = +7.5) first
    a = torch.rot90(alb, v, dims=(1, 2)).flip(2)
    alpha = 1 - torch.exp(-s)                             # voxel size 1
    T = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1 - alpha[..., :-1]], -1), -1)
    w = T * alpha
    return (w * a).sum(-1) + (1 - w.sum(-1)) * BG         # background -1

def make_object(cls, faint):
    """Class 0: ellipsoid 'figure' (eyes and mouth in front, ear spot on the right, stripe on
    the left). Class 1: 'box' (window in front, two dots on the right, band on the left).
    Backs are plain. The faint variant keeps a quarter of each mark's contrast."""
    base, m = 0.3, (0.25 if faint else 1.0)
    alb = torch.full_like(X, base)
    def mark(region, val):
        alb[region] = base + m * (val - base)
    if cls == 0:
        inside = (X / 5.5) ** 2 + (Y / 6.5) ** 2 + (Z / 4.0) ** 2 <= 1
        mark((Z > 0) & (((X - 2.2) ** 2 + (Y + 2) ** 2 < 1.5 ** 2) | ((X + 2.2) ** 2 + (Y + 2) ** 2 < 1.5 ** 2)), -0.9)
        mark((Z > 0) & (X.abs() < 2.5) & (Y > 1.5) & (Y < 3), -0.9)
        mark((X > 0) & ((Y + 1) ** 2 + Z ** 2 < 2.0 ** 2), 0.95)
        mark((X < 0) & (Z.abs() < 1) & (Y.abs() < 4.5), -0.9)
    else:
        inside = (X.abs() <= 5) & (Y.abs() <= 5) & (Z.abs() <= 4)
        mark((Z > 0) & (X.abs() < 2.5) & (Y > -3.5) & (Y < 0), 0.95)
        mark((X > 0) & (((Y + 2) ** 2 + (Z - 1.5) ** 2 < 1.2 ** 2) | ((Y + 2) ** 2 + (Z + 1.5) ** 2 < 1.2 ** 2)), -0.9)
        mark((X < 0) & ((Y - 1).abs() < 1) & (Z.abs() < 3.5), 0.95)
    return 5.0 * inside.to(dt), alb

# The "view prior": a Gaussian mixture over 16x16 renderings, one component per
# (class, variant, view), pixel noise std 0.1, front views over-represented.
PI = torch.tensor([0.55, 0.15, 0.15, 0.15], dtype=dt)
comps, lab = [], []                                        # lab: (class, variant 0 full / 1 faint, view)
for k in range(2):
    for u in range(2):
        sig, alb = make_object(k, u == 1)
        for v in range(4):
            comps.append(render(sig, alb, v).reshape(-1)); lab.append((k, u, v))
MU, LAB = torch.stack(comps), np.array(lab)                # MU: (16, 256)
D, S2 = 256, 0.1 ** 2
W_UNC = torch.tensor([0.25 * PI[v] for (k, u, v) in lab], dtype=dt)                 # unconditional
W_AGN = torch.tensor([(k == 0) * 0.5 * PI[v] for (k, u, v) in lab], dtype=dt)       # "class 0", any view
W_VIEW = [torch.tensor([(k == 0 and vv == v) * 0.5 for (k, u, vv) in lab], dtype=dt)
          for v in range(4)]                                                         # "class 0, view v"

beta = torch.linspace(1e-4, 0.02, 1000, dtype=dt)          # DDPM linear schedule
AB = torch.cumprod(1 - beta, 0)

def comp_terms(x, t, w):
    a = AB[t - 1][:, None]
    var = a * S2 + 1 - a                                   # each component: N(sqrt(a) mu, var I)
    d2 = ((x[:, None, :] - a.sqrt()[:, :, None] * MU[None]) ** 2).sum(-1)
    w = w.expand(len(x), -1)
    L = torch.where(w > 0, torch.log(w.clamp_min(1e-300)) - d2 / (2 * var), torch.tensor(-torch.inf, dtype=dt))
    return L, a, var

def logp(x, t, w):
    L, a, var = comp_terms(x, t, w)
    return torch.logsumexp(L, 1) - D / 2 * torch.log(2 * torch.pi * var[:, 0])

def eps_hat(x, t, w):
    """Exact noise prediction of the mixture, -sqrt(1 - ab_t) * score."""
    L, a, var = comp_terms(x, t, w)
    score = -(x - a.sqrt() * (torch.softmax(L, 1) @ MU)) / var
    return -(1 - a).sqrt() * score

# ---- 1. data
print("1. data")
for v in range(4):
    img = MU[v]
    print(f"   class 0, {VIEWS[v]:5s} view: {(img > 0.9).sum().item():2d} bright and "
          f"{((img < -0.5) & (img > -0.99)).sum().item():2d} dark mark pixels")
dd = torch.cdist(MU, MU)
back_pairs = [(k * 8 + 2, k * 8 + 6) for k in range(2)]    # full vs faint back view, per class
mask = ~torch.eye(16, dtype=bool)
for i, j in back_pairs:
    mask[i, j] = mask[j, i] = False
print(f"   min distance between components, back-view variant pairs excluded: {dd[mask].min():.2f}"
      f" (pixel-noise norm {0.1 * 16:.1f})")
print(f"   back views, full vs faint: class 0 {dd[back_pairs[0]]:.2f}, class 1 {dd[back_pairs[1]]:.3f}")
print(f"   data mean |x| over the four class-0 views: full {MU[0:4].abs().mean():.3f}, faint {MU[4:8].abs().mean():.3f}")

# ---- 2. the SDS direction is the gradient of a KL objective (exact scores)
print("2. mean(eps^) vs (sigma/alpha) d/dx KL, 500 common noises")
g = torch.Generator().manual_seed(0)
x0 = (MU[0] + 0.3 * torch.randn(D, generator=g, dtype=dt))[None]
M, h = 500, 1e-5
for t in [100, 500, 900]:
    al, sg = AB[t - 1].sqrt(), (1 - AB[t - 1]).sqrt()
    E = torch.randn(M, D, generator=g, dtype=dt)
    tt = torch.full((M,), t)
    eh = eps_hat(al * x0 + sg * E, tt, W_UNC).mean(0)
    fd = torch.zeros(D, dtype=dt)
    for j in range(D):
        e = torch.zeros(D, dtype=dt); e[j] = h
        fd[j] = -(logp(al * (x0 + e) + sg * E, tt, W_UNC).mean()
                  - logp(al * (x0 - e) + sg * E, tt, W_UNC).mean()) / (2 * h)
    ref = (sg / al) * fd
    print(f"   t={t}: rel. error {(eh - ref).abs().max() / ref.abs().max():.1e}, "
          f"max |grad| {ref.abs().max():.3f}, max |mean eps| {E.mean(0).abs().max():.3f}")

# ---- 3. SDS with classifier-free guidance, on a voxel object (3D) or on four free images (2D)
def sds(rep, prior, s, seed, steps=1500, lr=0.03):
    gen = torch.Generator().manual_seed(seed)
    rho = 3 * (1 - (X ** 2 + Y ** 2 + Z ** 2).sqrt() / 5) + 0.1 * torch.randn(X.shape, generator=gen, dtype=dt)
    a0 = 0.1 * torch.randn(X.shape, generator=gen, dtype=dt)       # density blob, random albedo
    def views(rho, a0):
        sig, alb = torch.nn.functional.softplus(rho), torch.tanh(a0)
        return torch.stack([render(sig, alb, v).reshape(-1) for v in range(4)])
    if rep == "3D":
        params = [rho.requires_grad_(True), a0.requires_grad_(True)]
        images = lambda: views(rho, a0)
    else:                                                  # 2D: free pixels, same start
        p = torch.atanh(views(rho, a0).clamp(-0.999, 0.999)).requires_grad_(True)
        params, images = [p], lambda: torch.tanh(p)
    opt = torch.optim.Adam(params, lr=lr)
    wc = torch.stack(W_VIEW) if prior == "view" else W_AGN[None].expand(4, -1)
    for _ in range(steps):
        x = images()
        t = torch.randint(20, 981, (4,), generator=gen)    # one noise level per view
        e = torch.randn(4, D, generator=gen, dtype=dt)
        a = AB[t - 1][:, None]
        xt = (a.sqrt() * x + (1 - a).sqrt() * e).detach()
        eu, ec = eps_hat(xt, t, W_UNC), eps_hat(xt, t, wc)
        grad = eu + s * (ec - eu) - e                      # w(t) = 1; no Jacobian of the prior
        opt.zero_grad(); (grad * x).sum().backward(); opt.step()
    with torch.no_grad():
        return images()

def measures(x):
    nn = ((x[:, None, :] - MU[None]) ** 2).sum(-1).argmin(1).numpy()
    correct = sum(LAB[nn[v]][0] == 0 and LAB[nn[v]][2] == v for v in range(4))
    janus = sum(LAB[nn[v]][2] == 0 for v in range(1, 4))   # side or back view nearest a front image
    full = int(((x[0] - MU[0]) ** 2).sum() < ((x[0] - MU[4]) ** 2).sum())
    rms = min(((x - MU[4 * u:4 * u + 4]) ** 2).mean().sqrt().item() for u in range(2))
    return correct, janus, full, x.abs().mean().item(), rms

print("3. SDS, seeds 0-4 (correct of 20 views, janus of 15 non-front views, full of 5 seeds)")
print("   rep prior        s  correct janus full  mean|x|  rms")
for rep in ["2D", "3D"]:
    for prior in ["agnostic", "view"]:
        for s in [1, 7.5, 30, 100]:
            R = np.array([measures(sds(rep, prior, s, seed)) for seed in range(5)])
            print(f"   {rep}  {prior:8s} {s:5.1f}   {int(R[:, 0].sum()):2d}    {int(R[:, 1].sum()):2d}"
                  f"    {int(R[:, 2].sum())}    {R[:, 3].mean():.3f}   {R[:, 4].mean():.3f}")

Output:

1. data
   class 0, front view:  2 bright and 18 dark mark pixels
   class 0, right view: 12 bright and  5 dark mark pixels
   class 0, back  view:  2 bright and  2 dark mark pixels
   class 0, left  view:  0 bright and 21 dark mark pixels
   min distance between components, back-view variant pairs excluded: 0.56 (pixel-noise norm 1.6)
   back views, full vs faint: class 0 1.45, class 1 0.022
   data mean |x| over the four class-0 views: full 0.780, faint 0.732
2. mean(eps^) vs (sigma/alpha) d/dx KL, 500 common noises
   t=100: rel. error 4.6e-10, max |grad| 2.580, max |mean eps| 0.169
   t=500: rel. error 8.2e-08, max |grad| 0.322, max |mean eps| 0.122
   t=900: rel. error 4.0e-06, max |grad| 0.136, max |mean eps| 0.123
3. SDS, seeds 0-4 (correct of 20 views, janus of 15 non-front views, full of 5 seeds)
   rep prior        s  correct janus full  mean|x|  rms
   2D  agnostic   1.0    5    15    0    0.675   0.309
   2D  agnostic   7.5    5    15    3    0.715   0.368
   2D  agnostic  30.0    5    15    5    0.768   0.471
   2D  agnostic 100.0    5    15    5    0.773   0.484
   2D  view       1.0   20     0    0    0.727   0.051
   2D  view       7.5   20     0    5    0.794   0.079
   2D  view      30.0   20     0    5    0.834   0.196
   2D  view     100.0   20     0    5    0.853   0.252
   3D  agnostic   1.0    5    15    0    0.679   0.308
   3D  agnostic   7.5    5    15    2    0.707   0.346
   3D  agnostic  30.0    5    15    5    0.773   0.471
   3D  agnostic 100.0    5    15    5    0.777   0.482
   3D  view       1.0   20     0    1    0.734   0.057
   3D  view       7.5   20     0    5    0.799   0.076
   3D  view      30.0   20     0    5    0.839   0.196
   3D  view     100.0   20     0    5    0.859   0.253

The data lines confirm the labels: the right view shows the bright ear spot, the left view the dark stripe, the front view the eyes and mouth. Components are separated by at least 0.56 in pixel distance; the two variants’ back views are close, since backs are plain (the class-0 pair is farther apart; in our reading, because the side marks are defined through the whole object and reach the silhouette’s rim). The second block checks the identity derived above on an exact prior: the average predicted noise matches the scaled gradient of the KL objective with a relative error of at most $4 \times 10^{-6}$ at every level. The size of the mean of $\epsilon$, up to 0.17 against gradients of 0.14 to 2.6, shows why the identity holds only in expectation: over 500 draws the subtracted noise is still comparable with the gradient at high noise levels, and a single SDS step sees one draw.

The table separates the two causes the article discusses. With the view-conditioned prior every view lands on its own view’s image, at every guidance scale, in 2D and in 3D. With the view-agnostic prior all 15 non-front views end nearest a front image, at every scale: the object shows its front on all four sides, the Janus problem, and its renderings sit far from any valid object (RMS about 0.31 against 0.057). The 2D rows show the same Janus count, so the failure is in the objective, not in the 3D optimization: with no view input, each rendering faces the same front-heavy mixture and seeks its strongest mode. In our reading, the 3D object here does little to resist it because the four axis-aligned views see largely different surfaces, and only the silhouettes are shared; in this example, then, consistency comes from the prior’s view conditioning, not from the shared geometry.

Guidance pushes renderings past the data. As $s$ rises from 1 to 100 with the view-conditioned prior, the mean absolute pixel value climbs from 0.734 to 0.859, beyond the 0.780 of the full-contrast object itself, and the RMS distance to a valid object grows from 0.057 to 0.253: guidance favors renderings that the unconditional mixture finds improbable, as derived above, and saturates them, the over-saturation that SV3D’s authors list among the artifacts of SDS. The variant column shows guidance deciding which variant SDS picks: from these near-identical starts, almost every seed picks the faint variant without guidance (the one exception is a 3D run with the view-conditioned prior) and every seed picks the full-contrast one at $s \ge 30$ (and at 7.5 with the view-conditioned prior). It does not show a loss of diversity from a diverse start, since the seeds differ only by small noise around the same initial blob; in our reading, that the seeds reach the same view and, at most scales, the same variant follows from the objective’s mode seeking. The Classifier-Free Guidance article measures guidance in sampling, where it also reduces variety; here it acts on an optimization.

Datasets and Benchmarks

There is no shared benchmark across the variants; each family is evaluated on its own data.

Data	Content	Used by
FFHQ	Face photographs, with camera poses from off-the-shelf estimators	EG3D training and FID
AFHQv2 Cats	A small set of cat face photographs	EG3D, with transfer learning from FFHQ (§5)
DreamFusion’s prompts	397 text prompts from DreamFusion’s website	Magic3D’s user study (§5)
Objaverse	800K+ artist-made 3D models (Zero-1-to-3 §3.4)	Training of Zero-1-to-3 and SV3D
Google Scanned Objects (GSO)	High-quality scans of household items (Zero-1-to-3 §4.3)	Evaluation; SV3D filters 300 objects to reduce near-duplicates (§3.1)
RTMV	Complex scenes, each composed of 20 random objects	Zero-1-to-3’s out-of-distribution evaluation
OmniObject3D	3D objects	SV3D’s view-synthesis evaluation

In the papers read here, image-to-3D benchmarks render held-out 3D models so that both images and geometry have references, while text-to-3D has no reference and relies on human judgment.

Evaluation Metrics

Image quality of 3D-aware GANs. EG3D reports the Fréchet Inception Distance between 50,000 generated images and all available real images, together with three consistency measures: identity consistency (mean ArcFace cosine similarity between two views of the same face), depth error and pose error against pseudo-ground truth estimated from the generated images (§5.1). On FFHQ at $256^2$ it reached FID 4.8, identity 0.76, depth 0.31 and pose 0.005, against FID 29.8 to 31.5 for GIRAFFE and Lifting StyleGAN at $256^2$ and π-GAN at $128^2$, and FID 4.7 at $512^2$, near the 2.97 of StyleGAN2 at $512^2$ (Table 2).

Human preference for text-to-3D. Magic3D’s raters on Amazon MTurk saw videos of two models side by side for each of the 397 prompts, three raters per prompt (1,191 comparisons); 61.7% preferred Magic3D over DreamFusion for realism and detail together (58.3% for realism, 66.0% for detail), and 87.7% preferred the fine models over the coarse ones (Table 1).

Views against held-out renderings. For image-to-3D, generated views or renderings of the generated model are compared with renderings of the true object using PSNR, SSIM, LPIPS, MSE and a CLIP score; SV3D notes that all these metrics evaluate multi-view consistency (§3.1). The Novel View Synthesis article discusses why pixel metrics penalize plausible invented content.

Geometry against held-out models. Chamfer distance and volumetric IoU between the generated and true meshes. On GSO, Zero-1-to-3 reached Chamfer distance 0.0717 and IoU 0.5052, against 0.0804 and 0.2944 for Point-E and 0.1230 and 0.2343 for MCC (Table 4). On 50 GSO objects, SV3Dp with masked SDS reached Chamfer distance 0.024 and 3D IoU 0.614, against 0.039 and 0.550 for Stable Zero123 and 0.021 and 0.689 for meshes fitted to ground-truth renderings, an upper reference (Table 6). The two tables use different object sets and protocols and are not comparable with each other.

User studies for image-to-3D. On 22 real-world images, 30 users preferred SV3D’s orbital videos over those of Zero123XL, Stable Zero123, EscherNet and Free3D 96%, 99%, 96% and 98% of the time (SV3D §3.1).

Applications

Magic3D’s authors list gaming, entertainment, architecture and robotics simulation as uses of 3D content, which they argue text-based creation could make accessible to novices and faster for artists (§1); SV3D adds AR and VR and e-commerce (§1). Generated meshes can be imported into standard graphics software (Magic3D §5). EG3D’s 3D-aware face model supports single-view 3D reconstruction and style mixing, and its authors note that these applications could be misused to generate edited imagery of real people (§5.3, §6). Related tasks with their own articles are text-to-image generation, whose models serve as priors here, video generation, whose models SV3D repurposes, and 3D reconstruction from real captures.

Open Problems

• Consistency without a view-aware prior. Score distillation from text-to-image models remains prone to the Janus problem and over-saturation (SV3D §2.2); view-conditioned and multi-view models address it by training on renderings of 3D models, which, in our reading, ties them to the coverage of Objaverse-like collections.
• Diversity. Score distillation seeks a mode, and the high guidance it uses trades diversity for fidelity (Zero-1-to-3 §3.3); in the example above, seeds started near the same blob reach the same variant at most guidance scales.
• Speed. Per-prompt optimization took 1.5 hours for DreamFusion (as reported by Magic3D) and 40 minutes for Magic3D on 8 A100s (§5); SV3D’s mesh extraction takes about 8 minutes without SDS and 20 with it (§4.2). Feed-forward models are fast but, in SV3D’s comparison, often less complete or detailed (§4.2).
• Lighting and materials. Generated textures tend to bake in illumination; SV3D fits a simple illumination model to reduce it, and lists mirror-like reflective surfaces as a remaining failure (§4, Limitations).
• Cameras and scenes. SV3D handles two degrees of freedom of the camera, elevation and azimuth (Limitations); Zero-1-to-3’s quality drops on cluttered scenes (§5.1); and EG3D needs the camera pose distribution of its data and cannot resolve ambiguities that geometry can explain, such as concave eye sockets that seem to follow the camera (§6).
• Evaluation. In the papers read here, text-to-3D results are compared by user studies and qualitative figures, and image-to-3D results against single held-out objects; in our reading, there is no common, automatic measure of plausibility and consistency for generated 3D content.