Differentiable Rendering

Differentiable rendering is rendering that returns, along with each pixel value, its derivatives with respect to the parameters of the scene: vertex positions, camera pose and intrinsics, materials, textures and lights. With those derivatives, an image loss can be backpropagated into the scene, and the scene can be fitted to photographs by gradient descent, or a network that predicts the scene can be trained from images alone. The idea is old under the names analysis by synthesis and inverse graphics; what made it a technique of its own were general-purpose renderers that supply the derivatives automatically. Loper and Black (2014) define a differentiable renderer as a process that supplies pixels as a function of model parameters, simulating a physical imaging system, and supplies the derivatives of those pixels with respect to the parameters.

The obstacle is visibility. A standard rasterizer decides, for every pixel, whether a triangle covers it and which triangle is in front, and both decisions are discrete. This article explains why that breaks gradients, and surveys the four answers: renderers that keep the hard forward pass and approximate the backward pass (OpenDR, the Neural Mesh Renderer), renderers that make the forward pass itself smooth (Soft Rasterizer), physically based renderers that differentiate the visibility discontinuities exactly in expectation (edge sampling, Mitsuba 3), and volume rendering, which is smooth by construction. Neural rendering places differentiable rendering among the learned rendering methods and shows, in its example, the zero gradient of a hard rasterizer next to a soft one; this article goes into the mechanisms.

Purpose

Inverse graphics poses vision as a search for the parameters $\Theta$ of a scene model whose rendering matches the observed image $I$. Loper and Black (2014) write the simplest objective as

$$E(\Theta) = \lVert f(\Theta) - I \rVert^2,$$

with $f$ the forward rendering function, and note that real objectives add robust penalties and priors. The scene may have thousands of parameters, so the objective is minimized with gradients, and the gradient needs the Jacobian of the renderer:

$$\frac{\partial E}{\partial \Theta} = 2\,\big(f(\Theta) - I\big)^{\top} \frac{\partial f}{\partial \Theta}.$$

Before general differentiable renderers, each application derived $\partial f / \partial \Theta$ by hand for its own model. Loper and Black list examples, among them Blanz and Vetter’s morphable face model, fitted by analysis by synthesis with explicitly computed derivatives, and de La Gorce et al.’s hand tracker, which paid special attention to derivatives at object boundaries. Their point is that differentiating the renderer once, instead of each objective, makes every problem that can be posed as rendering approximately differentiable by construction.

The same Jacobian serves learning. When a network predicts a mesh from one image and a differentiable renderer turns the mesh into a silhouette or color image, a 2D loss trains the network without 3D supervision; the Neural Mesh Renderer (Kato et al., 2018) and Soft Rasterizer (Liu et al., 2019) both demonstrate single-view mesh reconstruction this way.

When to Use It

Differentiable rendering fits problems in which a forward image model exists, the unknowns are continuous, and a reasonable starting point is available:

• Pose and shape fitting of a known or parametric model to images, such as rigid object pose, body pose and shape, or face shape and albedo.
• Training 3D predictors from 2D supervision, where the renderer closes the loop between a predicted mesh, point set or field and the training images.
• Inverse rendering of appearance: camera, material and light parameters recovered from photographs, which needs a physically based renderer when shadows and interreflections matter (Li et al., 2018).
• Per-scene fitting of radiance fields and Gaussians for novel view synthesis, where the renderer is volumetric.

It is a local method. Loper and Black (2014) describe OpenDR as addressing deterministic refinement, not stochastic search, and argue that a full solution combines sampling to get close with gradient-based refinement to finish. They also state its limitations: with differences of Gaussian pyramids, a simulated object that does not overlap the observed one records no gradient towards it, and, more fundamentally, a real feature does not pull on simulated features that are entirely occluded in the current rendering. Discrete changes are outside its reach as well; Liu et al. (2019) note that Soft Rasterizer, in its published form, handles neither shadows nor changes of topology.

How It Works

Why rasterization is not differentiable

A rasterizer projects each vertex with the pinhole camera model, then fills each triangle with the pixels whose centers it covers, keeping the nearest triangle per pixel in a depth buffer (polygon mesh). Projection is a smooth function of the vertices and the camera, and so is shading. The trouble is the sampling. Liu et al. (2019) identify rasterization and z-buffering as the two operations that are not differentiable with respect to the projected coordinates and depths, because they are discrete sampling steps.

Kato et al. (2018, §3.2) make this concrete for one vertex coordinate $x_i$ and one pixel $P_j$ outside the triangle. As $x_i$ moves right, the pixel keeps its color until the triangle’s edge crosses the pixel center at $x_1$, and then jumps to the triangle’s color. The pixel value $I_j(x_i)$ is a step function: its derivative is zero almost everywhere and undefined at the jump, so the information that moving the vertex by $x_1 - x_0$ would change the pixel never reaches the vertex.

Li et al. (2018) state the same problem in its exact form. A pixel’s value is an integral over the pixel filter $k$ of the incoming radiance $L$,

$$I = \iint k(x, y)\, L(x, y)\, dx\, dy,$$

and this integral is usually a differentiable function of the scene once radiance has been integrated over the filter. The integrand is not: each triangle edge splits the image plane into two half-spaces, and the integrand switches between them through a Heaviside step $\theta(\alpha(x, y))$, where $\alpha$ is the edge’s line equation. Differentiating the step produces a Dirac delta on the edge. Sampling the integrand at random points, or at the pixel center as a rasterizer does, has zero probability of landing on that delta, so automatic differentiation of the sampled image misses the change in covered area entirely (their Fig. 2). It follows, in our reading, that the two families below differ in what they repair: approximate and soft rasterizers change the function being differentiated, while edge sampling estimates the derivative of the true pixel integral.

Approximate gradients: OpenDR

OpenDR (Loper and Black, 2014) renders with OpenGL and approximates only the derivatives. Its forward model has three parts: vertex positions $V$, camera parameters $C$ and per-vertex brightness $A$. It approximates continuous pixel intensities by their sampled central value, uses OpenCV’s pinhole-plus-distortion camera, and does not support global illumination. Derivatives follow the chain rule through the projected 2D vertex coordinates $U$:

$$\frac{\partial f}{\partial V} = \frac{\partial f}{\partial U}\, \frac{\partial U}{\partial V}, \qquad \frac{\partial f}{\partial C} = \frac{\partial f}{\partial U}\, \frac{\partial U}{\partial C}.$$

The projection Jacobians are exact, and OpenCV provides them. The approximation lies in $\partial f / \partial U$, how pixels change when the image-space geometry moves. OpenDR first labels pixels with an extra rendering pass: occlusion-boundary pixels lie on edges that pass the depth test and join a triangle facing the camera to one facing away (§4.3).

• Interior pixels use a first-order Taylor expansion in image space: a patch translating right by one pixel replaces each pixel by its left neighbor, so the derivative is approximated by horizontal and vertical filtering with the kernel $\tfrac{1}{2}[-1, 0, 1]$, switched to one-sided kernels next to a boundary so that the filter never crosses it.
• Boundary pixels use the centered kernel across the boundary. This approximates the difference between the foreground and the surface behind it by the difference between the foreground and a pixel next to the hidden surface; the authors argue that the boundary gradient usually dominates and sets the direction of the optimization.

Derivatives cost three render passes beyond the image itself (boundary identification, triangle identification and barycentric coordinates), each read back from the GPU. The renderer is built on Chumpy, an automatic differentiation package the authors wrote around the NumPy API, so the objective around the renderer, such as a Gaussian pyramid of the residual, is differentiated automatically.

Approximate gradients: the Neural Mesh Renderer

The Neural Mesh Renderer (NMR; Kato et al., 2018) also keeps the standard rasterized image in the forward pass, because a blurred forward pass would let a face’s color leak outside the face (§3.2). In the backward pass, it replaces the step of $I_j(x_i)$ between the current position $x_0$ and the position $x_1$ where the edge reaches the pixel center by a linear ramp, so the derivative becomes $\delta^I_j / \delta^x_i$ with $\delta^x_i = x_1 - x_0$ and $\delta^I_j = I(x_1) - I(x_0)$. Because the ramp differs on either side of $x_0$, the error signal $\delta^P_j$ backpropagated to the pixel decides which side counts (their Eq. 1):

$$\left.\frac{\partial I_j(x_i)}{\partial x_i}\right|_{x_i = x_0} = \begin{cases} \delta^I_j / \delta^x_i, & \delta^P_j\, \delta^I_j < 0, \\ 0, & \delta^P_j\, \delta^I_j \ge 0. \end{cases}$$

The rule passes a gradient only when moving the vertex would change the pixel in the direction the loss wants. For a pixel inside the face, the derivatives from both sides are summed (Eqs. 2–4); with several faces, only the frontmost is drawn, and no gradient flows when the crossing point is occluded by a surface that does not include the vertex (§3.3). Textures are sampled bilinearly from per-face texture images, and ambient and directional lights are differentiable parameters (§3.4–3.5).

Liu et al. (2019, §3.4) contrast both renderers with their own: because OpenDR and NMR rasterize normally in the forward pass, they cannot send gradients to triangles that are occluded in the final image, and because their gradients act on the image plane, they cannot optimize triangle depths. A vertex receives OpenDR gradients only from pixels near it, because of the local filtering, and NMR defines no gradient from some pixels inside the face. Liu et al. also argue that the mismatch between a hard forward pass and a hand-crafted backward pass can make optimization behave in uncontrolled ways.

Soft rasterization

Soft Rasterizer (SoftRas; Liu et al., 2019) makes the forward pass itself smooth. It treats rasterization as a binary mask decided by where a pixel lies relative to a triangle, and z-buffering as a one-hot merge by depth, and replaces each with a soft version.

Probability maps. The influence of triangle $f_j$ on pixel $p_i$ is (their Eq. 1)

$$D^i_j = \operatorname{sigmoid}\!\left( \delta^i_j \, \frac{d^2(i, j)}{\sigma} \right),$$

where $d(i, j)$ is the closest distance from $p_i$ to the triangle’s edges, $\delta^i_j = +1$ inside the triangle and $-1$ outside, and $\sigma > 0$ sets the sharpness; the paper uses $\sigma = 10^{-4}$ unless stated otherwise. Inside pixels map to $(0.5, 1)$ and outside pixels to $(0, 0.5)$, with the boundary at 0.5. As $\sigma \to 0$ the map converges to the exact triangle, so ordinary rasterization is the limiting case. The authors use the Euclidean distance and note that barycentric or $L^1$ distances also work.

Aggregation. Colors are fused by a softmax-like aggregate function $A_S$ over the per-triangle color maps $C_j$, interpolated from vertex colors with barycentric coordinates clipped to $[0, 1]$ (their Eqs. 2–3):

$$I^i = \sum_j w^i_j C^i_j + w^i_b C_b, \qquad w^i_j = \frac{D^i_j \exp(z^i_j / \gamma)}{\sum_k D^i_k \exp(z^i_k / \gamma) + \exp(\epsilon / \gamma)},$$

where $z^i_j$ is the normalized negative depth of the point on triangle $j$ that projects to $p_i$ (larger means nearer), $C_b$ is the background color, $\epsilon$ is a small constant that admits the background, and $\gamma$ (also $10^{-4}$ by default) sets the sharpness. As $\gamma \to 0$, only the nearest triangle contributes, which is z-buffering. For silhouettes, which depend on neither color nor depth, a separate aggregate gives the probability that at least one triangle covers the pixel (their Eq. 4):

$$I^i_s = A_O(\{D_j\}) = 1 - \prod_j \big(1 - D^i_j\big).$$

Because every triangle contributes to every pixel through $D_j$ and $w_j$, gradients reach occluded triangles, triangle depths, and vertices far from the pixel (their Fig. 5). Liu et al. present the two smoothness parameters as a control over the loss landscape: larger $\sigma$ and $\gamma$ give smoother, partially transparent renderings and a smoother energy (their Fig. 10). The price, in our reasoning, is that a soft rendering is not the image a camera would record: a blurred silhouette compared with a sharp target need not be best matched at exactly the true geometry (near corners, or where several triangles overlap in Eq. 4, the soft silhouette at the true pose is not a symmetric blur of the target), which would be one reason to end an optimization at small $\sigma$ and $\gamma$. The authors’ schedule decays both (see Evidence and Impact); they motivate softness by the smoother loss landscape.

PyTorch3D’s mesh renderer implements the same blending with two engineering changes, visible in its code at release v0.7.9. The rasterizer keeps, per pixel, only the faces_per_pixel nearest faces among those that cover the pixel or lie within blur_radius of it (a squared distance in normalized device coordinates), and returns for each a signed squared distance; softmax_rgb_blend then computes $\operatorname{sigmoid}(-d/\sigma)$ over those $K$ faces, a softmax over the normalized reversed depth $(z_{\text{far}} - z)/(z_{\text{far}} - z_{\text{near}})$ (named z_inv in the code; larger means nearer, as in Eq. 3) with temperature $\gamma$, and the silhouette product of Eq. 4, with the same defaults $\sigma = \gamma = 10^{-4}$. Limiting the sum to $K$ faces bounds memory, which the documentation puts at 48 bytes per face per pixel for the rasterizer’s outputs and their gradients; in our reading, faces beyond the $K$ nearest then receive no gradient from that pixel. The PyTorch3D documentation credits Kato et al. and Liu et al. as the inspiration for the design, and SoftRas as the basis of its core rasterization step.

Physically based differentiable rendering

Rasterization handles only primary visibility, the first surface seen through each pixel. Li et al. (2018) point out that OpenDR and NMR approximate the primary-visibility gradients and assume Lambertian materials without shadows or global illumination. Their differentiable Monte Carlo ray tracer computes derivatives of a path-traced image with respect to camera pose, geometry, materials and lights, including the effects of shadows and interreflection.

Edge sampling. The scene function $f(x, y) = k(x, y) L(x, y)$ is written as a sum of terms $\theta(\alpha_i(x, y)) f_i(x, y)$, one per edge half-space. The product rule splits the gradient of the pixel integral into two parts (their Eq. 6):

$$\nabla \iint \theta(\alpha_i) f_i \, dx\, dy = \iint \delta(\alpha_i)\, \nabla\alpha_i\, f_i \, dx\, dy + \iint \nabla f_i\; \theta(\alpha_i)\, dx\, dy.$$

The second term is the ordinary integral with the smooth integrand replaced by its gradient, estimated by area sampling with automatic differentiation. The first term contains the Dirac delta; a change of variables turns it into an integral along the edge, where $\alpha_i = 0$ (Eq. 7). It is estimated by sampling points on edges and evaluating the scene function on both sides of each sampled point (Eq. 10):

$$\frac{1}{N} \sum_{j=1}^{N} \frac{\lVert E \rVert\, \nabla\alpha_i(x_j, y_j)\, \big(f_u(x_j, y_j) - f_l(x_j, y_j)\big)}{P(E)\, \lVert \nabla_{x_j, y_j}\alpha_i(x_j, y_j) \rVert},$$

with $\lVert E \rVert$ the edge’s length and $P(E)$ the probability of choosing it. With smooth shading, only silhouette edges have $f_u \ne f_l$; the renderer projects the meshes, clips them to the view frustum, picks a silhouette edge with probability proportional to its screen-space length and a point uniformly along it. Occlusion is handled for free: a sample on a hidden edge lands where the integrand is continuous, so both sides agree and it contributes zero (their Fig. 3b).

Shadows and indirect light are handled by the same construction in 3D. For a shading point $p$ and a mesh edge $(v_0, v_1)$, the edge equation becomes $\alpha(p, m) = (m - p) \cdot (v_0 - p) \times (v_1 - p)$ for points $m$ on the scene surfaces (Eq. 13), and the edge integral gains a correction for the angle between the edge plane and the surface it meets (Eq. 14). The authors’ errata page corrects the derivatives of $\alpha$ with respect to the edge endpoints $v_0$, $v_1$ and the shading point $p$ given in Eq. 16 of the paper; the corrected forms follow from differentiating Eq. 13, for example $\nabla_{v_0}\alpha = (v_1 - p) \times (m - p)$. Secondary edges cannot be pruned with one viewpoint, so the paper builds spatial hierarchies to importance-sample them (§4).

The authors compared their derivatives with central finite differences on low-resolution synthetic scenes and found agreement within 1% relative to the $L^1$ norm of the gradients, with finite differences needing two to three orders of magnitude more samples for the same error (their Fig. 7). They report an overhead of roughly 10–20 times the cost of rendering an image alone, and, for their CPU implementation, seconds to minutes for a 256 × 256 image at a few samples per pixel. They assume static scenes without participating media, meshes without interpenetration, and differentiable shaders (§5.5). The renderer was integrated with PyTorch, and the project page links its code, redner.

Mitsuba 3. Mitsuba 3 is a research renderer for forward and inverse light transport simulation, developed at EPFL, built on the just-in-time compiler Dr.Jit, and described by its README as able to compute derivatives of the entire simulation with respect to camera pose, geometry, BSDFs, textures and volumes. Differentiable rendering runs in the LLVM (CPU) and CUDA (GPU) variants, such as llvm_ad_rgb and cuda_ad_rgb. Its documentation at v3.9.1 shows how the visibility problem appears in practice. The prb integrator, Path Replay Backpropagation, has no projective sampling, and its documentation warns that it returns incorrect or biased gradients for geometric parameters such as vertex positions, so it cannot be used for shape optimization. The prb_projective and direct_projective integrators add projective sampling, which handles discontinuous visibility changes such as a moving shape’s gradients; their parameters set separate sample counts for the continuous derivatives, for primary-visibility discontinuities and for indirect-visibility discontinuities, with a guiding structure for the indirect ones. The documentation cites Zhang et al. (2023) for projective sampling, and Vicini et al. (2021) and Zeltner et al. (2021) for Path Replay Backpropagation and attached and detached sampling; those papers were not read for this article. Its inverse rendering tutorials include gradient-based optimization, caustics, object pose estimation, volume and shape optimization, and radiance field reconstruction.

Volume rendering as a differentiable renderer

A density field avoids the hard visibility decision. In volume rendering, a ray has some probability of interacting with the medium at every sample, and the pixel is the compositing sum $\hat C = \sum_i T_i \alpha_i \mathbf{c}_i$, a smooth function of every sample’s density and color; that article derives its gradients. In our reading, occlusion there is a transmittance factor rather than a test, so a sample behind another still receives a gradient, scaled by how much light reaches it, which is the property SoftRas built into its aggregate function. Neural radiance fields fit a network to photographs through this renderer, and 3D Gaussian splatting rasterizes semi-transparent Gaussians with a differentiable front-to-back blend whose backward pass lets every splat in a pixel receive a gradient. In our reading, both sidestep the problem that rasterizing opaque triangles creates, rather than solve it: their geometry is a soft field or a set of translucent blobs, and extracting a sharp surface is a further step (neural implicit representations describes surface renderers such as DVR, which differentiates the ray–surface intersection depth implicitly, and the NeuS authors’ argument that surface rendering gives gradients only at one point per ray).

How to Apply It

The workflow is the same whichever renderer is used:

1. Parameterize the scene with as few continuous unknowns as the problem allows: a rigid pose, the coefficients of a shape model, or vertex offsets from a template. NMR deforms a sphere of 642 vertices by per-vertex and global offsets, each vertex restricted to its original quadrant (§4.1); SoftRas’s shape fitting optimizes pose, translation and non-rigid deformation parameters (their Eq. 6). For rotations, an update of the form $R = \exp([\boldsymbol{\omega}]_\times) R_0$ keeps the estimate a rotation (rotation representations).
2. Choose what to compare. Silhouettes need only geometry and camera. NMR’s silhouette loss is a negative intersection over union between the rendered and target masks (Eq. 5), and SoftRas uses the same quantity as one minus IoU; color losses are $L^1$ or $L^2$ per pixel. OpenDR compares Gaussian pyramids of the residual and notes that any differentiable function of the image, such as edges or moments, can be compared instead.
3. Regularize. Image losses constrain only what is visible from the training views. NMR adds a smoothness term $\sum (\cos\theta_i + 1)^2$ over the dihedral angles of mesh edges (Eq. 6); SoftRas regularizes the Laplacian of shape and color, and reports that it helps, where NMR had reported a negative effect of a geometry regularizer on its scores.
4. Initialize and widen the basin. Gradients act only where rendered and observed features overlap (Loper and Black, 2014). OpenDR’s Kinect body fits converged from initial positions up to a meter away with Gaussian pyramids and background images. SoftRas decays $\sigma$ and $\gamma$ in five steps, and its baselines use five-level multi-resolution images (§5.2).
5. Check the gradient. Before trusting an optimization, compare the renderer’s derivatives with central finite differences on a small scene, as Li et al. did. Loper and Black warn that the right finite-difference step depends on the pixel: one epsilon can be too small for some pixels and too large for others (their Fig. 7).
6. Optimize with a first-order method, such as Adam, with a learning rate small enough that a single step does not move edges by more than a few pixels; this last point is our own rule of thumb.

Variants

Family	Forward pass	Gradient at visibility edges	Examples
Approximate rasterizers	Standard rasterization	Hand-designed: image-space filtering, or linear ramps gated by the error sign	OpenDR (2014), Neural Mesh Renderer (2018)
Soft rasterizers	Probabilistic coverage and depth aggregation	Exact gradient of a smoothed image	Soft Rasterizer (2019), PyTorch3D’s soft shaders
Physically based	Monte Carlo path tracing	Monte Carlo estimate of the true pixel integral’s gradient, by sampling edges (unbiased, Li et al.) or by projective sampling	Edge sampling and redner (2018), Mitsuba 3 (prb_projective)
Volumetric and splatting	Compositing of densities or translucent primitives	Smooth by construction	NeRF, 3D Gaussian splatting

The reference list of PyTorch3D’s rendering overview also names Differentiable Surface Splatting for point-based geometry (Yifan et al., 2019), the reparameterization of discontinuous integrands of Loubet et al. (2019), and the interpolation-based renderer of Chen et al. (2019); they are named here only as further variants.

Best Practices

• Match the renderer to the effects in the data. Silhouettes and diffuse shading are within reach of rasterizers; shadows, glossy reflections and interreflections need a physically based renderer, since Li et al. (2018) note that the rasterizing renderers ignore secondary effects.
• Use an integrator that handles visibility when geometry is optimized. Mitsuba 3’s documentation marks its non-projective prb integrator as giving incorrect or biased gradients for vertex positions.
• Anneal softness. Start soft enough for gradients to reach distant or occluded geometry, and decay $\sigma$ and $\gamma$ during the optimization, as Liu et al. (2019, §5.2) do; finishing sharp enough that the rendering is the image a camera would record is our reasoning for where the schedule should end.
• Combine with search. For large pose differences, a coarse search over candidates followed by gradient refinement follows Loper and Black’s division of labour between sampling and refinement.
• Keep forward and backward consistent when writing a renderer. Liu et al. (2019) argue that the mismatch may lead to uncontrolled optimization behaviour; in our reasoning, gradients of the function actually rendered are also easier to check than gradients of a different, hand-designed function.

Pitfalls

• Zero gradients from a hard renderer. Differentiating a point-sampled rasterized image with automatic differentiation gives zero with respect to geometry almost everywhere, as the step function of Kato et al. shows; a pipeline can train its colors and lights and silently never move its vertices.
• No gradient to hidden parts. With a hard forward pass, triangles occluded in the current rendering receive no gradient (Liu et al., 2019, §3.4); a body part hidden behind the torso cannot be pulled into place. SoftRas shows this on an SMPL body fit, in which NMR fails to move a hand that is fully occluded in the initial view (3D human pose estimation describes the SMPL body model).
• Local minima. Rotations and occlusions make image losses non-convex. In SoftRas’s cube experiment, NMR stays in a local minimum that a smoother rendering avoids (their Fig. 10). The NeRF authors judged mesh optimization through differentiable rasterizers or path tracers often difficult, likely because of local minima or poor conditioning (NeRF paper).
• Soft-renderer bias. In our reasoning, a rendering blurred by large $\sigma$ or $\gamma$ is a different image, so the best fit of a soft rendering to a sharp target need not sit at the true parameters; residual softness at the end of an optimization leaves this bias in the result.
• Truncation in implementations. Limiting soft blending to the $K$ nearest faces, as PyTorch3D does for memory, cuts the gradient from faces further back, which in our reading reintroduces part of the occlusion problem when $K$ is small.
• Finite differences as a substitute. Finite differences need one or two renderings per parameter, and the right step varies across pixels. Loper and Black measured forward differencing on 6 pose parameters as 1.75 times faster than their analytic derivatives at 640 × 480, but about 80 times slower when derivatives with respect to all 514 vertices were needed.

Evidence and Impact

Single-view mesh reconstruction without 3D supervision. NMR was evaluated on nearly the same data as the voxel-based method of Yan et al.: objects of 13 ShapeNet categories rendered from 24 azimuths at 64 × 64, with Yan et al.’s train, validation and test sets, and scored by the IoU of voxelized predicted and ground-truth meshes at $32^3$. SoftRas used the dataset provided by NMR with the same split and reports 3D IoU. NMR’s Table 1 gives a mean IoU of 0.6016 for its mesh-based method, against 0.5736 for the voxel-based method of Yan et al. and 0.4766 for retrieval, and the mesh method was better in 10 of the 13 categories. SoftRas’s Table 1, using the same mesh generator and training settings as NMR, reports a mean of 0.6234 trained with silhouettes only and 0.6464 with silhouette and color losses, and lists NMR at 0.6015, which differs in the last digit from NMR’s own table. The SoftRas text says its best setting beat NMR by more than 4.5 points; the table difference is 4.49 points. Its ablation (Table 2) gives 60.8 mean IoU (%) with barycentric distances and 62.0 with Euclidean ones, 62.4 after adding the Laplacian term, and 64.6 with the color aggregate, while a learned aggregate function reached 63.2 in place of $A_O$.

Rigid pose fitting. SoftRas’s Table 3 reports the mean relative angular error of cube rotation estimates over 100 randomly sampled initializations and targets: 82.80° for SoftRas against 93.40° for NMR and 95.02° for Li et al. without scheduling, and 63.57° against 80.94° and 78.56° with scheduling (decaying $\sigma$ and $\gamma$ for SoftRas, five-level multi-resolution images for the others), with 126.48° for a random guess. In our reading, even the best mean error is far from a solved pose, which shows how hard rotation from one colored, self-occluding image remains for gradient descent.

Body shape from depth and color. Loper and Black (2014) fitted a BlendSCAPE body model, with per-vertex albedo, spherical-harmonic lighting and the translation between the Kinect’s color and depth cameras, to the depth and color images of two views and a background capture per subject, estimating thousands of parameters against over a million residuals. They predicted body measurements from the fitted shapes of 23 subjects and, as an indication of accuracy, compared the errors with predictions from 3,803 meshes registered to CAESAR laser scans, noting that the two settings differ in subjects and method; each fit took around 7 minutes.

Physically based inverse rendering. Li et al. (2018) fitted camera pose, materials and light intensity of a manually recreated scene to a real photograph taken with a fisheye camera, with strong indirect illumination and non-Lambertian materials, and used the gradients to find scene configurations that fool image classifiers, a use they caution is not a robust attack, since the rendered scene’s statistics differ from real images.

The technique’s broader effect is that rendering became a layer. NMR and SoftRas made meshes trainable from images; libraries such as PyTorch3D and Mitsuba 3 made differentiable rasterization and path tracing available inside automatic differentiation frameworks; and the radiance-field methods of neural rendering rest on a renderer that is differentiable by design.

Historical Development

Year	Work	Contribution
2014	OpenDR (Loper and Black, ECCV)	General differentiable renderer on OpenGL with automatic differentiation of the objective; approximate image-space derivatives
2018	Neural 3D Mesh Renderer (Kato et al., CVPR)	Approximate rasterization gradient designed for training networks; single-image mesh reconstruction from silhouettes
2018	Edge sampling (Li et al., SIGGRAPH Asia)	Unbiased gradients of path-traced images, including shadows and interreflections, by sampling silhouette edges
2019	Soft Rasterizer (Liu et al., ICCV)	Smooth forward rasterization by probability maps and depth aggregation; gradients to occluded and distant triangles
2020	NeRF (Mildenhall et al., ECCV)	Scene fitted to photographs through differentiable volume rendering
2022	Mitsuba 3	Retargetable research renderer with differentiable variants on Dr.Jit
2023	3D Gaussian splatting (Kerbl et al., ACM TOG)	Differentiable rasterization of translucent Gaussians in real time

Loper and Black (2014) trace inverse graphics to the early days of the field, through Grenander’s analysis by synthesis and physics-based and regularization approaches, and claim no novelty for the idea itself, only for making it practical. They presented OpenDR as the first publicly available framework for differentiating the image generation process.