Neural Implicit Representations

A neural implicit representation stores a signal in the weights of a network that maps a coordinate to a value: a 3D point to its occupancy or signed distance, or a pixel position to a colour. Nothing is stored on a grid; the signal is queried at any continuous location. It is the implicit family of 3D representations with a neural network as the function. NeRF has its own article (neural radiance fields); the neural rendering hub places these networks among explicit, implicit and hybrid representations.

Definition

A neural implicit representation is a function $f_\theta : \mathbb{R}^d \to \mathbb{R}^k$, usually a multilayer perceptron with parameters $\theta$, trained so that $f_\theta(\mathbf{x})$ approximates a field at every coordinate $\mathbf{x}$: an occupancy probability or a signed distance function, whose level set is a surface; a colour at a pixel coordinate; or density and colour in a radiance field.

Xie et al. (2022) call these neural fields, fields parameterized fully or in part by a neural network, and list implicit neural representations, neural implicits and coordinate-based networks as other names.

Intuition

A digital image is a table of values at integer positions. A coordinate network replaces the table with a formula: ask for the colour at $(0.31, -0.72)$ and it computes one, between pixels as easily as on them. For a shape, the surface is a classifier’s decision boundary (inside or not) or a regressor’s zero crossing (signed distance).

The catch is that a plain network fed raw coordinates learns fine detail very slowly, so much of the design concerns the input encoding, the activation, or a feature grid.

Formal Definition

For one shape, $f_\theta(\mathbf{x})$ is fitted to samples $(\mathbf{x}_j, s_j)$ of the field. To represent many shapes, the network is conditioned on a latent code per shape, $f_\theta(\mathbf{x}, \mathbf{z}_i)$.

Occupancy. Occupancy Networks (Mescheder et al., 2019) output the probability that $\mathbf{x}$ is inside, conditioned through an encoder on an image, a point cloud or a coarse voxel grid, and train with a cross-entropy loss on points sampled in the bounding volume (their Eq. 3); the surface is the level set $f_\theta = \tau$. IM-NET (Chen & Zhang, 2019) regresses the same field, 1 inside and 0 outside (Eq. 1), with a mean squared error weighted to offset denser sampling near the surface (Eq. 2).

Signed distance and the auto-decoder. DeepSDF (Park et al., 2019) regresses the signed distance, negative inside, with a clamped loss (Eq. 4, $\delta = 0.1$):

$$\mathcal{L}\big(f_\theta(\mathbf{x}), s\big) = \big|\operatorname{clamp}(f_\theta(\mathbf{x}), \delta) - \operatorname{clamp}(s, \delta)\big|, \qquad \operatorname{clamp}(x, \delta) = \min(\delta, \max(-\delta, x)).$$

There is no encoder: each training shape owns a code $\mathbf{z}_i$, initialized from $\mathcal{N}(0, 0.01^2)$, and codes and network are optimized together under a Gaussian prior on the codes (Eq. 9):

$$\arg\min_{\theta, \{\mathbf{z}_i\}} \sum_{i=1}^{N} \Big( \sum_{j=1}^{K} \mathcal{L}\big(f_\theta(\mathbf{x}_j, \mathbf{z}_i), s_j\big) + \frac{1}{\sigma^2} \lVert \mathbf{z}_i \rVert_2^2 \Big).$$

At test time $\theta$ is frozen and a new shape’s code is found by minimizing over $\mathbf{z}$ alone (Eq. 10). The authors call this an auto-decoder; its advantage is that the observations can be any set of SDF samples, such as those from one depth map, while an encoder expects inputs like its training data.

Eikonal regularization. Without distance values, IGR (Gropp et al., 2020) fits a point cloud $\{\mathbf{x}_i\}$, optionally with normals $\mathbf{n}_i$ ($\tau = 1$; otherwise $\tau = 0$), by minimizing (their Eq. 2)

$$\ell(\theta) = \frac{1}{|I|} \sum_{i \in I} \Big( |f_\theta(\mathbf{x}_i)| + \tau \lVert \nabla_{\mathbf{x}} f_\theta(\mathbf{x}_i) - \mathbf{n}_i \rVert \Big) + \lambda\, \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \big( \lVert \nabla_{\mathbf{x}} f_\theta(\mathbf{x}) \rVert - 1 \big)^2 .$$

The eikonal term pushes the gradient towards unit length wherever $\mathcal{D}$ samples, the property an exact signed distance has almost everywhere.

Properties

Shape models and conditioning

Occupancy Networks, IM-NET and DeepSDF, all at CVPR 2019, each conditioned one network on a shape code to represent many ShapeNet shapes at any query resolution and with no fixed topology. DeepSDF uses eight fully connected layers of 512 units, with the code fed at the input and the fourth layer (§4).

A single global code must describe the whole object. Peng et al. (2020) blame the simple fully connected architecture, which acts only globally, for limiting such models to simple single objects and overly smooth surfaces. Their Convolutional Occupancy Networks give the decoder a feature interpolated at the query point from convolutionally processed feature planes or a coarse volume: a local, translation-equivariant code. For ShapeNet reconstruction from 3,000 noisy points, their Table 1 (mean over 13 classes) reports volumetric IoU 0.761 for Occupancy Networks against 0.884 with three $64^2$ feature planes; the first sentence of their §4 mentions only chairs. Xie et al. (2022, §2.1.2 in arXiv v4) give the general reason: a global code for a room of independent objects must cover every arrangement of them.

Supervision: 3D data, points, or images

• 3D ground truth: occupancy labels or signed distances computed from meshes, as above.
• Raw points: IGR. Infinitely many signed distance functions vanish on a finite point set, yet the authors observe that SGD on their loss reaches smooth, plausible zero sets; they prove this “plane reproduction” only for a linear model (§4). Their networks have eight Softplus layers of 512 units and $\lambda = 0.1$ (§2, §5).
• Images, through a differentiable renderer. DVR (Niemeyer et al., 2020) finds where each ray first crosses the occupancy level $\tau$ and differentiates the surface depth implicitly: with $\hat{\mathbf{p}} = \mathbf{r}_0 + \hat d\,\mathbf{w}$, $\partial \hat d / \partial \theta = -(\nabla_{\mathbf{p}} f_\theta \cdot \mathbf{w})^{-1}\, \partial f_\theta / \partial \theta$ (Eq. 8), so no volume of intermediate values is stored. IDR (Yariv et al., 2020) sphere-traces an SDF network, predicts colour from the surface point, normal and viewing direction with a second network, and refines rough camera poses, using object masks and IGR’s eikonal term. NeuS and VolSDF (2021) convert the SDF into a density for volume rendering; the NeuS authors argue that surface rendering gives gradients only at one intersection point per ray, which is too local.

Spectral bias and input encodings

Rahaman et al. (2019) analysed ReLU networks with Fourier tools and found a spectral bias: lower frequencies are learned first. In a coordinate network, edges and texture arrive last.

Fourier features (Tancik et al., 2020) pass the coordinates through sinusoids of random frequencies before the first layer; the positional encoding article covers the mapping and their neural tangent kernel argument for the frequency scale $\sigma$. SIREN (Sitzmann et al., 2020) keeps raw coordinates and uses $\sin$ as every activation. Its derivative is again a SIREN, so it can be supervised through its gradients, as in the eikonal problem. It needs its initialization: hidden weights from $\mathcal{U}(-\sqrt{6/n}, \sqrt{6/n})$ for $n$ inputs keep each sine’s input roughly standard normal through depth, and a factor $\omega_0 = 30$ in the first layer makes it span several periods over $[-1, 1]$ (NeurIPS version, §3.2). The supplement (arXiv v1, §1.5) applies $\omega_0$ in every layer with the weights divided by it, which speeds training; the official code does this.

Hybrid and fast variants

Hybrids move capacity into a structure of learned features with a small decoder. NGLOD (Takikawa et al., 2021) stores features at the corners of a sparse voxel octree around the surface, interpolates and sums them over the levels, and decodes them with a one-hidden-layer MLP; the octree depth sets the level of detail, and the authors report rendering two to three orders of magnitude faster than previous work (abstract). Instant-NGP (Müller et al., 2022) uses a multiresolution hash table instead, demonstrated on a gigapixel image, an SDF (§5.2) and radiance fields; the neural rendering hub compares such hybrids.

Extracting and using the field

• Meshing: sample $f_\theta$ on a grid and run marching cubes. Occupancy Networks start from $32^3$, subdivide only cells whose corners disagree, and refine the mesh with the network’s gradients (§3.3).
• Ray casting: sphere tracing (signed distance function) steps by $f_\theta$, safely only if the field never overestimates the distance, which nothing in training guarantees (our reasoning).
• Normals: $\nabla_{\mathbf{x}} f_\theta / \lVert \nabla_{\mathbf{x}} f_\theta \rVert$ on the zero level set, by backpropagation to the input (Park et al., 2019, §3).

Strengths and limitations

Memory scales with the parameter count rather than the resolution (Xie et al., 2022), any topology is possible, and the field is differentiable. But every query is a network evaluation, which made rendering slow before hybrids (Takikawa et al., 2021); DeepSDF’s authors note slow test-time latent optimization and a canonical-pose assumption (§7); and a global code generalizes poorly beyond single objects. The eikonal term is only a penalty, which IGR’s authors note need not yield unit gradients (§2).

Examples

The PyTorch code (version 2.14.1, CPU) compares three networks of about 8,600 trainable parameters each: a ReLU MLP on raw $(x, y)$, a ReLU MLP on 32 random Fourier features, and a SIREN with $\omega_0 = 30$. They fit a synthetic $128 \times 128$ image (disc, 8-pixel sine grating, square wave) on one pixel in four, tested on the rest, for three seeds.

import math, torch
import torch.nn as nn

torch.set_num_threads(4)

# Three coordinate networks with (almost) the same number of trainable parameters.
class Fourier(nn.Module):                     # gamma(v) = [cos 2 pi B v, sin 2 pi B v], B fixed
    def __init__(self, m, sigma):
        super().__init__()
        self.register_buffer("B", torch.randn(m, 2) * sigma)
    def forward(self, x):
        z = 2 * math.pi * x @ self.B.T
        return torch.cat([z.cos(), z.sin()], -1)

class Sine(nn.Linear):                        # sin(w0 (W x + b)); weights initialized as in the SIREN code
    def __init__(self, n_in, n_out, w0=30.0, first=False):
        super().__init__(n_in, n_out)
        self.w0 = w0
        bound = 1 / n_in if first else math.sqrt(6 / n_in) / w0
        nn.init.uniform_(self.weight, -bound, bound)
    def forward(self, x):
        return torch.sin(self.w0 * super().forward(x))

def relu_mlp(n_in, h, act=nn.ReLU):
    return [nn.Linear(n_in, h), act(), nn.Linear(h, h), act(), nn.Linear(h, h), act(), nn.Linear(h, 1)]

def make(kind, sigma=None):
    if kind == "ReLU, raw (x, y)":
        return nn.Sequential(*relu_mlp(2, 64))
    if kind == "ReLU, Fourier":
        return nn.Sequential(Fourier(32, sigma), *relu_mlp(64, 51))
    if kind == "SIREN":
        return nn.Sequential(Sine(2, 64, first=True), Sine(64, 64), Sine(64, 64), nn.Linear(64, 1))

def n_params(net):
    return sum(p.numel() for p in net.parameters())   # the Fourier matrix B is a buffer, not trained

def train(net, x, y, loss_fn, steps, lr=1e-3):
    opt = torch.optim.Adam(net.parameters(), lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        loss_fn(net, x, y).backward()
        opt.step()
    return net

# --- 1. An image as a function of pixel coordinates ------------------------------------
N = 128
g = torch.linspace(-1, 1, N)
yy, xx = torch.meshgrid(g, g, indexing="ij")
img = (0.3 + 0.4 * (xx**2 + yy**2 < 0.36)                         # a bright disc
       + 0.15 * torch.sin(16 * math.pi * xx) * (yy > 0)            # stripes, period 8 px
       + 0.15 * torch.sign(torch.sin(12 * math.pi * yy)) * (yy <= 0) * (xx < 0)).clamp(0, 1)
X, I = torch.stack([xx, yy], -1).reshape(-1, 2), img.reshape(-1)
even = (torch.arange(N) % 2 == 0)
train_px = (even[:, None] & even[None, :]).reshape(-1)             # fit on 1 pixel in 4
mse = lambda net, x, y: ((net(x).squeeze(1) - y) ** 2).mean()
psnr = lambda p, t: (10 * torch.log10(1 / ((p - t) ** 2).mean())).item()

print("image 128 x 128      params  PSNR fit pixels (dB), seeds 0-2   PSNR held-out pixels (dB)")
for kind, sigma in [("ReLU, raw (x, y)", None), ("ReLU, Fourier", 3.0), ("ReLU, Fourier", 6.0), ("SIREN", None)]:
    fit, held = [], []
    for seed in range(3):
        torch.manual_seed(seed)
        net = train(make(kind, sigma), X[train_px], I[train_px], mse, steps=1000)
        with torch.no_grad():
            p = net(X).squeeze(1)
        fit.append(psnr(p[train_px], I[train_px])); held.append(psnr(p[~train_px], I[~train_px]))
    name = kind + (f" s={sigma:g}" if sigma else "")
    print(f"{name:20s} {n_params(net):6d}  " + " ".join(f"{v:6.1f}" for v in fit)
          + "            " + " ".join(f"{v:5.1f}" for v in held))

image 128 x 128      params  PSNR fit pixels (dB), seeds 0-2   PSNR held-out pixels (dB)
ReLU, raw (x, y)       8577    19.6   19.6   19.3             19.3  19.3  19.2
ReLU, Fourier s=3      8671    39.9   40.4   40.6             23.7  24.3  24.3
ReLU, Fourier s=6      8671    38.1   41.1   38.7             16.1  18.9  18.4
SIREN                  8577    44.3   46.2   44.9             24.8  24.5  24.7

The raw ReLU network cannot even fit its training pixels (19.3–19.6 dB after 1,000 steps), as spectral bias predicts. Fourier features with $\sigma = 3$ and the SIREN fit them to 40–46 dB and reach about 24–25 dB on held-out pixels, limited, we think, by hard edges that samples two pixels apart do not determine. With $\sigma = 6$ the fit is similar but held-out PSNR falls to 16–19 dB, below the raw network: in our interpretation, frequencies the samples cannot support oscillate between them, the bandwidth trade-off Tancik et al. describe.

Continuing, the networks regress the exact signed distance of a 7-pointed star with an $L^1$ loss on 4,000 samples (half uniform, half noisy boundary points) and are tested on 20,000 held-out ones (“near”: $|d| < 0.02$). Last, an IGR-style network learns the star from 2,000 boundary points alone ($\tau = 0$, $\lambda = 0.1$). Unlike IGR, it samples $\mathcal{D}$ with Gaussians of fixed width 0.05, not the distance to the 50th neighbour, and starts from a network fitted to a circle’s SDF, our substitute for the geometric initialization IGR takes from Atzmon and Lipman (2020).

# --- 2. Signed distance of a 7-pointed star (negative inside) --------------------------
K, R_OUT, R_IN = 7, 0.75, 0.38
ang = torch.arange(2 * K, dtype=torch.float64) * math.pi / K + math.pi / 2
rad = torch.tensor([R_OUT, R_IN] * K, dtype=torch.float64)
A = torch.stack([rad * ang.cos(), rad * ang.sin()], 1)            # polygon vertices
B = A.roll(-1, 0)                                                 # edge ends

def star_sdf(p):                                                  # exact: nearest edge + even-odd sign
    p = p.double()
    ab, ap = B - A, p[:, None] - A
    t = ((ap * ab).sum(-1) / (ab * ab).sum(-1)).clamp(0, 1)
    d = (ap - t[..., None] * ab).norm(dim=-1).min(1).values
    up = (A[:, 1] > p[:, 1:]) != (B[:, 1] > p[:, 1:])
    xc = A[:, 0] + (p[:, 1:] - A[:, 1]) * (B[:, 0] - A[:, 0]) / (B[:, 1] - A[:, 1])
    inside = (up & (p[:, :1] < xc)).sum(1) % 2 == 1
    return torch.where(inside, -d, d).float()

def on_star(n, gen):                                              # uniform in arc length
    e = torch.multinomial((B - A).norm(dim=1), n, replacement=True, generator=gen)
    s = torch.rand(n, 1, generator=gen, dtype=torch.float64)
    return (A[e] + s * (B[e] - A[e])).float()

def grad_f(net, x, keep_graph=False):                             # spatial gradient by autograd
    x = x.detach().requires_grad_(True)
    g, = torch.autograd.grad(net(x).sum(), x, create_graph=keep_graph)
    return g

gen = torch.Generator().manual_seed(0)
def samples(n):                                                   # half uniform, half near the surface
    x = torch.cat([torch.rand(n // 2, 2, generator=gen) * 2 - 1,
                   on_star(n // 2, gen) + 0.05 * torch.randn(n // 2, 2, generator=gen)])
    return x, star_sdf(x)
x_tr, s_tr = samples(4000)
x_te, s_te = samples(20000)                                       # held-out points
near = s_te.abs() < 0.02
l1 = lambda net, x, y: (net(x).squeeze(1) - y).abs().mean()

print("\nSDF, 4,000 samples   params  |f - d| all, seeds 0-2   |f - d| near   median ||grad f| - 1|")
for kind, sigma in [("ReLU, raw (x, y)", None), ("ReLU, Fourier", 1.0), ("ReLU, Fourier", 3.0), ("SIREN", None)]:
    e_all, e_near, eik = [], [], []
    for seed in range(3):
        torch.manual_seed(seed)
        net = train(make(kind, sigma), x_tr, s_tr, l1, steps=1000)
        with torch.no_grad():
            err = (net(x_te).squeeze(1) - s_te).abs()
        e_all.append(err.mean().item()); e_near.append(err[near].mean().item())
        eik.append((grad_f(net, x_te).norm(dim=1) - 1).abs().median().item())
    name = kind + (f" s={sigma:g}" if sigma else "")
    print(f"{name:20s} {n_params(net):6d}  " + " ".join(f"{v:.4f}" for v in e_all) + "   "
          + " ".join(f"{v:.4f}" for v in e_near) + "   " + " ".join(f"{v:.3f}" for v in eik))

# --- 3. IGR-style fit from surface points only: |f| on the points + eikonal term --------
pts = on_star(2000, gen)
class Softplus100(nn.Softplus):
    def __init__(self): super().__init__(beta=100)
torch.manual_seed(0)
net = nn.Sequential(*relu_mlp(2, 64, act=Softplus100))
circle = lambda net, x, y: ((net(x).squeeze(1) - (x.norm(dim=1) - 0.5)) ** 2).mean()
train(net, torch.rand(4000, 2, generator=gen) * 2 - 1, None, circle, steps=300)  # start near a circle's SDF

def igr(net, x, _, lam=0.1):
    q = torch.cat([torch.rand(1000, 2) * 2 - 1, x[torch.randint(len(x), (1000,))] + 0.05 * torch.randn(1000, 2)])
    eikonal = ((grad_f(net, q, keep_graph=True).norm(dim=1) - 1) ** 2).mean()
    return net(x).abs().mean() + lam * eikonal
train(net, pts, None, igr, steps=1500)

M = 256                                                           # zero crossings along grid edges
gm = torch.linspace(-1, 1, M)
with torch.no_grad():
    F = net(torch.stack(torch.meshgrid(gm, gm, indexing="ij"), -1).reshape(-1, 2)).reshape(M, M)
cross = []
for Fa, Fb, axis in [(F[:-1], F[1:], 0), (F[:, :-1], F[:, 1:], 1)]:
    i, j = torch.nonzero((Fa > 0) != (Fb > 0), as_tuple=True)
    t = Fa[i, j] / (Fa[i, j] - Fb[i, j])
    p = torch.stack([gm[i], gm[j]], 1)
    p[:, axis] += t * (gm[1] - gm[0])
    cross.append(p)
Z = torch.cat(cross)                                              # points on the learned zero set
acc = star_sdf(Z).abs()                                           # their true distance to the star
comp = torch.cdist(on_star(5000, gen), Z).min(1).values           # true surface to learned zero set
with torch.no_grad():
    u = torch.rand(20000, 2, generator=gen) * 2 - 1
    f_u, d_u = net(u).squeeze(1), star_sdf(u)
gap = lambda q: torch.cdist(q.float(), Z).min(1).values.mean().item()
err_u, far = (f_u - d_u).abs(), d_u > 0.3
print(f"\nIGR fit from {len(pts)} surface points, no normals, no distances:")
print(f"  zero set -> star: mean {acc.mean():.4f}, max {acc.max():.4f};  "
      f"star -> zero set: mean {comp.mean():.4f}, max {comp.max():.4f}")
print(f"  gap at the 7 outer tips {gap(A[0::2]):.4f}, at the 7 inner corners {gap(A[1::2]):.4f}")
print(f"  uniform points: |f - d| mean {err_u.mean():.4f}, where d > 0.3 {err_u[far].mean():.4f}; "
      f"wrong sign {((f_u > 0) != (d_u > 0))[d_u.abs() > 0.02].float().mean():.2%}")
print(f"  median ||grad f| - 1| {(grad_f(net, u).norm(dim=1) - 1).abs().median():.3f}")

SDF, 4,000 samples   params  |f - d| all, seeds 0-2   |f - d| near   median ||grad f| - 1|
ReLU, raw (x, y)       8577  0.0117 0.0059 0.0057   0.0099 0.0041 0.0047   0.071 0.082 0.068
ReLU, Fourier s=1      8671  0.0034 0.0028 0.0037   0.0027 0.0020 0.0027   0.098 0.099 0.111
ReLU, Fourier s=3      8671  0.0168 0.0098 0.0197   0.0051 0.0036 0.0061   0.657 0.383 0.715
SIREN                  8577  0.0065 0.0080 0.0066   0.0044 0.0056 0.0054   0.142 0.155 0.150

IGR fit from 2000 surface points, no normals, no distances:
  zero set -> star: mean 0.0028, max 0.0108;  star -> zero set: mean 0.0040, max 0.0304
  gap at the 7 outer tips 0.0181, at the 7 inner corners 0.0119
  uniform points: |f - d| mean 0.0091, where d > 0.3 0.0142; wrong sign 0.00%
  median ||grad f| - 1| 0.038

On the smoother distance field the raw network does reasonably, and the best encoding differs: $\sigma = 1$ gives the lowest errors, about 0.003, while $\sigma = 3$ keeps the near-surface error small but has errors up to 0.020 overall and median eikonal residuals of 0.38–0.72, a field that is not distance-like between the samples. The SIREN, trained on values only, has residuals of about 0.15.

The IGR-style network never saw a distance value, yet its zero set lies 0.003 from the star on average, its field is off by 0.009 on average, and no sign is wrong outside $|d| \le 0.02$; the sign comes from the starting circle, since no normals were given (our reasoning). Its eikonal residual, 0.038, is the lowest here, yet its distance error is about three times the best supervised one (0.0028, on the mixed test points), and it rounds the corners (gaps of 0.018 at the tips, 0.012 at the inner corners), the smooth solutions IGR’s authors observed. Both blocks ran in about 35 s on our CPU.

Common Misconceptions

• “A small eikonal residual means an exact SDF.” A unit gradient is necessary, not sufficient (signed distance function); in the example the lowest residual did not give the lowest distance error.

Where It Is Used

• Shape priors and 3D reconstruction: completion, generation and single-view reconstruction (DeepSDF, Occupancy Networks, IM-NET); scenes (Convolutional Occupancy Networks).
• Surfaces from points: IGR and SIREN fit signed distances to point clouds, an alternative to Poisson surface reconstruction.
• Multi-view surface reconstruction: DVR, IDR, NeuS and VolSDF.