3D Representations

A 3D representation is the data structure that holds a shape or a scene: what is stored, where, and how a surface, an image or an inside/outside answer is recovered from it. Images have one dominant representation, the pixel grid; 3D has several; none is best for everything, and most pipelines convert between them.

Definition

A 3D representation describes geometry, and possibly appearance, in one of two ways:

• Explicit representations list the geometry: sampled depths, points, triangles, occupied cells, or primitives such as Gaussians. The surface is where the elements are.
• Implicit representations store a function over space and define the surface as a level set of it, such as the zero set of a signed distance function. The function can be sampled on a grid or computed by a formula or a neural network.

A second axis is discrete versus continuous: a grid or point set has a fixed resolution, while an analytic or neural function can be queried anywhere.

Intuition

A renderer asks “what is the first surface along this ray?”, a robot planner “is this point inside an obstacle?”, and a learning system “how do I output this shape from a fixed-size network?”. Each representation answers some of these questions in one step and the others only after a conversion: a mesh serves the renderer, a signed distance function the planner, a neural field the network.

Formal Definition

Let $n$ be the linear resolution, the number of samples along each side of the region of interest.

Depth map. A function $Z(u, v)$ on a pixel grid, read with known intrinsics $K$ and pose; each valid pixel back-projects to $Z\,K^{-1}(u, v, 1)^\top$ in the camera frame.

Point cloud. A set $\{\mathbf{p}_i\}_{i=1}^N \subset \mathbb{R}^3$ with optional attributes; covering a surface at spacing $1/n$ takes $N = O(n^2)$ points.

Polygon mesh. Vertices and faces, usually triangles stored as vertex-index triples. For a closed triangle mesh of genus $g$, Euler’s formula $|V| - |E| + |F| = 2 - 2g$ with $|E| = \tfrac{3}{2}|F|$ (each edge borders two triangles) gives $|F| = 2|V| - 4 + 4g$: about twice as many triangles as vertices.

Voxel grid. Values on an $n^3$ lattice of cells of side $h$: one occupancy bit, or a distance, colour or feature vector per cell, for $O(n^3)$ memory. An octree subdivides a cell into eight only where needed, and sparse voxel structures allocate only blocks near the surface; since a surface crosses $O(n^2)$ cells, both bring memory close to $O(n^2)$ (reasoning from the surface’s dimension).

Signed distance and occupancy fields. A signed distance function $f : \mathbb{R}^3 \to \mathbb{R}$ gives the distance to the nearest surface point, negative inside, positive outside and zero on the surface $\mathcal{S} = \{\mathbf{x} : f(\mathbf{x}) = 0\}$; where it is differentiable, $\lVert \nabla f \rVert = 1$. An occupancy function $o : \mathbb{R}^3 \to \{0, 1\}$ marks inside points. Either field can be a voxel grid ($O(n^3)$, or $O(n^2)$ in a band near the surface), a formula, or fixed-size network weights.

Radiance field. A function $(\mathbf{x}, \mathbf{d}) \mapsto (\sigma, \mathbf{c})$ giving a volume density $\sigma \ge 0$ and a colour $\mathbf{c}$ that may depend on the viewing direction $\mathbf{d}$. A pixel is rendered from samples $i$ along its ray, with spacing $\delta_i$:

$$\hat C = \sum_i T_i \alpha_i \mathbf{c}_i, \qquad \alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \exp\Big(-\sum_{j<i} \sigma_j \delta_j\Big).$$

3D Gaussians. A set of anisotropic Gaussians, each with a mean, a covariance $\Sigma = R S S^\top R^\top$ (rotation stored as a quaternion, scales as a 3-vector), an opacity, and colour as spherical-harmonic coefficients (Kerbl et al., 2023). They are projected into the image and composited front to back in the same form as $\hat C$. Memory grows with the number of Gaussians, which the optimization adapts to the scene.

Properties

Depth maps and point clouds

Both have their own articles (depth map, point cloud) and are the raw output of depth sensors, LiDAR, stereo and multi-view stereo. A depth map is 2.5D: silent about everything behind the first surface, with only a front-or-behind test along each ray. Point clouds merge views but have no surface between samples and no inside.

Polygon meshes

A mesh stores a piecewise-planar surface with explicit connectivity, which GPU rasterizers and simulation software consume natively. Moving vertices is easy; changing topology, such as opening a hole, is not. Inside tests need a closed (watertight) mesh and a ray-crossing or winding-number computation. For learning, Mescheder et al. (2019) note that mesh-predicting networks typically deform a template and so cannot change its topology.

Voxel grids, octrees and sparse voxels

A voxel grid is the 3D pixel grid: lookups, inside tests and 3D convolutions are immediate. Its weakness is memory. Riegler et al. (2017) reported that a dense 3D network with batch size 32 fitted in a 12 GB GPU only for inputs up to $64^3$, while their octree-based OctNet fitted at $256^3$. Octrees and sparse blocks skip the empty space and interior that fill most of a dense grid, at the cost of irregular memory access. Voxel hashing (Nießner et al., 2013) stores $8^3$-voxel blocks of distances in a hash table, allocated only within the truncation region around observed surfaces.

Signed distance and occupancy fields

A distance field answers inside/outside with one evaluation and says how far the surface is, which collision checking exploits; a ray can advance by $|f|$ per step without crossing the surface (sphere tracing; Hart, 1996, §2.2). Fields combine by Boolean operations, $\min(f_A, f_B)$ for union and $\max(f_A, f_B)$ for intersection; the sign stays correct, but the result is generally only a bound on the distance (reasoning from the definitions). Curless and Levoy (1996) fused range images into a voxel grid of weighted signed distances and extracted its zero isosurface, the approach behind TSDF fusion (truncation distance $\mu$).

Neural versions store the field in weights. DeepSDF (Park et al., 2019), with the same sign convention, conditions one network on a latent code per shape; the authors report representing a whole class, such as thousands of chairs, in 7.4 MB, less than half the 16.8 MB of one uncompressed $512^3$ bitmap. Occupancy networks (Mescheder et al., 2019) instead learn an occupancy probability and take the surface at a threshold. Neural fields handle arbitrary topology at any query resolution, but each query is a network evaluation, and local edits are hard because each weight can affect the whole shape.

Radiance fields

A radiance field stores how much each point absorbs and what colour it emits, not where a surface is, so it captures thin, fuzzy and semi-transparent structure and view-dependent appearance. NeRF (Mildenhall et al., 2020) encodes it in an MLP of eight 256-channel layers plus a small colour branch. The authors report 5 MB of weights per scene, against over 15 GB of voxel grids for the LLFF baseline on one synthetic scene (ECCV version, §6.3), but rendering took about 30 seconds per frame on an NVIDIA V100 (arXiv v2, appendix A). Grid-based fields trade memory for speed: in Kerbl et al.’s Table 1, on the Mip-NeRF360 scenes, Plenoxels used 2.1 GB and Instant-NGP 13 MB (Base) or 48 MB (Big), against 8.6 MB for the MLP-based Mip-NeRF360.

3D Gaussians

3D Gaussian splatting replaces the continuous field with explicit, semi-transparent ellipsoids, initialized from sparse structure-from-motion points and refined by cloning, splitting and removing Gaussians. Kerbl et al. (2023) report 1–5 million Gaussians per scene (§1; fewer on synthetic scenes, §7.2); in their Table 1, the fully trained models rendered the Mip-NeRF360 scenes at 134 frames per second against 0.06 for Mip-NeRF360, but stored 734 MB against 8.6 MB. With four bands of spherical harmonics (16 coefficients per colour channel), a Gaussian holds $3 + 3 + 4 + 1 + 48 = 59$ numbers: position, scale, quaternion, opacity and colour (our count). Gaussians are easy to edit but have no explicit surface; the authors name mesh reconstruction from them as a direction for future work.

Comparison

Representation	Stores	Memory vs $n$	Rendering	Inside test	Editing	Learning
Depth map	depth per pixel, one view	$O(n^2)$ per view	is a rendering (one view)	front/behind along ray	per view only	easy (2D networks)
Point cloud	surface samples	$O(n^2)$	splatting, holes	none	add/remove points	permutation-invariant networks
Mesh	vertices, faces	$O(n^2)$	rasterization, fast	needs watertight mesh	geometry easy, topology hard	hard (fixed topology)
Voxel grid	value per cell	$O(n^3)$ dense; near $O(n^2)$ octree/sparse	ray marching	lookup	easy, blocky	3D CNNs, memory-bound
SDF / occupancy grid	distance or occupancy per cell	$O(n^3)$; $O(n^2)$ in a band	sphere tracing or mesh first	sign	Boolean operations	3D CNNs
Neural SDF / occupancy	network weights	fixed by network size	many queries per ray	sign, one query	hard	natural output
Radiance field	density and colour	MLP fixed; grids large	volume rendering, slow	none (density)	hard (MLP)	per-scene optimization
3D Gaussians	per-Gaussian parameters	number of Gaussians	rasterization, real time	none	move/delete primitives	per-scene optimization

Converting Between Representations

• Depth map ↔ points or mesh: back-project with $K$, or render with a z-buffer.
• Depth maps → distance grid: volumetric fusion (Curless & Levoy, 1996), as in TSDF fusion and KinectFusion.
• Distance or occupancy grid → mesh: marching cubes (Lorensen & Cline, 1987) triangulates the level set cell by cell; neural fields are sampled on a grid and meshed the same way.
• Point cloud → mesh: through an implicit function, such as Poisson surface reconstruction, which needs oriented normals (point cloud).
• Mesh → distance field or points: nearest-triangle distance, signed by an inside test, or area-weighted sampling.
• Radiance field → depth or mesh: the weights $T_i \alpha_i$ give an expected depth per ray, and thresholding $\sigma$ on a grid gives a mesh; both are approximations, since density defines no single surface (reasoning from the volume rendering integral).

Every conversion to a discrete form fixes a resolution, and every conversion from depth maps or points must fill in what was not observed.

Examples

The code stores a torus (tube radius 0.25, circle radius 0.6) as an analytic SDF, occupancy grids, sampled SDF grids, point clouds, and depth maps rendered by sphere tracing from a camera with the world-to-camera convention $X_c = R X_w + t$ of the camera parameters article. Each form is turned into surface points and scored against 200,000 exact surface samples. Accuracy is the mean distance from the form’s points to the true surface, computed exactly as $|f|$; completeness is the mean distance from the true surface to the nearest point of the form; their average is the Chamfer-$L_1$ distance as defined by Mescheder et al. (2019). DeepSDF’s supplementary material (arXiv v1, §I.2) sums squared distances instead, so Chamfer values are comparable only under one definition. The last column is the fraction of the true surface within 0.02 (1% of the cube’s side) of the form.

import numpy as np
from scipy.spatial import cKDTree

R_MAJ, R_MIN = 0.6, 0.25           # torus: tube of radius 0.25 around a circle of radius 0.6

def sdf(p):
    """Exact signed distance to the torus: negative inside, positive outside."""
    q = np.sqrt(p[..., 0]**2 + p[..., 1]**2) - R_MAJ
    return np.sqrt(q**2 + p[..., 2]**2) - R_MIN

def sample_surface(n, rng):
    """Area-uniform samples on the torus (rejection sampling on the tube angle)."""
    out = []
    while sum(len(o) for o in out) < n:
        u, v = rng.uniform(0, 2 * np.pi, (2, 2 * n))
        keep = rng.uniform(0, 1, 2 * n) < (R_MAJ + R_MIN * np.cos(v)) / (R_MAJ + R_MIN)
        u, v = u[keep], v[keep]
        rho = R_MAJ + R_MIN * np.cos(v)
        out.append(np.stack([rho * np.cos(u), rho * np.sin(u), R_MIN * np.sin(v)], 1))
    return np.concatenate(out)[:n]

ref = sample_surface(200_000, np.random.default_rng(0))   # dense reference surface
ref_tree = cKDTree(ref)

def report(name, res, nbytes, pts):
    acc = np.abs(sdf(pts)).mean()                  # representation -> true surface (exact)
    d, _ = cKDTree(pts).query(ref)                 # true surface -> representation
    comp = d.mean()
    print(f"{name:10s} {res:>6} {nbytes:>10,d}  {acc:.1e}  {comp:.4f}  "
          f"{(acc + comp) / 2:.4f}  {np.mean(d < 0.02):6.1%}")

print("form          res      bytes  accuracy  complete  chamfer  cov<0.02")
report("analytic", "-", 16, ref)                   # two float64 radii; samples are exact

def grid(n):
    h = 2.0 / n                                    # voxels of side h in the cube [-1, 1]^3
    c = -1 + (np.arange(n) + 0.5) * h
    return h, np.stack(np.meshgrid(c, c, c, indexing="ij"), -1)

for n in [16, 32, 64, 128]:                        # occupancy: one bit per voxel
    h, P = grid(n)
    occ = sdf(P) < 0
    pad = np.pad(occ, 1)
    empty_nbr = np.zeros_like(occ)
    for ax in range(3):
        for s in (-1, 1):
            empty_nbr |= ~np.roll(pad, s, axis=ax)[1:-1, 1:-1, 1:-1]
    report("occupancy", f"{n}^3", np.packbits(occ).nbytes, P[occ & empty_nbr])

for n in [16, 32, 64, 128]:                        # sampled SDF: one float32 per voxel
    h, P = grid(n)
    D = sdf(P).astype(np.float32)
    g = np.stack(np.gradient(D, h), -1)            # central-difference gradient
    band = np.abs(D) < h                           # voxels near the zero level set
    nrm = g[band] / np.linalg.norm(g[band], axis=1, keepdims=True)
    pts = P[band] - D[band][:, None] * nrm         # project centres onto the surface
    report("sdf grid", f"{n}^3", D.nbytes, pts)

for m in [1_000, 4_000, 16_000, 64_000]:           # point cloud: three float32 per point
    pts = sample_surface(m, np.random.default_rng(1)).astype(np.float32)
    report("points", m, pts.nbytes, pts.astype(np.float64))

# Depth map from one camera: world-to-camera X_c = R X_w + t, looking at the origin.
C = np.array([0.0, -2.2, 1.6])                     # camera centre (world)
z = -C / np.linalg.norm(C)                         # optical axis
x = np.cross(z, [0.0, 0.0, 1.0]); x /= np.linalg.norm(x)
y = np.cross(z, x)                                 # image y points down
Rcw = np.stack([x, y, z])                          # rows: camera axes in world coordinates
t = -Rcw @ C
for w in [32, 64, 128, 256]:
    f = 0.5 * w / np.tan(np.radians(25))           # 50 degree field of view
    K = np.array([[f, 0, (w - 1) / 2], [0, f, (w - 1) / 2], [0, 0, 1]])
    u, v = np.meshgrid(np.arange(w), np.arange(w))
    rays_c = np.stack([u, v, np.ones_like(u)], -1).reshape(-1, 3) @ np.linalg.inv(K).T
    dirs = rays_c @ Rcw                            # camera -> world: R^T d
    dirs /= np.linalg.norm(dirs, axis=1, keepdims=True)
    s = np.zeros(len(dirs))                        # sphere tracing along each ray
    for _ in range(200):
        s += sdf(C + s[:, None] * dirs)
        s = np.minimum(s, 10.0)
    hit = np.abs(sdf(C + s[:, None] * dirs)) < 1e-4
    Z = np.where(hit, s * (dirs @ z), 0.0).astype(np.float32).reshape(w, w)  # 0 = no surface
    # Back-project valid pixels: X_c = Z K^-1 x, then X_w = R^T (X_c - t).
    Xc = rays_c[Z.ravel() > 0] * Z.ravel()[Z.ravel() > 0, None]
    report("depth map", f"{w}^2", Z.nbytes, (Xc - t) @ Rcw)

Output:

form          res      bytes  accuracy  complete  chamfer  cov<0.02
analytic        -         16  2.4e-17  0.0000  0.0000  100.0%
occupancy    16^3        512  3.8e-02  0.0696  0.0536    1.4%
occupancy    32^3      4,096  2.4e-02  0.0380  0.0313    6.8%
occupancy    64^3     32,768  1.2e-02  0.0186  0.0154   57.7%
occupancy   128^3    262,144  6.6e-03  0.0096  0.0081  100.0%
sdf grid     16^3     16,384  2.6e-05  0.0381  0.0191   15.2%
sdf grid     32^3    131,072  5.3e-07  0.0191  0.0096   53.2%
sdf grid     64^3  1,048,576  1.7e-08  0.0096  0.0048   99.9%
sdf grid    128^3  8,388,608  5.7e-10  0.0048  0.0024  100.0%
points       1000     12,000  8.3e-09  0.0382  0.0191   19.2%
points       4000     48,000  8.3e-09  0.0192  0.0096   57.3%
points      16000    192,000  8.1e-09  0.0096  0.0048   96.7%
points      64000    768,000  8.2e-09  0.0048  0.0024  100.0%
depth map    32^2      4,096  3.7e-08  0.1828  0.0914    5.5%
depth map    64^2     16,384  9.3e-08  0.1495  0.0748   20.5%
depth map   128^2     65,536  4.2e-08  0.1332  0.0666   36.7%
depth map   256^2    262,144  5.3e-08  0.1222  0.0611   41.9%

The analytic SDF is exact in 16 bytes. Doubling a grid’s resolution multiplies its memory by eight and roughly halves its error. Occupancy boundary voxels sit 0.3 to 0.42 of a voxel off the surface (0.038 at $h = 0.125$), because one bit cannot say where in the cell the surface passes. The SDF grid stores the distance itself, so moving each near-surface voxel centre along the gradient by its value lands it within $3 \times 10^{-5}$ of the surface even at $16^3$, at 32 times the memory of the packed occupancy grid; its remaining error is the spacing of the extracted points. Exact samples flatter this projection; fused depth is noisier. Point completeness halves each time the count quadruples, as spacing $\propto N^{-1/2}$ predicts. Depth-map points also lie on the surface, but no resolution makes one view complete: the far side is never seen, and coverage stops near 42%.

Common Misconceptions

• “Higher resolution fixes a representation’s limits.” It reduces sampling error, not what the representation cannot express: a depth map never shows the back of an object, and an occupancy grid never places the surface inside a cell.
• “A radiance field contains a surface.” It contains a density; any surface taken from it depends on a threshold or on a depth estimate from the compositing weights.

Where It Is Used

• Registration and mapping: points for iterative closest point; sparse or hashed TSDF grids in dense RGB-D SLAM systems such as KinectFusion.
• Graphics and robotics: meshes for rendering and physics; occupancy grids and distance fields for collision checking and planning.
• Learning 3D shape: voxel grids and octrees for 3D convolutional networks, points for permutation-invariant networks, and neural fields.
• Novel view synthesis: radiance fields and 3D Gaussians.