Time-of-Flight Camera

A time-of-flight (ToF) camera is a depth camera that carries its own light source and measures, at every pixel at once, how long that light takes to travel to the scene and back. Light covers the distance twice, so a round-trip delay $\Delta t$ means a distance $c\,\Delta t/2$ . A delay of one nanosecond is 15 cm of range, which is far too short to time with an ordinary camera exposure, and the two families of ToF camera solve this in different ways. A continuous-wave (CW) camera, also called amplitude-modulated continuous-wave (AMCW) or indirect ToF, modulates the brightness of its light periodically and measures the phase shift of the returning modulation by correlating it, inside each pixel, with a reference signal. Su et al. (2018) give Microsoft’s Kinect One as an example of this kind. A direct ToF camera times individual photons with single-photon avalanche diodes (SPADs) and builds a histogram of arrival times. The camera’s output is a range image: one distance per pixel, together with a brightness image of the returned light.

The LiDAR article covers pulsed laser ranging, beam steering and flash LiDAR. This article covers the per-pixel camera: the continuous-wave pixel and how it turns correlation samples into range, why the range wraps and how two frequencies unwrap it, how noise and ambient light set its precision, what multipath interference does to it, and the SPAD camera. The example simulates a continuous-wave camera looking into the corner of a room.

How It Works

Modulated light and a correlating pixel

Gutierrez-Barragan et al. (2019) give the image formation model of a continuous-wave camera. The light source’s intensity follows a periodic modulation function $M(t)$ . Ignoring indirect light, the radiance reaching a pixel is

R(t) = s\, M\!\left(t - \frac{2d}{c}\right) + A_\text{amb},

where $d$ is the distance of the scene point, $s$ a scale factor for the point’s albedo, shading and intensity fall-off, and $A_\text{amb}$ ambient light such as sunlight. The pixel multiplies $R(t)$ by a demodulation function $D(t)$ with the same fundamental frequency (a homodyne system) and integrates the product over the exposure time $T$ . The demodulation can be done by photonic mixer devices or fast optical shutters. The recorded value is $I = I_s\,\chi(d) + I_a$ , where $\chi$ is the normalized cross-correlation of $M$ and $D$ , $I_s$ the effective albedo and $I_a$ the ambient contribution. With three unknowns ( $d$ , $I_s$ , $I_a$ ), at least three measurements with different pairs of modulation and demodulation functions are needed (Gutierrez-Barragan et al., 2019, Eqs. 1–4). They also describe two-bucket sensors, which emulate a zero-mean demodulation function by taking the difference of two complementary measurements.

Su et al. (2018) write the same measurement in a model they attribute to Lange (2000): a raw correlation measurement $b_{\omega,\psi} = \int_0^T E(t)\, f_\omega(t - \psi/\omega)\,dt$ of the irradiance $E$ against a programmable reference $f_\omega$ with angular frequency $\omega$ and phase offset $\psi$ . They note that $f$ is typically zero-mean, which makes the measurement robust to ambient light. In our reading, this is why the two papers count differently: with a zero-mean reference the ambient term drops out and two phase offsets determine the phase, while Gutierrez-Barragan et al. consider positive demodulation functions ( $0 \le D \le 1$ ), which leave the ambient term as a third unknown.

The four-bucket phase

With sinusoidal modulation the correlation is itself a sinusoid of the delay. Naik et al. (2015) write the received signal as $s(t) = \alpha \cos(f_M t + \varphi) + \beta$ and its cross-correlation with the emitted signal as

c(\tau) = \frac{\alpha}{2}\cos(f_M \tau + \varphi) + \beta .

The “four bucket principle” samples $c(\tau)$ four times per period, at $\tau = 0, \pi/2, \pi, 3\pi/2$ , and gives the phase as $\arctan\bigl[(c(\tau_4) - c(\tau_2))/(c(\tau_1) - c(\tau_3))\bigr]$ (Naik et al., 2015, Eq. 5). The example uses this convention, written for samples $c_k$ with $k = 0, \dots, 3$ :

c_k = B + A\cos\!\left(\varphi + \tfrac{k\pi}{2}\right), \qquad I = c_0 - c_2 = 2A\cos\varphi, \qquad Q = c_3 - c_1 = 2A\sin\varphi,

\hat\varphi = \operatorname{atan2}(Q, I) \bmod 2\pi, \qquad \hat A = \tfrac12\sqrt{I^2 + Q^2}, \qquad \hat B = \tfrac14\textstyle\sum_k c_k,

with $\varphi = 4\pi f r/c$ for modulation frequency $f$ and range $r$ . The offset $B$ cancels from both differences, which is why ambient light does not shift the noise-free phase. The range is

r = \frac{c\,\varphi}{4\pi f}.

Papers differ in sign and argument order, and the difference matters. Su et al. (2018, Eqs. 2–3) write $d = c\varphi/2\omega$ with $\varphi = \operatorname{atan2}(b_{\omega,\pi/2}, b_{\omega,0})$ , where $\omega$ is the angular frequency, so their reference is delayed by $\psi$ and their samples go as $\cos(\varphi - \psi)$ . Naik et al.’s samples go as $\cos(f_M\tau + \varphi)$ . Applying one paper’s formula to samples taken in the other’s convention returns $-\varphi$ , a range of $U - r$ within each period $U$ ; the example prints this case. Naik et al.’s paper also has two slips, read from the rendered page. Its amplitude formula (Eq. 6) prints a minus sign between the two squared differences inside the square root, where the sum of squares needs a plus; with the plus, it gives $\alpha/2$ , the amplitude of $c(\tau)$ , rather than the $\alpha$ the text says it equals. And its Eq. 2 writes the emitted signal as $\cos(f_M t)$ without the factor $2\pi$ , while Eq. 1, $z = c\varphi/(4\pi f_M)$ , treats $f_M$ as a frequency in hertz.

Four phases are not the only choice. Guo et al. (2018, Eq. 3) recover the phase from $K \ge 2$ phase offsets $\psi_k$ as the arctangent of the ratio of the $\sin\psi$ - and $\cos\psi$ -weighted sums of the measurements. They report that the Kinect 2 uses three modulation frequencies with three phases each, nine camera functions in all (Guo et al., 2018, §3.4).

Ambiguity distance and unwrapping

The phase repeats every $2\pi$ , so the range is known only modulo the ambiguity distance

U = \frac{c}{2f},

which Su et al. write as the interval $[0, c\pi/\omega]$ , half a modulation wavelength. At 80 MHz it is 1.874 m and at 16 MHz 9.369 m (our arithmetic). A surface at 2.5 m seen at 80 MHz reports 0.63 m. Guo et al. (2018) describe the trade-off: longer modulation periods extend the unambiguous range, while shorter ones resolve finer detail.

The usual remedy is to measure at two or more frequencies. Su et al. (2018) give the extended range of a frequency pair as $d_\text{max} = c\pi/\gcd(\omega_1, \omega_2)$ , set by the greatest common divisor of the frequencies, and recover the range by a one-dimensional search through a lookup table of the phase pairs that each candidate distance would produce (their Eq. 4). Guo et al. (2018) cite an analytic solution by Goshov and Solodkin based on the Chinese remainder theorem. Multiple frequencies have costs. The measurements are taken one after another, so moving objects shift between them, and Guo et al. report errors that are strongest along depth discontinuities; some cameras instead assign different frequencies to neighbouring pixels, which costs spatial resolution. Gutierrez-Barragan et al. (2019) describe multi-frequency sinusoid coding as prone to large errors in low signal-to-noise ratio scenarios.

Precision

Noise in the four samples becomes noise in the phase. The following first-order derivation is ours. If each sample has Poisson shot noise (variance equal to its mean, see image noise) and Gaussian read noise of variance $\sigma^2$ , then $\operatorname{var}(I) = c_0 + c_2 + 2\sigma^2 = 2(B + \sigma^2)$ , and the same holds for $Q$ . Linearizing $\operatorname{atan2}$ around the true phase gives

\sigma_\varphi \approx \frac{\sqrt{(B + \sigma^2)/2}}{A}, \qquad \sigma_r = \frac{c\,\sigma_\varphi}{4\pi f}.

Three consequences follow. Precision improves with the returned amplitude $A$ , which is proportional to albedo and falls with the square of distance. Ambient light adds to $B$ and therefore to the noise, even though it cancels from the noise-free phase. And for a fixed phase noise, range noise falls as $1/f$ , which is why cameras want high modulation frequencies. The frequency is limited in practice. Gutierrez-Barragan et al. (2019) describe the bandwidth of the coding functions in most current CW-ToF cameras as typically below 200 MHz, and argue that the depth resolution of a CW-ToF system is determined by its coding functions; they design coding functions under peak-power, bandwidth and binary constraints and report, from simulations and hardware experiments, performance considerably higher than sinusoid and square coding. Su et al. (2018) chose 40 and 70 MHz for their camera because it “achieves a high modulation contrast within this range”, which in our reading means that the modulation depth, and with it $A$ , depends on the frequency; the derivation above assumes it does not.

Direct time of flight with single-photon detectors

A direct ToF camera measures arrival times instead of phases. Gupta et al. (2019) describe the SPAD-based system as a laser sending periodic short pulses and a SPAD that detects the first incident photon in each laser period, after which it is blind for a dead time (about 100 ns in their description). The arrival times of these first photons, recorded over many periods with time-correlated single-photon counting (TCSPC), form a histogram with bins of width $\Delta$ , and with $B$ bins per laser period the unambiguous range is $c B \Delta/2$ . At low photon flux the histogram is a scaled copy of the returning waveform, and the depth comes from its peak. Under strong ambient light, however, the detector is likely to fire on an ambient photon in an early bin of every period, so the histogram is skewed towards earlier bins and the laser peak can shrink to a small blip on a decaying tail. This distortion is called pile-up. It can be inverted computationally by the Coates correction, which Gupta et al. describe as working for relatively low pile-up but amplifying noise at high flux; they derive instead an optimal incident flux and reach it by attenuating the light according to an estimate of the ambient level. Their analysis is framed for SPAD-based pulsed LiDAR, which shares the detector with SPAD array cameras; LiDAR covers flash LiDAR with SPAD receivers.

In our reading, a third design sits between the two. Schober et al. (2017) distinguish modulated ToF cameras, whose illumination and integration profiles are sinusoidal, from pulsed ToF cameras, whose integration profile is more flexible and can change from frame to frame. Both capture a set of raw infrared frames under active illumination and compute depth from them, and Schober et al. state that the two are computationally the same: the camera’s response curve is sine-like for modulated ToF and more general for pulsed ToF.

Gutierrez-Barragan et al. (2019) use the term Impulse ToF (I-ToF) for pulsed systems such as LiDAR and contrast it with CW-ToF; abbreviations of this kind differ between authors, so a source’s own definition should be checked.

Data It Produces

A ToF camera reports a range image, normally with companion images:

Range. A ToF camera measures the length of a path along the pixel’s ray. The GenICam Standard Features Naming Convention (SFNC) gives ToF devices as the example of devices that “operate naturally in spherical coordinates”, reporting a radial distance rather than a Cartesian $Z$ , although the measurement is captured on a planar pixel sensor (SFNC 2.8, §21.2.6, Example 1). A depth map usually stores $Z$ , and the conversion needs only the intrinsics: for a unit ray $\hat{\mathbf d}$ through the pixel, $Z = r\,\hat d_z$ . Su et al. (2018) call the conversion trivial given the intrinsic matrix, and use depth, distance and path length interchangeably in their paper. A dataset’s documentation must therefore say which quantity it stores.
Amplitude. $\hat A$ , the strength of the modulated return, which depends on albedo, incidence angle and range. Naik et al. (2015) note that the correlation keeps the amplitude, phase and ambient light of the received signal.
Offset or intensity. $\hat B$ , the mean of the samples, which includes ambient light. It gives an infrared image registered to the range image by construction.
Validity. The SFNC’s 3D chapter marks invalid pixels with a confidence (mask) image or with an out-of-range value such as 0 or a high value, and discourages NaN (SFNC 2.8, Chapter 21).
Raw correlation frames, when the camera exposes them. Guo et al. (2018) describe the Kinect 2 as having an open-source SDK, libfreenect2, that exposes its raw correlation measurements. libfreenect2’s README describes it as a driver for Kinect for Windows v2 devices supporting IR and depth image transfer and the registration of RGB and depth images (OpenKinect, libfreenect2).

The SFNC classes ToF cameras, with stereo cameras and fixed-pattern triangulation cameras, as Areascan 3D devices that typically acquire a complete range map in a single shot, a 2.5D range image with one range value per sensor coordinate. Back-projected through the intrinsics, the range image becomes an organized point cloud.

Characteristics

The parameters that describe a continuous-wave camera follow from the model above:

Property	What sets it	Source
Ambiguity distance	$c/(2f)$ for one frequency; $c/(2\gcd(f_1, f_2))$ for a pair	Su et al. (2018)
Range precision	$\sigma_r = c\,\sigma_\varphi/(4\pi f)$ , with $\sigma_\varphi$ set by amplitude, offset and read noise	Our derivation
Modulation frequency	Coding-function bandwidth, typically below 200 MHz	Gutierrez-Barragan et al. (2019)
Ambient tolerance	Ambient light adds shot noise through the offset	Our derivation; Gutierrez-Barragan et al. (2019) model it as $I_a$
Motion tolerance	Raw frames are taken in sequence	Guo et al. (2018); Schober et al. (2017)

The modulation frequencies reported in the papers read here vary. Su et al. (2018) used a Texas Instruments OPT8241 development camera that runs at 48 MHz by default and set it to 40 and 70 MHz. Naik et al. (2015) state that their physical experiments used the Kinect One at a modulation frequency of 120 MHz, while Guo et al. (2018) state that the Kinect 2 uses frequencies below 50 MHz. In our reading both papers mean the second Kinect, so the two statements do not agree; neither paper cites a source for its figure, and the papers read here do not settle the difference. Naik et al. describe ToF cameras as delivering depth maps at full frame rates of 30 frames per second. As an example of an older device, Human3.6M recorded a Mesa SR4000 at 176 × 144 pixels and 25 Hz.

Calibration

A ToF camera needs geometric calibration like any camera, to convert range to $Z$ and to back-project pixels (camera calibration). Its range measurement needs calibration of its own. Guo et al. (2018) report that systematic errors are often associated with imperfect sinusoidal modulation and can be reduced through calibration. For the Kinect 2 they calibrated each camera function by placing a Lambertian plane normal to the optical axis at a series of known distances, with a light-absorbing tube in front of the light source to narrow the beam and limit multipath, and dividing out the squared-distance fall-off. The three fitted functions shown in their Fig. 2 were cosines, one of them clipped, and the paper’s characterization also covers shot noise (as a lookup table of empirical distributions), vignetting and a per-pixel time delay, with details in its supplement. Su et al. (2018) calibrated the phase non-linearity of their camera at each of the two frequencies and then treated the measured signal as sinusoidal.

Processing the Data

Su et al. (2018) describe the conventional ToF pipeline as a cascade: denoising, phase unwrapping and multipath correction, each step seeing only its predecessor’s output. Guo et al. (2018) group the methods that address the individual artifacts:

Noise. Shot noise and other random noise are typically addressed by bilateral filtering of the raw measurements, the depth map or both, sometimes guided by other images.
Motion. Moving objects shift between the sequential raw measurements. Guo et al. cite three remedies: fewer sequential measurements, detecting and correcting the affected pixels, or estimating motion between the raw frames and warping them. Schober et al. (2017) instead integrate raw frames over several depth frames with an explicit model of camera and object motion.
Multipath interference (MPI). Light that reaches a pixel along more than one path mixes phases. Naik et al. (2015) define MPI as multiple optical reflections returning to a single pixel, give a two-bounce reflection and subsurface scattering as examples, and survey earlier corrections, most of which assume that the returns are sparse (a few discrete paths), and many of which use measurements at several frequencies. Their own method combines the camera’s measurements with direct and global components of light transport measured with an external near-infrared projector, giving a closed-form correction under a two-bounce approximation. Guo et al. (2018) describe the phasor imaging of Gupta, Nayar, Hullin and Martin (2015) as exploiting the fact that MPI is diminished at much higher modulation frequencies, with two such frequencies but a reduced unambiguous range. Guo et al. also point out that with a single frequency, measurements affected by MPI lie on the same manifold as single-bounce measurements and cannot be identified, while with multiple frequencies they leave it.
Learned reconstruction. Su et al. (2018) train a network that maps dual-frequency raw correlation images to depth, jointly doing denoising, unwrapping and MPI correction, and Guo et al. (2018) a two-stage network for motion and MPI; both train on simulated data, Su et al. because large raw ToF datasets with ground-truth depth did not exist and Guo et al. because ground truth for real scenes is difficult to capture. On their synthetic test set of 9,400 images, Su et al. report an overall mean absolute error of 0.029 m for their combined network on correlation inputs against 0.347 m for a conventional pipeline of bilateral denoising, lookup-table unwrapping and non-linearity correction, which they label PHASOR (their Table 1); the conventional pipeline was more accurate on the pixels they class as low-MPI (0.011 m against 0.025 m).

Mixed pixels. A pixel that straddles a depth edge collects light from both surfaces. For a continuous-wave camera the two returns add as phasors, as in the multipath case of the example below, so the reported range is the phase of the sum, not an average of the two ranges (our derivation). Depth map shows flying pixels from averaging across an edge in the depth domain.

Example: a continuous-wave pixel in a room corner

The script simulates a 160 × 120 continuous-wave ToF camera with a co-located light source, looking into the concave corner of a room (two walls at $z = 5 - |x|$ m), with a floor 1.2 m below the camera and a box in front. Ray casting gives the exact radial range of every pixel. Each pixel’s four correlation samples follow the convention above, with an amplitude proportional to albedo times the cosine of incidence over $r^2$ , an offset equal to the amplitude plus ambient light, Poisson shot noise and 5 electrons of read noise. The camera is run at 80 MHz, at 16 MHz, and with both, unwrapping the 80 MHz range by a lookup over its wrap counts $n = 0, \dots, 4$ , choosing the candidate nearest the 16 MHz range. Light levels are set by an exposure scale $s \in \{1, 0.1, 0.01\}$ , which scales signal and ambient light together, and ambient light $E \in \{0, 20000\}$ electrons per unit albedo. The script compares the measured precision and wrap-error counts with the first-order prediction above, using a Gaussian phase-noise model for the expected counts. It also adds a second light path to the wall pixels of one image row: light that reaches the mirror point on the opposite wall first, travelling $\Delta = 2|x|$ further, with a relative amplitude $a = 0.2$ that we chose rather than computed. It compares the resulting bias with the phasor closed form

b = \frac{c}{4\pi f}\,\arg\!\bigl(1 + a\,e^{i\theta}\bigr), \qquad \theta = \frac{2\pi f \Delta}{c}.

Since the real part of $1 + a e^{i\theta}$ is positive for $a < 1$ , the bias has the sign of $\sin\theta$ and is bounded by $|b| \le c \arcsin(a)/(4\pi f)$ (our derivation). Ranges are radial, along each pixel’s ray; one printed line converts a corner pixel’s range to depth $Z$ .

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"        # one BLAS thread, set before NumPy loads
import math
import numpy as np
from scipy.stats import norm

C = 299_792_458.0                                # speed of light (m/s)
F80, F16 = 80e6, 16e6                            # modulation frequencies (Hz)
U = {F80: C / (2 * F80), F16: C / (2 * F16)}     # ambiguity distance c/(2f) per frequency
DMAX = C / (2 * math.gcd(int(F80), int(F16)))    # unambiguous range of the pair

# Scene: pinhole camera at the origin (x right, y down, z forward), light source at
# the camera. A concave room corner (walls z = 5 - |x|), a floor 1.2 m below the
# camera, and a box. Every pixel stores the radial range r along its ray.
W, H, FPX, CX, CY = 160, 120, 150.0, 79.5, 59.5
u, v = np.meshgrid(np.arange(W), np.arange(H))
ray = np.stack([(u - CX) / FPX, (v - CY) / FPX, np.ones((H, W))], -1)
d = ray / np.linalg.norm(ray, axis=-1, keepdims=True)           # unit viewing rays
PLANES = {"left wall": ((-1, 0, 1), 5.0), "right wall": ((1, 0, 1), 5.0),
          "floor": ((0, 1, 0), 1.2)}                             # n . X = k
BOX = np.array([-0.8, 0.3, 1.4]), np.array([0.0, 1.2, 2.0])     # min and max corners
ALBEDO = {"left wall": 0.7, "right wall": 0.7, "floor": 0.4, "box": 0.25}

def raycast(d):
    r = np.full((H, W), np.inf)
    lab = np.full((H, W), "", object)
    nrm = np.zeros((H, W, 3))
    for name, (n, k) in PLANES.items():
        n = np.array(n, float)
        k, n = k / np.linalg.norm(n), n / np.linalg.norm(n)
        nd = d @ n
        t = np.where(nd > 0, k / np.where(nd > 0, nd, 1.0), np.inf)
        hit = t < r
        r[hit], lab[hit], nrm[hit] = t[hit], name, n
    t0, t1 = BOX[0] / d, BOX[1] / d                               # slab test
    tin, tout = np.minimum(t0, t1).max(-1), np.maximum(t0, t1).min(-1)
    hit = (tin <= tout) & (tin > 0) & (tin < r)
    face = np.minimum(t0, t1).argmax(-1)
    n_box = -np.sign(d) * (np.arange(3) == face[..., None])
    r[hit], lab[hit], nrm[hit] = tin[hit], "box", n_box[hit]
    return r, lab, nrm

r, lab, nrm = raycast(d)
X = r[..., None] * d                                              # 3D points (m)
res = max(np.abs(X[lab == s] @ (np.array(n) / np.linalg.norm(n)) - k / np.linalg.norm(n)).max()
          for s, (n, k) in PLANES.items())
inside = np.all((X[lab == "box"] >= BOX[0] - 1e-9) & (X[lab == "box"] <= BOX[1] + 1e-9))
print("pixels per surface:", {s: int((lab == s).sum()) for s in ALBEDO})
print(f"range {r.min():.3f}-{r.max():.3f} m; U80 = {U[F80]:.4f} m, U16 = {U[F16]:.4f} m, "
      f"pair range c/(2 gcd) = {DMAX:.4f} m; pixels with r >= U80: {int((r >= U[F80]).sum())}")
print(f"max plane residual {res:.1e} m; box points inside the box: {inside}")
print(f"corner pixel (0, 0): range r = {r[0, 0]:.4f} m, depth Z = r * d_z = {X[0, 0, 2]:.4f} m")

# Radiometry (electrons per sample): amplitude A falls as 1/r^2 with albedo and the
# cosine of incidence; offset B = A (full modulation depth) + ambient E * albedo.
A0, READ = 1e5, 5.0
rho = np.vectorize(ALBEDO.get)(lab).astype(float)
cosi = np.abs(np.sum(nrm * d, -1))
A1 = A0 * rho * cosi / r**2

def samples(r, A, B, f, rng=None, a=0.0, delta=0.0):
    """Four correlation samples c_k = B + A cos(phi + k pi/2), phi = 4 pi f r / c,
    plus an optional second path of relative amplitude a and extra length delta."""
    phi = 4 * np.pi * f * r / C
    th = 2 * np.pi * f * delta / C
    c = np.stack([B + A * (np.cos(phi + k * np.pi / 2) + a * np.cos(phi + th + k * np.pi / 2))
                  for k in range(4)])
    if rng is not None:                       # Poisson shot noise + Gaussian read noise
        c = rng.poisson(c).astype(float) + rng.normal(0.0, READ, c.shape)
    return c

def decode(c, f):
    I, Q = c[0] - c[2], c[3] - c[1]           # 2A cos(phi), 2A sin(phi)
    phi = np.mod(np.arctan2(Q, I), 2 * np.pi)
    return U[f] * phi / (2 * np.pi), np.hypot(I, Q) / 2, c.mean(0)

def dual(r80, r16):
    """Lookup over the wrap counts n of the 80 MHz range, nearest to the 16 MHz range."""
    cand = r80[..., None] + np.arange(round(DMAX / U[F80])) * U[F80]
    k = np.abs(cand - r16[..., None]).argmin(-1)
    return np.take_along_axis(cand, k[..., None], -1)[..., 0]

def werr(e, Uf):                              # error folded into [-U/2, U/2)
    return np.mod(e + Uf / 2, Uf) - Uf / 2

print("\nnoise-free checks (largest absolute errors)")
for E in (0.0, 20000.0):
    B = A1 + E * rho
    c80 = samples(r, A1, B, F80)
    r80, Ah, Bh = decode(c80, F80)
    r16 = decode(samples(r, A1, B, F16), F16)[0]
    rd = dual(r80, r16)
    flip = U[F80] * np.mod(np.arctan2(c80[1] - c80[3], c80[0] - c80[2]), 2 * np.pi) / (2 * np.pi)
    print(f"E={E:>7.0f}: 80 MHz |r^-(r mod U)| {np.abs(werr(r80 - r, U[F80])).max():.1e} m, "
          f"wrap errors {int((np.abs(r80 - r) > U[F80] / 2).sum())}; 16 MHz {np.abs(r16 - r).max():.1e} m; "
          f"dual {np.abs(rd - r).max():.1e} m; amplitude {np.abs(Ah / A1 - 1).max():.1e}, "
          f"offset {np.abs(Bh / B - 1).max():.1e} (relative)")
print(f"swapped arctan2 order: |r_swap - (U - r^)| {np.abs(werr(flip + r80, U[F80])).max():.1e} m, "
      f"RMS error against truth {np.sqrt(np.mean(werr(flip - r, U[F80])**2)):.3f} m")

def expected_wraps(phi_w, n, sphi, sdiff):
    """Expected wrap-error counts and 3-sigma + 1 tolerances (Gaussian phase noise)."""
    lo, hi = norm.cdf(-phi_w / sphi), norm.cdf(-(2 * np.pi - phi_w) / sphi)
    p = {F80: np.where(n == 0, lo + hi, np.where(n == 1, 1 - lo, 1.0)),
         F16: norm.cdf(-4 * np.pi * F16 * r / C / sphi)
              + norm.cdf(-(2 * np.pi - 4 * np.pi * F16 * r / C) / sphi),
         "dual": 2 * norm.cdf(-(U[F80] / 2) / sdiff)}
    return {m: (q.sum(), 3 * np.sqrt(np.sum(q * (1 - q))) + 1) for m, q in p.items()}

rng = np.random.default_rng(0)
phi80_true, n80 = 2 * np.pi * np.mod(r, U[F80]) / U[F80], np.floor(r / U[F80])
prec, wraps = [], []
for E in (0.0, 20000.0):
    for s in (1.0, 0.1, 0.01):                # exposure scale: signal and ambient both scale
        A = s * A1
        B = A + s * E * rho
        sphi = np.sqrt((B + READ**2) / 2) / A                    # predicted phase noise (rad)
        est = {f: decode(samples(r, A, B, f, rng), f)[0] for f in (F80, F16)}
        est["dual"] = dual(est[F80], est[F16])
        rms = {f: np.sqrt(np.mean(werr(est[f] - r, U[f])**2)) for f in (F80, F16)}
        pred = {f: np.sqrt(np.mean((U[f] / (2 * np.pi) * sphi)**2)) for f in (F80, F16)}
        sd = np.sqrt(sum((U[f] / (2 * np.pi) * sphi)**2 for f in (F80, F16)))
        exp = expected_wraps(phi80_true, n80, sphi, sd)
        bad = {m: np.abs(est[m] - r) > (U[F16] if m == F16 else U[F80]) / 2 for m in (F80, F16, "dual")}
        ok = ~bad["dual"]
        prec.append(f"{E:>7.0f} {s:>5} | {1e3*rms[F80]:8.3f} {1e3*pred[F80]:8.3f} {rms[F80]/pred[F80]:6.3f} | "
                    f"{1e3*rms[F16]:8.3f} {1e3*pred[F16]:8.3f} {rms[F16]/pred[F16]:6.3f} | {rms[F16]/rms[F80]:6.3f}")
        wraps.append(f"{E:>7.0f} {s:>5} | " + " | ".join(
            f"{int(bad[m].sum()):5d} {exp[m][0]:8.1f} {exp[m][1]:5.1f}" for m in (F80, F16, "dual"))
            + f" | {1e3*np.sqrt(np.mean((est['dual'][ok] - r[ok])**2)):7.2f} {1e3*np.sqrt(np.mean((est['dual'] - r)**2)):8.1f}")
print("\nprecision: RMS of the folded range error, measured and predicted (mm)")
print("      E     s |  80 MHz     pred  ratio |  16 MHz     pred  ratio | 16/80")
print("\n".join(prec))
print("\nwrap errors: observed, expected, tolerance; dual-frequency RMS (mm) without and with them")
print("      E     s |  80 MHz             |  16 MHz             |  dual               | RMS ok  RMS all")
print("\n".join(wraps))

# Multipath: pixels of image row 40 on the two walls. The second path goes camera ->
# mirror point on the other wall -> this point -> camera, Delta = 2|x| longer.
row, a = 40, 0.2
m = np.isin(lab[row], ["left wall", "right wall"])
rr, delta = r[row, m], 2 * np.abs(X[row, m, 0])
print(f"\nmultipath on row {row}: {m.sum()} wall pixels, a = {a}, Delta {delta.min():.3f}-{delta.max():.3f} m")
bias, closed = {}, {}
for f in (F80, F16):
    est = decode(samples(rr, A1[row, m], A1[row, m], f, a=a, delta=delta), f)[0]
    bias[f] = werr(est - rr, U[f])
    closed[f] = U[f] / (2 * np.pi) * np.angle(1 + a * np.exp(2j * np.pi * f * delta / C))
pick = [np.abs(delta - t).argmin() for t in (0.1, 0.5, 1.0, 1.5, 2.5, 3.4)]
print(" Delta (m) | bias 80 MHz  closed form | bias 16 MHz  closed form   (mm)")
for i in pick:
    print(f"   {delta[i]:6.3f}  | {1e3*bias[F80][i]:10.3f} {1e3*closed[F80][i]:11.3f} | "
          f"{1e3*bias[F16][i]:10.3f} {1e3*closed[F16][i]:11.3f}")
for f in (F80, F16):
    th = 2 * np.pi * f * delta / C
    print(f" {f/1e6:.0f} MHz: max |sim - closed| {np.abs(bias[f] - closed[f]).max():.1e} m; "
          f"negative biases {int((bias[f] < 0).sum())} (pixels with sin(theta) < 0: {int((np.sin(th) < 0).sum())}); "
          f"max |bias| {1e3*np.abs(bias[f]).max():.3f} mm, bound c asin(a)/(4 pi f) {1e3*U[f]*np.arcsin(a)/(2*np.pi):.3f} mm")

Output (NumPy 2.3.5, SciPy 1.16.3, seed 0; the script runs in under half a second):

pixels per surface: {'left wall': 6645, 'right wall': 8448, 'floor': 1152, 'box': 2955}
range 1.432-5.361 m; U80 = 1.8737 m, U16 = 9.3685 m, pair range c/(2 gcd) = 9.3685 m; pixels with r >= U80: 16442
max plane residual 8.9e-16 m; box points inside the box: True
corner pixel (0, 0): range r = 3.9192 m, depth Z = r * d_z = 3.2680 m

noise-free checks (largest absolute errors)
E=      0: 80 MHz |r^-(r mod U)| 8.9e-16 m, wrap errors 16442; 16 MHz 1.8e-15 m; dual 8.9e-16 m; amplitude 8.9e-16, offset 1.1e-15 (relative)
E=  20000: 80 MHz |r^-(r mod U)| 1.3e-15 m, wrap errors 16442; 16 MHz 4.4e-15 m; dual 1.8e-15 m; amplitude 1.7e-15, offset 4.4e-16 (relative)
swapped arctan2 order: |r_swap - (U - r^)| 4.4e-16 m, RMS error against truth 0.531 m

precision: RMS of the folded range error, measured and predicted (mm)
      E     s |  80 MHz     pred  ratio |  16 MHz     pred  ratio | 16/80
      0   1.0 |    3.983    4.025  0.990 |   20.204   20.125  1.004 |  5.072
      0   0.1 |   13.461   13.499  0.997 |   68.682   67.494  1.018 |  5.102
      0  0.01 |   67.094   61.998  1.082 |  346.396  309.990  1.117 |  5.163
  20000   1.0 |   10.353   10.366  0.999 |   51.702   51.829  0.998 |  4.994
  20000   0.1 |   33.802   33.086  1.022 |  170.812  165.432  1.033 |  5.053
  20000  0.01 |  127.042  113.880  1.116 |  635.658  569.402  1.116 |  5.004

wrap errors: observed, expected, tolerance; dual-frequency RMS (mm) without and with them
      E     s |  80 MHz             |  16 MHz             |  dual               | RMS ok  RMS all
      0   1.0 | 16441  16443.2   7.4 |     0      0.0   1.0 |     0      0.0   1.0 |    3.98      4.0
      0   0.1 | 16449  16444.7  12.0 |     0      0.0   1.0 |     0      0.0   1.1 |   13.46     13.5
      0  0.01 | 16445  16442.4  24.4 |     4      0.3   2.7 |   361    303.3  47.2 |   62.13    286.7
  20000   1.0 | 16444  16444.3  10.5 |     0      0.0   1.0 |     0      0.0   1.2 |   10.35     10.4
  20000   0.1 | 16450  16443.6  16.4 |     0      0.0   1.0 |    47     33.2  17.4 |   33.12     98.9
  20000  0.01 | 16441  16446.2  28.4 |    36     14.5  12.2 |  1809   1595.5 103.8 |  110.41    654.6

multipath on row 40: 160 wall pixels, a = 0.2, Delta 0.033-3.464 m
 Delta (m) | bias 80 MHz  closed form | bias 16 MHz  closed form   (mm)
    0.099  |      8.230       8.230 |      8.250       8.250
    0.476  |     37.291      37.291 |     39.589      39.589
    0.991  |     59.683      59.683 |     81.735      81.735
    1.501  |     41.344      41.344 |    122.157     122.157
    2.519  |    -57.386     -57.386 |    195.702     195.702
    3.407  |    -27.533     -27.533 |    248.129     248.129
 80 MHz: max |sim - closed| 1.2e-15 m; negative biases 90 (pixels with sin(theta) < 0: 90); max |bias| 60.038 mm, bound c asin(a)/(4 pi f) 60.047 mm
 16 MHz: max |sim - closed| 1.4e-15 m; negative biases 0 (pixels with sin(theta) < 0: 0); max |bias| 251.027 mm, bound c asin(a)/(4 pi f) 300.234 mm

Noise-free checks. Without noise every decoder is exact to rounding error, with and without ambient light, so the ambient offset cancels as the derivation says. At 80 MHz the noise-free camera is nonetheless wrong on 16,442 of the 19,200 pixels, every pixel beyond 1.874 m, by a whole multiple of the ambiguity distance. The 16 MHz phase and the dual scheme recover the true range everywhere, since the room fits within 9.369 m. Decoding with the arctangent arguments in the other order returns exactly $U - \hat r$ , which here is a 0.531 m RMS error: the phase convention has to match the one the samples were taken in. The amplitude and offset images are also recovered exactly. Every data and noise-free check held.

Precision. At every light level the 16 MHz error is close to five times the 80 MHz error, and each printed 16/80 ratio lies within the planned band of 4.5 to 5.5 (hypothesis H1 held). Both frequencies see the same phase noise, so the range noise scales with $c/(4\pi f)$ . Ambient light at the full exposure raises the 80 MHz error from about 4 mm to about 10 mm, because it adds shot noise to the offset. The measured-to-predicted ratio was planned to lie between 0.9 and 1.1. At 80 MHz (H2a) it held at five conditions and failed at $E = 20000$ , $s = 0.01$ (1.116). At 16 MHz (H2b) it held at four conditions and failed at both $s = 0.01$ conditions (1.117 without ambient light, 1.116 with it).

Wrap errors. Observed counts were planned to lie within $3\sqrt{\sum p(1-p)} + 1$ of the counts expected from the Gaussian phase-noise model. At 80 MHz alone the count stays near 16,442 at every light level, since almost all of it comes from the true wrap, and it held within tolerance at all six conditions (H3a held). At 16 MHz alone (H3b) the count was 0 as expected at the four brighter conditions (held) and failed at both $s = 0.01$ conditions: 4 against an expected 0.3 (tolerance 2.7) without ambient light, and 36 against 14.5 (tolerance 12.2) with it. The dual scheme (H4a) made no wrap errors at $s = 1$ and at $s = 0.1$ without ambient light, and 47 against 33.2 (tolerance 17.4) at $s = 0.1$ with ambient light, all held; it failed at both $s = 0.01$ conditions, 361 against 303.3 (tolerance 47.2) and 1,809 against 1,595.5 (tolerance 103.8). The count rose as the exposure fell under both ambient settings (H4b held). A wrong wrap count costs a multiple of 1.874 m, so a small number of such pixels dominates the RMS error: at $s = 0.1$ with ambient light, 47 wrong pixels raise the dual-frequency RMS from 33.12 mm to 98.9 mm.

Every failure is at the lowest exposure, and in every failing case the measured value is larger than predicted. The plan recorded before the pilot that “the first-order model is not expected to hold where σ_φ approaches 1 rad” and that “the two s = 0.01 conditions may fail, and that is reported as printed”. In our reading, the predictions linearize $\operatorname{atan2}$ around the true phase and treat the phase error as Gaussian, which by construction describes only small phase errors; the dimmest pixels at $s = 0.01$ are where the noise is no longer small compared with the amplitude.

Multipath. The simulated biases match the closed form to rounding error at both frequencies (H5a held). For a short extra path the bias is the same at both frequencies (8.230 and 8.250 mm at $\Delta = 0.099$ m): for small $\theta$ the closed form tends to $a\Delta/(2(1+a))$ , independent of $f$ (our derivation). In our reading, comparing the two frequencies therefore reveals little about short extra paths. For longer paths the two estimates separate. At 16 MHz the bias keeps growing, to 248 mm at $\Delta = 3.4$ m; at 80 MHz it stays within its bound of 60.047 mm and changes sign once $\Delta$ exceeds the 80 MHz ambiguity distance, so that 90 pixels are biased short (H5b and H5c held). A higher frequency thus bounds the multipath error more tightly in this two-path model, but the error does not vanish, and its sign depends on the geometry.

How the example was planned. The scene, light levels and criteria were fixed in a plan before the pilot run, and the first run is the pilot script unchanged, with the same seed and output. Three planning scripts ran before the criteria were written, none of them drawing noise. The first used a box farther away, which put every pixel beyond the 80 MHz ambiguity distance; the plan’s log records “Changed before any criterion: the box was moved nearer and made taller”. The second computed the predicted precision and the expected wrap counts with their tolerances, and the third evaluated the noise-free model, including the multipath closed form. Because the predicted values behind H1–H5 were printed before those criteria were written (the second script printed H1’s predicted ratio of 5, which also follows from the derivation), these five hypotheses are illustrations of the derivations rather than blind tests. The example departs from the batch plan’s description of it in eight ways:

Wrap errors are printed as counts out of 19,200 pixels, each with an expected count and tolerance from a Gaussian phase-noise model.
The noise model and light levels (Poisson plus read noise, three exposures, two ambient levels) are our choices.
Single-frequency RMS errors are folded into $\pm U/2$ to measure precision; the dual-frequency RMS error is given without and with wrap errors.
Precision against frequency is shown with the two frequencies only, not with a sweep.
The second light path is our geometry (the mirror point on the other wall, one image row, $a = 0.2$ ) and is noise-free.
A printed line checks the phase convention by swapping the arctangent’s arguments; the Kinect 2’s three phases per frequency appear in the text only.
Data checks, an amplitude and offset check, and the range-to-depth line were added.
The box was moved between the first two planning scripts, as described above.

Strengths

Dense depth from one viewpoint. Every pixel measures range directly, with no matching between views, so textureless surfaces are measured as well as textured ones (our reading of the measurement model). The light source sits beside the lens, so the baseline between illumination and sensor is small (Su et al., 2018).
Compact and low-power. Su et al. (2018) cite low power requirements, low-cost CMOS sensor technology and a small baseline between sensor and illumination; Gutierrez-Barragan et al. (2019) contrast low-power, low-cost CW-ToF cameras with pulsed systems that need high-speed lasers and detectors.
Registered intensity. The amplitude and offset images come from the same pixels as the range.
Precision that does not degrade with baseline geometry. Range noise depends on the returned light and the modulation frequency (derivation above), not on a triangulation angle, unlike stereo, whose depth error grows with the square of depth.

Limitations

Phase wrapping. A single frequency gives range only modulo $c/(2f)$ ; the example’s 80 MHz camera misplaces 16,442 of its 19,200 pixels without noise. Two-frequency unwrapping fails when the noise in the coarse-frequency range approaches half the fine frequency’s ambiguity distance (our derivation), and each failure is a gross error.
Multipath interference. Concave corners and translucent materials (Naik et al.’s subsurface scattering) return light along several paths. Naik et al. (2015) call MPI perhaps the main source of error in ToF cameras.
Ambient light. Sunlight adds shot noise to every sample. In SPAD cameras it also causes pile-up (Gupta et al., 2019).
Motion. Sequential raw frames see a moving scene at different times, which causes errors at moving depth edges (Guo et al., 2018).
Dark, distant and oblique surfaces. The amplitude falls with albedo, the cosine of incidence and $1/r^2$ , and precision with it. Su et al. (2018) note that their amplitude normalization amplifies noise in regions where reflectivity is low or distances are large.
Mixed pixels at depth edges (above).
Several cameras. Cameras of the same kind can interfere; Gutierrez-Barragan et al. (2019) list coding designs that mitigate multi-camera interference among earlier work.

Typical Applications

Su et al. (2018) list interactive commercial applications, scene reconstruction from RGB-D data and the 3D datasets built with it, action recognition, face and pose tracking, human–computer interaction, robotics and augmented and virtual reality; Gutierrez-Barragan et al. (2019) name autonomous navigation, augmented reality and user interfaces. Examples on this site include the time-of-flight depth camera of HoloLens 2 in augmented reality, the ToF stream of Human3.6M, and depth cameras as input to 3D reconstruction. The second Kinect, a continuous-wave ToF camera, is described in depth estimation.

Comparison with Other Sensors

Sensor	Principle	Typical strengths	Typical weaknesses
Continuous-wave ToF camera	Phase of modulated light, per pixel	Dense, compact, no texture needed	Wrapping, multipath, ambient light, motion between raw frames
SPAD (direct ToF) camera	Photon arrival-time histograms	Fine timing resolution	Pile-up under strong ambient light
LiDAR	Pulsed or FMCW laser ranging, scanned or flash	Long range, accurate	Sparse, mechanical or costly scanning
Stereo camera	Triangulation of matched pixels	Can work passively, including in sunlight	Needs texture, error grows with $Z^2$
Structured light	Triangulation against a projected pattern	High accuracy at short range	Projector–camera baseline, shadows

The LiDAR article describes AMCW ranging as limited to moderate ranges by its short ambiguity distance and mostly suited to indoor use, after Royo and Ballesta-Garcia, and as the principle of most time-of-flight cameras. Su et al. (2018) note that hybrid structured-light and ToF systems need coded, synchronized illumination and a projector–camera baseline, which removes many of ToF’s inherent benefits. For the families of depth sensing, active and passive, see depth estimation.