Foundations

Thermal and Infrared Camera

Cameras that image infrared light, from near-infrared cameras built on silicon to long-wave thermal cameras that see the heat objects emit, with Planck's law, emissivity, NETD and temperature measurement

beginner

An infrared camera images light with wavelengths longer than red. The name covers two quite different devices. A near-infrared (NIR) camera is an ordinary silicon image sensor used just beyond the visible, sometimes with its own infrared illumination; it still images reflected light. A thermal camera, also called a long-wave infrared (LWIR) or far-infrared (FIR) camera, works at wavelengths around 10 µm, where objects at everyday temperatures emit most strongly, so it images the heat that surfaces radiate themselves and works without any light source. A thermal image looks like a grey-level photograph, but its values depend on temperature and on a surface property, emissivity, that a colour camera never sees.

How It Works

Bands

Sources divide the infrared differently. Hwang et al. (2015), describing the cameras used for pedestrian detection in cars, give near infrared as 0.75–1.3 µm and long-wavelength infrared as 7.5–13 µm, “also known as the thermal band”. Gade et al. (2013) give the mid-wavelength infrared (MWIR) as 3–5 µm and the LWIR as 8–15 µm, and Saponaro et al. (2015) work between 8 and 14 µm (printed “8-14m” in their paper, a typo for micrometres). Bijelic et al. (2020) call their thermal camera an FIR camera. The example below uses 8–14 µm.

Silicon stops at about 1.1 µm: image sensor derives this from the band gap, so a silicon chip sees visible and near-infrared light but not the thermal infrared. Levoy notes that silicon is sensitive to NIR, that most sensors carry a filter to block it, and that removing the filter turns the camera into a NIR camera (Photons and Sensors, slide 30). Iris cameras are an example: biometrics describes Daugman’s systems imaging the iris under 700–900 nm illumination. Bijelic et al. use a gated NIR camera: a flash laser at 808 nm illuminates the scene, and the camera captures the echo after an adjustable delay, which, they write, significantly reduces backscatter from particles in adverse weather. Many LiDAR units also work in the near infrared. Between the NIR and the thermal bands lies the short-wave infrared, which multispectral satellite imaging uses for reflective bands.

Emitted radiation and Planck’s law

Every object above absolute zero emits infrared radiation, described by Planck’s law (Gade et al., 2013; Saponaro et al., 2015). For an ideal emitter, a blackbody, at temperature TT, the spectral radiance is

Bλ(T)=2hc2λ5 1exp⁡ ⁣(hcλkT)−1,B_\lambda(T) = \frac{2hc^2}{\lambda^5}\,\frac{1}{\exp\!\left(\dfrac{hc}{\lambda k T}\right) - 1},

in watts per square metre per steradian per metre of wavelength, with Planck’s constant hh, the speed of light cc and Boltzmann’s constant kk. Gade et al. write the same law with a factor 2π2\pi instead of 2 and call the quantity intensity; in our derivation, their form is the spectral radiant exitance, the power leaving a unit area into the whole hemisphere, which for a surface whose radiance is the same in all directions is π\pi times the radiance (radiometry and photometry defines both). This article uses radiance, which the lens turns into the irradiance a pixel integrates (image formation). The spectrum peaks at λmax⁡=b/T\lambda_{\max} = b/T with Wien’s constant b≈2.898×10−3b \approx 2.898 \times 10^{-3} m K, about 9.7 µm at 300 K. Hwang et al. state, citing other work, that the human body radiates at 9.3 µm, and give this as support for thermal cameras in capturing people. As Gade et al. put it, the peak moves to shorter wavelengths as temperature rises, and for extremely hot objects the radiation extends into the visible.

A thermal camera integrates the radiance over its band. In our derivation, a band radiance L(T)=∫8 μm14 μmBλ(T) dλL(T) = \int_{8\,\mu\text{m}}^{14\,\mu\text{m}} B_\lambda(T)\,d\lambda near room temperature changes by about 1.5% per kelvin (computed below), so temperature contrasts of a fraction of a degree are small changes on a large signal.

Emissivity and reflection

Real surfaces emit less than a blackbody. Saponaro et al. define emissivity as a material’s effectiveness at emitting thermal radiation compared with a blackbody at the same temperature, and note that the camera receives emitted radiation and also reflected and transmitted radiation; metals have very low emissivity and high reflectance, so many look mirror-like in LWIR images, while wood has high emissivity. Gade et al. likewise report reflections in the floor in their thermal images of sports halls.

The example uses the simplest model consistent with this, which is ours: an opaque grey surface (emissivity ε\varepsilon independent of wavelength and direction, no transmission) seen through no atmosphere, which reflects the fraction 1−ε1-\varepsilon of radiation arriving from surroundings at a reflected apparent temperature TrT_r:

Lmeas=ε L(T)+(1−ε) L(Tr).L_{\text{meas}} = \varepsilon\,L(T) + (1-\varepsilon)\,L(T_r).

That reflectivity equals 1−ε1-\varepsilon for an opaque body is Kirchhoff’s law, standard physics in our derivation. Measuring temperature means inverting this equation, which needs ε\varepsilon and TrT_r.

Detectors

Gade et al. (Section 1.2) divide thermal-camera detectors into two kinds. Photon detectors convert absorbed radiation directly into a change of free charge carriers in a semiconductor; they typically work in the MWIR, are very sensitive, and need cooling, which makes them more expensive and adds maintenance. Thermal detectors absorb the radiation as heat and sense the rise in their own temperature through a material property, such as the temperature-dependent resistance of a bolometer; they work in the LWIR without cooling, and of the two uncooled types, ferroelectric detectors and microbolometers, the authors write that the microbolometer has shown more advantages. Among the cameras in the papers cited here, the Axis Q1922 uses an uncooled microbolometer (Gade et al.; Bijelic et al. use the same model), and Saponaro et al. describe their Xenics Gobi 640 as an uncooled LWIR camera; Hwang et al. used a FLIR A35. A survey by Gade and Moeslund (2014) covers thermal cameras and their applications.

Data It Produces

A thermal camera delivers a single-channel image whose values encode received radiance; Gade et al. describe the result as an image whose pixel values represent temperature; in our reading this holds only after a radiometric calibration and under the emissivity assumptions above. The thermal cameras in the cited papers have modest resolutions: 640 × 480 pixels at 30 Hz with a 17 µm pixel pitch for the Axis Q1922 (Bijelic et al.), 640 × 480 for the Xenics Gobi 640, and 320 × 256 pixels at 20 frames per second with a 39° vertical field of view for the FLIR A35 (Hwang et al.). Gade et al. wrote in 2013 that commercial thermal lenses had a field of view of at most around 60°, so they stitched three cameras to cover a sports court. As histogram equalization notes, high-bit-depth thermal data must be mapped to 8 bits for display.

Characteristics

The usual figure of merit is the noise-equivalent temperature difference (NETD). Bijelic et al. give the Axis Q1922 an NETD below 100 mK, and Saponaro et al. give the Gobi 640 a “50mC sensitivity at 30°C” (as printed: 50 millidegrees, which as a temperature difference equals 50 mK). In our reading of the name, NETD is the blackbody temperature change whose signal change equals the standard deviation of the noise, at a stated scene temperature. Converting it to radiance needs the slope of the band radiance, σL=NETD⋅dL/dT\sigma_L = \text{NETD} \cdot dL/dT, and converting radiance noise back to temperature on a surface of emissivity ε\varepsilon gives, in our derivation,

σT=σLε dL/dT(T),\sigma_T = \frac{\sigma_L}{\varepsilon \, dL/dT(T)},

because only the emitted part of the signal carries the temperature. A low-emissivity surface is therefore both hard to measure (a wrong ε\varepsilon or TrT_r biases it) and noisy once corrected.

Calibration

Geometric calibration follows camera calibration, but, as both Gade et al. and Hwang et al. note, a conventional checkerboard is not visible to a thermal camera. Gade et al. built a board with a 5 × 4 grid of small incandescent bulbs; Hwang et al. used a board with holes, which, once heated, differs in temperature from the holes. Hwang et al. also aligned a colour and a thermal camera physically: a beam splitter of zinc-coated silicon transmits the thermal band to the thermal camera and reflects the visible band to the colour camera, and after a stereo calibration the two cameras were moved on a three-axis jig until their translation was zero, so that the two images share a focal length and principal point and have no parallax. The splitter’s uneven visible reflectance tints the colour image, which they corrected by dividing by an image of a white plane.

Radiometric calibration maps counts to radiance and temperature. For a satellite thermal instrument, multispectral satellite imaging describes how Landsat’s TIRS subtracts its own thermal emission by viewing cold space and checks its calibration against an onboard blackbody; the example below assumes this step has been done and starts from radiance.

Processing the Data

Thermal images are often paired with colour images. Unless the cameras share an optical path, as with Hwang et al.’s beam splitter, the pair must be registered, a case of multimodal image registration, and feature matching lists visible–thermal matching among its cases of extreme appearance change. Paired data then feed pedestrian detection and multimodal fusion: Hwang et al. annotated 95,328 aligned colour–thermal pairs recorded from a car by day and night, and Bijelic et al. recorded an FIR camera alongside colour cameras, a gated camera, lidar and radar in their adverse-weather driving dataset, although their fusion network uses the colour camera, lidar, gated camera and radar streams. Thermal images are also translated into colour images by image-to-image translation. For temperature measurement, the per-pixel inversion below is the core step.

From Radiance to Temperature

The script builds a 240 × 320 synthetic thermal image with five regions: a painted wall (295 K, ε=0.95\varepsilon = 0.95), a road (290 K, 0.93), a warm disk with a skin-like emissivity (307 K, 0.98), a warm bare-metal plate (330 K, 0.10) and a bare-metal plate at room temperature (295 K, 0.10). The temperatures and emissivities are illustrative values, not measurements. It computes the 8–14 µm band radiance by Gauss–Legendre quadrature, forms each pixel’s measured radiance with the model above for a reflected temperature of 260 K (outdoors, an illustrative cold sky) and of 295 K (indoors, surroundings at the wall’s temperature), and adds Gaussian noise equal to a 50 mK NETD referred to 30 °C. It then recovers temperature by bisection, once with the true ε\varepsilon and TrT_r and once assuming ε=1\varepsilon = 1, as an uncalibrated reading would. It also checks Wien’s law, the quadrature against scipy.integrate.quad, the analytic slope against a finite difference and the bisection against scipy.optimize.brentq, and compares the radiance a silicon sensor could collect, below 1.1 µm, with the thermal band. Constants come from scipy.constants (CODATA 2022 in SciPy 1.16.3); the code uses NumPy 2.3.5 and runs in about 5 seconds.

import os
os.environ["OMP_NUM_THREADS"] = "1"   # one BLAS thread, before NumPy is imported
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np
from scipy import constants as sc
from scipy.integrate import quad
from scipy.optimize import brentq

h, c, k = sc.h, sc.c, sc.k            # CODATA 2022 values in SciPy 1.16.3

def B(lam, T):
    """Planck spectral radiance, W m^-2 sr^-1 per metre of wavelength."""
    return 2 * h * c**2 / lam**5 / np.expm1(h * c / (lam * k * T))

def dBdT(lam, T):
    x = h * c / (lam * k * T)
    return B(lam, T) * x * np.exp(x) / np.expm1(x) / T

# Band radiance over 8-14 um by 64-node Gauss-Legendre quadrature, vectorized over pixels.
a, b = 8e-6, 14e-6
xg, wg = np.polynomial.legendre.leggauss(64)
lam = 0.5 * (b - a) * xg + 0.5 * (b + a)
wq = 0.5 * (b - a) * wg

def L(T):                              # W m^-2 sr^-1
    T = np.asarray(T, float)
    return (B(lam, T[..., None]) * wq).sum(-1)

def dL(T):                             # W m^-2 sr^-1 K^-1
    T = np.asarray(T, float)
    return (dBdT(lam, T[..., None]) * wq).sum(-1)

def invert(target):
    """Temperature whose band radiance equals target: bisection on [150, 450] K."""
    lo = np.full(np.shape(target), 150.0)
    hi = np.full(np.shape(target), 450.0)
    for _ in range(60):
        mid = 0.5 * (lo + hi)
        up = L(mid) < target
        lo, hi = np.where(up, mid, lo), np.where(up, hi, mid)
    return 0.5 * (lo + hi)

# 1. Wien peak: argmax of B on a 1e-5 um grid against b/T.
grid = np.arange(5.0, 15.0, 1e-5) * 1e-6
for T in (300.0, 307.0, 310.0):
    peak = grid[np.argmax(B(grid, T))] * 1e6
    wien = sc.Wien / T * 1e6
    print(f"Wien peak {T:.0f} K: grid {peak:.6f} um, b/T {wien:.6f} um, diff {abs(peak - wien):.3e}")

# 2. Band radiance, its slope, and the radiance step of a stated NETD at 30 C.
T0 = 303.15
Lq = quad(lambda l: B(l, T0), a, b, epsabs=0, epsrel=1e-13, limit=200)[0]
fd = (L(T0 + 1e-3) - L(T0 - 1e-3)) / 2e-3
print(f"L(303.15 K) = {L(T0):.4f} W/m^2/sr, rel diff to quad {abs(L(T0) - Lq) / Lq:.3e}")
print(f"dL/dT = {dL(T0):.5f} W/m^2/sr/K, rel diff to central difference {abs(dL(T0) - fd) / fd:.3e}")
print(f"relative sensitivity {100 * dL(T0) / L(T0):.3f} %/K")
for netd in (0.050, 0.100):
    step = dL(T0) * netd
    print(f"NETD {1000 * netd:.0f} mK -> radiance step {step:.5f} W/m^2/sr ({100 * step / L(T0):.4f} % of L)")

# 3. What a silicon sensor would see: radiance below 1.1 um against the 8-14 um band.
for T in (290.0, 295.0, 307.0, 330.0, 1000.0):
    Ls = quad(lambda l: B(l, T), 0.1e-6, 1.1e-6, epsabs=0, epsrel=1e-10, limit=200)[0]
    Lb = quad(lambda l: B(l, T), a, b, epsabs=0, epsrel=1e-10, limit=200)[0]
    print(f"T {T:6.0f} K: L(0.1-1.1 um) {Ls:.3e}, L(8-14 um) {Lb:.3e}, ratio {Ls / Lb:.3e}")

# 4. A synthetic thermal scene: labels, true temperatures and emissivities.
H, W = 240, 320
rr, cc = np.mgrid[:H, :W]
lab = np.zeros((H, W), int)                       # 0: painted wall (background)
lab[180:, :] = 1                                  # 1: road
lab[(rr - 100)**2 + (cc - 80)**2 <= 45**2] = 2    # 2: warm disk, skin-like emissivity
lab[40:120, 150:230] = 3                          # 3: warm bare-metal plate
lab[40:120, 245:315] = 4                          # 4: bare-metal plate at room temperature
names = ["wall", "road", "warm disk", "warm metal", "ambient metal"]
T_reg = np.array([295.0, 290.0, 307.0, 330.0, 295.0])
e_reg = np.array([0.95, 0.93, 0.98, 0.10, 0.10])
T_img, e_img = T_reg[lab], e_reg[lab]
counts = np.bincount(lab.ravel(), minlength=5)
print("pixels per region:", dict(zip(names, counts.tolist())),
      "| one label per pixel:", counts.sum() == H * W,
      "| maps match table:", all(np.all(T_img[lab == i] == T_reg[i]) and np.all(e_img[lab == i] == e_reg[i])
                               for i in range(5)))

sigma_L = dL(303.15) * 0.050                      # noise: NETD 50 mK referred to 30 C
for T_refl in (260.0, 295.0):
    print(f"\nreflected apparent temperature {T_refl:.0f} K")
    # Noise-free region values: opaque grey surface, no atmosphere.
    L_reg = e_reg * L(T_reg) + (1 - e_reg) * L(T_refl)
    T_known = invert((L_reg - (1 - e_reg) * L(T_refl)) / e_reg)
    T_naive = invert(L_reg)
    T_brent = np.array([brentq(lambda T: float(L(T)) - t, 150, 450, xtol=1e-12)
                        for t in (L_reg - (1 - e_reg) * L(T_refl)) / e_reg])
    print(f"max |brentq - bisection| = {np.abs(T_brent - T_known).max():.3e} K")
    # Noisy image, same seed for both reflected temperatures.
    rng = np.random.default_rng(0)
    L_img = e_img * L(T_img) + (1 - e_img) * L(T_refl) + rng.normal(0, sigma_L, (H, W))
    Tk_img = invert((L_img - (1 - e_img) * L(T_refl)) / e_img)
    Tn_img = invert(L_img)
    print("region         T(K)   eps   err known eps  err eps=1   std known  predicted  ratio   std eps=1")
    for i, n in enumerate(names):
        m = lab == i
        sk, sn = Tk_img[m].std(), Tn_img[m].std()
        pred = sigma_L / (e_reg[i] * dL(T_reg[i]))
        print(f"{n:13s} {T_reg[i]:5.0f} {e_reg[i]:5.2f} {T_known[i] - T_reg[i]:12.3e} "
              f"{T_naive[i] - T_reg[i]:10.3e} {sk:10.4f} {pred:10.4f} {sk / pred:7.4f} {sn:10.4f}")

Output:

Wien peak 300 K: grid 9.659240 um, b/T 9.659240 um, diff 1.492e-07
Wien peak 307 K: grid 9.439000 um, b/T 9.438997 um, diff 3.403e-06
Wien peak 310 K: grid 9.347650 um, b/T 9.347651 um, diff 1.469e-06
L(303.15 K) = 57.6105 W/m^2/sr, rel diff to quad 2.467e-16
dL/dT = 0.86189 W/m^2/sr/K, rel diff to central difference 1.708e-11
relative sensitivity 1.496 %/K
NETD 50 mK -> radiance step 0.04309 W/m^2/sr (0.0748 % of L)
NETD 100 mK -> radiance step 0.08619 W/m^2/sr (0.1496 % of L)
T    290 K: L(0.1-1.1 um) 4.983e-14, L(8-14 um) 4.694e+01, ratio 1.062e-15
T    295 K: L(0.1-1.1 um) 1.090e-13, L(8-14 um) 5.084e+01, ratio 2.144e-15
T    307 K: L(0.1-1.1 um) 6.436e-13, L(8-14 um) 6.099e+01, ratio 1.055e-14
T    330 K: L(0.1-1.1 um) 1.355e-11, L(8-14 um) 8.353e+01, ratio 1.622e-13
T   1000 K: L(0.1-1.1 um) 1.645e+01, L(8-14 um) 1.924e+03, ratio 8.548e-03
pixels per region: {'wall': 39239, 'road': 19200, 'warm disk': 6361, 'warm metal': 6400, 'ambient metal': 5600} | one label per pixel: True | maps match table: True

reflected apparent temperature 260 K
max |brentq - bisection| = 0.000e+00 K
region         T(K)   eps   err known eps  err eps=1   std known  predicted  ratio   std eps=1
wall            295  0.95    0.000e+00 -1.476e+00     0.0567     0.0567  0.9991     0.0546
road            290  0.93    0.000e+00 -1.812e+00     0.0608     0.0608  1.0002     0.0576
warm disk       307  0.98    0.000e+00 -7.562e-01     0.0490     0.0493  0.9942     0.0484
warm metal      330  0.10    0.000e+00 -6.029e+01     0.4067     0.4032  1.0087     0.0710
ambient metal   295  0.10    0.000e+00 -3.080e+01     0.5420     0.5388  1.0060     0.0756

reflected apparent temperature 295 K
max |brentq - bisection| = 5.684e-14 K
region         T(K)   eps   err known eps  err eps=1   std known  predicted  ratio   std eps=1
wall            295  0.95    0.000e+00  0.000e+00     0.0567     0.0567  0.9991     0.0538
road            290  0.93    0.000e+00  3.580e-01     0.0608     0.0608  1.0002     0.0564
warm disk       307  0.98    0.000e+00 -2.279e-01     0.0490     0.0493  0.9942     0.0482
warm metal      330  0.10    0.000e+00 -3.099e+01     0.4067     0.4032  1.0087     0.0524
ambient metal   295  0.10    0.000e+00  0.000e+00     0.5420     0.5388  1.0060     0.0542

Planck’s law in practice. The peaks fall on Wien’s law to within the grid step, and at 310 K the peak is at 9.35 µm, close to the 9.3 µm Hwang et al. quote for the human body. The band radiance changes by 1.50% per kelvin at 30 °C, so a 50 mK NETD corresponds to a radiance change of 0.075%: the camera must resolve several parts in ten thousand of its signal.

Silicon sees nothing of a room-temperature scene. Below 1.1 µm, a 330 K object emits about 10−1310^{-13} of its 8–14 µm radiance, and cooler ones less, so a CMOS camera has no emitted signal to work with at these temperatures. At 1000 K the ratio is 8.5×10−38.5 \times 10^{-3}, more than ten orders of magnitude higher, because the short-wavelength tail of Planck’s law grows exponentially with temperature: glowing hot objects do register on silicon. (The silicon integral ignores the sensor’s quantum efficiency, so it is an upper bound on what the chip could use.)

Ignoring emissivity pulls readings toward the surroundings. With ε\varepsilon and TrT_r known, the inversion is exact in every region. Assuming ε=1\varepsilon = 1 reads every surface warmer than its surroundings too cold and every colder one too warm, because a surface with ε<1\varepsilon < 1 mixes its own emission with the reflection of TrT_r. For the high-emissivity regions the error stays below 2 K. For bare metal it dominates: under a 260 K sky the 330 K plate reads about 270 K, colder than the wall, and the room-temperature plate reads about 264 K. Indoors, with surroundings at 295 K, the room-temperature plate and the wall read correctly whatever their emissivity, since emission and reflection then come from the same temperature, while the 330 K plate still reads 31 K low. The error grows as ε\varepsilon falls and as the gap between TT and TrT_r widens.

Correcting emissivity amplifies noise. With the correction, the noise in temperature on each region matches σL/(ε dL/dT)\sigma_L/(\varepsilon\,dL/dT) to within 1%, so the metal plates, at ε=0.10\varepsilon = 0.10, are about ten times noisier than the high-emissivity regions (0.41–0.54 K against 0.05–0.06 K). The room-temperature plate is noisier than the warm one because dL/dTdL/dT is smaller at 295 K than at 330 K. These noise values are identical under both reflected temperatures by construction: the correction removes the reflected term exactly, and both conditions draw the same noise (seed 0). Without the correction the metal looks almost as quiet as the rest of the image, at 0.05–0.08 K, but wherever its temperature differs from the surroundings’ it is wrong by tens of kelvin. In this model, low-emissivity surfaces cannot be measured well either way.

All planned checks and hypotheses held: the Wien peaks within the grid half-step, the quadrature, slope and root-finding cross-checks, the data checks (region counts, one label per pixel, maps matching the table), exact recovery with known emissivity, the sign of the ε=1\varepsilon = 1 errors, zero error at T=TrT = T_r, errors above 10 K for the warm plate and below 3 K for ε≥0.93\varepsilon \ge 0.93 under both reflected temperatures, errors growing as ε\varepsilon falls and as ∣T−Tr∣|T - T_r| widens, the noise ratios between 0.95 and 1.05, the silicon ratios below 10−1010^{-10} up to 330 K and more than 10610^6 times higher at 1000 K, and the sensitivity between 1.3 and 2.0% per kelvin. The hypotheses, checks and their tolerances were written in the plan before any computation (its “Part 1 (written before any script ran): hypotheses and set-up”). A planning script then computed, on the same set-up, the values of the exact checks (Wien grid error, quadrature and slope agreement, exact inversion, the bisection against brentq, and the T=TrT = T_r case), and the plan’s “Part 2 (after reachable.py; before the pilot): reachable values, tolerances, precision, runtime” tabulated them beside the tolerances Part 1 had already fixed. The batch plan named one reflected temperature; the example adds a second, adds image noise and the noise measure, adds the 1000 K row, and adds the library cross-checks. The script ran as a pilot and then, unchanged, as the first run, with identical output.

Strengths

  • No illumination needed. A thermal camera images emitted radiation, so it works in darkness; Saponaro et al. also point to its relative invariance to colour changes and its invariance to lighting conditions.
  • Contrast from temperature. Warm people and animals stand out against cooler backgrounds; Gade et al. note that detection in thermal images can often be reduced to a trivial task. Hwang et al. add that LWIR cameras are less disturbed than NIR cameras by headlights and traffic signals.
  • Physical measurement. With calibration and known emissivity, pixel values give surface temperature.
  • Privacy. Gade et al. chose thermal cameras for long-term analysis in sports arenas because people cannot be identified in their images.
  • NIR on silicon is cheap. A NIR camera is an ordinary sensor without its infrared-blocking filter (Levoy), and biometrics gives the unobtrusiveness of NIR illumination as one reason iris cameras use it.

Limitations

  • Emissivity and reflections. Low-emissivity surfaces read wrong unless ε\varepsilon and TrT_r are known, and correcting them amplifies noise (example above); reflections can also add candidate detections, as of people reflected in Gade et al.’s floors.
  • Low resolution. Thermal sensors in the cited papers have 320 × 256 to 640 × 480 pixels, against 1920 × 1024 for Bijelic et al.’s colour cameras.
  • No colour or texture. Objects at the same temperature merge; Gade et al. report that people are often fragmented into small parts. The model above also leaves out the atmosphere and treats emissivity as independent of wavelength and direction; both are our simplifications.
  • Calibration targets. Geometric calibration needs heated or emitting targets rather than printed patterns.
  • Detector cost. Sensitive photon detectors need cooling (Gade et al.).

Typical Applications

  • Automotive night vision and fusion: pedestrian detection on paired colour–thermal data (Hwang et al.), and an FIR camera among the sensors of an adverse-weather driving dataset built for sensor fusion in autonomous driving (Bijelic et al.).
  • Surveillance and tracking: video surveillance data such as MEVA include thermal infrared cameras, and single-object tracking benchmarks have run thermal and RGB-thermal challenges (VOT).
  • Occupancy and people counting in sports halls (Gade et al.).
  • Material recognition from heating, cooling and water permeation seen in LWIR video (Saponaro et al.).
  • Aerial and satellite thermal mapping: temperature orthophotos from drone LWIR cameras (aerial image analysis) and Landsat’s thermal bands (remote sensing).
  • Biometrics and gated imaging with active NIR (iris recognition; Bijelic et al.’s gated camera).

Comparison with Other Sensors

Sensor Light it measures Needs illumination Typical output
Visible camera (digital camera) Reflected visible light Yes (daylight or lamps) Colour image, high resolution
NIR camera Reflected near infrared, below about 1.1 µm on silicon Sometimes its own NIR source (iris, gated cameras) Monochrome image
Thermal (LWIR) camera Emitted and reflected radiation around 8–14 µm No Single-channel radiance or temperature image, lower resolution
LiDAR Its own NIR laser pulses Its own laser Point cloud with range and intensity

Related

  • Image Formation

    How light leaving a scene becomes an array of pixel values, through illumination and reflectance, the lens, the sensor and its colour filters, sampling and quantization, and the camera's own processing.

  • Pedestrian Detection

    Finding every person on foot in an image, usually from a vehicle or a fixed camera; from HOG with a linear SVM and boosted channel features to adapted convolutional detectors and crowd-aware losses and suppression, with the Caltech, CityPersons and CrowdHuman benchmarks and the log-average miss rate.

  • Image Registration

    Aligning two images of the same scene or object, taken at different times, from different viewpoints or with different sensors, by finding the geometric transformation that maps points in one onto the corresponding points in the other.

  • Sensor Fusion for Autonomous Driving

    Combining camera, LiDAR and radar measurements into one estimate of the scene around a vehicle, covering calibration and projection between sensors, time synchronization and ego-motion compensation, early, mid-level, bird's-eye-view and late fusion, and fusion over time through tracking.

  • Multispectral Satellite Imaging

    How passive optical satellites record the Earth in a handful of spectral bands, from whiskbroom and pushbroom scanners and spectral response functions to the four resolutions, calibration from digital numbers to radiance and reflectance, atmospheric correction, band-to-band registration and spectral indices, with hyperspectral imaging in outline and the instruments of Landsat 8 and 9, Sentinel-2 and EnMAP.

  • Video Surveillance and Security

    Computer vision applied to video from fixed security and traffic cameras as its research literature describes it; the processing chain from detection to multi-camera tracking, re-identification, attributes, events and search, the PETS, DukeMTMC, CityFlow, AI City Challenge, VIRAT and MEVA benchmarks with their protocols and metrics, documented failure modes, NIST's accuracy and demographic findings, and the privacy concerns stated in official documents.

References

  1. Hwang, S., Park, J., Kim, N., Choi, Y. & Kweon, I. S. (2015). Multispectral Pedestrian Detection: Benchmark Dataset and Baseline. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  2. Bijelic, M., Gruber, T., Mannan, F., Kraus, F., Ritter, W., Dietmayer, K. & Heide, F. (2020). Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  3. Gade, R., Jørgensen, A. & Moeslund, T. B. (2013). Long-term Occupancy Analysis using Graph-Based Optimisation in Thermal Imagery. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  4. Saponaro, P., Sorensen, S., Kolagunda, A. & Kambhamettu, C. (2015). Material Classification with Thermal Imagery. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  5. Levoy, M. (2013). Photons and Sensors. Lecture slides, CS 178: Digital Photography, Stanford University, Spring 2013 (cited by slide number).
  6. SciPy 1.16.3 source: scipy/constants/_codata.py and _constants.py (CODATA 2022 values), scipy/scipy at tag v1.16.3.
  7. Gade, R. & Moeslund, T. B. (2014). Thermal Cameras and Applications: A Survey. Machine Vision and Applications (not read).