Omnidirectional Camera

An omnidirectional camera (from omni, all) is, in Scaramuzza’s definition, a camera with a 360° field of view in the horizontal plane, or with a visual field that covers a hemisphere or approximately the whole sphere. Such cameras are also called panoramic, spherical, catadioptric or fisheye cameras, depending on how they are built. What sets them apart is geometric: rays at 90° or more from the optical axis cannot reach the image plane of a pinhole camera, so an omnidirectional camera needs a projection model built on the sphere of viewing directions rather than on the plane. This article covers the ways such cameras are built, the single-viewpoint condition, the unified sphere model and its successors, spherical image representations and calibration. The classical fisheye mappings and the Kannala–Brandt polynomial belong to lens distortion, and stitching several views into a panorama belongs to image stitching.

How It Works

Three ways to build one

Scaramuzza’s chapter names three constructions:

• Dioptric cameras use lenses alone, typically a fisheye lens, and can reach a field of view somewhat larger than 180°, slightly more than a hemisphere.
• Catadioptric cameras combine a standard camera with a shaped mirror (parabolic, hyperbolic or elliptical). They give 360° in the horizontal plane and, in his words, more than 100° in elevation.
• Polydioptric cameras use several cameras with overlapping fields of view. According to the chapter, they were at the time the only cameras that cover the full sphere, 4π steradians.

The chapter dates catadioptric cameras in robotics to Yagi and Kawato (1990), who used them to localize robots. It says that fisheye cameras spread from 2000, when manufacturing techniques pushed their field of view to 180° and beyond, and that from 2005 they were miniaturized to 1–2 cm with fields of view of 190° or more. Usenko, Demmel and Cremers (2018) give the practical contrast that motivates their work: catadioptric cameras are mechanically complex and expensive, while fisheye lenses are small, light and widely available, so their paper concentrates on fisheye lenses.

A rectilinear lens, the subject of focal length and field of view, cannot reach a full angle of 180°; a fisheye lens on an ordinary sensor uses another mapping from angle to image radius, described by the projection functions of lens distortion (equidistance $r = f\theta$, stereographic, equisolid-angle, orthogonal, and Kannala–Brandt’s polynomial in $\theta$). An omnidirectional camera model has to cover all three constructions, or at least the first two, with one parametric form.

Central projection and the single effective viewpoint

A vision system is central when all the optical rays to the viewed points meet in one point, the projection centre or single effective viewpoint. A perspective camera is central. Scaramuzza states that all modern fisheye cameras are central. Catadioptric cameras are central only for particular mirrors and placements: as he describes Baker and Nayar’s (1999) result, the mirrors with a single viewpoint are the rotated (swept) conic sections, namely hyperbolic, parabolic and elliptical mirrors. For a hyperbolic or elliptical mirror, the camera’s centre of projection must coincide with one focus of the conic (the example below traces this case, with the viewpoint at the other focus). A parabolic mirror reflects rays aimed at its focus into rays parallel to its axis, so it needs an orthographic lens between camera and mirror. In a non-central system, the envelope of the rays forms a caustic rather than a point (the chapter’s Fig. 2).

The chapter gives two reasons why the property matters. Every pixel then measures the light passing through one point in one direction, so a calibrated image can be remapped onto a plane at any distance, giving a geometrically correct perspective view, or onto a sphere centred at the viewpoint. And epipolar geometry, with everything structure from motion builds on it, holds for any central camera, perspective or omnidirectional. Scaramuzza, Martinelli and Siegwart (2006, ICVS) add a practical caveat: the camera and mirror axes must be aligned and the mirror’s focus must coincide with the camera’s centre, which is hard to achieve exactly, so a misaligned catadioptric camera is only approximately single-viewpoint. A polydioptric rig is not central in general, since its cameras’ centres lie at different points (our reasoning), which is why panoramas stitched from such rigs suffer from parallax.

Camera Models

All models below map a 3D point $\mathbf{x} = (x, y, z)$ in the camera frame to pixels $\mathbf{u} = (u, v)$ through a projection $\pi$, and back through an unprojection $\pi^{-1}$ that returns a unit bearing vector. Usenko et al. fix the frame as here: $z$ along the principal axis and $x, y$ along the image axes. For each model they give the set $\Omega$ of 3D points with a valid projection and the image domain $\Theta$ on which the unprojection is defined.

The unified sphere model

Geyer and Daniilidis (2000) showed, as Scaramuzza summarizes their paper, that every central catadioptric projection and the perspective projection are equivalent to a two-step mapping: points are first projected onto a unit sphere centred at the single viewpoint, then from a point on the sphere’s axis at distance $\epsilon$ from its centre onto a plane. In the chapter’s four steps, a scene point $P$ is normalized to $P_s = P/\lVert P\rVert = (x_s, y_s, z_s)$, shifted to a frame centred at $(0, 0, -\epsilon)$, projected to the normalized plane,

$$\tilde{\mathbf{m}} = \left(\frac{x_s}{z_s + \epsilon},\; \frac{y_s}{z_s + \epsilon},\; 1\right),$$

and mapped to pixels by an intrinsic matrix $K$. The parameter $\epsilon$ ranges from 0 (a planar mirror, which is the perspective case) to 1 (a parabolic mirror). The inverse (the chapter’s Eq. 6, read from the rendered page) is

$$P_s \propto \left(x_m,\; y_m,\; 1 - \epsilon\,\frac{x_m^2 + y_m^2 + 1}{\epsilon + \sqrt{1 + (1 - \epsilon^2)(x_m^2 + y_m^2)}}\right),$$

normalized to unit length. The chapter’s Table 1 gives $\epsilon$ for each mirror: 1 for the parabola, 0 for the perspective case, and $d/\sqrt{d^2 + 4l^2}$ for both the hyperbola and the ellipse, where $d$ is the distance between the foci and $l$ the latus rectum, which the chapter defines as the chord through a focus parallel to the directrix. The example below traces a hyperbolic mirror to test this formula.

Scaramuzza stresses that this model was developed for central catadioptric cameras and is not valid for fisheye lenses. An extension to fisheye lenses by Ying and Hu (2004), as he describes it, works only with limited accuracy, because the projection of a fisheye lens varies from lens to lens and with the field of view. Usenko et al. cite Mei and Rives (2007) for the model’s standard formulation; OpenCV’s omnidir module in ccalib implements Mei and Rives’s model, according to its tutorial, which states that the model handles both catadioptric and fisheye cameras.

Usenko’s form of the unified camera model

Usenko et al. call the model the unified camera model (UCM) and write it, with five parameters $[\gamma_x, \gamma_y, c_x, c_y, \xi]$, as

$$\pi(\mathbf{x}) = \begin{bmatrix} \gamma_x \dfrac{x}{\xi d + z} \\[2mm] \gamma_y \dfrac{y}{\xi d + z} \end{bmatrix} + \begin{bmatrix} c_x \\ c_y \end{bmatrix}, \qquad d = \sqrt{x^2 + y^2 + z^2},$$

which is the chapter’s model with $\xi = \epsilon$. They propose an equivalent parametrization with $[f_x, f_y, c_x, c_y, \alpha]$, $\alpha \in [0, 1]$:

$$\pi(\mathbf{x}) = \begin{bmatrix} f_x \dfrac{x}{\alpha d + (1 - \alpha) z} \\[2mm] f_y \dfrac{y}{\alpha d + (1 - \alpha) z} \end{bmatrix} + \begin{bmatrix} c_x \\ c_y \end{bmatrix}, \qquad \xi = \frac{\alpha}{1 - \alpha}, \quad \gamma = \frac{f}{1 - \alpha}.$$

For $\alpha = 0$ the model is the pinhole camera, and $\alpha = 0.5$ corresponds to $\xi = 1$, the parabolic mirror. Their Table 3, which calibrates each of five lenses on three sequences, shows smaller standard deviations of the intrinsics in the $\alpha$ form than in the $\xi$ form, which they explain by the strong coupling between $\gamma$ and $\xi$; the $\alpha$ form also keeps $f$ close to the focal lengths of the other models. Three symbols are therefore in use for one quantity, and converting between them is the first thing to check when moving calibrations between tools:

Source	Symbol	Relation
Scaramuzza’s chapter (Geyer–Daniilidis model)	$\epsilon$	$\epsilon = \xi$
Mei and Rives; OpenCV omnidir (xi)	$\xi$	$\xi = \alpha/(1 - \alpha)$
Usenko et al.	$\alpha$	$\alpha = \xi/(1 + \xi)$, focal length $f = (1 - \alpha)\gamma$

The valid set is $\Omega = \{\mathbf{x} \mid z > -w d\}$ with $w = \alpha/(1 - \alpha)$ for $\alpha \le 0.5$ and $w = (1 - \alpha)/\alpha$ for $\alpha > 0.5$. Usenko et al. note that, for $\alpha > 0.5$, $w$ is the sine of the angle between the horizontal axis of their schematic (Fig. 2) and the perpendicular to the tangent drawn from the pinhole’s focal point to the circle; in our reading, points beyond that tangent ray do not project one to one. The unprojection is defined on all of $\mathbb{R}^2$ for $\alpha \le 0.5$ and on $r^2 \le (1 - \alpha)^2/(2\alpha - 1)$ otherwise, with $r$ measured in the $\xi$-form normalized coordinates. Usenko et al. add that the UCM can also be applied to fisheye lenses but does not fit most of them perfectly, so a distortion model is often added.

Extended unified and double sphere models

The extended unified camera model (EUCM) of Khomutenko, Garcia and Martinet (2016), as Usenko et al. present it, replaces the sphere by an ellipsoid symmetric about the $z$ axis, with a sixth parameter $\beta > 0$:

$$d = \sqrt{\beta (x^2 + y^2) + z^2},$$

in the same projection formula as the $\alpha$-form UCM, to which it reduces for $\beta = 1$. Its unprojection needs a final normalization because the model projects through an ellipsoid. (COLMAP lists it as the Enhanced Unified Camera Model, EUCM, after the original paper’s title; see COLMAP.)

Usenko et al.’s own double sphere (DS) model projects a point onto two unit spheres whose centres are $\xi$ apart, then onto the image plane of a pinhole camera shifted by $\alpha/(1 - \alpha)$ from the second sphere:

$$\pi(\mathbf{x}) = \begin{bmatrix} f_x \dfrac{x}{\alpha d_2 + (1 - \alpha)(\xi d_1 + z)} \\[2mm] f_y \dfrac{y}{\alpha d_2 + (1 - \alpha)(\xi d_1 + z)} \end{bmatrix} + \begin{bmatrix} c_x \\ c_y \end{bmatrix}, \quad d_1 = \lVert \mathbf{x} \rVert, \quad d_2 = \sqrt{x^2 + y^2 + (\xi d_1 + z)^2}.$$

Its valid set is $z > -w_2 d_1$, with $w_2 = (w_1 + \xi)/\sqrt{2 w_1 \xi + \xi^2 + 1}$ and $w_1$ the UCM’s $w$; its unprojection is in closed form, defined on $r^2 \le 1/(2\alpha - 1)$ for $\alpha > 0.5$. Usenko et al. list three advantages: the model fits fisheye lenses well, it needs no trigonometric functions, and its inverse is closed-form, unlike higher-order polynomial models that unproject by iteration.

Polynomial models

Polynomial models trade closed forms for flexibility. Kannala and Brandt’s model, described in lens distortion, maps the angle $\theta$ to an image radius by an odd polynomial and must be inverted by iterative root finding, for example Newton’s method, as Usenko et al. note. Usenko et al. point out that when it is implemented as a distortion applied after a pinhole projection, as in Kalibr’s equidistant model or OpenCV’s fisheye model, it inherits a singularity at $z = 0$ that makes it unsuitable for lenses near 180°. They also evaluate the field-of-view (FOV) model of Devernay and Faugeras, whose single parameter $w$ approximately corresponds to the field of view of an ideal fisheye lens.

Scaramuzza’s Taylor model goes the other way: it maps an image point to a ray. The 2006 ICVS paper assumes that the lifting function can be written as a polynomial in the distance $\rho$ from the image centre,

$$P_s \propto \left(x_m,\; y_m,\; a_0 + a_1 \rho + a_2 \rho^2 + \dots + a_N \rho^N\right),$$

with coefficients and degree found by calibration. The IROS 2006 paper drops $a_1$, on the grounds that the derivative of the function at $\rho = 0$ vanishes for the hyperbolic and parabolic mirror models and the fisheye models it cites; the chapter gives the same argument for its form without the first-order term, and reports that polynomials of order three or four model all catadioptric cameras and many fisheye cameras very accurately. Because one polynomial covers mirrors and lenses, the model describes catadioptric, fisheye and perspective cameras with one form.

How the models compare

Usenko et al. calibrated every model on 16 sequences: three each from five lenses with fields of view of 195°, 183°, 150°, 126° and 122°, and one sequence from the EuRoC dataset. On mean reprojection error, Kannala–Brandt with eight parameters was best on most sequences and the double sphere model second, with less than 1% greater error than the eight-parameter model on every sequence despite having six parameters. Their text describes EUCM’s error as slightly greater than DS’s and smaller than that of Kannala–Brandt with six parameters; in their Table 1, DS has the smaller overhead on 11 of the 16 sequences and EUCM on the other 5 (BF2M2020S23-1, the three BM2820 sequences and BM4018S118-2), where the two models’ mean errors are equal to the table’s three decimals. UCM and FOV had the largest errors. In their timing (Table 2, measured on one Intel Xeon E5-1620 core), the eight-parameter Kannala–Brandt projection was about nine times slower than EUCM’s and five times slower than DS’s, which they attribute to its trigonometric operations. They conclude that sphere-based models are a good alternative to high-order polynomials when fast projection and a closed-form inverse are needed.

Data It Produces

A fisheye camera produces a circular or cropped-circular image on an ordinary rectangular sensor; a catadioptric camera produces an annular image of the mirror, which is, in the caption of Scaramuzza’s Fig. 7, typically unwrapped into a cylindrical panorama of about 360° in azimuth and 100° in elevation. Consumer 360° cameras, of which Tateno, Navab and Tombari (2018) name the Ricoh Theta, Gear360 and Insta360 One, deliver whole-sphere images.

Images that cover the sphere are stored in a planar layout:

• Equirectangular (longitude–latitude) images map longitude and latitude linearly to columns and rows. Coors, Condurache and Geiger (2018) call this the most popular representation of 360° images and note its heavy distortion near the poles, so that an object looks different depending on its latitude. COLMAP has had an EQUIRECTANGULAR camera model since version 4.1.
• Cube maps project the sphere onto the six faces of a cube, each the image plane of a virtual perspective camera. SphereNet’s authors observe that this does not remove distortion but only reduces its effect, and adds discontinuities at the face boundaries.
• Icosahedral representations subdivide an icosahedron. Eder, Shvets, Lim and Frahm (2020) render a spherical image to a set of locally planar “tangent images” on the faces of a subdivided icosahedron.

Angular resolution is the quantity to compare across layouts. Eder et al. give 0.352° per pixel for a 512 × 1024 equirectangular image and 0.088° for 2048 × 4096, against 0.094° for a 480 × 640 perspective image with a 45° × 60° field of view.

Characteristics

• Field of view and resolution. The pixels of a sensor are spread over a much wider field than in a perspective camera, so each pixel covers a larger angle. Usenko et al. present the other side of this trade-off as an advantage: large camera motions map to small pixel motions, which keeps the optical flow between frames small, a benefit for direct methods.
• Valid domains. Every sphere-based model has a set $\Omega$ of directions it can project and an image region $\Theta$ it can unproject; for $\alpha > 0.5$ both are bounded, and a lens whose field exceeds the model’s $\Omega$ cannot be represented by it.
• Single viewpoint. Fisheye lenses are central according to Scaramuzza; catadioptric cameras are central only with the right mirror and alignment; multi-camera rigs are not central.
• Sampling on the sphere. No planar layout samples the sphere uniformly; the example below measures by how much the equirectangular and cube-map layouts differ.

Calibration

Omnidirectional cameras are calibrated like perspective ones, from images of a planar target in several poses (camera calibration), with one addition from Scaramuzza’s chapter: the target should be imaged all around the camera, not on one side only, to compensate for misalignment between camera and mirror. The chapter describes three open-source MATLAB toolboxes:

• Mei’s toolbox uses checkerboard images and the Geyer–Daniilidis model, and suits catadioptric cameras with hyperbolic, parabolic, folded and spherical mirrors.
• Barreto’s toolbox uses images of lines instead of checkerboards with the same model, and suits parabolic mirrors.
• Scaramuzza’s toolbox (OCamCalib) uses checkerboards and the Taylor model, works with catadioptric cameras and with fisheye lenses of fields of view up to 195°, and detects the centre of distortion and the calibration points automatically.

The ICVS 2006 paper estimates the Taylor coefficients by a two-step linear least-squares minimization and chooses the polynomial degree by reprojection error; on a catadioptric camera with a vertical field of view above 200° and 900 × 1200 images, their experiments section reports an RMS reprojection error of 1.2 pixels with a fourth-order polynomial, while its introduction gives an average reprojection error of 1 pixel.

OpenCV’s cv::omnidir::calibrate returns the intrinsic matrix K, the single parameter xi of Mei’s model and distortion coefficients D, with flags that fix the skew, individual distortion terms, xi, the parameters named by CALIB_FIX_GAMMA, or the centre; according to the tutorial, its calibration follows Li et al.’s multiple-camera calibration toolbox (IROS 2013). Usenko et al. calibrate all their models with a grid of AprilTag markers, a Huber-robust reprojection error minimized by Gauss–Newton over the intrinsics and every image’s pose, an initialization of the intrinsics by Heng et al.’s method (with $\beta = 1$ for EUCM and $\xi = 0$ for DS) and initial poses from UPnP. COLMAP’s camera models, used throughout its bundle adjustment, include several fisheye models, EUCM and the equirectangular model.

Processing the Data

Rectification. OpenCV’s omnidir tutorial offers four rectifications: perspective, which loses much of the field of view; cylindrical, which keeps all of it and looks unnatural only in the middle of the bottom; stereographic, which may lose a little view; and longitude–latitude, which is not friendly to view but makes epipolar lines horizontal, so that standard stereo matching can be applied. For omnidirectional stereo, the tutorial recommends longitude–latitude rectification over perspective rectification, which wastes the wide field of view. Rectification to a perspective view is possible only for a central camera, by the chapter’s argument above. Each rectification is an image warping driven by the model’s unprojection.

Geometry on bearings. Because a calibrated central camera gives a unit bearing vector for every pixel, two-view geometry and visual odometry can work on bearings instead of normalized image coordinates; in our reading this is what makes the essential matrix usable for directions behind the camera, where $z \le 0$ and a pinhole’s normalized coordinates do not exist.

Learning on spherical images. A standard convolutional neural network applied to an equirectangular image meets the latitude-dependent distortion described above. SphereNet adapts the sampling locations of each filter, projecting the kernel’s sampling pattern onto the plane tangent to the sphere (a gnomonic projection) so that filter outputs are invariant to latitudinal rotation, and lets networks trained on perspective images transfer to omnidirectional ones. Tateno et al. compute a similar distortion-aware sampling grid on the tangent plane of each pixel’s direction and use it to predict depth from a single equirectangular panorama with a network trained on perspective images. Eder et al. instead change the data: they train standard networks on tangent images and report that this compares favourably with specialized spherical convolutions and scales to higher resolutions.

Example: Models, Mirrors and Sphere Layouts on Exact Rays

The example checks the models above on synthetic rays whose images are known exactly, with no images and no fitting noise. It has four parts:

1. Conventions, round trips and valid domains. The UCM, EUCM and DS models are implemented from Usenko et al.’s equations at fixed parameters of the size their Table 3 reports ($f = 300$ px, UCM $\alpha = 0.65$; EUCM $\alpha = 0.6$, $\beta = 1.1$; DS $\xi = -0.2$, $\alpha = 0.59$). The checks are:
   • K1: the $\alpha$ and $\xi$ forms of the UCM agree.
   • K2: the chapter’s Eq. 6 and Usenko’s Eq. 10 lift the same normalized point to the same direction.
   • K3: unprojection inverts projection inside $\Omega$.
   • K4: the round trip stops working where $\Omega$ ends. A scan along one meridian from 90° to 150° in steps of 0.001° finds the last angle whose round trip is accurate to $10^{-9}$ rad, compares it with the analytic boundary $\theta_\Omega$, and reports the fraction of scanned angles beyond $\theta_\Omega + 0.001°$ that fail.
   • K5: the boundary of $\Omega$ projects onto the boundary of $\Theta$.
2. Fits to a fisheye. An ideal equidistant fisheye ($r = f\theta$, $f = 300$ px, a 190° field, 3,421 rays) is fitted by each model’s full intrinsic vector with SciPy’s least_squares, starting from the lens’s focal length and centre, $\alpha = 0.5$, and Usenko et al.’s initial values $\beta = 1$ for EUCM and $\xi = 0$ for DS. A self-fit to each model’s own pixels (K6) first confirms that the fitter recovers known parameters.
3. Mirrors traced exactly. Rays are reflected off a parabolic mirror viewed by an orthographic camera, a hyperbolic mirror viewed by a pinhole at its outer focus, and, as a negative control, the parabolic mirror viewed by a pinhole at $-5h$. The checks are:
   • K7: whether the reflected rays meet in one point, the focus $F_1$, measured by the least-squares common point of the reflected lines and their largest distance from it.
   • K8: whether the image radius obeys the unified model exactly, by a linear least-squares fit of $r = \gamma \sin\theta / (\xi + \cos\theta)$.
   • For the hyperbola, the fitted $\xi$ is compared with Table 1, using $l = L = 2b^2/a$, the length of the chord through a focus perpendicular to the axis; $l^*$ is the value of $l$ for which Table 1 would give the fitted $\xi$.
4. Sphere layouts. For a 2048 × 1024 equirectangular image and a cube map with six 592 × 592 faces (2,097,152 and 2,102,784 pixels), the exact solid angle of every pixel is computed, with the total checked against $4\pi$ (K9).

The hypotheses were written before any code ran:

• H1 (from Usenko et al.’s comparison): UCM fits the fisheye worse than EUCM and DS.
• H2: DS fits better than EUCM, as Usenko et al.’s text reports for their lenses. Our ideal lens is an analogue of their real ones, so H2 tests the analogue.
• H3: $\Omega$ is exactly the domain where the round trip works.
• H4: the boundary of $\Omega$ maps to the boundary of $\Theta$.
• H5 and H6: the parabola and hyperbola set-ups are central and obey the unified model, with $\xi = 1$ for the parabola; the negative control is not central.
• H7: Table 1’s formula holds for the hyperbola with $l$ as the chapter defines it.
• H8: the ratio of largest to smallest pixel solid angle equals $\cot(\pi/2048)$ for the equirectangular image (our derivation: the equator row covers $\Delta\lambda \sin\Delta$, the pole row $\Delta\lambda(1 - \cos\Delta)$, with $\Delta = \pi/H$) and lies just below $3\sqrt{3}$ for the cube map (our derivation: $d\Omega = dx\,dy/(1 + x^2 + y^2)^{3/2}$ on a face at unit distance, compared at the centre and the corner).

The camera frame has $z$ along the optical axis, $x$ right and $y$ down; $\theta$ is measured from $+z$. For the mirrors, the model axis points from the mirror toward the camera ($z_{\text{model}} = -z_{\text{mirror}}$), so the image centre sees the direction of the camera itself. The code uses NumPy 2.3.5 and SciPy 1.16.3, uses no random numbers, and runs in well under a second.

import os
for k in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[k] = "1"
import numpy as np
from scipy.optimize import least_squares

# Camera frame: z along the optical axis, x right, y down; theta = angle from +z.
def angle(a, b):
    return np.arctan2(np.linalg.norm(np.cross(a, b), axis=-1), np.sum(a * b, axis=-1))

def rays(theta_deg, az_deg):
    t, p = np.meshgrid(np.radians(theta_deg), np.radians(az_deg), indexing="ij")
    t, p = t.ravel(), p.ravel()
    return np.stack([np.sin(t) * np.cos(p), np.sin(t) * np.sin(p), np.cos(t)], axis=1)

def pix(X, fx, fy, cx, cy, den):
    return np.stack([fx * X[:, 0] / den + cx, fy * X[:, 1] / den + cy], axis=1)

def w_of(a):                                   # Usenko Eqs. 9 and 45
    return a / (1 - a) if a <= 0.5 else (1 - a) / a

# --- Unified camera model (UCM), Usenko's alpha form, Eqs. 7-15
def ucm_project(X, q):
    fx, fy, cx, cy, a = q
    d = np.linalg.norm(X, axis=1)
    return pix(X, fx, fy, cx, cy, a * d + (1 - a) * X[:, 2])

def ucm_unproject(u, q):
    fx, fy, cx, cy, a = q
    mx, my = (u[:, 0] - cx) / fx * (1 - a), (u[:, 1] - cy) / fy * (1 - a)
    r2, xi = mx**2 + my**2, a / (1 - a)
    s = (xi + np.sqrt(1 + (1 - xi**2) * r2)) / (1 + r2)
    return np.stack([s * mx, s * my, s - xi], axis=1)

def ucm_omega(X, q):
    return X[:, 2] > -w_of(q[4]) * np.linalg.norm(X, axis=1)

def ucm_r2(u, q):
    fx, fy, cx, cy, a = q
    return (((u[:, 0] - cx) / fx)**2 + ((u[:, 1] - cy) / fy)**2) * (1 - a)**2

def ucm_bound(q):
    return (1 - q[4])**2 / (2 * q[4] - 1)

# --- Extended unified camera model (EUCM), Eqs. 16-23
def eucm_project(X, q):
    fx, fy, cx, cy, a, b = q
    d = np.sqrt(b * (X[:, 0]**2 + X[:, 1]**2) + X[:, 2]**2)
    return pix(X, fx, fy, cx, cy, a * d + (1 - a) * X[:, 2])

def eucm_unproject(u, q):
    fx, fy, cx, cy, a, b = q
    mx, my = (u[:, 0] - cx) / fx, (u[:, 1] - cy) / fy
    r2 = mx**2 + my**2
    mz = (1 - b * a**2 * r2) / (a * np.sqrt(1 - (2 * a - 1) * b * r2) + (1 - a))
    M = np.stack([mx, my, mz], axis=1)
    return M / np.linalg.norm(M, axis=1, keepdims=True)

def eucm_omega(X, q):
    d = np.sqrt(q[5] * (X[:, 0]**2 + X[:, 1]**2) + X[:, 2]**2)
    return X[:, 2] > -w_of(q[4]) * d

def eucm_r2(u, q):
    return ((u[:, 0] - q[2]) / q[0])**2 + ((u[:, 1] - q[3]) / q[1])**2

def eucm_bound(q):
    return 1 / (q[5] * (2 * q[4] - 1))

# --- Double sphere model (DS), Eqs. 40-51
def ds_project(X, q):
    fx, fy, cx, cy, xi, a = q
    d1 = np.linalg.norm(X, axis=1)
    k = xi * d1 + X[:, 2]
    d2 = np.sqrt(X[:, 0]**2 + X[:, 1]**2 + k**2)
    return pix(X, fx, fy, cx, cy, a * d2 + (1 - a) * k)

def ds_unproject(u, q):
    fx, fy, cx, cy, xi, a = q
    mx, my = (u[:, 0] - cx) / fx, (u[:, 1] - cy) / fy
    r2 = mx**2 + my**2
    mz = (1 - a**2 * r2) / (a * np.sqrt(1 - (2 * a - 1) * r2) + 1 - a)
    s = (mz * xi + np.sqrt(mz**2 + (1 - xi**2) * r2)) / (mz**2 + r2)
    return np.stack([s * mx, s * my, s * mz - xi], axis=1)

def ds_omega(X, q):
    w1 = w_of(q[5])
    w2 = (w1 + q[4]) / np.sqrt(2 * w1 * q[4] + q[4]**2 + 1)
    return X[:, 2] > -w2 * np.linalg.norm(X, axis=1)

def ds_r2(u, q):
    return ((u[:, 0] - q[2]) / q[0])**2 + ((u[:, 1] - q[3]) / q[1])**2

def ds_bound(q):
    return 1 / (2 * q[5] - 1)

def theta_omega(name, q):                      # analytic boundary of Omega, degrees
    if name == "UCM":
        return 90 + np.degrees(np.arcsin(w_of(q[4])))
    if name == "EUCM":
        w, b = w_of(q[4]), q[5]
        return np.degrees(np.arccos(-np.sqrt(w**2 * b / (1 + w**2 * (b - 1)))))
    w1 = w_of(q[5])
    return 90 + np.degrees(np.arcsin((w1 + q[4]) / np.sqrt(2 * w1 * q[4] + q[4]**2 + 1)))

MODELS = {  # name: project, unproject, omega, r2, bound, test params, init, bounds
    "UCM": (ucm_project, ucm_unproject, ucm_omega, ucm_r2, ucm_bound,
            [300, 300, 640, 512, 0.65], [300, 300, 640, 512, 0.5],
            ([-np.inf] * 4 + [0.0], [np.inf] * 4 + [1.0])),
    "EUCM": (eucm_project, eucm_unproject, eucm_omega, eucm_r2, eucm_bound,
             [300, 300, 640, 512, 0.60, 1.10], [300, 300, 640, 512, 0.5, 1.0],
             ([-np.inf] * 4 + [0.0, 1e-6], [np.inf] * 4 + [1.0, 100.0])),
    "DS": (ds_project, ds_unproject, ds_omega, ds_r2, ds_bound,
           [300, 300, 640, 512, -0.20, 0.59], [300, 300, 640, 512, 0.0, 0.5],
           ([-np.inf] * 4 + [-1.0, 0.0], [np.inf] * 4 + [1.0, 1.0])),
}

print("Part 1: conventions, round trips and valid domains (test parameters)")
X1 = rays(np.arange(0, 121, 1.0), np.arange(0, 360, 10.0))
qU = np.array(MODELS["UCM"][5], float)
xi, gam = qU[4] / (1 - qU[4]), qU[0] / (1 - qU[4])
inU = ucm_omega(X1, qU)
d = np.linalg.norm(X1[inU], axis=1)
uxi = pix(X1[inU], gam, gam, qU[2], qU[3], xi * d + X1[inU, 2])        # Usenko Eq. 5
print(f"K1 UCM alpha form vs xi form (xi = {xi:.4f}): max |du| = "
      f"{np.abs(ucm_project(X1[inU], qU) - uxi).max():.2e} px")
u_in = ucm_project(X1[inU], [1, 1, 0, 0, qU[4]])                      # f = 1, c = 0 pixels
m = u_in * (1 - qU[4])                                                 # xi-form normalized m
print(f"K2 same point to both formulas: max |(1 - alpha) u_in - m| = "
      f"{np.abs((1 - qU[4]) * u_in - m).max():.2e}")
r2 = (m**2).sum(1)
g6 = np.stack([m[:, 0], m[:, 1],
               1 - xi * (r2 + 1) / (xi + np.sqrt(1 + (1 - xi**2) * r2))], axis=1)  # Scaramuzza Eq. 6
g6 /= np.linalg.norm(g6, axis=1, keepdims=True)
print(f"K2 Scaramuzza Eq. 6 vs Usenko Eq. 10: max angle = "
      f"{angle(g6, ucm_unproject(u_in, [1, 1, 0, 0, qU[4]])).max():.2e} rad")
scan = np.arange(90000, 150001) / 1000.0                               # theta, 0.001 deg steps
Xs = rays(scan, [30.0])
for name, (P, U, Om, R2, B, q, _, _) in MODELS.items():
    q = np.array(q, float)
    inside = Om(X1, q)
    k3 = angle(X1[inside], U(P(X1[inside], q), q)).max()
    with np.errstate(invalid="ignore", divide="ignore"):
        err = angle(Xs, U(P(Xs, q), q))
    ok = err < 1e-9
    th = theta_omega(name, q)
    last = scan[ok].max()
    beyond = scan > th + 0.001
    tb = np.radians(th)
    xb = np.array([[np.sin(tb) * np.cos(np.radians(30)), np.sin(tb) * np.sin(np.radians(30)), np.cos(tb)]])
    k5 = R2(P(xb, q), q)[0] / B(q) - 1
    print(f"{name:4s} K3 round trip in Omega: {k3:.2e} rad | K4 theta_Omega {th:.4f}, "
          f"last valid {last:.4f} (diff {last - th:+.5f} deg), "
          f"fail beyond {np.mean(~ok[beyond]):.4f} | K5 r2/bound - 1 = {k5:.2e}")

print("\nPart 2: fits to an ideal equidistant fisheye, r = f*theta, 190 deg field")
F = rays(np.arange(1, 96, 1.0), np.arange(0, 360, 10.0))
F = np.vstack([[0.0, 0.0, 1.0], F])
tF = np.degrees(np.arccos(np.clip(F[:, 2], -1, 1)))
rF = 300.0 * np.radians(tF)
phi = np.arctan2(F[:, 1], F[:, 0])
target = np.stack([640 + rF * np.cos(phi), 512 + rF * np.sin(phi)], axis=1)
print(f"rays: {len(F)}, theta max {tF.max():.1f} deg")
for name, (P, U, Om, R2, B, q, init, bnd) in MODELS.items():
    fit = lambda tgt: least_squares(lambda p: (P(F, p) - tgt).ravel(), np.array(init, float),
                                    bounds=bnd, method="trf", x_scale="jac", ftol=1e-15,
                                    xtol=1e-15, gtol=1e-15, max_nfev=2000)
    own = P(F, np.array(q, float))                                     # K6: model's own pixels
    s6 = fit(own)
    rms6 = np.sqrt(np.mean(np.sum((P(F, s6.x) - own)**2, axis=1)))
    rel6 = np.max(np.abs(s6.x - q) / np.abs(q))
    s = fit(target)
    e = np.linalg.norm(P(F, s.x) - target, axis=1)
    band = [np.sqrt(np.mean(e[(tF >= lo) & (tF <= hi)]**2)) for lo, hi in ((0, 60), (60, 90), (90, 95))]
    inside = Om(F, s.x)
    rt = angle(F[inside], U(P(F[inside], s.x), s.x)).max()
    print(f"{name:4s} K6 self-fit RMS {rms6:.2e} px, rel param err {rel6:.2e} "
          f"(rays in test Omega {Om(F, np.array(q, float)).mean():.4f})")
    print(f"     fisheye fit RMS {np.sqrt(np.mean(e**2)):.6e} px, max {e.max():.4e} px, "
          f"band RMS 0-60/60-90/90-95: {band[0]:.4e} / {band[1]:.4e} / {band[2]:.4e} px")
    print(f"     params {np.array2string(s.x, precision=4, suppress_small=True)}, "
          f"in Omega {inside.mean():.4f}, round trip {rt:.2e} rad")

print("\nPart 3: catadioptric mirrors traced exactly")
rho = np.linspace(0.01, 2.0, 200)
az = np.radians(np.arange(0, 360, 10.0))
Rg, Ag = np.meshgrid(rho, az, indexing="ij")
xs, ys = Rg.ravel() * np.cos(Ag.ravel()), Rg.ravel() * np.sin(Ag.ravel())
print(f"mirror points per setup: {xs.size}")

def reflect(din, grad):
    n = grad / np.linalg.norm(grad, axis=1, keepdims=True)
    return din - 2 * np.sum(din * n, axis=1, keepdims=True) * n

def common_point(Pt, D):
    D = D / np.linalg.norm(D, axis=1, keepdims=True)
    M = np.eye(3)[None] - D[:, :, None] * D[:, None, :]
    p = np.linalg.solve(M.sum(0), (M @ Pt[:, :, None]).sum(0)).ravel()
    return p, np.linalg.norm((M @ (p - Pt)[:, :, None])[:, :, 0], axis=1).max()

def ucm_fit(r_img, s):                         # r = g sin(t) / (xi + cos t), model z = -z_mirror
    ct, st = -s[:, 2], np.hypot(s[:, 0], s[:, 1])
    xi_, g_ = np.linalg.lstsq(np.stack([r_img, -st], 1), -r_img * ct, rcond=None)[0]
    return xi_, g_, np.abs(g_ * st / (xi_ + ct) - r_img).max()

h = 1.0                                        # parabola z = (rho^2 - h^2) / (2h), focus at 0
Pp = np.stack([xs, ys, (Rg.ravel()**2 - h**2) / (2 * h)], 1)
gp = np.stack([2 * xs, 2 * ys, -2 * h * np.ones_like(xs)], 1)
a_, b_ = 1.0, 1.0                              # hyperbola, foci at 0 and (0, 0, -d)
c_ = np.sqrt(a_**2 + b_**2)
dd = 2 * c_
Ph = np.stack([xs, ys, -c_ + a_ * np.sqrt(1 + Rg.ravel()**2 / b_**2)], 1)
gh = np.stack([-2 * xs / b_**2, -2 * ys / b_**2, 2 * (Ph[:, 2] + c_) / a_**2], 1)
F2 = np.array([0.0, 0.0, -dd])
C5 = np.array([0.0, 0.0, -5 * h])
setups = {
    "parabola + orthographic": (Pp, np.tile([0.0, 0.0, 1.0], (len(Pp), 1)), gp, np.hypot(xs, ys)),
    "hyperbola + pinhole at F2": (Ph, Ph - F2, gh, np.hypot(Ph[:, 0], Ph[:, 1]) / (Ph[:, 2] + dd)),
    "parabola + pinhole at -5h": (Pp, Pp - C5, gp, None),
}
for name, (Pt, din, grad, r_img) in setups.items():
    din = din / np.linalg.norm(din, axis=1, keepdims=True)
    dout = reflect(din, grad)
    p, dmax = common_point(Pt, dout)
    line = f"{name:26s} K7 common point |p - F1| = {np.linalg.norm(p):.2e}, max line distance {dmax:.2e}"
    if r_img is not None:
        s = dout / np.linalg.norm(dout, axis=1, keepdims=True)
        xi_, g_, res = ucm_fit(r_img, s)
        line += f"\n{'':26s} K8 UCM fit xi = {xi_:.12f}, gamma = {g_:.12f}, max |residual| {res:.2e}"
        if name.startswith("hyperbola"):
            L = 2 * b_**2 / a_                 # latus rectum: chord through a focus
            xi_T = dd / np.sqrt(dd**2 + 4 * L**2)
            lstar = dd / 2 * np.sqrt(1 / xi_**2 - 1)
            line += (f"\n{'':26s} H7 Table 1 xi (l = latus rectum {L:.1f}) = {xi_T:.12f}, "
                     f"l*/L - 1 = {lstar / L - 1:.2e}")
    print(line)

print("\nPart 4: solid angle per pixel")
W, H = 2048, 1024
lat = np.pi / 2 - np.arange(H + 1) * np.pi / H                         # row edges, top to bottom
om_e = (2 * np.pi / W) * (np.sin(lat[:-1]) - np.sin(lat[1:]))          # one value per row
tot_e = np.sum(np.repeat(om_e, W))
ratio_e = om_e.max() / om_e.min()
print(f"equirectangular {W}x{H}: total/4pi - 1 = {tot_e / (4 * np.pi) - 1:.2e}, max/min = {ratio_e:.6f}, "
      f"cot(pi/2H) = {1 / np.tan(np.pi / (2 * H)):.6f} (rel diff {ratio_e * np.tan(np.pi / (2 * H)) - 1:.2e})")
N = 592
e = np.linspace(-1, 1, N + 1)
Fq = lambda x, y: np.arctan(x * y / np.sqrt(1 + x**2 + y**2))
x1, y1 = np.meshgrid(e[:-1], e[:-1], indexing="ij")
x2, y2 = np.meshgrid(e[1:], e[1:], indexing="ij")
om_c = Fq(x2, y2) - Fq(x1, y2) - Fq(x2, y1) + Fq(x1, y1)
tot_c = 6 * np.sum(om_c)
print(f"cube map 6x{N}x{N}: total/4pi - 1 = {tot_c / (4 * np.pi) - 1:.2e}, max/min = "
      f"{om_c.max() / om_c.min():.6f}, limit 3*sqrt(3) = {3 * np.sqrt(3):.6f}")
for nm, npix in (("equirectangular", W * H), ("cube map", 6 * N * N)):
    print(f"{nm:16s} pixels {npix:,}, mean pixel size {np.degrees(np.sqrt(4 * np.pi / npix)):.4f} deg")

Output:

Part 1: conventions, round trips and valid domains (test parameters)
K1 UCM alpha form vs xi form (xi = 1.8571): max |du| = 2.27e-13 px
K2 same point to both formulas: max |(1 - alpha) u_in - m| = 0.00e+00
K2 Scaramuzza Eq. 6 vs Usenko Eq. 10: max angle = 4.34e-16 rad
UCM  K3 round trip in Omega: 5.37e-15 rad | K4 theta_Omega 122.5790, last valid 122.5780 (diff -0.00097 deg), fail beyond 1.0000 | K5 r2/bound - 1 = 2.22e-16
EUCM K3 round trip in Omega: 1.56e-15 rad | K4 theta_Omega 133.1702, last valid 133.1700 (diff -0.00017 deg), fail beyond 1.0000 | K5 r2/bound - 1 = 4.44e-16
DS   K3 round trip in Omega: 3.28e-15 rad | K4 theta_Omega 124.5377, last valid 125.7510 (diff +1.21328 deg), fail beyond 0.9524 | K5 r2/bound - 1 = -6.57e-04

Part 2: fits to an ideal equidistant fisheye, r = f*theta, 190 deg field
rays: 3421, theta max 95.0 deg
UCM  K6 self-fit RMS 0.00e+00 px, rel param err 0.00e+00 (rays in test Omega 1.0000)
     fisheye fit RMS 1.120691e+00 px, max 3.6013e+00 px, band RMS 0-60/60-90/90-95: 8.5648e-01 / 1.1925e+00 / 2.3637e+00 px
     params [296.1919 296.1919 640.     512.       0.6298], in Omega 1.0000, round trip 8.65e-16 rad
EUCM K6 self-fit RMS 4.18e-14 px, rel param err 4.04e-16 (rays in test Omega 1.0000)
     fisheye fit RMS 1.231030e-01 px, max 4.3060e-01 px, band RMS 0-60/60-90/90-95: 1.0279e-01 / 1.2675e-01 / 2.3290e-01 px
     params [300.6036 300.6036 640.     512.       0.5983   1.1364], in Omega 1.0000, round trip 7.91e-16 rad
DS   K6 self-fit RMS 8.97e-14 px, rel param err 3.19e-15 (rays in test Omega 1.0000)
     fisheye fit RMS 2.119149e-01 px, max 7.5168e-01 px, band RMS 0-60/60-90/90-95: 1.7438e-01 / 2.1579e-01 / 4.1751e-01 px
     params [216.9507 216.9507 640.     512.      -0.2794   0.5614], in Omega 1.0000, round trip 8.52e-16 rad

Part 3: catadioptric mirrors traced exactly
mirror points per setup: 7200
parabola + orthographic    K7 common point |p - F1| = 1.85e-17, max line distance 1.07e-15
                           K8 UCM fit xi = 1.000000000000, gamma = 1.000000000000, max |residual| 1.29e-14
hyperbola + pinhole at F2  K7 common point |p - F1| = 7.55e-17, max line distance 1.21e-15
                           K8 UCM fit xi = 0.942809041582, gamma = 0.333333333333, max |residual| 1.05e-15
                           H7 Table 1 xi (l = latus rectum 2.0) = 0.577350269190, l*/L - 1 = -7.50e-01
parabola + pinhole at -5h  K7 common point |p - F1| = 3.15e-01, max line distance 5.50e-01

Part 4: solid angle per pixel
equirectangular 2048x1024: total/4pi - 1 = 0.00e+00, max/min = 651.898136, cot(pi/2H) = 651.898136 (rel diff 4.21e-12)
cube map 6x592x592: total/4pi - 1 = 2.22e-16, max/min = 5.178560, limit 3*sqrt(3) = 5.196152
equirectangular  pixels 2,097,152, mean pixel size 0.1403 deg
cube map         pixels 2,102,784, mean pixel size 0.1401 deg

Conventions and valid domains. K1 held: the two forms of the UCM agree to rounding error, so the conversion in the table above is exact. K2 held: the chapter’s Eq. 6 is Usenko et al.’s Eq. 10 written as a direction, and the two lift the same normalized point to the same ray. K3 held for all three models. For UCM and EUCM, H3 and H4 held: the last angle whose round trip worked lies within one 0.001° step of the analytic boundary, every scanned angle beyond it failed, and the boundary direction projects onto the edge of $\Theta$. For the double sphere model, K4 and K5 failed, and with them H3 and H4. The last angle whose round trip held lies +1.21328° beyond the boundary given by Usenko et al.’s Eqs. 43–45, only 0.9524 of the angles beyond the boundary failed, and the boundary direction projects to $r^2/\text{bound} - 1 = -6.57 \times 10^{-4}$, inside the edge of $\Theta$. At these parameters, the closed-form inverse recovered the ray on directions that the paper’s $\Omega$ excludes; this run does not show which of the paper’s two domains, if either, is the exact injective set for DS.

Fits. The data checks held: the fisheye set has 3,421 rays up to $\theta = 95°$, all inside each test model’s $\Omega$, and each mirror set-up has 7,200 points. K6 held: each model fitted to its own pixels recovered its parameters to rounding error, so the fitter can reach the exact answer. H1 held: UCM’s RMS error on the ideal fisheye is about 1.12 px, against 0.123 px for EUCM and 0.212 px for DS. The band errors show where UCM’s error lies: 2.36 px between 90° and 95° against 0.86 px within 60° of the axis. H2 failed: DS (0.2119 px) fitted our ideal equidistant lens worse than EUCM (0.1231 px). This tests our analogue, an ideal lens with no manufacturing deviations, and does not contradict Usenko et al.’s results on real lenses. Every ray stayed inside every fitted model’s $\Omega$. DS’s fitted focal length (216.95 px) is far from the lens’s 300 px; Usenko et al.‘s Table 3 likewise shows a smaller DS focal length than the other models’ for each lens; in our reading, $f$ plays a different role in each model, so the fitted values are not comparable across models.

Mirrors. K7, K8, H5 and H6 held. The parabola with an orthographic camera and the hyperbola with a pinhole at its outer focus send every reflected ray through the focus $F_1$ to rounding error, and the unified model reproduces their images exactly, with $\xi = 1$ and $\gamma = h$ for the parabola. The negative control is detected: the same parabola seen by a pinhole has no common point, and its reflected lines miss the best one by up to 0.55 $h$, which is why the chapter requires an orthographic lens. H7 failed: for $a = b = 1$ the traced hyperbola has $\xi = 0.942809$, while Table 1 with $l$ equal to the latus rectum ($L = 2$) gives 0.577350, and $l^*/L - 1 = -0.75$. In our reading of these numbers, the printed formula $d/\sqrt{d^2 + 4l^2}$ agrees with the traced mirror only if $l$ is a quarter of the chord through the focus: with $d = 2\sqrt{2}$, $d/\sqrt{d^2 + L^2/4} = 2\sqrt{2}/3 = 0.942809$. Readers converting a mirror’s geometry to $\xi$ should check which length a source means by its symbol.

Sphere layouts. K9 and H8 held. Both layouts cover the sphere exactly, and at almost equal pixel counts they have the same mean pixel size, 0.14°. Their uniformity differs widely: an equirectangular pixel at the equator covers 652 times the solid angle of one at the pole, exactly $\cot(\pi/2048)$, whereas cube-map pixels vary by a factor of 5.18, just under the limit $3\sqrt{3} \approx 5.196$ between a face’s centre and its corners. This non-uniformity is the distortion that SphereNet, distortion-aware filters and tangent images address.

Disclosures. A planning script computed the float-precision references and the self-fit used to set the tolerances before the pilot; it computed none of the hypotheses’ measures. The output above comes from the second, final pilot. In the first pilot, the K2 check passed differently scaled points to the two formulas: the normalized point was multiplied by $(1 - \alpha)$ before being passed to ucm_unproject, which multiplies by $(1 - \alpha)$ again. K2 failed there. The repair passes the unscaled $f = 1$, $c = 0$ projection to ucm_unproject, forms the chapter’s $m$ from it by one multiplication, and adds the printed check that both formulas receive the same point (the first K2 line). The repair was approved by the editor before the second pilot ran. Between the two pilots, only the K2 outcome changed, from failed to held; every other printed line is identical.

Strengths

• Coverage. One sensor sees a hemisphere or more; Scaramuzza notes that landmarks can be tracked in all directions and over longer periods, which improves motion estimation and mapping, and that place recognition becomes easier than with perspective cameras.
• Motion estimation. Usenko et al. cite studies showing the benefit of a large field of view for visual odometry, and note that more textured regions are visible and that large motions give small pixel motions.
• Compact models with closed forms. EUCM and DS fit fisheye lenses closely with six parameters and have closed-form inverses, according to Usenko et al.’s evaluation.

Limitations

• Resolution. Spreading a sensor over a wide field lowers the angular resolution per pixel, as Eder et al.’s comparison of equirectangular and perspective images shows.
• Non-uniform sampling. Planar layouts of the sphere distort shapes and sample unevenly (the example’s 652-fold range for equirectangular pixels), which standard CNNs and feature detectors do not expect.
• Model validity. Sphere-based models are exact only for central catadioptric cameras; fisheye lenses are approximated, and each model has a bounded valid domain. Misalignment between camera and mirror breaks the single viewpoint.
• Mechanics. Catadioptric cameras are, in Usenko et al.’s words, mechanically complex and expensive; multi-camera rigs need synchronization and are not central.
• Tool support. Methods built for perspective images often exclude these cameras: 3D foundation models such as VGGT do not support fisheye or panoramic images, and visual servoing needs new feature models for fisheye or catadioptric cameras.

Typical Applications

Scaramuzza lists automotive surround views for drivers, miniature fisheye cameras in surgical endoscopes and on micro aerial vehicles for pipeline inspection and rescue, sky observation in meteorology, and photorealistic street views and city reconstructions. In robotics, omnidirectional cameras serve localization, mapping and navigation of ground and aerial robots (robot perception); the visual odometry benchmark used with Direct Sparse Odometry was recorded with fisheye lenses, and ORB-SLAM3 supports fisheye cameras (ORB-SLAM). Virtual reality and 360° video motivate the spherical-learning work above.

Comparison with Other Sensors

• Perspective and wide-angle cameras are central with a simple model and uniform image-plane sampling, but are limited to well under 180° (focal length and field of view); a fisheye lens beyond that limit is an omnidirectional camera described by the models above or by lens distortion’s fisheye projections.
• Multi-camera rigs stitched into panoramas (image stitching) give the full sphere at higher resolution, at the cost of parallax between camera centres and of synchronization.
• LiDAR scanners also cover 360° in azimuth, but measure range directly at sparse angles; an omnidirectional camera gives dense appearance without depth, which must be recovered by stereo or motion.