Vision Systems & Engineering
Multi-Camera Systems
Rigs of several cameras treated as one instrument, covering topologies, the overlap graph between views, joint extrinsic calibration of more than two cameras and its upkeep in service, the synchronization budget, clock mapping of many streams, and bandwidth on shared links.
intermediate
A multi-camera system is a set of cameras whose images are used together, so that the set behaves as one sensor: a stereo camera, the surround rig of a car, a ring of cameras around a capture volume, or a factory cell with a camera on each side of a part. What makes it a system rather than several cameras is shared knowledge: where every camera sits relative to the others, which frames describe the same moment, and how all the streams reach the computer that uses them. Each of the three can fail on its own, and each fails more often as cameras are added.
This article treats the rig as a whole. Calibrating one camera, and the two-camera case that OpenCV’s stereoCalibrate handles, are in camera calibration; the levels of synchronization, capture timestamps and clock domains are in real-time vision pipelines; trigger wiring, action signals and the machine-vision interface standards are in industrial machine vision.
Definition
A multi-camera system (or rig) is a set of cameras with known intrinsic parameters, known poses relative to a common reference frame, and a known relation between the times at which their frames are exposed. Its calibration is the set of extrinsic transforms, expressed here, as in OpenCV’s calibrateMultiview, in the frame of one reference camera; its synchronization is the rule that groups frames from different cameras into sets that describe one instant.
Two cameras overlap when some part of the scene, or a calibration target, can be seen by both at once. The overlap graph has a node per camera and an edge between every pair that overlaps, weighted by how much they share (for calibration, by the number of target views both cameras saw).
Intuition
A rig is calibrated by showing it things that several cameras see at the same time. With two cameras this is a single relation. With eight cameras in a ring, where each camera sees only what its neighbours see, no single target view relates camera 0 to camera 4: the relation has to be passed along a chain of neighbours, and every link adds its own error. A chain is also redundant in places. Going round a ring from camera 0 back to camera 0, the composed relative poses should give the identity; in practice they do not quite, and the leftover rotation measures how inconsistent the pairwise estimates are.
The answer OpenCV’s multiview calibration uses is to start from a chain, then fit everything at once. A chain along a spanning tree of the overlap graph gives every camera a first pose. A joint least-squares fit of all camera poses and all target poses to all observations then lets the far cameras borrow constraints from both sides of the ring, as bundle adjustment does for a moving camera.
Once in service, a rig can drift: a camera is knocked, a mount relaxes, a lens is refocused. A rig that stays calibrated is one that is checked, with fresh observations compared against the stored calibration, and repaired one camera at a time when only one has moved.
Formal Definition
Write each camera’s pose as world-to-camera, , with OpenCV’s camera axes (x right, y down, z forward) and centre . Calibration is only defined up to a rigid motion of the whole rig (the gauge), so one camera is chosen as the reference: , , and every other pose is relative to it. With a target of known size, the metric scale comes from the target; without one, it is free as well.
A planar target in view has a pose in camera 0’s frame and points in its own plane. Camera observes when it sees view . Joint calibration minimizes the reprojection error over all cameras except the reference and all target poses,
where is the set of (camera, view) pairs in which the target was detected and the pinhole projection. The reference camera’s pose does not enter as an unknown, but its observations do.
Relative poses compose along the graph. If camera ’s pose relative to camera is , then
For a cycle of cameras with independently estimated relative rotations , the loop closure discrepancy is the angle of , which is zero for exact estimates. For any solution expressed as absolute poses, composing its own relative poses around the loop gives the identity exactly, so a loop measures the consistency of pairwise estimates, not of a final calibration.
Properties
Topologies
Rigs differ in what the cameras share:
- Stereo pairs and small clusters share most of their fields of view and are calibrated as pairs; the stereo camera article covers the device.
- Surround rigs on vehicles point outward and, in our reading, overlap only near their edges. In BEV perception, the usual input is such a rig with its calibration; nuScenes has six cameras covering 360 degrees, as that article reports.
- Inward-looking rings and domes around a capture volume see a central object from all sides. In our reasoning, opposite cameras see opposite faces of a planar target, so a target is seen by a few neighbouring cameras at a time, and the overlap graph is a ring with short chords.
- Networks of fixed views, such as surveillance installations, may overlap partly or not at all; video surveillance describes datasets of both kinds.
The topology fixes the overlap graph, and the overlap graph fixes how calibration error is distributed. Cameras that share views with the reference are tied to it directly; cameras that do not are tied through others.
The overlap graph and the spanning-tree initialization
OpenCV 5.0.0’s calibrateMultiview, in the calib module, follows the scheme above (OpenCV, multiview_calibration.cpp). It checks that the detection mask connects every camera, calibrates or poses each camera on its own, and builds a weighted overlap graph. The function maximumSpanningTree weights each camera pair by the number of frames both detected plus, for each shared frame, the two views’ ratios of point-hull area to image area, 0.5 times the number of the two views that pass a 160° test on the angle between the camera’s optical axis and the pattern normal (the test compares for that angle with 160°, which in our reading of the code every view passes, since that minimum never exceeds 90°), and 1.0 if the two cameras’ optical axes are less than 160° apart; it then finds a maximum spanning tree with Prim’s algorithm, starting from camera 0. A breadth-first traversal from camera 0 orders the tree’s pairs, a comment in the source noting that this order makes it easier to find views relative to camera 0. By default each pair is registered with registerCameras, with the intrinsics fixed (a flag selects a pairwise stereo calibration instead), and the relative poses are chained with the composition formula above. Each frame’s target pose is initialized from the camera whose single-camera fit of that frame had the smallest error. A Levenberg–Marquardt fit then refines the camera poses and the frame poses together; a comment states that camera 0’s pose is not optimized but its reprojection error is taken into account. The errors pass through a robust exponential weighting function (scale 30 in the call).
A maximum spanning tree keeps the strongest overlaps, but it makes a far camera’s initial pose depend on every link of its path to the reference, measured in hops (tree depth). The joint fit is what removes that dependence. DUSt3R’s official implementation uses a spanning tree for a similar purpose, to initialize the global alignment of pairwise pointmaps (init="mst", as that article describes), and ORB-SLAM keeps a spanning tree of keyframes inside its Essential Graph. Distributing a loop’s discrepancy over a pose graph is the subject of pose graph optimization and loop closure; a calibration rig is the special case in which the graph is fixed and every camera is in view of a known target.
Calibration upkeep
A stored calibration is a hypothesis about the rig that stays true only while nothing moves. The practical test, in our reasoning, is to observe a target again and compare: with the stored poses fixed, fit each new target pose to all cameras that see it and look at each camera’s reprojection RMS. A camera that has moved shows an RMS well above the noise floor. Its neighbours can show one too, because a target seen by the moved camera and a neighbour is fitted to cameras that disagree. The example below measures both effects. Once the moved camera is known, it can be re-estimated alone against the others, keeping the reference frame and every other camera unchanged, which preserves anything already computed in that frame.
The synchronization budget
The real-time vision pipelines article sorts synchronization into hardware triggering, network triggering with synchronized clocks, software-paced capture and matching by timestamp (its own grouping). A rig needs a budget for the offset it can tolerate, and the budget is set by the scene, not by the cameras. In our arithmetic, a point moving at speed at distance from a camera of focal length (in pixels) moves in the image by about
during an offset . With px, m and m/s, an offset of 1 ms gives 0.33 px, about the noise level used in the example below; free-running 30 fps cameras matched by nearest timestamp can be up to half a frame period apart, 16.7 ms, or about 5.6 px. For cameras the budget applies to the worst pair, not the average. The stereo camera article derives what such an offset does to stereo depth.
GenICam’s naming convention gives the controls a rig uses. Action signals, which industrial machine vision describes, can raise a FrameStart trigger in every camera of a group at once (SFNC 2.8, Section 14.1). The same convention defines ActionQueueSize, the maximum number of scheduled action commands that can be pending at once (Section 14.2.4), and an ActionLate event for a valid scheduled action command whose execution time is already past (Section 15.2); in our reading, a trigger can thus be sent ahead of time and executed when the device’s clock reaches the scheduled time. PtpEnable turns on the Precision Time Protocol and PtpStatus returns the latched state of the PTP clock (Sections 27.3.2 and 27.3.5).
Mapping the clocks of many streams
Each camera stamps its frames with its own timestamp counter, in nanoseconds, the same counter used for image, chunk and event data (SFNC 2.8, Section 3.66). Without PTP, cameras give clock domains, each with its own offset and rate relative to the host. TimestampLatch copies the current counter into TimestampLatchValue (Sections 3.68 and 3.69); in our reasoning, latching each camera repeatedly while reading the host’s monotonic clock gives pairs of readings to which an offset and a drift can be fitted per camera, mapping every stream into one domain before frames are grouped. In our reasoning, the uncertainty of each reading, including the time the latch command takes to reach the device, sets a floor on how well the streams can be aligned this way, and a budget tighter than that floor needs a hardware trigger or PTP.
Bandwidth on shared links
In our arithmetic, synchronized cameras deliver their frames at the same moment, and eight cameras of 8-bit pixels at 30 fps produce MB/s, about 2.36 Gbit/s of pixel data before protocol overhead, more than one Gigabit Ethernet link carries. Even when the average fits, a common trigger produces a burst: the eight frames total 9.8 MB, which take about 79 ms to cross a 1 Gbit/s link, longer than the 33 ms frame period. The data must be spread, either over several links or in time. SFNC’s DeviceLinkThroughputLimit limits the bandwidth a device streams on its link, inserting delays uniformly between transport-layer packets if necessary to control the peak (SFNC 2.8, Section 3.37), and DeviceLinkThroughputLimitMode switches it on or off (Section 3.36). On GigE Vision, GevSCPD sets the delay between packets of a stream channel in units of the GEV timestamp counter, which the convention calls a crude flow-control mechanism (Section 27.4.70). In our reading, limiting each camera’s peak trades latency for not losing packets; the interfaces themselves are compared in industrial machine vision.
Examples
Calibrating a ring of eight cameras, and finding the camera that moved
The script below builds an inward-looking ring of eight cameras of radius 3 m, at heights of +0.3 m (even cameras) and −0.3 m (odd cameras), each looking at the centre, with known ( px, pixels, no distortion). A planar board of points at 0.1 m pitch is shown at 40 seeded poses near the centre; each pose is kept only if exactly two or three cameras see its front within 50° of its normal with every point at least 10 pixels inside the image, so the overlap graph is a ring in which three-camera views add edges between second neighbours. Image points get 0.3-pixel Gaussian noise per coordinate. All poses are world-to-camera and expressed in camera 0’s frame; errors are the rotation angle of in degrees and the camera-centre distance in millimetres; RMS is the square root of the mean squared residual length per point, as OpenCV’s computeReprojectionMSE divides the squared norm by the number of points.
Part A follows the steps of OpenCV 5.0.0’s calibrateMultiview, re-implemented from its description rather than ported, with three differences, each fixed in the plan before the pilot ran. The spanning tree is weighted by overlap counts alone, without OpenCV’s area-ratio and angle terms. Each tree edge’s relative pose is the chordal mean of the per-view relative rotations, with the mean of the translations, from board poses obtained by decomposing each view’s homography with known, as in Zhang’s method in camera calibration and homography; OpenCV registers each pair with registerCameras. The joint fit uses the plain squared loss with SciPy’s least_squares, given the Jacobian’s sparsity pattern, where OpenCV uses its own Levenberg–Marquardt with a robust weighting. The outcomes therefore test this re-implementation, not OpenCV. Board poses are initialized from the camera with the lowest per-view RMS, as OpenCV does, and camera 0 is fixed.
Part B turns camera 4 by 0.5° about its own y axis, keeping its centre, and shows the rig 10 new boards, one in each 36° band of directions. It reports each camera’s RMS with the true geometry and the boards fitted (the floor), under the old calibration from Part A with the boards fitted, and after re-estimating camera 4 alone with the others fixed; and camera 4’s rotation error against its true new pose. The ring is generated five times (seeds 0–4); seed 0 is also run without noise, where every step must recover the truth.
The plan fixed these checks and hypotheses, with their bars, before any code ran. C0: without noise every step recovers the ground truth within (degrees, millimetres, pixels), and camera 4’s RMS under the old calibration is above zero. C1: in every seed each board is seen by two or three cameras, every pair of ring neighbours shares at least one board, no pair more than two apart around the ring shares one, the tree has seven edges, and camera 4 sees at least two of the new boards. H1: the chained solution’s error grows with hop count (positive least-squares slope of error on hops, pooled over cameras 1–7 and the five seeds). H2: joint refinement removes most of the growth (joint slope at most half the chained slope). H3a: in each seed, the chained solution’s largest disagreement with the eight ring-edge pairwise estimates exceeds the joint solution’s; H3b: the joint solution’s largest disagreement is at most three times , the RMS over the ring edges of the pairwise estimates’ rotation error against the truth. H4a: camera 4’s RMS under the old calibration exceeds three times its floor; H4b: it is the largest of the eight; H4c: no other camera exceeds three times its floor. H5a: after re-estimation camera 4’s RMS is at most 1.2 times its floor; H5b: its rotation error is below 0.05°, a tenth of the move. The script runs in about 20 seconds with NumPy 2.3.5 and SciPy 1.16.3 (Python 3.13).
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np
from scipy.optimize import least_squares
from scipy.sparse import lil_matrix
from scipy.spatial.transform import Rotation
# ---------- the rig and the board (world-to-camera: x_cam = R X + t) ----------
K = np.array([[1000.0, 0, 640], [0, 1000.0, 480], [0, 0, 1]])
W, H, N_CAM, SIGMA = 1280, 960, 8, 0.3
gx, gy = np.meshgrid(np.arange(9) * 0.1 - 0.4, np.arange(6) * 0.1 - 0.25)
BOARD = np.c_[gx.ravel(), gy.ravel(), np.zeros(54)] # 9 x 6 points, 0.1 m pitch
def look_at(C): # camera at C looking at the origin, y down
z = -C / np.linalg.norm(C)
x = np.cross(z, [0, 0, 1.0]); x /= np.linalg.norm(x)
R = np.vstack([x, np.cross(z, x), z])
return R, -R @ C
def ring():
cams = []
for k in range(N_CAM):
a = np.deg2rad(45 * k)
cams.append(look_at(np.array([3 * np.cos(a), 3 * np.sin(a), 0.3 if k % 2 == 0 else -0.3])))
return cams
def project(R, t, X):
x = X @ R.T + t
return x[:, :2] / x[:, 2:] * K[0, 0] + K[:2, 2], x[:, 2]
def visible(cam, Rb, tb): # front side within 50 deg, all points in image
R, t = cam
C = -R.T @ t
v = C - tb
if np.dot(Rb[:, 2], v) / np.linalg.norm(v) < np.cos(np.deg2rad(50)):
return False
uv, z = project(R @ Rb, R @ tb + t, BOARD)
return bool(np.all(z > 0) and np.all((uv > 10) & (uv < [W - 10, H - 10])))
def random_board(rng, cams, az_range=(0, 360)): # board-to-world pose seen by 2 or 3 cameras
while True:
az = np.deg2rad(rng.uniform(*az_range)); el = np.deg2rad(rng.uniform(-20, 20))
n = np.array([np.cos(el) * np.cos(az), np.cos(el) * np.sin(az), np.sin(el)])
roll = np.deg2rad(rng.uniform(-30, 30))
r = rng.uniform(0, 0.3); phi = rng.uniform(0, 2 * np.pi)
tb = np.array([r * np.cos(phi), r * np.sin(phi), rng.uniform(-0.2, 0.2)])
x0 = np.cross([0, 0, 1.0], n); x0 /= np.linalg.norm(x0)
y0 = np.cross(n, x0)
x1 = np.cos(roll) * x0 + np.sin(roll) * y0
Rb = np.c_[x1, np.cross(n, x1), n] # board z axis = front normal
seen = [k for k in range(N_CAM) if visible(cams[k], Rb, tb)]
if 2 <= len(seen) <= 3:
return Rb, tb, seen
def observe(cams, boards, rng, sigma):
obs = {}
for f, (Rb, tb, seen) in enumerate(boards):
for k in seen:
R, t = cams[k]
uv, _ = project(R @ Rb, R @ tb + t, BOARD)
obs[(k, f)] = uv + rng.normal(0, sigma, uv.shape)
return obs
def to_cam0(cams): # gauge: express every camera in camera 0's frame
R0, t0 = cams[0]
return [(R @ R0.T, t - R @ R0.T @ t0) for R, t in cams]
# ---------- step 1: one board pose per camera and view, from the homography ----------
def homography(xy, uv):
def norm(p):
c = p.mean(0); s = np.sqrt(2) / np.mean(np.linalg.norm(p - c, axis=1))
return np.array([[s, 0, -s * c[0]], [0, s, -s * c[1]], [0, 0, 1]])
T1, T2 = norm(xy), norm(uv)
a = (T1 @ np.c_[xy, np.ones(len(xy))].T).T; b = (T2 @ np.c_[uv, np.ones(len(uv))].T).T
A = []
for (x, y, _), (u, v, _) in zip(a, b):
A.append([-x, -y, -1, 0, 0, 0, u * x, u * y, u])
A.append([0, 0, 0, -x, -y, -1, v * x, v * y, v])
Hn = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
return np.linalg.inv(T2) @ Hn @ T1
def pose_from_homography(Hm): # Zhang's decomposition with K known
B = np.linalg.inv(K) @ Hm
lam = 1 / np.linalg.norm(B[:, 0])
if B[2, 2] < 0:
lam = -lam # board in front of the camera
r1, r2, t = lam * B[:, 0], lam * B[:, 1], lam * B[:, 2]
U, _, Vt = np.linalg.svd(np.c_[r1, r2, np.cross(r1, r2)])
return U @ np.diag([1, 1, np.linalg.det(U @ Vt)]) @ Vt, t
def rms(res):
return np.sqrt(np.mean(np.sum(res.reshape(-1, 2) ** 2, axis=1)))
def view_poses(obs):
poses, err = {}, {}
for (k, f), uv in obs.items():
R, t = pose_from_homography(homography(BOARD[:, :2], uv))
poses[(k, f)] = (R, t)
err[(k, f)] = rms(project(R, t, BOARD)[0] - uv)
return poses, err
# ---------- step 2: maximum spanning tree on overlap counts (Prim, from camera 0) ----------
def overlap_counts(obs, n_boards):
seen = np.zeros((N_CAM, n_boards), bool)
for k, f in obs:
seen[k, f] = True
O = (seen[:, None, :] & seen[None, :, :]).sum(2)
np.fill_diagonal(O, 0)
return O, seen
def prim_tree(O):
parent = [-1] * N_CAM
best = np.full(N_CAM, -np.inf); best[0] = np.inf
done = np.zeros(N_CAM, bool)
for _ in range(N_CAM):
cand = np.where(~done, best, -np.inf)
u = int(np.argmax(cand)) # ties go to the lowest index
if cand[u] == -np.inf:
raise RuntimeError("overlap graph is not connected")
done[u] = True
for v in range(N_CAM):
if not done[v] and O[u, v] > best[v]:
best[v], parent[v] = O[u, v], u
return parent
def bfs_pairs(parent):
pairs, level, hops = [], [0], {0: 0}
while level:
nxt = []
for p in level:
for c in range(N_CAM):
if parent[c] == p:
pairs.append((p, c)); hops[c] = hops[p] + 1; nxt.append(c)
level = nxt
return pairs, hops
# ---------- step 3: chained initialization along the tree ----------
def relative_pose(poses, seen, a, b): # camera a -> camera b from shared boards
Rs, ts = [], []
for f in np.where(seen[a] & seen[b])[0]:
Ra, ta = poses[(a, f)]; Rb_, tb_ = poses[(b, f)]
Rr = Rb_ @ Ra.T
Rs.append(Rr); ts.append(tb_ - Rr @ ta)
U, _, Vt = np.linalg.svd(np.sum(Rs, axis=0)) # chordal mean of the rotations
return U @ np.diag([1, 1, np.linalg.det(U @ Vt)]) @ Vt, np.mean(ts, axis=0)
def chain(poses, seen, pairs):
cams = {0: (np.eye(3), np.zeros(3))}
for p, c in pairs:
Rr, tr = relative_pose(poses, seen, p, c)
Rp, tp = cams[p]
cams[c] = (Rr @ Rp, tr + Rr @ tp)
return [cams[k] for k in range(N_CAM)]
def board_init(poses, err, cams, n_boards): # each board from the camera that fits it best
out = []
for f in range(n_boards):
k = min((e, k) for (k, g), e in err.items() if g == f)[1]
Rk, tk = cams[k]; Rv, tv = poses[(k, f)]
out.append((Rk.T @ Rv, Rk.T @ (tv - tk)))
return out
# ---------- step 4: joint refinement (cameras not listed in `free` stay fixed) ----------
def refine(obs, cams, boards, free):
keys = sorted(obs)
nb, nf = len(boards), len(free)
kk = np.array([k for k, _ in keys]); ff = np.array([f for _, f in keys])
UV = np.stack([obs[key] for key in keys]) # (n_obs, 54, 2)
Rc0 = np.stack([c[0] for c in cams]); tc0 = np.stack([c[1] for c in cams])
def unpack(x):
Rc, tc = Rc0.copy(), tc0.copy()
if nf:
p = x[:6 * nf].reshape(nf, 6)
Rc[free] = Rotation.from_rotvec(p[:, :3]).as_matrix(); tc[free] = p[:, 3:]
q = x[6 * nf:].reshape(nb, 6)
return Rc, tc, Rotation.from_rotvec(q[:, :3]).as_matrix(), q[:, 3:]
def resid(x):
Rc, tc, Rb, tb = unpack(x)
Xw = np.einsum("bij,pj->bpi", Rb, BOARD) + tb[:, None, :] # board points in camera 0's frame
Xc = np.einsum("oij,opj->opi", Rc[kk], Xw[ff]) + tc[kk][:, None, :]
uv = Xc[..., :2] / Xc[..., 2:] * K[0, 0] + K[:2, 2]
return (uv - UV).ravel()
x0 = np.concatenate([np.r_[Rotation.from_matrix(cams[k][0]).as_rotvec(), cams[k][1]] for k in free] +
[np.r_[Rotation.from_matrix(R).as_rotvec(), t] for R, t in boards])
S = lil_matrix((108 * len(keys), len(x0)), dtype=int)
for i, (k, f) in enumerate(keys):
if k in free:
j = free.index(k); S[108 * i:108 * i + 108, 6 * j:6 * j + 6] = 1
o = 6 * nf + 6 * f; S[108 * i:108 * i + 108, o:o + 6] = 1
sol = least_squares(resid, x0, jac_sparsity=S, x_scale="jac", ftol=1e-10, xtol=1e-10, gtol=1e-10)
Rc, tc, Rb, tb = unpack(sol.x)
return [(Rc[k], tc[k]) for k in range(N_CAM)], [(Rb[f], tb[f]) for f in range(nb)], resid(sol.x)
def per_camera_rms(obs, cams, boards):
out = {}
for k in range(N_CAM):
r = [project(cams[k][0] @ boards[f][0], cams[k][0] @ boards[f][1] + cams[k][1], BOARD)[0] - uv
for (kk, f), uv in obs.items() if kk == k]
out[k] = rms(np.concatenate(r).ravel()) if r else np.nan
return out
# ---------- measures ----------
def rot_err(Ra, Rb):
return np.rad2deg(Rotation.from_matrix(Ra @ Rb.T).magnitude())
def centre_err(ca, cb):
return 1000 * np.linalg.norm(-ca[0].T @ ca[1] + cb[0].T @ cb[1])
def ring_disagreement(cams, poses, seen):
d, P = [], []
for k in range(N_CAM):
a, b = k, (k + 1) % N_CAM
Rr, _ = relative_pose(poses, seen, a, b)
P.append(Rr)
d.append(rot_err(cams[b][0] @ cams[a][0].T, Rr))
return np.array(d), P
def run(seed, sigma):
rng = np.random.default_rng(seed)
truth_world = ring()
boards_w = [random_board(rng, truth_world) for _ in range(40)]
obs = observe(truth_world, boards_w, np.random.default_rng(seed + 1000), sigma)
truth = to_cam0(truth_world)
R0w, t0w = truth_world[0]
boards_true = [(R0w @ Rb, R0w @ tb + t0w) for Rb, tb, _ in boards_w]
poses, err = view_poses(obs)
O, seen = overlap_counts(obs, 40)
parent = prim_tree(O)
pairs, hops = bfs_pairs(parent)
chained = chain(poses, seen, pairs)
binit = board_init(poses, err, chained, 40)
joint, bjoint, res_joint = refine(obs, chained, binit, list(range(1, N_CAM)))
res_chained = np.concatenate([(project(chained[k][0] @ binit[f][0], chained[k][0] @ binit[f][1] + chained[k][1], BOARD)[0] - obs[(k, f)]).ravel() for k, f in sorted(obs)])
res_true = np.concatenate([(project(truth[k][0] @ boards_true[f][0], truth[k][0] @ boards_true[f][1] + truth[k][1], BOARD)[0] - obs[(k, f)]).ravel() for k, f in sorted(obs)])
A = dict(truth=truth, poses=poses, seen=seen, O=O, pairs=pairs, hops=hops, chained=chained, joint=joint,
boards_true=boards_true, bjoint=bjoint, binit=binit,
rms=(rms(res_true), rms(res_chained), rms(res_joint)),
counts=[len(s) for _, _, s in boards_w])
# ----- Part B: camera 4 turns by 0.5 deg about its own y axis; 10 new boards -----
Rd = Rotation.from_rotvec(np.deg2rad(0.5) * np.array([0, 1.0, 0])).as_matrix()
moved_world = list(truth_world); moved_world[4] = (Rd @ truth_world[4][0], Rd @ truth_world[4][1])
rngB = np.random.default_rng(seed + 2000)
newb = [random_board(rngB, moved_world, (36 * i, 36 * (i + 1))) for i in range(10)]
obsB = observe(moved_world, newb, np.random.default_rng(seed + 3000), sigma)
moved = to_cam0(moved_world)
posesB, errB = view_poses(obsB)
_, b_floor, _ = refine(obsB, moved, board_init(posesB, errB, moved, 10), [])
_, b_old, _ = refine(obsB, joint, board_init(posesB, errB, joint, 10), [])
new, b_new, _ = refine(obsB, joint, b_old, [4])
B = dict(floor=per_camera_rms(obsB, moved, b_floor), before=per_camera_rms(obsB, joint, b_old),
after=per_camera_rms(obsB, new, b_new), n4=sum(1 for k, _ in obsB if k == 4),
err4_before=rot_err(joint[4][0], moved[4][0]), err4_after=rot_err(new[4][0], moved[4][0]),
c4_after=centre_err(new[4], moved[4]), counts=[len(s) for _, _, s in newb])
return A, B
def slope(h, e):
h, e = np.asarray(h, float), np.asarray(e, float)
return np.polyfit(h, e, 1)[0]
# ---------- C0: noise-free, seed 0 ----------
A, B = run(0, 0.0)
c0 = {
"homography poses, rot deg": max(rot_err(A["poses"][(k, f)][0], A["truth"][k][0] @ A["boards_true"][f][0]) for k, f in A["poses"]),
"homography poses, trans mm": max(1000 * np.linalg.norm(A["poses"][(k, f)][1] - (A["truth"][k][0] @ A["boards_true"][f][1] + A["truth"][k][1])) for k, f in A["poses"]),
"chained, rot deg": max(rot_err(A["chained"][k][0], A["truth"][k][0]) for k in range(N_CAM)),
"chained, centre mm": max(centre_err(A["chained"][k], A["truth"][k]) for k in range(N_CAM)),
"joint, rot deg": max(rot_err(A["joint"][k][0], A["truth"][k][0]) for k in range(N_CAM)),
"joint, centre mm": max(centre_err(A["joint"][k], A["truth"][k]) for k in range(N_CAM)),
"joint RMS px": A["rms"][2],
"B: camera 4 re-estimated, rot deg": B["err4_after"],
"B: camera 4 re-estimated, centre mm": B["c4_after"],
"B: camera 4 RMS after px": B["after"][4],
}
print("C0, noise-free (tolerance 1e-6):")
for name, v in c0.items():
print(f" {name:38s} {v:.1e} {'held' if v < 1e-6 else 'failed'}")
print(f" B: camera 4 RMS before px {B['before'][4]:.4f} {'held' if B['before'][4] > 0 else 'failed'} (> 0)")
# ---------- noisy runs, seeds 0-4 ----------
HOP, ER = [], {m: [] for m in ("cr", "cc", "jr", "jc")}
H3, RMS_A, PB = [], [], []
print("\nC1 data checks per seed: boards seen by 2/3 cameras, min boards on a ring edge,"
" max ring distance sharing a board, tree edges, Part B boards seen by camera 4")
for seed in range(5):
A, B = run(seed, SIGMA)
O = A["O"]
ring_min = min(O[k, (k + 1) % N_CAM] for k in range(N_CAM))
far = max([min(abs(a - b), N_CAM - abs(a - b)) for a in range(N_CAM) for b in range(N_CAM) if O[a, b] > 0])
print(f" seed {seed}: {A['counts'].count(2)}/{A['counts'].count(3)}, {ring_min}, {far}, {len(A['pairs'])}, {B['n4']}"
f" (Part B boards seen by 2/3: {B['counts'].count(2)}/{B['counts'].count(3)})")
for k in range(1, N_CAM):
HOP.append(A["hops"][k])
ER["cr"].append(rot_err(A["chained"][k][0], A["truth"][k][0])); ER["cc"].append(centre_err(A["chained"][k], A["truth"][k]))
ER["jr"].append(rot_err(A["joint"][k][0], A["truth"][k][0])); ER["jc"].append(centre_err(A["joint"][k], A["truth"][k]))
dc, P = ring_disagreement(A["chained"], A["poses"], A["seen"])
dj, _ = ring_disagreement(A["joint"], A["poses"], A["seen"])
eps = np.sqrt(np.mean([rot_err(P[k], A["truth"][(k + 1) % N_CAM][0] @ A["truth"][k][0].T) ** 2 for k in range(N_CAM)]))
loop = np.eye(3)
for k in range(N_CAM):
loop = P[k] @ loop
H3.append((seed, rot_err(loop, np.eye(3)), eps, dc.max(), dj.max()))
RMS_A.append((seed,) + A["rms"])
PB.append((seed, B))
if seed == 0:
A0, B0 = A, B
print("\nSeed 0: tree pairs (parent, child):", A0["pairs"])
print("camera hops chained rot deg centre mm joint rot deg centre mm")
for k in range(1, N_CAM):
print(f" {k} {A0['hops'][k]} {rot_err(A0['chained'][k][0], A0['truth'][k][0]):.5f} {centre_err(A0['chained'][k], A0['truth'][k]):8.3f}"
f" {rot_err(A0['joint'][k][0], A0['truth'][k][0]):.5f} {centre_err(A0['joint'][k], A0['truth'][k]):8.3f}")
print("\nSeeds 0-4 pooled: mean error by hop count (n = cameras)")
print("hops n chained rot deg centre mm joint rot deg centre mm")
Hn = np.array(HOP)
for h in sorted(set(HOP)):
m = Hn == h
print(f" {h} {m.sum():2d} {np.mean(np.array(ER['cr'])[m]):.5f} {np.mean(np.array(ER['cc'])[m]):8.3f}"
f" {np.mean(np.array(ER['jr'])[m]):.5f} {np.mean(np.array(ER['jc'])[m]):8.3f}")
s = {m: slope(HOP, ER[m]) for m in ER}
print(f"slope per hop: chained {s['cr']:.6f} deg, {s['cc']:.4f} mm; joint {s['jr']:.6f} deg, {s['jc']:.4f} mm")
print(f"H1 chained slope > 0: rot {'held' if s['cr'] > 0 else 'failed'}, centre {'held' if s['cc'] > 0 else 'failed'}")
print(f"H2 joint/chained slope <= 0.5: rot {s['jr'] / s['cr']:.4f} {'held' if s['jr'] <= 0.5 * s['cr'] else 'failed'},"
f" centre {s['jc'] / s['cc']:.4f} {'held' if s['jc'] <= 0.5 * s['cc'] else 'failed'}")
print("\nRing closure: seed, loop angle of the pairwise estimates deg, eps_pair deg, largest ring-edge disagreement chained, joint deg")
for seed, lo, eps, mc, mj in H3:
print(f" {seed} {lo:.5f} {eps:.5f} {mc:.5f} {mj:.5f} H3a {'held' if mc > mj else 'failed'}"
f" H3b ratio {mj / eps:.3f} {'held' if mj <= 3 * eps else 'failed'}")
print("\nPart A reprojection RMS px: seed, at the true parameters, chained, joint")
for seed, rt, rc, rj in RMS_A:
print(f" {seed} {rt:.4f} {rc:.4f} {rj:.4f}")
print("\nPart B, seed 0: per-camera RMS px on the 10 new boards")
print("camera floor old calibration camera 4 re-estimated")
for k in range(N_CAM):
print(f" {k} {B0['floor'][k]:.4f} {B0['before'][k]:.4f} {B0['after'][k]:.4f}")
print("\nPart B, seeds 0-4: seed, cam 4 before/floor, largest other before/floor (camera), cam 4 after/floor, cam 4 rot err before, after deg")
for seed, B in PB:
ratio = {k: B["before"][k] / B["floor"][k] for k in range(N_CAM)}
other = max((r, k) for k, r in ratio.items() if k != 4)
top = max(B["before"], key=B["before"].get)
a4 = B["after"][4] / B["floor"][4]
print(f" {seed} {ratio[4]:.3f} {other[0]:.3f} ({other[1]}) {a4:.4f} {B['err4_before']:.5f} {B['err4_after']:.5f}"
f" H4a {'held' if ratio[4] > 3 else 'failed'} H4b {'held' if top == 4 else 'failed'} (largest: {top})"
f" H4c {'held' if other[0] <= 3 else 'failed'} H5a {'held' if a4 <= 1.2 else 'failed'}"
f" H5b {'held' if B['err4_after'] < 0.05 else 'failed'}")
C0, noise-free (tolerance 1e-6):
homography poses, rot deg 2.9e-13 held
homography poses, trans mm 9.8e-12 held
chained, rot deg 2.1e-13 held
chained, centre mm 1.3e-11 held
joint, rot deg 4.0e-14 held
joint, centre mm 2.9e-12 held
joint RMS px 1.1e-13 held
B: camera 4 re-estimated, rot deg 1.1e-14 held
B: camera 4 re-estimated, centre mm 1.0e-12 held
B: camera 4 RMS after px 2.8e-13 held
B: camera 4 RMS before px 1.8458 held (> 0)
C1 data checks per seed: boards seen by 2/3 cameras, min boards on a ring edge, max ring distance sharing a board, tree edges, Part B boards seen by camera 4
seed 0: 36/4, 3, 2, 7, 3 (Part B boards seen by 2/3: 9/1)
seed 1: 31/9, 1, 2, 7, 3 (Part B boards seen by 2/3: 9/1)
seed 2: 37/3, 3, 2, 7, 3 (Part B boards seen by 2/3: 9/1)
seed 3: 36/4, 2, 2, 7, 3 (Part B boards seen by 2/3: 7/3)
seed 4: 32/8, 2, 2, 7, 4 (Part B boards seen by 2/3: 8/2)
Seed 0: tree pairs (parent, child): [(0, 1), (0, 7), (1, 2), (7, 6), (6, 5), (5, 4), (4, 3)]
camera hops chained rot deg centre mm joint rot deg centre mm
1 1 0.37440 19.088 0.03599 1.980
2 2 0.33429 16.578 0.04642 2.598
3 5 0.21553 11.946 0.05380 3.493
4 4 0.36766 14.512 0.03697 1.858
5 3 0.38494 3.659 0.13403 6.199
6 2 0.29837 11.696 0.02577 0.777
7 1 0.18317 8.402 0.04495 1.873
Seeds 0-4 pooled: mean error by hop count (n = cameras)
hops n chained rot deg centre mm joint rot deg centre mm
1 10 0.25038 12.286 0.04267 2.129
2 9 0.30254 13.106 0.05121 2.324
3 6 0.56479 21.985 0.06900 2.424
4 5 0.64202 29.107 0.06817 2.941
5 4 0.69226 29.533 0.09428 4.671
6 1 1.24476 69.776 0.10924 5.794
slope per hop: chained 0.143388 deg, 6.6027 mm; joint 0.012123 deg, 0.5608 mm
H1 chained slope > 0: rot held, centre held
H2 joint/chained slope <= 0.5: rot 0.0845 held, centre 0.0849 held
Ring closure: seed, loop angle of the pairwise estimates deg, eps_pair deg, largest ring-edge disagreement chained, joint deg
0 0.14424 0.31451 0.14424 0.63423 H3a failed H3b ratio 2.017 held
1 0.46853 0.30908 0.46853 0.52103 H3a failed H3b ratio 1.686 held
2 1.25004 0.32806 1.25004 0.49470 H3a held H3b ratio 1.508 held
3 0.84542 0.27111 0.84542 0.38506 H3a held H3b ratio 1.420 held
4 1.13050 0.32743 1.13050 0.64274 H3a held H3b ratio 1.963 held
Part A reprojection RMS px: seed, at the true parameters, chained, joint
0 0.4207 1.0969 0.4140
1 0.4245 1.1324 0.4186
2 0.4199 0.9378 0.4141
3 0.4240 0.9843 0.4169
4 0.4237 0.9779 0.4178
Part B, seed 0: per-camera RMS px on the 10 new boards
camera floor old calibration camera 4 re-estimated
0 0.4000 0.3993 0.3993
1 0.4351 0.4359 0.4359
2 0.4117 0.4138 0.4138
3 0.3967 1.0864 0.3966
4 0.4036 1.9149 0.4021
5 0.4249 2.1827 0.4217
6 0.4032 1.2323 0.4020
7 0.3996 0.4012 0.4012
Part B, seeds 0-4: seed, cam 4 before/floor, largest other before/floor (camera), cam 4 after/floor, cam 4 rot err before, after deg
0 4.744 5.137 (5) 0.9962 0.52288 0.09685 H4a held H4b failed (largest: 5) H4c failed H5a held H5b failed
1 4.157 5.885 (3) 0.9878 0.51028 0.05087 H4a held H4b failed (largest: 3) H4c failed H5a held H5b failed
2 6.129 5.623 (3) 0.9964 0.54675 0.23659 H4a held H4b held (largest: 4) H4c failed H5a held H5b failed
3 7.254 6.422 (5) 0.9927 0.56969 0.05248 H4a held H4b held (largest: 4) H4c failed H5a held H5b failed
4 3.945 5.229 (5) 0.9972 0.43728 0.04580 H4a held H4b failed (largest: 5) H4c failed H5a held H5b held
C0 held for every quantity: without noise the homography poses, the chain, the joint fit and the re-estimated camera agree with the truth to about or better, and the moved camera is visible (1.8458 px) without any noise. C1 held in every seed. On seed 0 the tree has two branches from camera 0, one through camera 1 to camera 2 and one through camera 7 round to camera 3, so camera 3 is five hops away although it is three cameras from camera 0 around the ring; the tree follows the overlap counts, not the ring distance.
H1 held: pooled over the five seeds, the chained rotation error rises from 0.25° at one hop to 0.69° at five hops (and 1.24° for the single camera at six), and the centre error from 12 mm to 30 mm (70 mm at six), with slopes of 0.143 degrees and 6.6 mm per hop. H2 held: the joint slopes are 0.0845 and 0.0849 times the chained ones. The joint errors still grow slightly with hops, from 0.043° to 0.094° at five hops, while the joint centre errors stay below 6 mm at every hop count. In the per-camera table of seed 0 the chained errors do not rise monotonically along each branch; the slope is a property of the pool, not of every chain. The reprojection RMS shows the same thing in the image: in every seed the chained calibration’s RMS is above 0.9 px, and the joint fit ends slightly below the RMS at the true parameters, since, in our reading, a least-squares fit with free parameters absorbs part of the noise.
By construction, the chained solution reproduces the pairwise estimate on each of its tree edges exactly, so when the tree’s seven edges are all ring edges, as in seed 0, its disagreement on them is zero and its largest ring-edge disagreement equals the loop angle; the two printed columns are identical in every seed. H3a held in seeds 2, 3 and 4 and failed in seeds 0 and 1, where the joint solution’s largest disagreement (0.63423° and 0.52103°) exceeded the chained one (0.14424° and 0.46853°). H3b held in every seed; the largest ratio was 2.017, in seed 0.
In Part B, H4a held in every seed: camera 4’s RMS under the old calibration was more than three times its floor, the smallest ratio being 3.945, in seed 4. H4b failed in seeds 0, 1 and 4, where camera 5, camera 3 and camera 5 had the largest RMS, and held in seeds 2 and 3. H4c failed in every seed: in each, another camera’s ratio exceeded 5. The plan’s hypotheses file, written before any code ran, had named H4c as a hypothesis that might fail for a neighbour of camera 4.
H5a held in every seed: re-estimating camera 4 alone brought its RMS just below its floor, and in seed 0, the one seed whose per-camera table is printed, every camera’s RMS after re-estimation differs from its floor only from the third decimal on, cameras 3, 5 and 6 included. H5b failed in seeds 0 to 3, with rotation errors of 0.09685°, 0.05087°, 0.23659° and 0.05248° against the true new pose, and held in seed 4 (0.04580°).
The example is our own construction and the ring, board and noise are synthetic; it shows the behaviour of the re-implementation on this rig, not a general accuracy figure for multiview calibration.
All hypotheses, checks and bars were written in the plan before any script ran, and the printed output is that of the first run, which was identical to the pilot and reproduced exactly on a second run. Before the pilot, a planning script computed reachable values on the same set-up, starting the joint fit at the true parameters, and printed its own runtime, used only to size the pilot; its first attempt stopped with an error because a list was named like the image height, a second attempt was stopped for running too long before any output was read, and, before any result, the residual function was rewritten to evaluate all observations in one NumPy expression and the least_squares tolerances were changed from to , for runtime. The departures from the batch plan listed above (count-only tree weights, the pairwise estimator, OpenCV’s board initialization with a plain squared loss, five seeds, measuring ring closure on the pairwise estimates, the data check on the second-neighbour edges that three-camera views add, the added rotation-error measure for the re-estimated camera, and the split of H4) were approved by the editor at the pilot, and H3 was kept and reported as printed.
Common Misconceptions
- “Calibrate each pair and chain them.” A chain is an initialization. Its error depends on the path from the reference camera, as the example shows; a joint fit over all cameras and target views is the calibration.
- “A closed loop means a consistent calibration.” Any calibration written as absolute poses closes every loop exactly. A loop discrepancy describes the pairwise measurements that went into the calibration, not the result.
- “The camera with the highest error is the one that moved.” Re-checking a calibration fits each target view to all the cameras that see it, so a moved camera spreads error to the neighbours that share views with it; in the example a neighbour had the largest RMS in three seeds of five.
- “Synchronized clocks mean synchronized exposures.” PTP gives cameras a common time base, which SFNC exposes through its PTP controls (Section 27.3). With the frame trigger off, a camera captures at its own
AcquisitionFrameRate(Section 5.5.10); exposures start together when FrameStart is triggered in every camera, for example from a physical input line or an action signal (Sections 5.6.4 and 14.1), so in our reading a rig’s timing depends on both. - “If the average bandwidth fits, the link is fine.” A common trigger makes all cameras send at once, and the burst, not the average, decides whether packets are lost.
Where It Is Used
- Stereo and depth: two-camera rigs, calibrated and synchronized as one device (stereo camera, stereo vision).
- Autonomous driving: surround rigs for BEV perception, and camera–LiDAR timing in sensor fusion for autonomous driving.
- Capture volumes and reconstruction: rings and domes for 3D reconstruction and photogrammetry, where a calibrated rig replaces structure from motion’s estimation of moving cameras.
- Inspection cells: several cameras on one part, triggered together (industrial machine vision).
- Surveillance networks: overlapping and disjoint views across a site (video surveillance).
- Global alignment of many views: an initialization followed by joint refinement also appears in DUSt3R (from a spanning tree), pose graph optimization (from chained odometry or rotation averaging), and ORB-SLAM, whose Essential Graph contains a spanning tree of keyframes.
Related
- Stereo Camera
A depth sensor built from two or more matched, synchronized and factory-calibrated imagers, sometimes with an on-board matcher and an infrared texture projector, that outputs disparity or depth computed by triangulation.
- Bundle Adjustment
The joint nonlinear least-squares refinement of all camera parameters and 3D points in a reconstruction, made practical by the sparse structure of the problem and the Schur complement.
- Pose Graph Optimization
Estimating a trajectory of poses from noisy relative-pose measurements by nonlinear least squares on SE(2), SE(3) or Sim(3), with the initialization, robust back-ends and certifiable solvers that decide whether the optimizer finds the right answer.
- Industrial Machine Vision
The system engineering of vision on the factory floor, covering machine-vision cameras and their interfaces, GenICam, triggering from encoders and line-scan acquisition, strobed lighting and optics selection, integration with PLCs and robots, gauging and measurement uncertainty in production, and smart cameras against PC-based systems.
- Bird's-Eye-View Perception
Turning images from one or more cameras into a metric top-down grid around the vehicle, by inverse perspective mapping, depth-based lifting or attention, and using that grid for 3D detection, map segmentation and planning.
- Video Surveillance and Security
Computer vision applied to video from fixed security and traffic cameras as its research literature describes it; the processing chain from detection to multi-camera tracking, re-identification, attributes, events and search, the PETS, DukeMTMC, CityFlow, AI City Challenge, VIRAT and MEVA benchmarks with their protocols and metrics, documented failure modes, NIST's accuracy and demographic findings, and the privacy concerns stated in official documents.