Rolling Shutter

A rolling shutter exposes and reads out an image sensor one row at a time, so the rows of a single frame are captured at different instants. When the camera or the scene moves during the readout, each row sees the world from a slightly different pose, and straight vertical edges lean, lines bend and fast objects are stretched or squeezed. The alternative, a global shutter, exposes every pixel over the same interval. The image sensor article explains why most CMOS pixels force a rolling readout and gives the basic row timing and skew; this article takes the next step. It describes the projection models that make a rolling-shutter image usable for geometry, the rolling-shutter constraint that links a point’s row to its capture time, how the line delay is calibrated, rolling-shutter-aware pose estimation and visual odometry, correcting images to a global-shutter geometry, and the flicker bands a rolling shutter records under mains lighting.

Definition

On a rolling-shutter camera, row $v$ of a frame starts its exposure a fixed time after row $v-1$. Oth, Furgale, Kneip and Siegwart (2013) call this time the line delay $d$, “the time between the start of integration of two consecutive lines”, and note that on most rolling-shutter cameras it is fixed, so the timestamp of every line follows from the line delay and the frame time. Other sources call the same quantity the line time (the image sensor article) or the row time; Schubert et al. (2019) give their camera’s “row time difference” as about 29.47 µs. The readout time of a frame is, without blanking rows, the number of rows times the line delay: for Schubert et al.’s 1024-row images this is about 30 ms (our arithmetic).

Every row has the same exposure time; only the start of exposure moves down the image. Rolling shutter is therefore a matter of when each row is captured, not of how long. It is distinct from two related mechanisms. GenICam’s global reset mode starts exposure for all pixels at once but ends it line by line, as the digital camera article describes. Nor is it the “rolling shutter” of a spinning lidar, whose time skew across a scan has the same consequence for moving platforms but a different cause.

Intuition

Think of the sensor as a scanner that sweeps from the top of the image to the bottom during the readout. A static camera looking at a static scene records the same image as a global shutter would. If the camera pans to the right during the sweep, the lower rows are recorded later, after the scene has moved further left in the image, so a vertical pole comes out leaning. Albl, Kukelova and Pajdla (2015) and Liu, Cui, Larsson and Pollefeys (2020) name the resulting distortions skew, smear and wobble; rotation about the optical axis bends straight lines, as the image sensor article describes.

Two consequences follow. First, the distortion depends on the motion during each frame, so no static calibration can remove it; it has to be modelled with the motion or avoided. Second, the distortion carries information. A rolling-shutter image of a known object encodes not only its pose but also its velocity relative to the camera; Oth et al., following Ait-Aider and colleagues, describe the rolling-shutter camera as a high-frequency sensor with dense temporal coverage, and Schubert et al. (2019) use the effect to estimate velocities alongside an IMU.

Formal Definition

Let $T(t)$ be the camera’s world-to-camera pose at time $t$ and $\pi$ the camera’s projection. With a global shutter, a point $\mathbf{X}$ is imaged at $\mathbf{u} = \pi(T(t_0)\mathbf{X})$ for one frame time $t_0$. With a rolling shutter, the time depends on the row the point lands in. Oth et al. (2013, Eq. 6) write it with the first line starting at the frame time $\bar t$:

$$\mathbf{u} = (u, v) = \pi\big(T(\bar t + v\,d)\,\mathbf{X}\big).$$

The row $v$ appears on both sides: the point’s row fixes its capture time, and the capture time fixes the pose that decides its row. Schubert et al. (2018, Eq. 8) call the condition that “pose time equals row time” the rolling-shutter constraint, citing Meingast, Geyer and Sastry (2005, not read) for it. Apart from special cases there is no closed form for the time, and Schubert et al. report that iterating $t^* \leftarrow t(\mathbf{u}'(t^*))$ a few times suffices. When the row is observed, as for a detected keypoint, the constraint is already satisfied and the capture time can be read off the measurement; this is why Schubert et al. note that indirect (keypoint) methods know each keypoint’s capture time from extraction, whereas direct methods, which warp pixels into another image, must solve the constraint.

The trajectory $T(t)$ needs a model. The common choices are:

• Constant velocity within a frame. Schubert et al. (2018, Eqs. 1–2) set $T_i(t) = \exp(\hat{\boldsymbol\xi}_i t)\,T_{0,i}$ with a twist $\boldsymbol\xi_i$ and measure time in rows from the middle of the image, $t(x, y) = y - y_0$, so that the reference pose $T_{0,i}$ belongs to the middle row; they choose the middle because the initial pose comes from tracking that assumes a global shutter. Oth et al. reference the first line instead.
• Rotation and translation separately. Albl et al. (2015, Eq. 5) write the pose of row $r_i$ as $\mathrm{R}(r_i\mathbf{w})\mathrm{R}(\mathbf{v})\mathbf{X}_i + \mathbf{C} + r_i\mathbf{t}$, with a Cayley parametrization; to obtain low-degree polynomials they linearize the rolling-shutter rotation, $(\mathrm{I} + (r_i - r_0)[\mathbf{w}]_\times)\mathrm{R}(\mathbf{v})$ (Eq. 6), and then also the initial rotation (Eq. 8).
• Interpolation between frames. As Oth et al. describe them, Hedborg et al.’s rolling-shutter structure from motion interpolates between the camera poses at the beginning of each frame, linearly in position and by SLERP in rotation, and their rolling-shutter bundle adjustment keeps this interpolation between consecutive frames.
• Continuous-time splines. Oth et al. represent the pose as a fourth-order B-spline over a whole video, which allows motion more general than constant velocity between frames.

Two conventions change the numbers and should be stated with any model: where time zero is (first row or middle row), and which way the shutter runs. Schubert et al. (2018) note that the shutter may run along columns, in which case the image can simply be rotated; Albl et al. (2020) rely on two cameras with opposite readout directions.

Properties

Only motion times line delay is observable from one frame. Time enters the projection only as $d\,(v - v_0)$, so an angular velocity $\boldsymbol\omega$ and a linear velocity $\mathbf{u}$ appear only as the products $\boldsymbol\omega d$ and $\mathbf{u} d$ (our derivation; consistent with Schubert et al. 2019, who note that they are free to measure time in pixel units until the twist is compared with IMU measurements). Doubling the line delay and halving the velocities gives the same image. Recovering $d$ itself needs a time scale from outside the frame: the frame rate across several frames, an IMU, or a known light frequency.

The distortion grows with motion times readout. A pan at $\omega$ rad/s shifts the image by about $f\omega$ pixels per second, so the bottom row of a frame is shifted by $f\omega\,H d$ pixels relative to the top row ($H$ rows); the image sensor article gives the same skew for an object moving across the image. For Schubert et al.’s (2019) recordings, whose ten sequences have mean rotational velocities from 39 to 137°/s (their Table I), a 30 ms readout and an assumed focal length of 300 pixels give shifts from about 6 to 22 pixels per frame if all of that rotation were a pan (our arithmetic).

The pose of a frame is ambiguous without a reference row. A global-shutter estimate fitted to a rolling-shutter image describes no single instant; it is closest to some average pose. Comparing it with the pose at the first row or at the middle row gives different errors, which the example below prints.

Linearized models are local. Albl et al. (2015) state that their linear approximation of the camera orientation is meaningful only around the identity rotation. They therefore first run the standard P3P, which in their synthetic experiments brought the orientation within 6° even at a rolling-shutter angular velocity of 30° per frame, rotate the data by that estimate, and then solve. In their solver they discard solutions with $\lVert\mathbf{w}\rVert > 0.2$, far from the linearization point.

Degeneracies. Albl et al. (2020) summarize earlier results that structure from motion with rolling shutter has degenerate configurations: constant translation along the baseline, as for a side-looking camera on a vehicle, admits no solution, and (nearly) parallel readout directions are in general degenerate. Schubert et al. (2018) summarize an earlier finding that reconstructions can collapse to a plane when the readout directions of the frames are parallel, and that softly constraining the velocities with a prior made their optimization more stable.

Noise couples to time. Because the capture time is computed from the measured row, noise in the row coordinate also perturbs the pose at which the point is predicted. Oth et al. derive the resulting covariance of the reprojection error, which is no longer diagonal and depends on the motion: for slow motion it tends to the global-shutter case, and for fast motion along the scanning direction it can become degenerate.

Calibrating the Line Delay

A rolling-shutter model needs $d$, and datasheets are not always available. Oth et al. (2013) describe the earlier standard method, which they attribute to Geyer et al. with the reference Meingast, Geyer and Sastry (2005): the sensor is lit by an LED flashing at a known high frequency, and the line delay follows from the spacing of the light and dark bands it produces. They state that the method requires expensive hardware and is prone to imprecision, especially if the camera has a fixed lens.

Their own method needs only video of a known calibration pattern. The intrinsics and lens distortion are first calibrated from still images, as in ordinary camera calibration; the camera is then moved in front of the pattern, and the B-spline trajectory and the line delay are estimated jointly by nonlinear least squares, initialized from a global-shutter PnP solution for every frame and from $d_0 = (1/\text{fps})/N_R$ for $N_R$ rows, which assumes no gap between frames. With too many uniformly spaced knots the spline over-fits and the line-delay estimate begins to diverge, so they start with too few knots and split a segment whenever its cost exceeds its expected value. On a Matrix Vision BlueFOX camera at four pixel-clock settings, the median of their estimates was within 0.21 µs of the nominal line delay in every case (for example 41.14 against 41.25 µs at 40 MHz, Table 1, read from the page image), with smaller standard deviations than the LED method; they also calibrated three smartphones (Table 3), finding 64.41 µs for an iPhone 3GS at 640 × 480 and 24.12 and 30.25 µs for an iPhone 4S and a Galaxy S3 at 1920 × 1080.

Rolling-Shutter-Aware Geometry

Absolute pose. Perspective-n-point with a rolling shutter estimates the pose of the reference row together with the velocities. Albl et al. (2015) give a minimal, non-iterative solver from six correspondences (R6P) based on their double-linearized model, with up to 20 solutions, and report a significant improvement in the number of RANSAC inliers over P3P on real rolling-shutter images; their abstract reports orientation errors under half a degree and relative position errors under 2%. On real data, Oth et al. report mean absolute errors of 189 mm against 8.34 mm in position and 0.34 against 0.03 rad in orientation for a global-shutter PnP and their continuous-time rolling-shutter localization, on a camera running at about 7 fps with a line delay near 137 µs (their Figure 5).

Odometry and SLAM. Direct methods are especially sensitive, since a rolling-shutter displacement moves pixels outside the region where the photometric linearization holds, as direct methods describes. Schubert et al. (2018) extend DSO with a constant-velocity rolling-shutter projection model and a velocity prior at each keyframe, solving the rolling-shutter constraint for every warped point; they report better results than global-shutter DSO on rolling-shutter sequences. Schubert et al. (2019) carry the model into direct visual–inertial odometry and record a dataset with a rig of two uEye UI-3241LE-M-GL cameras at 20 Hz, one in global-shutter and one in rolling-shutter mode. Their Table II (absolute trajectory error after SE(3) alignment, median of six runs, read from the page image) shows how badly an unmodelled rolling shutter can fail: on sequence 1 the global-shutter method reached 0.038 m on global-shutter data and 79.591 m on rolling-shutter data, and their rolling-shutter method 0.040 m on rolling-shutter data. Across the ten sequences the global-shutter method on global-shutter data was more accurate on seven and the rolling-shutter method on rolling-shutter data on three; they suggest the constant-twist assumption under large accelerations as one reason, and note that their shutter is at the slow end of the scale.

Datasets. Several benchmarks record with rolling-shutter cameras and say so: TUM RGB-D’s Kinect colour camera, Cityscapes’ stereo rig and the cameras used for Tanks and Temples, whose authors found COLMAP no worse on their rolling-shutter footage. EuRoC MAV and ETH3D’s SLAM benchmark use global shutters. Drone mapping is a well-documented case: aerial image analysis describes Vautherin et al.’s (2016) flights, in which a rolling-shutter camera model removed most of the error for the slow-readout Phantom 2 Vision+ at 8 m/s.

Correcting Rolling-Shutter Images

Instead of changing every algorithm, one can synthesize a global-shutter image and feed it to standard methods. Albl et al. (2020) argue that a single rolling-shutter image cannot be undistorted without fairly restrictive assumptions about the scene or the motion, pure rotation being the best-understood case. They show that two cameras mounted with different, ideally opposite, readout directions, with a known (ideally negligible) baseline and known synchronization, provide the constraints to recover the motion from a sparse set of point correspondences and to undo the distortion regardless of image content. Liu et al. (2020) train a network on pairs of consecutive rolling-shutter frames to predict the global-shutter image of the latest frame, by estimating a dense displacement field and warping learned features; with no public dataset for the task they built two, Carla-RS (2,500 image pairs from a simulator with six-degree-of-freedom motion) and Fastec-RS (2,584 pairs synthesized from a 2,400 fps camera on a ground vehicle). Oth et al. point to evidence that structure from motion on wobble-corrected images can be less accurate than modelling the rolling shutter on the raw images.

Examples

Pose under readout motion, and the line delay

The first script simulates a rolling-shutter camera and compares a global-shutter PnP with a rolling-shutter PnP; it then recovers the line delay from a short video. It uses NumPy 2.3.5 and SciPy 1.16.3 and runs in about 4 seconds.

The camera is a pinhole with $f = 800$ pixels and a 640 × 480 image read from top to bottom with a line delay of 62.5 µs, a 30 ms readout. The pose convention is world-to-camera, $\mathbf{x}_c(t) = \operatorname{Exp}(\boldsymbol\omega t)\,\mathrm{R}\mathbf{X} + \mathbf{p} + \mathbf{u}t$, with the angular velocity $\boldsymbol\omega$ in rad/s in the camera frame and the linear velocity $\mathbf{u}$ in m/s: the form of Albl et al.’s (2015) Eq. 5 with an exact rotation exponential instead of the Cayley map. Each point’s row is found by fixed-point iteration on the rolling-shutter constraint, and checked against SciPy’s brentq root finder. The amount of motion is expressed as $S$, the horizontal image shift in pixels that the pan produces during one readout; the rotation and translation speeds scale with $S$ (at $S = 30$, a pan of 1.25 rad/s).

• Part (a) draws 100 scenes of 40 points at depths of 4–8 m, with 0.5-pixel noise (seeds 1000–1099; each scene’s noise is reused for every $S$). Time is measured from the middle row, Schubert et al.’s convention. The global-shutter (GS) PnP is a DLT followed by Levenberg–Marquardt on the reprojection error, as in the perspective-n-point article. The rolling-shutter (RS) PnP is our own constant-velocity fit of the Eq. 5 form: twelve parameters (pose, $\boldsymbol\omega$, $\mathbf{u}$), the capture time taken from each observed row, the line delay known, started from the GS estimate. It is not Albl et al.’s minimal R6P solver. The GS error is scored against the pose of the middle row and against the pose of the first row, because the sources leave the reference row open. “GS bias” is the GS rotation error on noise-free data.
• Part (b) estimates the line delay jointly with pose and velocities from five frames at 20 fps of a known target (seeds 2000–2099, $S = 10$), with time measured from the first row of the first frame (Oth et al.’s convention) and $d$ started at Oth et al.’s $d_0 = 1/(\text{fps}\cdot N_R)$. It also prints the ratio of the smallest to the largest singular value of the Jacobian at the true parameters, for five frames, for one frame, and for five frames without motion.

import os
for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[k] = "1"
import numpy as np
from scipy.optimize import least_squares, brentq
from scipy.spatial.transform import Rotation

F, CX, CY, W, H = 800.0, 320.0, 240.0, 640, 480   # pinhole camera, pixels
D = 62.5e-6            # line delay in seconds: readout H*D = 30 ms
SIGMA, N_PTS, TRIALS = 0.5, 40, 100
S_LIST = [0, 1, 3, 10, 30]   # horizontal image shift (px) from panning during one readout

def rot(w):
    return Rotation.from_rotvec(w).as_matrix()

def logrot(R):
    return Rotation.from_matrix(R).as_rotvec()

def project(Xc):
    return np.c_[F * Xc[:, 0] / Xc[:, 2] + CX, F * Xc[:, 1] / Xc[:, 2] + CY]

def to_camera(X, t, R, p, w, u):
    # world-to-camera pose at time t: x_c = Exp(w t) R X + p + u t
    Rt = Rotation.from_rotvec(np.outer(t, w)).as_matrix()
    return np.einsum("nij,nj->ni", Rt, X @ R.T) + p + np.outer(t, u)

def rs_project(X, R, p, w, u, d, t_of_row):
    # each point's row fixes its capture time: solve v = proj_v(pose(t(v)) X) by fixed-point iteration
    v = project(X @ R.T + p)[:, 1]
    for it in range(1, 101):
        v_new = project(to_camera(X, t_of_row(v), R, p, w, u))[:, 1]
        done = np.max(np.abs(v_new - v)) < 1e-12
        v = v_new
        if done:
            break
    uv = project(to_camera(X, t_of_row(v), R, p, w, u))
    return uv, it, np.max(np.abs(uv[:, 1] - v))

def motion(S, d):
    k = S / (F * H * d)            # pan rate (rad/s) that shifts the image by S px per readout
    return k * np.array([0.3, 1.0, 0.2]), k * np.array([1.5, 0.0, 0.5])   # rad/s, m/s

def make_scene(rng, umin, umax, vmin, vmax):
    uv = rng.uniform([umin, vmin], [umax, vmax], (N_PTS, 2))
    Z = rng.uniform(4.0, 8.0, N_PTS)
    Xc = np.c_[(uv[:, 0] - CX) / F * Z, (uv[:, 1] - CY) / F * Z, Z]
    R = Rotation.random(rng=rng).as_matrix()
    p = -R @ rng.uniform(-2.0, 2.0, 3)
    return (Xc - p) @ R, R, p          # world points, reference pose

def dlt_pose(X, uv):
    x = (uv - [CX, CY]) / F
    A = []
    for (Xw, Yw, Zw), (a, b) in zip(X, x):
        A.append([Xw, Yw, Zw, 1, 0, 0, 0, 0, -a * Xw, -a * Yw, -a * Zw, -a])
        A.append([0, 0, 0, 0, Xw, Yw, Zw, 1, -b * Xw, -b * Yw, -b * Zw, -b])
    P = np.linalg.svd(np.array(A))[2][-1].reshape(3, 4)
    if np.linalg.det(P[:, :3]) < 0:
        P = -P
    U, s, Vt = np.linalg.svd(P[:, :3])
    return U @ Vt, P[:, 3] / s.mean()

def fit_gs(X, uv):
    R0, p0 = dlt_pose(X, uv)
    res = lambda q: (project(X @ rot(q[:3]).T + q[3:]) - uv).ravel()
    q = least_squares(res, np.r_[logrot(R0), p0], method="lm", x_scale="jac",
                      xtol=1e-15, ftol=1e-15, gtol=1e-15, max_nfev=5000).x
    return rot(q[:3]), q[3:], np.sqrt(np.mean(res(q) ** 2)), (R0, p0)

def fit_rs(X, uv, d, v0, R0, p0):
    t = d * (uv[:, 1] - v0)            # capture time from the observed row
    res = lambda q: (project(to_camera(X, t, rot(q[:3]), q[3:6], q[6:9], q[9:])) - uv).ravel()
    q = least_squares(res, np.r_[logrot(R0), p0, np.zeros(6)], method="lm", x_scale="jac",
                      xtol=1e-15, ftol=1e-15, gtol=1e-15, max_nfev=5000).x
    return rot(q[:3]), q[3:6], q[6:9], q[9:], np.sqrt(np.mean(res(q) ** 2))

def rot_err(Ra, Rb):
    return np.degrees(np.linalg.norm(logrot(Ra @ Rb.T)))

def centre_err_cm(Ra, pa, Rb, pb):
    return 100 * np.linalg.norm(Ra.T @ pa - Rb.T @ pb)

# ---------- (a) global-shutter PnP against rolling-shutter PnP ----------
V0 = 240.0                       # pose reference row: middle of the frame
t_mid = lambda v: D * (v - V0)
print("(a) PnP on one rolling-shutter frame: 40 points, 0.5 px noise, 100 scenes, readout 30 ms")
rows = {S: [] for S in S_LIST}
fp_res, fp_its, brent_diff, outside, base = 0.0, 0, 0.0, 0, []
for i in range(TRIALS):
    rng = np.random.default_rng(1000 + i)
    X, Rm, pm = make_scene(rng, 40, 600, 40, 440)
    noise = rng.normal(0.0, SIGMA, (N_PTS, 2))
    for S in S_LIST:
        w, u = motion(S, D)
        uv0, its, r = rs_project(X, Rm, pm, w, u, D, t_mid)
        fp_res, fp_its = max(fp_res, r), max(fp_its, its)
        outside += np.sum((uv0[:, 0] < 0) | (uv0[:, 0] > W) | (uv0[:, 1] < 0) | (uv0[:, 1] > H))
        if i < 5:   # library cross-check of the fixed point with a bracketing root finder
            for j in range(N_PTS):
                g = lambda v: v - project(to_camera(X[j:j + 1], t_mid(np.array([v])), Rm, pm, w, u))[0, 1]
                vb = brentq(g, uv0[j, 1] - 100, uv0[j, 1] + 100, xtol=1e-12)
                brent_diff = max(brent_diff, abs(vb - uv0[j, 1]))
        t1 = D * (0 - V0)                          # capture time of the first row
        R1, p1 = rot(w * t1) @ Rm, pm + u * t1
        # noise-free fits: base checks and the global-shutter bias
        Rg0, pg0, _, (Rd0, pd0) = fit_gs(X, uv0)
        Rr0, pr0, _, _, _ = fit_rs(X, uv0, D, V0, Rg0, pg0)
        base.append((S, rot_err(Rd0, Rm), rot_err(Rg0, Rm), rot_err(Rr0, Rm),
                     centre_err_cm(Rr0, pr0, Rm, pm)))
        # noisy fits
        uv = uv0 + noise
        Rg, pg, rms_g, _ = fit_gs(X, uv)
        Rr, pr, wr, ur, rms_r = fit_rs(X, uv, D, V0, Rg, pg)
        rows[S].append([rot_err(Rg0, Rm), rot_err(Rg, Rm), rot_err(Rg, R1), rot_err(Rr, Rm),
                        centre_err_cm(Rg, pg, Rm, pm), centre_err_cm(Rr, pr, Rm, pm),
                        rms_g, rms_r, np.linalg.norm(wr - w) / max(np.linalg.norm(w), 1e-300)])
print(f"data: fixed point max {fp_its} iterations, residual {fp_res:.1e} px; "
      f"brentq difference {brent_diff:.1e} px; points outside image {outside}")
b = np.array(base)
print(f"noise-free, S=0: DLT rotation error {b[b[:, 0] == 0, 1].max():.1e} deg, "
      f"GS {b[b[:, 0] == 0, 2].max():.1e} deg")
print(f"noise-free, all S: RS rotation error {b[:, 3].max():.1e} deg, centre error {b[:, 4].max():.1e} cm")
print("medians over 100 scenes (90th percentile in brackets)")
print("   S | GS bias | GS rot vs mid    | GS rot vs row 0  | RS rot           |"
      " GS centre cm    | RS centre cm    | GS rms | RS rms | RS w rel.err")
for S in S_LIST:
    a = np.array(rows[S])
    m, q = np.median(a, 0), np.percentile(a, 90, 0)
    cell = lambda k, f: f"{m[k]:{f}} [{q[k]:{f}}]"
    werr = "      -" if S == 0 else f"{m[8]:.4f}"
    print(f"{S:4d} | {m[0]:.5f} | {cell(1, '.5f')} | {cell(2, '.5f')} | {cell(3, '.5f')} | "
          f"{cell(4, '6.3f')} | {cell(5, '6.3f')} | {m[6]:.4f} | {m[7]:.4f} | {werr}")

# ---------- (b) line-delay recovery from a short video of a known target ----------
FPS, K, S_B = 20.0, 5, 10
print(f"\n(b) line delay from {K} frames at {FPS:.0f} fps of a known target, true d = {D * 1e6:.4f} us")

def frames_uv(X, R0, p0, w, u, d):
    out = []
    for k in range(K):
        uv, _, _ = rs_project(X, R0, p0, w, u, d, lambda v, k=k: k / FPS + d * v)
        out.append(uv)
    return out

def resid_b(q, X, uvs):
    R0, p0, w, u, d = rot(q[:3]), q[3:6], q[6:9], q[9:12], q[12] * 1e-6
    return np.concatenate([(project(to_camera(X, k / FPS + d * uv[:, 1], R0, p0, w, u)) - uv).ravel()
                           for k, uv in enumerate(uvs)])

def fit_b(X, uvs):
    gs = [fit_gs(X, uv)[:2] for uv in uvs]
    T = (K - 1) / FPS
    w0 = logrot(gs[-1][0] @ gs[0][0].T) / T
    u0 = (gs[-1][1] - gs[0][1]) / T
    d0 = 1.0 / (FPS * H)                       # Oth et al.'s initial value: 1/fps / rows
    q0 = np.r_[logrot(gs[0][0]), gs[0][1], w0, u0, d0 * 1e6]
    q = least_squares(resid_b, q0, method="lm", x_scale="jac", args=(X, uvs),
                      xtol=1e-15, ftol=1e-15, gtol=1e-15, max_nfev=5000).x
    return q[12], d0 * 1e6

def jac_sv_ratio(q, X, uvs):
    J = np.empty((2 * N_PTS * len(uvs), q.size))
    for j in range(q.size):
        h = 1e-6 * max(1.0, abs(q[j]))
        e = np.zeros(q.size); e[j] = h
        J[:, j] = (resid_b(q + e, X, uvs) - resid_b(q - e, X, uvs)) / (2 * h)
    s = np.linalg.svd(J, compute_uv=False)
    return s[-1] / s[0]

errs, dhat, outside_b = [], [], 0
for i in range(TRIALS):
    rng = np.random.default_rng(2000 + i)
    X, R0, p0 = make_scene(rng, 140, 500, 80, 400)
    w, u = motion(S_B, D)
    uvs = frames_uv(X, R0, p0, w, u, D)
    outside_b += sum(np.sum((a[:, 0] < 0) | (a[:, 0] > W) | (a[:, 1] < 0) | (a[:, 1] > H)) for a in uvs)
    if i == 0:
        q_true = np.r_[logrot(R0), p0, w, u, D * 1e6]
        d_nf, d0 = fit_b(X, uvs)
        print(f"initial d0 = 1/(fps*rows) = {d0:.4f} us; noise-free estimate error {abs(d_nf - D * 1e6):.1e} us")
        print(f"Jacobian s_min/s_max at the truth: 5 frames {jac_sv_ratio(q_true, X, uvs):.1e}, "
              f"1 frame {jac_sv_ratio(q_true, X, uvs[:1]):.1e}, "
              f"5 frames without motion {jac_sv_ratio(np.r_[q_true[:6], np.zeros(6), D * 1e6], X, frames_uv(X, R0, p0, 0 * w, 0 * u, D)):.1e}")
    noisy = [a + rng.normal(0.0, SIGMA, a.shape) for a in uvs]
    dh, _ = fit_b(X, noisy)
    dhat.append(dh)
    errs.append(abs(dh - D * 1e6))
print(f"points outside image: {outside_b}")
print(f"noisy (0.5 px, 100 scenes): median estimate {np.median(dhat):.4f} us, "
      f"median |error| {np.median(errs):.4f} us, 90th percentile {np.percentile(errs, 90):.4f} us")

(a) PnP on one rolling-shutter frame: 40 points, 0.5 px noise, 100 scenes, readout 30 ms
data: fixed point max 11 iterations, residual 1.1e-13 px; brentq difference 3.4e-13 px; points outside image 0
noise-free, S=0: DLT rotation error 1.5e-13 deg, GS 4.0e-14 deg
noise-free, all S: RS rotation error 3.4e-14 deg, centre error 3.3e-13 cm
medians over 100 scenes (90th percentile in brackets)
   S | GS bias | GS rot vs mid    | GS rot vs row 0  | RS rot           | GS centre cm    | RS centre cm    | GS rms | RS rms | RS w rel.err
   0 | 0.00000 | 0.03467 [0.06293] | 0.03467 [0.06293] | 0.04177 [0.07266] |  0.359 [ 0.616] |  0.440 [ 0.759] | 0.4847 | 0.4664 |       -
   1 | 0.05419 | 0.06325 [0.09075] | 0.06487 [0.10258] | 0.04177 [0.07266] |  0.424 [ 0.748] |  0.440 [ 0.760] | 0.5185 | 0.4665 | 2.3672
   3 | 0.16241 | 0.16170 [0.21319] | 0.17348 [0.24289] | 0.04177 [0.07266] |  0.739 [ 1.201] |  0.440 [ 0.761] | 0.7220 | 0.4667 | 0.7894
  10 | 0.53956 | 0.54251 [0.70195] | 0.57987 [0.75451] | 0.04190 [0.07266] |  2.039 [ 3.348] |  0.445 [ 0.768] | 1.8463 | 0.4676 | 0.2378
  30 | 1.60335 | 1.60906 [2.09239] | 1.70841 [2.28250] | 0.04244 [0.07267] |  5.817 [10.094] |  0.448 [ 0.784] | 5.2718 | 0.4674 | 0.0816

(b) line delay from 5 frames at 20 fps of a known target, true d = 62.5000 us
initial d0 = 1/(fps*rows) = 104.1667 us; noise-free estimate error 3.3e-13 us
Jacobian s_min/s_max at the truth: 5 frames 3.0e-05, 1 frame 6.0e-13, 5 frames without motion 0.0e+00
points outside image: 0
noisy (0.5 px, 100 scenes): median estimate 62.4896 us, median |error| 0.8874 us, 90th percentile 2.4020 us

The data checks held: the fixed point converged within 11 iterations to a residual of about $10^{-13}$ pixels, agreed with brentq to the same level, and kept every point inside the image. On noise-free data the DLT and the GS fit at $S = 0$, and the RS fit at every $S$, returned the true pose to rounding error, as they must, since each fits the model that generated the data; those exactness checks held.

The global-shutter fit is biased as soon as the camera moves. Its noise-free error grows from 0.054° at a 1-pixel shift to 1.60° at 30 pixels, and the ratio of the 30-pixel to the 10-pixel bias is 2.97, close to the factor of 3 that a bias linear in the shift would give (our planned hypothesis that this ratio lies between 2.7 and 3.3 held). The noisy medians follow the bias: the GS rotation error rises strictly with $S$ (0.063°, 0.162°, 0.543°, 1.609°) and so does its camera-centre error (0.42 to 5.8 cm), so the hypothesis of a strict increase held. The RS fit stays at its noise floor: its median rotation error is between 0.04177° and 0.04244° at every $S$, which met the planned criterion of staying within a factor of 1.5 of the $S = 0$ value, and its centre error stays between 0.440 and 0.448 cm. Already at $S = 3$ the GS error is larger than the RS error, and at $S = 30$ it is 38 times larger; that hypothesis (larger at $S \ge 3$, at least ten times at $S = 30$) held.

Without motion the extra parameters cost precision. At $S = 0$ the GS fit, with six parameters, has a smaller median rotation error (0.0347° against 0.0418°) and centre error (0.359 against 0.440 cm) than the twelve-parameter RS fit; the hypothesis that GS is more precise there held. The reprojection error tells the same story from the residuals: the RS fit’s RMS stays at about 0.467 pixels for every $S$, close to $0.5\sqrt{(80-12)/80} = 0.461$ pixels expected from the noise (the planned range of 0.43–0.49 held), while the GS fit’s RMS grows to 5.27 pixels at $S = 30$; the hypothesis that the GS RMS is at least twice the RS RMS at $S = 30$ and within 5% of it at $S = 0$ (0.4847 against 0.4664) held. In our reading, a residual that grows with camera motion is thus a cheap sign of an unmodelled rolling shutter.

The reference row matters for the GS fit. Measured against the first-row pose instead of the middle-row pose, its median error is larger at $S = 10$ (0.580° against 0.543°) and at $S = 30$ (1.708° against 1.609°), so the hypothesis that the first-row reference gives the larger error at $S \ge 10$ held; at $S = 0$ the two references coincide. Under either reference the comparison with the RS fit comes out the same way. The RS fit’s angular velocity is poorly determined from one frame: its relative error falls from 2.37 at $S = 1$ to 0.082 at $S = 30$, and the relative error times $S$ is nearly the same in every row (2.37, 2.37, 2.38, 2.45), so in our reading the absolute velocity error hardly changes with the motion and only its size relative to the motion does.

In part (b), all three identifiability checks held. Over five frames the Jacobian has full rank (ratio $3.0 \times 10^{-5}$); from one frame it has a null direction (ratio $6.0 \times 10^{-13}$), the exact trade-off between line delay and velocities derived under Properties; and without motion the line delay has no effect on the image at all (ratio 0). With motion and a known frame rate, the joint fit recovered the line delay exactly on noise-free data, starting from Oth et al.’s initial value of 104.17 µs, and with 0.5-pixel noise the median estimate was 62.49 µs with a median absolute error of 0.89 µs (90th percentile 2.40 µs). The planned criterion on this error (median below 1.33 µs, and median estimate within 1.33 µs of 62.5 µs) held, but it is not an independent test: the bar was set from a reachable-value run of the same model on the same seeds before the pilot, and the plan says that this criterion “is met by construction once the reach run is reproduced; it is stated as an illustration of the reachable precision, not as a test.”

Flicker bands

Lamps on an alternating-current grid flicker at twice the grid frequency, and the exposure article derives how much of the flicker an exposure of length $t_e$ passes: a relative modulation of $m\,\lvert\operatorname{sinc}(\pi\nu t_e)\rvert$, with phase set by the start of the exposure. Because each row of a rolling shutter starts its exposure $d$ later than the one above, the phase advances down the image and the flicker becomes horizontal bands with a period of $1/(\nu d)$ rows (our application of that formula). The second script renders a 480 × 640 image of a grey ramp under a light $1 + 0.3\cos(2\pi\nu t + 0.7)$, integrating each row over its own exposure window, for 100 Hz and 120 Hz flicker (50 Hz and 60 Hz grids) and four exposure times. It fits a sinusoid to the row profile and compares the fitted period and amplitude with those predictions. It runs in about a second.

import os
for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[k] = "1"
import numpy as np
from scipy.integrate import quad
from scipy.optimize import minimize_scalar

W, H, D = 640, 480, 62.5e-6        # image size and line delay (s), as in the geometry example
M, PHI = 0.3, 0.7                  # flicker modulation depth and phase
albedo = 0.3 + 0.4 * np.linspace(0, 1, W)          # a grey ramp across the columns

def light(t, nu):
    return 1.0 + M * np.cos(2 * np.pi * nu * t + PHI)

def render(nu, te, n_sub=4000):
    # each row integrates the flickering light over its own window [v*D, v*D + te]
    start = np.arange(H) * D
    s = (np.arange(n_sub) + 0.5) / n_sub * te       # midpoint rule over the exposure
    row_light = light(start[:, None] + s[None, :], nu).mean(1)
    return row_light[:, None] * albedo[None, :], row_light

def fit_sinusoid(profile, period):
    v = np.arange(H)
    A = np.c_[np.ones(H), np.cos(2 * np.pi * v / period), np.sin(2 * np.pi * v / period)]
    c, *_ = np.linalg.lstsq(A, profile, rcond=None)
    return np.sum((A @ c - profile) ** 2), np.hypot(c[1], c[2]) / c[0]

print("flicker bands on a rolling shutter: line delay 62.5 us, 480 rows, modulation 0.3")
print("  nu Hz | exposure ms | band period rows: fitted  1/(nu d) | amplitude: fitted  m|sinc(pi nu te)|")
worst_quad = 0.0
for nu in (100.0, 120.0):
    for te in (1e-3, 5e-3, 1e-2, 1 / 120):
        img, row = render(nu, te)
        profile = img.mean(1) / albedo.mean()
        q = quad(lambda t: light(t, nu), 100 * D, 100 * D + te, epsabs=1e-14, epsrel=1e-14)[0] / te
        worst_quad = max(worst_quad, abs(q - row[100]))
        predicted_amp = M * abs(np.sinc(nu * te))     # np.sinc(x) = sin(pi x)/(pi x)
        period_pred = 1.0 / (nu * D)
        grid = np.linspace(20, 1000, 4901)
        p0 = grid[np.argmin([fit_sinusoid(profile, p)[0] for p in grid])]
        p_fit = minimize_scalar(lambda p: fit_sinusoid(profile, p)[0], bounds=(p0 - 1, p0 + 1),
                                method="bounded", options={"xatol": 1e-9}).x
        amp = fit_sinusoid(profile, p_fit)[1]
        per = f"{p_fit:9.4f}" if predicted_amp > 1e-6 else "        -"
        print(f"  {nu:5.0f} | {te * 1e3:11.4f} | {per}  {period_pred:9.4f} | {amp:.6f}  {predicted_amp:.6f}")
print(f"midpoint rule against scipy quad, row 100: max difference {worst_quad:.1e}")

flicker bands on a rolling shutter: line delay 62.5 us, 480 rows, modulation 0.3
  nu Hz | exposure ms | band period rows: fitted  1/(nu d) | amplitude: fitted  m|sinc(pi nu te)|
    100 |      1.0000 |  160.0000   160.0000 | 0.295089  0.295089
    100 |      5.0000 |  160.0000   160.0000 | 0.190986  0.190986
    100 |     10.0000 |         -   160.0000 | 0.000000  0.000000
    100 |      8.3333 |  160.0000   160.0000 | 0.057296  0.057296
    120 |      1.0000 |  133.3333   133.3333 | 0.292944  0.292944
    120 |      5.0000 |  133.3333   133.3333 | 0.151365  0.151365
    120 |     10.0000 |  133.3333   133.3333 | 0.046774  0.046774
    120 |      8.3333 |         -   133.3333 | 0.000000  0.000000
midpoint rule against scipy quad, row 100: max difference 6.7e-09

The fitted band period is 160 rows at 100 Hz and 133.33 rows at 120 Hz, as $1/(\nu d)$ predicts, and the fitted amplitudes agree with Exposure’s sinc factor to all six printed decimals; the planned checks on period, amplitude and quadrature (midpoint rule against SciPy’s quad, $6.7 \times 10^{-9}$) held. An exposure of a whole number of flicker periods, 10 ms at 100 Hz or 8.33 ms at 120 Hz, removes the bands entirely, but an exposure tuned to one grid leaves bands on the other (amplitude 0.047 for 10 ms at 120 Hz). The bands also turn the image into a clock: their spacing measures the line delay if the flicker frequency is known, which is the principle of the LED calibration described above. This part is an exact computation, and its planning run was identical to its pilot; the plan records it as “an illustration of Exposure’s formula”, not a test of it.

Our plan for these examples departed from the batch plan’s outline in five ways, each decided before any run: motion times readout is expressed as the image shift $S$ with the readout fixed, since only the products enter the projections; part (a) adds the global-shutter bias, the two reference rows, the reprojection RMS and the velocity error; the line delay is recovered jointly over five frames, with the identifiability checks, because one frame cannot determine it; the optional flicker part is included as a rendered image; and the rolling-shutter PnP is our own Levenberg–Marquardt fit of the Eq. 5 form rather than R6P. The editor approved both scripts unchanged after their pilots, and the first run, with the same seeds and settings, printed the same output as the pilot; nothing was changed after the first run. Both scripts were run twice with identical output.

Common Misconceptions

• “A rolling shutter shortens or lengthens the exposure of some rows.” Every row has the same exposure time; only its start moves. Unequal exposure per row is the global-reset mode, not rolling shutter.
• “The distortion is a fixed property of the camera and can be calibrated once.” The line delay is fixed (Oth et al. note that some cameras change it with the exposure setting), but the distortion depends on the motion in each frame. The image sensor article makes the same point.
• “A global-shutter algorithm on rolling-shutter data just becomes noisier.” The error is a bias that grows with motion times readout, not noise: in the example the global-shutter fit’s noise-free error alone exceeds the rolling-shutter fit’s noisy error from a 1-pixel shift on, and in Schubert et al.’s (2019) visual–inertial experiments the global-shutter method diverged on rolling-shutter data in most runs.
• “Modelling the rolling shutter is always better.” With little motion, the extra velocity parameters cost precision (the example’s $S = 0$ row), and the models rest on assumptions such as constant velocity during a frame, which Schubert et al. (2019) suggest is violated under large accelerations.
• “The line delay can be read from a single image of a moving target.” From one frame only velocity times line delay is observable; a time scale such as the frame rate, an IMU or a known flicker frequency is needed.

Where It Is Used

Rolling-shutter models appear wherever geometry is computed from consumer or automotive cameras: structure from motion, bundle adjustment and SLAM with a pose per row, visual odometry and visual–inertial odometry, perspective-n-point for localization, drone mapping and sensor fusion, where projecting lidar points into rolling-shutter images requires solving for each point’s capture time. Real-time pipelines must account for the readout in timestamps and latency, and drone vision recommends global-shutter cameras where geometry matters, or modelling the rolling shutter. The choice of shutter is part of camera calibration and of choosing a camera: a global shutter avoids the problem, and a rolling shutter that is modelled, with a calibrated line delay, can match it and even provide velocity information.