Dense Trajectories

Dense trajectories are a hand-crafted video representation for action recognition. Points are sampled on a dense grid in every frame, tracked through the next 15 frames with dense optical flow, and each track is described by its own shape and by histograms of gradients and flow computed in the space-time tube around it. Wang, Kläser, Schmid and Liu introduced it at CVPR 2011 and extended it in a 2013 journal version; Wang and Schmid’s improved dense trajectories (iDT, ICCV 2013) added camera-motion compensation. Encoded with Fisher vectors and classified by a linear SVM, iDT was what Tran et al. (2015) still called the state-of-the-art hand-crafted feature, and the baseline that early deep video networks were measured against.

Problem

An action such as “get out of car” is defined by motion over a few seconds. Before deep networks, a video was described by many local space-time descriptors, pooled into one vector and classified, the video counterpart of bag-of-visual-words image classification. Dense trajectories answer where to compute those descriptors, and in what volume.

The earlier answer was space-time interest points (STIP). Laptev and Lindeberg (2003) extended the Harris corner detector to space-time. In Laptev’s journal version (2005), a $3 \times 3$ second-moment matrix $\mu$ of the derivatives $L_x, L_y, L_t$ is computed at separate spatial and temporal scales, and points are positive local maxima of $H = \det\mu - k\,\operatorname{trace}^3\mu$, with $k \approx 0.005$, so that points fire only where image values vary strongly in both space and time. Such points are sparse, and a cuboid at a fixed position does not follow the moving structure. Wang et al. (2011) cite an evaluation in which dense sampling at regular positions outperformed interest point detectors, and combine dense sampling with tracking.

Inputs and Outputs

• Input: grayscale video frames; for iDT, also human bounding boxes per frame.
• Output: for each trajectory, its position and scale and four descriptors: trajectory shape (30 dimensions), HOG (96), HOF (108) and MBH (192, as MBHx and MBHy of 96 each), as given in the journal version (Wang et al., 2013) and the iDT paper.

Intuition

KLT feature tracking follows sparse corners one by one; a dense flow field moves any number of points at little extra cost, and its smoothness makes tracks robust to fast, irregular motion. Limiting tracks to 15 frames keeps drift small, and a descriptor tube that bends with the track keeps the same structure in view, unlike a straight cuboid.

The motion descriptors must also separate the actor’s motion from the camera’s: MBH does so implicitly by using flow gradients, and iDT explicitly by estimating the camera motion.

Algorithm

Dense trajectories (Wang et al., 2011; 2013), for each frame $t$:

1. Dense sampling. Sample points on a grid with step $W = 5$ pixels, on up to 8 spatial scales spaced by a factor of $1/\sqrt{2}$; each scale is tracked separately.
2. Prune homogeneous areas. Drop points whose smaller structure-tensor eigenvalue is below a threshold relative to the frame’s largest (Mathematical Formulation).
3. Track. Compute dense flow between frames $t$ and $t+1$ with Farnebäck’s polynomial-expansion method in OpenCV (see optical flow), median-filter it and move each point by the filtered flow at its rounded position.
4. Re-sample. Wherever no tracked point lies in a $W \times W$ cell, start a new point.
5. End and prune. End each track after $L = 15$ frames. Remove static trajectories and erratic ones: in the journal version, those in which one frame-to-frame displacement exceeds 70% of the overall displacement.
6. Describe. Compute the shape descriptor, and HOG, HOF and MBH in an $N \times N \times L$ tube along the track ($N = 32$), divided into $n_\sigma \times n_\sigma \times n_\tau = 2 \times 2 \times 3$ cells, one histogram per cell.

Improved trajectories (Wang & Schmid, 2013) keep steps 1–5 and change the flow that the descriptors and pruning use:

1. Match frames. Combine SURF matches between frames $t$ and $t+1$ with flow vectors at Shi–Tomasi good features to track.
2. Mask humans. Discard matches inside human-detector boxes, since people do not move with the camera.
3. Estimate the camera motion as a homography fitted with RANSAC, independently for each pair of frames.
4. Warp and recompute. Warp frame $t+1$ by the homography and recompute the flow between frame $t$ and the warped frame.
5. Describe and prune. Compute HOF, MBH and the shape descriptor from the warped flow; HOG is unchanged. Remove trajectories whose largest warped displacement is below 1 pixel, as consistent with camera motion.

Mathematical Formulation

Sampling threshold. In the journal version, a grid point $i$ with structure-tensor eigenvalues $\lambda_i^1, \lambda_i^2$ (see image gradients) is kept if $\min(\lambda_i^1, \lambda_i^2) \ge T$, with

$$T = 0.001 \times \max_{i \in I} \min(\lambda_i^1, \lambda_i^2)$$

over the frame $I$. The CVPR paper only says “a threshold”.

Tracking. With dense flow $\omega_t = (u_t, v_t)$ and a $3 \times 3$ median kernel $M$,

$$P_{t+1} = (x_{t+1}, y_{t+1}) = (x_t, y_t) + (M * \omega_t)\big|_{(\bar{x}_t, \bar{y}_t)},$$

where $(\bar{x}_t, \bar{y}_t)$ is the rounded position, as written in the CVPR version; the kernel size is from the journal version. The median keeps motion boundaries sharp, where bilinear interpolation would mix foreground and background motion.

Shape descriptor. With displacements $\Delta P_t = P_{t+1} - P_t$,

$$S' = \frac{(\Delta P_t, \ldots, \Delta P_{t+L-1})}{\sum_{j=t}^{t+L-1} \lVert \Delta P_j \rVert},$$

which has $2L = 30$ entries.

Histograms. HOG and HOF quantize orientations into 8 full-circle bins weighted by magnitude; HOF adds a ninth bin for flow below a magnitude threshold. Over $2 \times 2 \times 3$ cells this gives 96 and 108 dimensions. MBH, from Dalal, Triggs and Schmid’s human detector (2006), applies the HOG computation to $\nabla u$ and to $\nabla v$ separately: MBHx and MBHy, 96 each. A constant flow $(u, v) = (c_1, c_2)$ has $\nabla u = \nabla v = 0$, so it contributes nothing to MBH, whereas HOF moves its mass to the bin of $\operatorname{atan2}(c_2, c_1)$. DT normalized each histogram descriptor by its L2 norm; iDT used RootSIFT instead (L1 normalization followed by a square root), which its authors report gains about 0.5%.

Encoding and Classification

Bag of features (DT; the iDT paper reuses its codebook settings): a k-means codebook of 4,000 words per descriptor type, trained on 100,000 random training features; each video becomes a histogram of word counts, and channels are combined in an SVM kernel

$$K(x_i, x_j) = \exp\Big(-\sum_c \frac{1}{A_c} D(x_i^c, x_j^c)\Big),$$

with $D$ the $\chi^2$ distance for channel $c$ and $A_c$ its mean over training pairs.

Fisher vectors (iDT’s default): reduce each descriptor type to half its dimension with PCA, fit a Gaussian mixture with $K = 256$ components to 256,000 sampled features, and encode a video by first- and second-order statistics (gradients with respect to the Gaussians’ means and standard deviations), $2DK$ values per descriptor type. Following Perronnin, Sánchez and Mensink (2010), apply power normalization $f(z) = \operatorname{sign}(z)\,|z|^\alpha$ to each dimension, then L2 normalization; the descriptor types are concatenated and a linear one-versus-rest SVM with $C = 100$ classifies. By our arithmetic, the four halved descriptors (15 + 48 + 54 + 96 dimensions) give a Fisher vector of $2 \times 256 \times 213 = 109{,}056$ values per video. Both encodings are described under image classification.

Parameters

Parameter	Value	Effect (journal version, Hollywood2 and YouTube)
Trajectory length $L$	15 frames	Accuracy rises up to 20 frames; longer tracks drift or cross shot boundaries
Sampling step $W$	5 pixels	$W = 2$ is slightly better (58.9% vs 58.2% mAP on Hollywood2) but much slower
Spatial scales $S$	up to 8, factor $1/\sqrt{2}$	Gains saturate at 5–6 scales
Neighborhood $N$	32 pixels	Stable between 24 and 48 on Hollywood2
Cell grid $n_\sigma \times n_\sigma \times n_\tau$	$2 \times 2 \times 3$	Going from $n_\sigma = 1$ to 2 helps most
Eigenvalue factor	0.001	Trades density for saliency
iDT camera threshold	1 pixel	Maximal warped displacement below which a track is removed

Complexity

Dense flow dominates. On the Hollywood2 clips of one movie (528 × 224 pixels, about 15,000 frames) on one 2.6 GHz CPU core, the journal version reports 52% of the time in optical flow, 26% in descriptors, 15% in writing features to disk and 6% in tracking, at 1.8 frames per second, slightly faster than STIP (1.5 frames per second) while producing about ten times as many features (205.1 versus 21.5 per frame). iDT adds SURF matching, a human detector, a RANSAC homography and a second flow computation per frame pair.

Implementation

The example builds two 16-frame clips of a textured square crossing a textured background with a flat band at the top, one with a static camera and one panning by (2.0, 0.8) pixels per frame; both share the first frame and the square’s own motion. A dense Lucas–Kanade estimate on a two-level pyramid stands in for Farnebäck’s flow. The code samples one scale, tracks with median-filtered flow, prunes static tracks (our threshold, as the papers give none: no step of 1 pixel or more) and erratic ones (the 70% rule), and runs the iDT step: a normalized-DLT homography with RANSAC from flow matches at good features, warping and recomputed flow. HOF and MBH use a 24-pixel tube. “Swept” counts background points that the square passes over.

import numpy as np
from scipy import ndimage as ndi

H = W = 96
L = 15                      # trajectory length (frames)
STEP = 5                    # dense sampling step W (pixels)
PAN = np.array([2.0, 0.8])     # camera pan: image shift of the scene, px/frame (x, y)
OBJ_V = np.array([-0.6, -1.4])  # object's own motion, px/frame


def make_clip(pan, seed=0):
    """Textured square moving over a textured background with a flat band."""
    rng = np.random.default_rng(seed)
    bg = ndi.gaussian_filter(rng.standard_normal((200, 200)), 2.0)
    bg[:70, :] = 0.0                                  # flat, untextured band
    obj = ndi.gaussian_filter(rng.standard_normal((24, 24)), 1.5)
    bg, obj = bg / bg.std(), 1.5 * obj / obj.std()
    yy, xx = np.mgrid[0:H, 0:W].astype(float)
    frames, masks = [], []
    for t in range(L + 1):
        sx, sy = pan * t                              # scene shift at frame t
        f = ndi.map_coordinates(bg, [yy - sy + 50, xx - sx + 50], order=3)
        ox, oy = np.array([30.0, 55.0]) + (pan + OBJ_V) * t   # object corner
        alpha = ndi.map_coordinates(np.pad(np.ones((24, 24)), 1),
                                    [yy - oy + 1, xx - ox + 1], order=1)  # anti-aliased mask
        tex = ndi.map_coordinates(obj, [yy - oy, xx - ox], order=3, mode="nearest")
        frames.append((1 - alpha) * f + alpha * tex)
        masks.append(alpha > 0.5)
    return np.array(frames), np.array(masks)


def lk_flow(a, b, sigma=3.0, iters=5, levels=2):
    """Dense Lucas-Kanade with iterative warping on a small pyramid."""
    pa, pb = [a], [b]
    for _ in range(levels - 1):
        pa.append(ndi.zoom(ndi.gaussian_filter(pa[-1], 1), 0.5, order=1))
        pb.append(ndi.zoom(ndi.gaussian_filter(pb[-1], 1), 0.5, order=1))
    u = np.zeros_like(pa[-1]); v = np.zeros_like(pa[-1])
    for lev in range(levels - 1, -1, -1):
        A, B = pa[lev], pb[lev]
        if u.shape != A.shape:
            u = 2 * ndi.zoom(u, np.array(A.shape) / u.shape, order=1)
            v = 2 * ndi.zoom(v, np.array(A.shape) / v.shape, order=1)
        Iy, Ix = np.gradient(A)
        g = lambda z: ndi.gaussian_filter(z, sigma)
        Sxx, Sxy, Syy = g(Ix * Ix), g(Ix * Iy), g(Iy * Iy)
        det = Sxx * Syy - Sxy ** 2 + 1e-9
        yy, xx = np.mgrid[0:A.shape[0], 0:A.shape[1]].astype(float)
        for _ in range(iters):
            Bw = ndi.map_coordinates(B, [yy + v, xx + u], order=1, mode="nearest")
            It = Bw - A
            bx, by = g(Ix * It), g(Iy * It)
            u -= (Syy * bx - Sxy * by) / det
            v -= (Sxx * by - Sxy * bx) / det
    return u, v


def min_eig(img, sigma=1.5):
    Iy, Ix = np.gradient(img)
    a, b, c = (ndi.gaussian_filter(z, sigma) for z in (Ix * Ix, Ix * Iy, Iy * Iy))
    return (a + c) / 2 - np.sqrt(((a - c) / 2) ** 2 + b ** 2)


def sample_grid(img):
    lam = min_eig(img)
    ys, xs = np.mgrid[STEP // 2:H:STEP, STEP // 2:W:STEP]
    keep = lam[ys, xs] >= 0.001 * lam.max()          # T = 0.001 x max min-eigenvalue
    return np.c_[xs[keep], ys[keep]].astype(float), ys.size


def at(field, p):
    ix = np.clip(np.rint(p[:, 0]).astype(int), 0, W - 1)
    iy = np.clip(np.rint(p[:, 1]).astype(int), 0, H - 1)
    return field[iy, ix]


def track(points, flows):
    """Median-filtered flow, read at the rounded point position."""
    traj = [points.copy()]
    alive = np.ones(len(points), bool)
    for u, v in flows:
        um, vm = ndi.median_filter(u, 3), ndi.median_filter(v, 3)
        p = traj[-1]
        q = p + np.c_[at(um, p), at(vm, p)]
        alive &= (q[:, 0] >= 0) & (q[:, 0] <= W - 1) & (q[:, 1] >= 0) & (q[:, 1] <= H - 1)
        traj.append(q)
    return np.stack(traj, 1), alive                  # (n, L+1, 2)


def prune(traj, min_motion=1.0):
    d = np.linalg.norm(np.diff(traj, axis=1), axis=2)  # (n, L) step lengths
    static = d.max(1) < min_motion                     # our threshold (not from the paper)
    erratic = d.max(1) > 0.7 * d.sum(1)                # one step > 70% of the total
    return static, erratic & ~static


def shape_descriptor(traj):
    dp = np.diff(traj, axis=1)                         # (n, L, 2)
    return dp.reshape(len(dp), -1) / np.linalg.norm(dp, axis=2).sum(1, keepdims=True)


def orient_hist(dx, dy, nbins=8, zero_thr=None):
    mag, ang = np.hypot(dx, dy), np.arctan2(dy, dx) % (2 * np.pi)
    b = (ang / (2 * np.pi) * nbins).astype(int) % nbins
    if zero_thr is None:
        return np.bincount(b.ravel(), mag.ravel(), nbins)
    h = np.bincount(b.ravel(), (mag * (mag >= zero_thr)).ravel(), nbins)
    return np.r_[h, (mag < zero_thr).sum()]          # 9th bin: near-zero flow


def tube_descriptor(traj, fields, kind, N=24, ns=2, nt=3):
    """HOF or MBH in an N x N x L tube along each trajectory, ns x ns x nt cells."""
    out = []
    c = N // ns
    for tr in traj:
        cells = []
        for k in range(nt):
            acc = [0.0] * (ns * ns)
            for t in range(k * L // nt, (k + 1) * L // nt):
                x, y = np.rint(tr[t]).astype(int) + N // 2   # patch corner in padded field
                pu, pv = (np.pad(f, N)[y:y + N, x:x + N] for f in fields[t])
                for i in range(ns):
                    for j in range(ns):
                        su, sv = pu[i*c:(i+1)*c, j*c:(j+1)*c], pv[i*c:(i+1)*c, j*c:(j+1)*c]
                        if kind == "HOF":
                            h = orient_hist(su, sv, zero_thr=0.25)
                        else:                          # MBHx and MBHy: gradients of u and of v
                            gy, gx = np.gradient(su); hy, hx = np.gradient(sv)
                            h = np.r_[orient_hist(gx, gy), orient_hist(hx, hy)]
                        acc[i * ns + j] = acc[i * ns + j] + h
            cells += acc
        d = np.concatenate(cells)
        out.append(d / (np.linalg.norm(d) + 1e-12))
    return np.array(out)


def ransac_homography(src, dst, rng, iters=200, thr=0.5):
    def norm(p):                                       # Hartley normalization
        c = p.mean(0); f = np.sqrt(2) / np.linalg.norm(p - c, axis=1).mean()
        return np.array([[f, 0, -f * c[0]], [0, f, -f * c[1]], [0, 0, 1]])
    def dlt(s, d):
        Ts, Td = norm(s), norm(d)
        s, d = apply(Ts, s), apply(Td, d)
        A = []
        for (x, y), (xp, yp) in zip(s, d):
            A += [[-x, -y, -1, 0, 0, 0, x * xp, y * xp, xp],
                  [0, 0, 0, -x, -y, -1, x * yp, y * yp, yp]]
        Hn = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
        return np.linalg.inv(Td) @ Hn @ Ts
    def apply(Hm, p):
        q = np.c_[p, np.ones(len(p))] @ Hm.T
        return q[:, :2] / q[:, 2:]
    best = None
    for _ in range(iters):
        i = rng.choice(len(src), 4, replace=False)
        with np.errstate(divide="ignore", invalid="ignore"):   # degenerate samples
            inl = np.linalg.norm(apply(dlt(src[i], dst[i]), src) - dst, axis=1) < thr
        if best is None or inl.sum() > best.sum():
            best = inl
    Hm = dlt(src[best], dst[best])
    return Hm / Hm[2, 2], best, apply


def camera_compensated_flows(frames, flows, rng):
    """iDT step: homography from flow matches at good features, warp, recompute flow."""
    yy, xx = np.mgrid[0:H, 0:W].astype(float)
    out, stats = [], []
    for t, (u, v) in enumerate(flows):
        lam = min_eig(frames[t])
        ys, xs = np.nonzero(lam > 0.01 * lam.max())
        sel = rng.choice(len(xs), min(300, len(xs)), replace=False)
        src = np.c_[xs[sel], ys[sel]].astype(float)
        dst = src + np.c_[u[ys[sel], xs[sel]], v[ys[sel], xs[sel]]]
        Hm, inl, apply = ransac_homography(src, dst, rng)
        # warp frame t+1 back onto frame t with H, then recompute the flow
        g = np.c_[xx.ravel(), yy.ravel()]
        p = apply(Hm, g)
        warped = ndi.map_coordinates(frames[t + 1], [p[:, 1].reshape(H, W),
                                     p[:, 0].reshape(H, W)], order=3, mode="nearest")
        out.append(lk_flow(frames[t], warped))
        stats.append((*(apply(Hm, g) - g).mean(0), inl.mean()))
    return out, np.array(stats)


rng = np.random.default_rng(0)
R = {}
for name, pan in [("static camera", np.zeros(2)), ("panning camera", PAN)]:
    frames, masks = make_clip(pan)
    flows = [lk_flow(frames[t], frames[t + 1]) for t in range(L)]
    pts, n_grid = sample_grid(frames[0])
    traj, alive = track(pts, flows)
    obj = at(masks[0], pts)                           # ground truth: starts on the square
    # background points the square passes over (from the generator's true motion)
    swept = ~obj & np.any([at(ndi.binary_dilation(masks[t], iterations=2), pts + pan * t)
                           for t in range(L + 1)], axis=0)
    static, erratic = prune(traj)
    kept = alive & ~static & ~erratic
    bg = ~obj & ~swept
    print(f"{name}: {len(pts)} of {n_grid} grid points pass the eigenvalue test; "
          f"{np.sum(~alive)} left the frame, {np.sum(alive & static)} static and "
          f"{np.sum(alive & erratic)} erratic pruned; kept: "
          f"{np.sum(kept & obj)} object, {np.sum(kept & swept)} swept, {np.sum(kept & bg)} background")
    R[name] = dict(frames=frames, flows=flows, traj=traj, kept=kept, obj=obj, bg=bg, alive=alive)

r = R["static camera"]
s = shape_descriptor(r["traj"][r["kept"] & r["obj"]])
print("shape descriptor, object, first 2 steps:", np.round(s.mean(0)[:4], 3), "dim", s.shape[1])

# iDT: camera-motion compensation on the panning clip
r = R["panning camera"]
wflows, st = camera_compensated_flows(r["frames"], r["flows"], rng)
print(f"homography motion: mean ({st[:, 0].mean():.2f}, {st[:, 1].mean():.2f}) px/frame, "
      f"true ({PAN[0]}, {PAN[1]}); mean inlier fraction {st[:, 2].mean():.2f}")
mag = np.stack([np.hypot(at(u, r["traj"][:, t]), at(v, r["traj"][:, t]))
                for t, (u, v) in enumerate(wflows)], 1)
camera = mag.max(1) < 1.0                             # iDT: max stabilized motion < 1 px
k = r["kept"]
print(f"kept background trajectories: {np.sum(k & r['bg'])} before, "
      f"{np.sum(k & r['bg'] & ~camera)} after camera-motion removal; "
      f"object: {np.sum(k & r['obj'])} before, {np.sum(k & r['obj'] & ~camera)} after")

# HOF vs MBH on object trajectories, static vs panning camera (same start points)
sel = R["static camera"]["alive"] & r["alive"] & r["obj"]
D = {name: {kd: tube_descriptor(R[name]["traj"][sel], R[name]["flows"], kd) for kd in ("HOF", "MBH")}
     for name in R}
hof_w = tube_descriptor(r["traj"][sel], wflows, "HOF")
cos = lambda a, b: np.mean(np.sum(a * b, 1))
ref = D["static camera"]
print(f"{sel.sum()} object trajectories, mean cosine similarity to the static-camera descriptor: "
      f"HOF {cos(ref['HOF'], D['panning camera']['HOF']):.3f}, "
      f"MBH {cos(ref['MBH'], D['panning camera']['MBH']):.3f}, "
      f"HOF on warped flow {cos(ref['HOF'], hof_w):.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3, about 1 second):

static camera: 300 of 361 grid points pass the eigenvalue test; 0 left the frame, 254 static and 0 erratic pruned; kept: 25 object, 21 swept, 0 background
panning camera: 300 of 361 grid points pass the eigenvalue test; 125 left the frame, 0 static and 0 erratic pruned; kept: 25 object, 36 swept, 114 background
shape descriptor, object, first 2 steps: [-0.027 -0.063 -0.027 -0.064] dim 30
homography motion: mean (2.00, 0.79) px/frame, true (2.0, 0.8); mean inlier fraction 0.80
kept background trajectories: 114 before, 3 after camera-motion removal; object: 25 before, 25 after
25 object trajectories, mean cosine similarity to the static-camera descriptor: HOF 0.003, MBH 0.999, HOF on warped flow 1.000

The eigenvalue test removes the 61 grid points in the flat band. With a static camera, the static test removes every background track except 21 that the square drags along as it passes. With the pan, no background track is static, so 114 survive DT’s pruning. The homography recovers the pan to within 0.01 pixel per frame, the square’s flow vectors being RANSAC outliers, and the 1-pixel rule on the warped flow removes 111 of these 114 tracks while keeping all 25 on the square. The square’s shape descriptor is close to the true $(-0.6, -1.4)/(15 \times 1.52) = (-0.026, -0.061)$ per step. HOF of the same 25 tracks is almost orthogonal between the clips, because the pan turns the square’s flow from about 247° to 337°, into another bin; MBH barely changes, because the pan adds the same constant on both sides of the square’s boundary. After warping, HOF matches the static-camera clip. The camera here is a pure translation and the flow clean; real videos add blur, zoom and several depth planes.

Properties and Behavior

• Descriptors (DT, CVPR 2011). With bag of features, the four descriptors combined reached 58.3% mAP on Hollywood2 and 84.2% average accuracy on the YouTube dataset (25-fold leave-one-out). MBH was the best single descriptor on all four datasets and beat HOF by 11.1 points on the hand-held YouTube videos (83.9% vs 72.8%). For the combined descriptor, dense trajectories beat KLT trajectories by 0.8 (KTH) to 6.1 points (Table 1; the text says 2% to 6%).
• Journal version (2013). Among nine datasets, the combined descriptor reached 46.6% average accuracy over the three HMDB51 splits (48.3% with spatio-temporal pyramids).
• Improved trajectories (ICCV 2013). With Fisher vectors and human detection, iDT reached 64.3% mAP on Hollywood2, 57.2% average accuracy over the three HMDB51 splits, 91.1% mAP on Olympic Sports and 91.2% average accuracy on UCF50 (leave-one-group-out), against 60.1%, 52.2%, 84.7% and 88.6% for DT with the same encoding and RootSIFT (Tables 1–2). HOF gained most: on HMDB51 it rose from 39.5% to 48.9%, close to MBH’s 52.1%, and the trajectory shape descriptor from 25.4% to 32.4%. Fisher vectors beat bag of features for both DT and iDT, and human detection added 1.3 points on Hollywood2 and HMDB51 (Table 4).
• UCF101. The iDT paper has no UCF101 result. Simonyan and Zisserman (2014) list iDT at 85.9% mean accuracy over the three UCF101 splits, citing the ICCV paper and Wang and Schmid’s THUMOS 2013 workshop entry.

Limitations

• Camera models. iDT assumes one homography between consecutive frames. Its authors show failures under severe motion blur, when people fill the frame, and when the background consists of two planes at different depths.
• Local motion only (our reading). Fifteen-frame tracks and orderless pooling, apart from coarse spatio-temporal pyramids, do not represent the order of events across a video.
• No learning of features. Each stage is designed or trained separately, so the descriptors cannot adapt to the classes; this is what end-to-end networks changed.

Variants

Deep networks replaced DT and iDT but used them as a baseline and a complement. Simonyan and Zisserman (2014) presented the temporal stream of their two-stream network, which takes stacked optical flow, as a learned generalization of HOF, MBH and the trajectory descriptor; the network reached 88.0% on UCF101 and 59.4% on HMDB51 against iDT’s 85.9% and 57.2%, and they named pooling along trajectories and explicit camera-motion handling as iDT ingredients their model still lacked. Combining networks with iDT still helped. The 3D convolutional network C3D rose from 85.2% to 90.4% on UCF101 (three-split setting) when its features were combined with iDT (Tran et al., 2015), and Feichtenhofer, Pinz and Zisserman (2016) raised their two-stream fusion network from 92.5% to 93.5% on UCF101 and from 65.4% to 69.2% on HMDB51 (mean accuracy, their Tables 5–6) by averaging its scores with those of an SVM on Fisher-encoded iDT features. They read the gain as complementarity and suspected it might vanish with far more training data.