Video Understanding

Action Recognition with Improved Trajectories

Wang and Schmid's ICCV 2013 paper that estimated camera motion as a homography from SURF and optical-flow matches, masked out detected humans, and used the estimate to warp the flow and prune background trajectories; winner of the 2023 Helmholtz Prize.

advanced

“Action Recognition with Improved Trajectories” by Heng Wang and Cordelia Schmid of the LEAR team at INRIA, France, was presented at the IEEE International Conference on Computer Vision (ICCV) in Sydney in 2013. It made dense trajectories, then the strongest local features for action recognition, robust to camera motion: the camera’s motion between two frames is estimated as a homography, the optical flow is corrected with it, and trajectories that merely follow the camera are discarded. The resulting features, known as improved dense trajectories (iDT), set new records on four benchmarks, won the THUMOS 2013 challenge on UCF101, and remained the reference hand-crafted video representation in the first years of deep video networks. The paper received the Helmholtz Prize in 2023. This article covers the paper as published; the dense trajectories article explains the algorithm, its descriptors and its encodings.

Problem

By 2013, action recognition on realistic video (movies, YouTube clips, TV shows) was dominated by local space-time features pooled into one vector per video and classified with an SVM. Many image descriptors had been extended to video, such as 3D-SIFT and HOG3D, and the authors cite dense trajectories (Wang, Kläser, Schmid and Liu) as the best of them across datasets.

Camera motion was the weak point. When the camera pans, tilts or zooms, many trajectories are spent on the moving background, and HOF, which the earlier work had shown to degrade under camera motion, mixes the actor’s motion with the camera’s. The authors attribute MBH’s lead among the descriptors to its robustness to camera motion, since it uses the derivatives of the flow, and argued that explicit camera-motion estimation would still help. They describe very few action-recognition methods as having tried it, among them Jain, Jégou and Bouthemy (CVPR 2013), who in their account decomposed visual motion into dominant and residual parts to stabilize dense trajectories.

Contribution

The abstract lists the first four contributions below; the fifth comes from the experimental setup:

  • Camera-motion estimation from two kinds of matches. Consecutive frames are assumed to be related by a homography, justified by the small motion between two frames. Correspondences come from SURF descriptors matched by nearest neighbour, chosen for robustness to motion blur, and from dense optical flow sampled at Shi–Tomasi good features to track. SURF fires on blobs and good features on corners and edges, so their union spreads the points more evenly, which the authors call critical for a good fit. The homography is fitted with RANSAC.
  • Human detection to remove inconsistent matches. People often fill much of the frame in action datasets, and their motion does not follow the camera’s. A human detector by Prest, Schmid and Ferrari, which the authors describe as an adaptation of a part-based detector to action datasets, masks out matches inside its boxes; to fill in missed detections, each box is propagated forward and backward with the mean flow for up to 15 frames.
  • Flow warping. The second frame is warped by the homography and the flow recomputed, so that HOF, MBH and the trajectory descriptor see motion relative to the background; HOG is unchanged.
  • Trajectory pruning. A trajectory whose largest displacement in the warped flow is below 1 pixel is treated as camera motion and removed.
  • Encoding. Fisher vectors, with power and L2 normalization following Perronnin, Sánchez and Mensink (2010), are compared with the bag-of-features pipeline of the earlier work and adopted as the default.

The code and the human bounding boxes were released on the authors’ web page. Each step, with a worked example on a synthetic panning clip, is explained in the dense trajectories article.

Experiments

The evaluation used four datasets, each with its standard protocol:

  • Hollywood2: 12 classes, 1,707 clips from 69 movies (823 for training, 884 for testing, from different movies), scored by mean average precision (mAP).
  • HMDB51: 51 classes, 6,766 clips from movies and YouTube, three splits with 70 training and 30 test clips per class, scored by mean accuracy over the splits. The authors used the original videos rather than the stabilized versions distributed with the dataset.
  • Olympic Sports: 16 sports, 783 YouTube sequences (649 for training, 134 for testing), mAP.
  • UCF50: 50 classes, 6,618 YouTube clips in 25 groups, leave-one-group-out cross-validation, mean accuracy.

The baseline throughout is dense trajectories with the same Fisher-vector encoding and RootSIFT normalization (square root after L1 normalization, which the authors credit with about 0.5%), so the gains measure camera-motion handling alone.

Which component helps (Table 1). The authors split the method into “WarpFlow” (descriptors from the warped flow, all trajectories kept) and “RmTrack” (background trajectories removed, descriptors from the original flow). Warping accounts for most of the gain. With all descriptors combined on Hollywood2, the baseline’s 60.1% mAP rises to 63.6% with WarpFlow, 61.7% with RmTrack and 64.3% with both; on Olympic Sports the figures are 84.7%, 88.9%, 87.0% and 91.1%. Olympic Sports, which the authors describe as having significant camera motion, shows the largest single gains: the trajectory descriptor rises from 62.4% to 77.2% and HOF from 74.5% to 87.6%, close to MBH. MBH itself still gains 3.1 and 3.0 points on Hollywood2 and HMDB51 (the text says over 3%) and 6.7 on Olympic Sports, which the authors attribute to clearer motion boundaries. HOG gains at most 1.8 points, probably from the removal of background trajectories, in the authors’ reading.

Encoding (Table 2). The gain from camera-motion handling is about the same under either encoding: with bag of features, improved trajectories reach 62.2% on Hollywood2, 52.1% on HMDB51, 83.3% on Olympic Sports and 87.2% on UCF50. Fisher vectors beat bag of features in every case, by about 2 points on Hollywood2 and 7.8 points (improved) to 9.3 points (baseline) on Olympic Sports; the bag-of-features results use an RBF-χ2\chi^2 kernel SVM and the Fisher vectors a linear SVM.

Human detection (Tables 3 and 4). To compare the detector with an ideal one, the authors annotated humans by hand in 20 training and 20 test videos per class of Hollywood2. On this subset, all descriptors combined scored 48.9% mAP without human masking, 50.7% with the automatic detector and 51.9% with manual boxes. On the full datasets, results without human detection were 63.0%, 55.9%, 90.2% and 90.5%, the gain being larger on Hollywood2 and HMDB51, where people often fill the movie frame.

State of the art (Table 4). The final results, 64.3% mAP on Hollywood2, 57.2% accuracy on HMDB51, 91.1% mAP on Olympic Sports and 91.2% accuracy on UCF50, exceeded the best results listed: 62.5%, 52.1% and 83.2% from Jain et al. on the first three, and 83.3% from Shi et al. on UCF50. On Hollywood2, the authors note, every compared result was itself a modification of dense trajectories.

Reception

THUMOS 2013. The first THUMOS challenge, held with ICCV 2013, evaluated recognition of UCF101’s 101 classes over its three splits. According to the official results page, the LEAR-INRIA entry by Wang and Schmid ranked first of 16 teams with 85.900% overall accuracy, ahead of Florence (85.708%) and Canberra (85.437%). Its notebook paper describes the ICCV method with two changes: human detection was dropped as too expensive for a dataset of that size, and spatio-temporal pyramids (two temporal blocks, three horizontal strips) were added; without them, HOG, HOF and MBH combined reached 84.8%. The third- and fourth-placed entries, Canberra’s and CAS-SIAT’s, also name improved trajectories in their titles.

The baseline for deep networks. Simonyan and Zisserman (2014) list iDT at 85.9% on UCF101 and 57.2% on HMDB51, citing the ICCV paper and the THUMOS entry; their two-stream network, at 88.0% and 59.4%, was comparable. They cite Wang and Schmid on the value of camera-motion compensation but used a simpler substitute, subtracting each flow field’s mean vector, and named explicit camera-motion handling as an ingredient their network still lacked. Tran et al. (2015) still called iDT the state-of-the-art hand-crafted feature when introducing C3D (3D convolutional networks). Networks were also combined with it: Feichtenhofer, Pinz and Zisserman (2016) list C3D, TDD and their own two-stream fusion network, each combined with Fisher-encoded iDT, among the strongest results (their Table 6), and read the gain for their own network as complementarity; the numbers are given in the dense trajectories article.

Helmholtz Prize. The IEEE PAMI Technical Committee lists “Action Recognition With Improved Trajectories” by H. Wang and C. Schmid as a 2023 winner of the Helmholtz Prize, a biennial prize that it presents at each ICCV to ICCV papers from ten years earlier with significant impact on computer vision research.

Limitations

The authors show three failure cases: severe motion blur, which makes both SURF matching and flow unreliable and which they flag for further work since blur is common in realistic video; people dominating the frame, so that the homography fits them, which the detector fixes only when it fires (it misses unusual poses and viewpoints); and a background of two planes at different depths, which no single homography fits.

Cost was the main limitation identified later. The ICCV paper reports no running times; its journal extension (Wang, Oneata, Verbeek & Schmid, 2016) gives 10 to 15 seconds per frame on one core for the human detector, which was therefore left out for the 1,000 hours of TRECVID MED video, as it had been for THUMOS, and says that computing improved rather than original trajectories, without the detector, roughly doubled the feature-extraction time. In Tran et al.’s UCF101 comparison (their Table 6), iDT ran at 3.5 frames per second on a CPU, 202.2 hours for the dataset, against 2.2 hours for C3D on a GPU, a comparison they note is unfair to a CPU-only method.

More fundamentally, in our reading, iDT is a chain of separately designed or trained stages, so its descriptors cannot adapt to the classes. Feichtenhofer et al. suspected that the complementarity of iDT and networks might vanish with far more training data, and networks pretrained on Kinetics, such as I3D in 2017, went far beyond iDT on UCF101 and HMDB51 on their own.

What Came After

The journal version, with Dan Oneata and Jakob Verbeek as coauthors, extends the evaluation to six classification datasets, action localization in feature-length movies and event recognition on TRECVID MED, and adds spatial layout to the Fisher-vector encoding. In deep models, camera-motion compensation reappeared in simplified form as the two-stream network’s mean-flow subtraction, and in Wang, Qiao and Tang’s trajectory-pooled deep-convolutional descriptors (TDD, CVPR 2015), which pool two-stream ConvNet feature maps over tubes centred on improved trajectories. The action recognition article follows the shift from hand-crafted to learned representations.

Related

  • Dense Trajectories

    A hand-crafted video representation that samples points densely, tracks them with median-filtered optical flow for 15 frames, and describes the space-time tube around each track; the improved version cancels camera motion with a homography.

  • Action Recognition

    Naming the action that a video clip shows, also called video classification; the task's definition and variants, the model families, how models are trained and evaluated on Kinetics, UCF101 and HMDB51, representative results, and the pitfalls in comparing them.

  • Two-Stream Networks

    Video recognition networks that run one CNN on single RGB frames for appearance and a second on stacked optical flow for motion, then fuse the two, from Simonyan and Zisserman's 2014 design to convolutional fusion, temporal segment networks and two-stream I3D.

  • 3D Convolutional Networks

    Video networks whose convolutions span time as well as space, from C3D's homogeneous 3×3×3 kernels to mixed and factorized (2+1)D, pseudo-3D and separable designs, inflation of image networks, and efficient channel-separated and expanded architectures such as CSN and X3D.

  • Homography

    An invertible projective transformation of the plane, a 3×3 matrix defined up to scale, that relates two images of a planar scene or two images taken by a camera rotating about its center.

  • RANSAC

    Random sample consensus, a randomized method that fits a model to data containing many outliers by repeatedly fitting minimal random samples and keeping the model that the most data points agree with.

References

  1. Wang, H. & Schmid, C. (2013). Action Recognition with Improved Trajectories. IEEE International Conference on Computer Vision (ICCV), 3551–3558.
  2. Wang, H., Kläser, A., Schmid, C. & Liu, C.-L. (2013). Dense Trajectories and Motion Boundary Descriptors for Action Recognition. International Journal of Computer Vision, 103(1), 60–79.
  3. Perronnin, F., Sánchez, J. & Mensink, T. (2010). Improving the Fisher Kernel for Large-Scale Image Classification. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 143–156.
  4. Wang, H. & Schmid, C. (2013). LEAR-INRIA Submission for the THUMOS Workshop. Notebook paper, ICCV Workshop on Action Recognition with a Large Number of Classes (THUMOS 2013) (official workshop copy; archived copy of 3 January 2014).
  5. THUMOS'13 Action Recognition Challenge Results (official workshop results page; archived copy of 27 February 2015).
  6. IEEE Computer Society, Technical Committee on Pattern Analysis and Machine Intelligence. The Helmholtz Prize.
  7. Simonyan, K. & Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. Advances in Neural Information Processing Systems 27 (NIPS).
  8. Tran, D., Bourdev, L., Fergus, R., Torresani, L. & Paluri, M. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks. IEEE International Conference on Computer Vision (ICCV), 4489–4497.
  9. Feichtenhofer, C., Pinz, A. & Zisserman, A. (2016). Convolutional Two-Stream Network Fusion for Video Action Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1933–1941.
  10. Wang, L., Qiao, Y. & Tang, X. (2015). Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4305–4314.
  11. Wang, H., Oneata, D., Verbeek, J. & Schmid, C. (2016). A Robust and Efficient Video Representation for Action Recognition. International Journal of Computer Vision, 119(3), 219–238.