Simple Online and Realtime Tracking

“Simple Online and Realtime Tracking” by Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft, of Queensland University of Technology and the University of Sydney, was presented at the IEEE International Conference on Image Processing (ICIP) in 2016 after appearing on arXiv in February of that year. It introduced SORT, a multi-object tracking method assembled from two classical components, a Kalman filter and the Hungarian algorithm, and argued that with a modern CNN detector this minimal tracker could match far more elaborate online trackers while running much faster. This article covers the paper as published; the SORT article explains the algorithm in detail.

Problem

By 2015, visual multi-object tracking was benchmarked on the newly established MOTChallenge, and the authors drew three observations from its leaderboard. First, classical data association methods that defer hard decisions, Multiple Hypothesis Tracking and Joint Probabilistic Data Association, had returned in updated forms and held many of the top positions. Their combinatorial cost and delayed decisions made them ill-suited to online use. Second, the top-ranked tracker was the only one not built on the Aggregate Channel Features (ACF) pedestrian detector, suggesting that detection quality, not association, was limiting the rest. Third, accuracy and speed traded off sharply: the most accurate trackers were too slow for real-time applications such as collision avoidance for autonomous vehicles.

Many online trackers of the time added appearance models learned online, motion context between objects, and dedicated handling of occlusions and detector errors. The paper asked how simple a multi-object tracker could be, and how well it could perform, if those components were dropped and a better detector was used instead.

Contribution

The authors listed three contributions:

• CNN detection for tracking. They brought a Faster R-CNN detector into the multi-object tracking pipeline and quantified how much the detector alone changes tracking accuracy.
• A pragmatic tracker. They presented an online tracking-by-detection method built only from a constant-velocity Kalman filter and Hungarian assignment on box overlap, with no appearance features, and evaluated it on the MOTChallenge benchmark.
• An open baseline. They released the code (github.com/abewley/sort) so that the method could serve as a baseline for research and be adopted in applications.

The design was deliberately minimal, in the spirit of Occam’s razor: only the position and size of each box are used, for both motion prediction and association, and short- and long-term occlusion are not handled explicitly, on the grounds that such cases occur rarely and that re-identification would add cost incompatible with real-time use.

Method

The paper describes four components, each in a few paragraphs:

• Detection. Faster R-CNN, with either the ZF or the deeper VGG16 network, using the default weights trained for PASCAL VOC. Only person detections with probability above 50% are passed to the tracker.
• Estimation model. Each target is a Kalman filter on the box center, area, and aspect ratio, plus the velocities of center and area, $\mathbf{x} = [u, v, s, r, \dot{u}, \dot{v}, \dot{s}]^\top$, under a linear constant-velocity model that ignores other objects and camera motion. The aspect ratio is held constant.
• Data association. Predicted boxes are compared with detections by intersection-over-union (IoU), the assignment is solved optimally with the Hungarian algorithm, and pairs with overlap below $\mathrm{IoU}_{\min}$ are rejected. The authors observed that IoU implicitly handles short occlusions by passing objects, because only the occluder is detected and the hidden track is simply left unassigned.
• Track management. A detection that overlaps no track starts a new one with zero velocity and large velocity variance, followed by a probationary period to filter out false positives. A track is deleted after $T_{\text{lost}}$ frames without a detection, set to 1 in all experiments, both because the constant-velocity model predicts poorly over longer gaps and because re-identification was out of scope. An object that reappears gets a new identity.

The initial covariances, $\mathrm{IoU}_{\min}$, and $T_{\text{lost}}$ were tuned on a training and validation split taken from earlier work. The SORT article gives the full state-space model, the defaults of the reference implementation, and a worked example.

Results

Detector comparison. On validation sequences, swapping ACF for Faster R-CNN (VGG16) raised SORT’s multi-object tracking accuracy (MOTA) from 15.1 to 34.0, the gain the abstract summarizes as up to 18.9%. The same swap raised the MOTA of MDP, a more complex online tracker, from 24.0 to 33.5, supporting the thesis that the detector dominates.

MOTChallenge 2015 test set. On the 11 withheld test sequences, with Faster R-CNN (VGG16) detections, SORT reached a MOTA of 33.4 and a multi-object tracking precision (MOTP) of 72.1. Among the trackers listed in the paper, this was the highest MOTA of any online method, ahead of TDAM (33.0) and MDP (30.3), and close to the near-online NOMT (33.7), which uses future frames. SORT also had the fewest mostly lost trajectories (30.9%) and the fewest false positives in the table. Its 1,001 identity switches, however, were more than any other online tracker listed, more than twice TDAM’s 464.

Speed. The tracking component ran at 260 Hz on a single core of a 2.5 GHz Intel i7, which the authors describe as over 20 times faster than other state-of-the-art trackers. In their speed–accuracy plot of MOTChallenge entries, SORT occupied the accurate, real-time corner; the only faster tracker shown was far less accurate.

Impact

SORT’s main influence was to reset expectations about what a tracker needs. By showing that a few classical components on top of a strong detector were competitive, it shifted attention toward detection quality and toward association designs that stay simple and fast. Its released code made it easy to reuse, and the paper has been widely cited. Several later trackers adopted its Kalman-plus-assignment structure and its name, from DeepSORT to OC-SORT.

Limitations

The authors were explicit about what they left out. With no appearance model and $T_{\text{lost}} = 1$, any occlusion longer than a frame or two ends a track, and a returning object receives a new identity; the high identity-switch count in their own results reflects this. They also called the constant-velocity model a poor predictor of true dynamics, and they evaluated only on pedestrians, although they noted that the approach extends to other classes the detector can find.

Some caveats follow from the setup. The 260 Hz figure excludes detection, so end-to-end speed depends on the detector, and Faster R-CNN in 2016 was far slower than the tracker. The benchmark comparison is between complete systems: SORT ran on its own Faster R-CNN (VGG16) detections, and the paper does not list the detections behind the other entries in its table, although its introduction notes that ACF detections were the norm on the benchmark. Part of SORT’s advantage therefore likely comes from the detector, which is the paper’s point but limits what the table says about association alone. Later work found that overlap-only matching also suffers under camera motion, low frame rates, and fast or nonlinear motion, where predicted and detected boxes stop overlapping.

What Came After

The paper’s conclusion suggested that its simplicity would let new methods concentrate on re-identification for long-term occlusion, and the most direct follow-up did exactly that. DeepSORT (Wojke, Bewley, and Paulus, ICIP 2017), with Bewley as a co-author, added a CNN appearance descriptor trained for person re-identification and a matching cascade that favors recently seen tracks, and reported roughly 45% fewer identity switches on MOT16 with the same detections.

Later trackers kept SORT’s structure while revisiting its weak points. ByteTrack (Zhang et al., ECCV 2022) also associates low-confidence detections, which SORT discards, in a second matching round, recovering occluded objects. OC-SORT (Cao et al., CVPR 2023) corrects the error a track accumulates while it coasts unobserved and adds observation-based motion cues to the association cost, targeting nonlinear motion and occlusion. Other descendants, such as BoT-SORT and StrongSORT, add camera-motion compensation and stronger appearance models; the SORT article describes these variants.