ByteTrack

ByteTrack is a multi-object tracking method introduced by Zhang et al. (arXiv 2021, ECCV 2022). Its core is BYTE, an association rule for tracking by detection: instead of discarding detections below a confidence threshold, it matches high-score detections to tracks first and then gives the tracks left over a second chance with the low-score detections, which often belong to partially occluded or blurred objects. ByteTrack itself is BYTE combined with a SORT-style Kalman filter and the YOLOX detector. It adds almost no computation to SORT, and it became a common baseline that later trackers such as BoT-SORT extend.

Problem

Tracking-by-detection systems usually keep only detections whose score exceeds a threshold, often around 0.5, because low-score boxes contain many background false positives. But a detector’s score also falls when an object is partly hidden, blurred, or unusually sized. Thresholding then deletes true objects exactly when they are hardest to see: the track is missed for those frames, and if the gap is long, the object returns under a new identity. Lowering the threshold recovers those objects but admits background boxes that become spurious tracks. The ByteTrack authors observed that similarity to an existing track separates the two cases: a low-score box that overlaps where a track is predicted to be is probably that object; one that matches no track is probably background.

Inputs and Outputs

Input: per frame, detected boxes with confidence scores, from any detector. The tracker uses the boxes, the scores, and optionally appearance features.

Output: per frame, the boxes of active tracks with integer identities. Lost tracks are kept internally but not reported.

Intuition

High-score detections are trustworthy enough to drive everything: they update tracks and, when unmatched, start new ones. Low-score detections are trusted only as evidence for objects the tracker already follows. A track that found no high-score box in this frame probably belongs to an object that just became harder to see, so a low-score box near its predicted position is likely the same object. Low-score boxes never start tracks, so background clutter at low confidence cannot create identities.

Algorithm

Detections are split by two thresholds, $\tau_{\text{high}}$ and $\tau_{\text{low}}$; boxes below $\tau_{\text{low}}$ are ignored. Tracks are either tracked (matched recently) or lost (unmatched, kept for a buffer of frames). For each frame:

1. Split. $\mathcal{D}_{\text{high}}$ holds boxes with score above $\tau_{\text{high}}$; $\mathcal{D}_{\text{low}}$ holds boxes with score between $\tau_{\text{low}}$ and $\tau_{\text{high}}$.
2. Predict. Propagate every track, tracked and lost, with its Kalman filter.
3. First association. Match $\mathcal{D}_{\text{high}}$ to all tracks, lost ones included, using IoU or an appearance distance, solved as an assignment with the Hungarian algorithm. Matched tracks are updated; a lost track that matches is recovered under its old identity.
4. Second association. Match $\mathcal{D}_{\text{low}}$ to the tracks still unmatched, using IoU only. Matched tracks are updated.
5. Discard. Low-score boxes left unmatched are treated as background and dropped.
6. Lose and delete. Tracks unmatched after both rounds become lost; lost tracks older than the buffer are deleted.
7. Create. Each unmatched high-score box starts a new track.

for each frame:
    D = detector(frame)
    D_high = {d in D : d.score > tau_high}
    D_low  = {d in D : tau_low < d.score <= tau_high}
    predict all tracks T (tracked and lost) with the Kalman filter
    first:  match T to D_high by IoU (or appearance); update matches
    T_rem, D_rem = unmatched tracks, unmatched high-score boxes
    second: match T_rem to D_low by IoU only; update matches
    drop unmatched boxes of D_low
    mark tracks still unmatched as lost; delete those lost > buffer frames
    start a new track for each box in D_rem
    output tracked (not lost) tracks

The second round uses IoU alone on purpose: the authors argue that appearance features of heavily occluded or blurred boxes are unreliable, and their ablation found IoU better than re-identification features for that round. The official implementation restricts the second round to tracks that were matched in the previous frame, so lost tracks can be recovered only by high-score boxes.

Mathematical Formulation

Let $\mathcal{T}$ be the tracks with predicted boxes $\hat{b}_i$ and $\mathcal{D}$ the detections $d_j$ with scores $s_j$. The two detection sets are

$$\mathcal{D}_{\text{high}} = \{ d_j : s_j > \tau_{\text{high}} \}, \qquad \mathcal{D}_{\text{low}} = \{ d_j : \tau_{\text{low}} < s_j \le \tau_{\text{high}} \}.$$

Each round solves an assignment on the IoU cost

$$C_{ij} = 1 - \mathrm{IoU}(\hat{b}_i, d_j),$$

rejecting assigned pairs whose IoU is below a minimum. In the first round, the official code by default multiplies the IoU by the detection score, using $1 - s_j \, \mathrm{IoU}(\hat{b}_i, d_j)$, so confident detections are preferred; the MOT20 setting disables this. With re-identification, the first-round cost is replaced or combined with an appearance distance.

The motion model is the constant-velocity Kalman filter of SORT, with one difference in parameterization: the official code uses DeepSORT’s filter, whose state holds the box center, aspect ratio, and height, $[x, y, a, h]$, with their velocities, and whose noise scales with box height.

Parameters

Values from the paper and from the official repository (github.com/ifzhang/ByteTrack):

• High threshold $\tau_{\text{high}}$ (track_thresh). The paper’s default is 0.6, as in the evaluation script; the demo script uses 0.5. The authors report that BYTE is less sensitive to this threshold than SORT, because objects below it are still recovered in the second round.
• Low threshold $\tau_{\text{low}}$. Fixed at 0.1 in the official code. The paper’s pseudocode has a single threshold and sends every box below $\tau_{\text{high}}$ to the second round.
• New-track threshold. The code starts a track only from an unmatched high-score box with score above $\tau_{\text{high}} + 0.1$, and confirms it only if it is matched again in the next frame (except in the first frame of a sequence).
• First-round matching threshold (match_thresh). The paper rejects pairs with IoU below 0.2. The code expresses this as a maximum cost on the score-weighted cost: 0.9 in the evaluation script and 0.8 in the demo.
• Second-round matching threshold. A maximum IoU distance of 0.5 in the code, so pairs need an IoU above about 0.5.
• Track buffer (track_buffer, default 30 frames). How long a lost track is kept for recovery, scaled by frame rate relative to 30 frames per second.

Complexity

BYTE adds a second assignment to SORT’s per-frame work. With $N$ detections and $M$ tracks, both rounds together cost $O(NM)$ for the IoU matrices plus the assignment, $O(n^3)$ worst case with $n = \max(N, M)$, which for tens of objects is negligible next to the detector. On MOT17, the authors measured about 4 ms per frame for association and 17.9 to 30.0 ms for YOLOX detection, depending on the input resolution, on a V100 GPU.

Implementation

A compact tracker following the official code’s association rules, with SORT’s Kalman filter. Object B is half hidden in frames 7–12: its score drops to 0.35 and it nearly stops. A low-score background box appears in frames 9–10. The same tracker is run with and without the second association:

import numpy as np
from scipy.optimize import linear_sum_assignment

# SORT's constant-velocity Kalman filter on z = [u, v, s, r].
F = np.eye(7)
F[0, 4] = F[1, 5] = F[2, 6] = 1.0
H = np.eye(4, 7)
Q = np.diag([1, 1, 1, 1, 0.01, 0.01, 0.0001])
R = np.diag([1, 1, 10, 10])


def box_to_z(b):  # [x1, y1, x2, y2] -> [u, v, s, r]
    w, h = b[2] - b[0], b[3] - b[1]
    return np.array([b[0] + w / 2, b[1] + h / 2, w * h, w / h])


def x_to_box(x):
    w = np.sqrt(x[2] * x[3])
    h = x[2] / w
    return np.array([x[0] - w / 2, x[1] - h / 2, x[0] + w / 2, x[1] + h / 2])


def iou(a, b):  # pairwise IoU between box arrays of shape (n, 4) and (m, 4)
    tl = np.maximum(a[:, None, :2], b[None, :, :2])
    br = np.minimum(a[:, None, 2:], b[None, :, 2:])
    inter = np.prod(np.clip(br - tl, 0, None), axis=2)
    area = lambda c: (c[:, 2] - c[:, 0]) * (c[:, 3] - c[:, 1])
    return inter / (area(a)[:, None] + area(b)[None, :] - inter)


def match(tracks, boxes, iou_min):
    """Hungarian matching on IoU; returns pairs and unmatched indices."""
    if not tracks or len(boxes) == 0:
        return [], list(range(len(tracks))), list(range(len(boxes)))
    cost = iou(np.array([t.box for t in tracks]), boxes)
    rows, cols = linear_sum_assignment(-cost)
    pairs = [(r, c) for r, c in zip(rows, cols) if cost[r, c] >= iou_min]
    mt, md = {r for r, _ in pairs}, {c for _, c in pairs}
    return (pairs, [i for i in range(len(tracks)) if i not in mt],
            [j for j in range(len(boxes)) if j not in md])


class Track:
    count = 0

    def __init__(self, box):
        self.x = np.r_[box_to_z(box), 0, 0, 0]
        self.P = np.diag([10, 10, 10, 10, 1e4, 1e4, 1e4])
        Track.count += 1
        self.id, self.misses = Track.count, 0

    def predict(self):
        self.x, self.P = F @ self.x, F @ self.P @ F.T + Q
        self.misses += 1
        self.box = x_to_box(self.x)

    def update(self, box):
        y = box_to_z(box) - H @ self.x
        S = H @ self.P @ H.T + R
        K = self.P @ H.T @ np.linalg.inv(S)
        self.x, self.P = self.x + K @ y, (np.eye(7) - K @ H) @ self.P
        self.misses = 0


class Byte:
    def __init__(self, high=0.6, low=0.1, iou1=0.2, iou2=0.5, buffer=30, second=True):
        self.high, self.low, self.iou1, self.iou2 = high, low, iou1, iou2
        self.buffer, self.second, self.tracks = buffer, second, []

    def update(self, boxes, scores):
        d_high = boxes[scores > self.high]
        d_low = boxes[(scores > self.low) & (scores <= self.high)]
        for t in self.tracks:
            t.predict()
        # First association: high-score boxes against all tracks, lost ones included.
        pairs, t_rem, d_rem = match(self.tracks, d_high, self.iou1)
        for i, j in pairs:
            self.tracks[i].update(d_high[j])
        # Second association: low-score boxes against the remaining tracks that
        # were matched in the previous frame, by IoU only.
        if self.second:
            cand = [self.tracks[i] for i in t_rem if self.tracks[i].misses == 1]
            pairs2, _, _ = match(cand, d_low, self.iou2)
            for i, j in pairs2:
                cand[i].update(d_low[j])
        # Unmatched low-score boxes are dropped as background; unmatched
        # high-score boxes start tracks; lost tracks survive `buffer` frames.
        self.tracks += [Track(d_high[j]) for j in d_rem]
        self.tracks = [t for t in self.tracks if t.misses <= self.buffer]
        return sorted(t.id for t in self.tracks if t.misses == 0)


# A walks right with high scores. B walks left, then is half hidden in
# frames 7-12: its score drops to 0.35 and it slows almost to a stop.
# A low-score background box appears in frames 9-10.
def detections(f):
    bx = 200 - 6 * min(f, 6) - 1 * max(0, min(f, 12) - 6) - 2 * max(0, f - 12)
    boxes = [[10 + 8 * f, 20, 40 + 8 * f, 80], [bx, 30, bx + 30, 90]]
    scores = [0.9, 0.35 if 7 <= f <= 12 else 0.9]
    if f in (9, 10):
        boxes.append([300, 200, 320, 240])
        scores.append(0.2)
    return np.array(boxes, float), np.array(scores)


rng = np.random.default_rng(0)
noise = rng.normal(0, 1.0, (16, 3, 4))
for name, second in [("high-score only", False), ("BYTE", True)]:
    Track.count = 0
    tracker = Byte(second=second)
    print(name)
    for f in range(1, 17):
        boxes, scores = detections(f)
        ids = tracker.update(boxes + noise[f - 1, :len(boxes)], scores)
        if f >= 5:
            print(f"  frame {f:2d}: scores {np.round(scores, 2).tolist()} -> ids {ids}")

Output:

high-score only
  frame  5: scores [0.9, 0.9] -> ids [1, 2]
  frame  6: scores [0.9, 0.9] -> ids [1, 2]
  frame  7: scores [0.9, 0.35] -> ids [1]
  frame  8: scores [0.9, 0.35] -> ids [1]
  frame  9: scores [0.9, 0.35, 0.2] -> ids [1]
  frame 10: scores [0.9, 0.35, 0.2] -> ids [1]
  frame 11: scores [0.9, 0.35] -> ids [1]
  frame 12: scores [0.9, 0.35] -> ids [1]
  frame 13: scores [0.9, 0.9] -> ids [1, 3]
  frame 14: scores [0.9, 0.9] -> ids [1, 3]
  frame 15: scores [0.9, 0.9] -> ids [1, 3]
  frame 16: scores [0.9, 0.9] -> ids [1, 3]
BYTE
  frame  5: scores [0.9, 0.9] -> ids [1, 2]
  frame  6: scores [0.9, 0.9] -> ids [1, 2]
  frame  7: scores [0.9, 0.35] -> ids [1, 2]
  frame  8: scores [0.9, 0.35] -> ids [1, 2]
  frame  9: scores [0.9, 0.35, 0.2] -> ids [1, 2]
  frame 10: scores [0.9, 0.35, 0.2] -> ids [1, 2]
  frame 11: scores [0.9, 0.35] -> ids [1, 2]
  frame 12: scores [0.9, 0.35] -> ids [1, 2]
  frame 13: scores [0.9, 0.9] -> ids [1, 2]
  frame 14: scores [0.9, 0.9] -> ids [1, 2]
  frame 15: scores [0.9, 0.9] -> ids [1, 2]
  frame 16: scores [0.9, 0.9] -> ids [1, 2]

Without the second round, B’s detections are discarded while its score is low. Its lost track coasts at the old walking speed and has drifted far ahead by frame 13, so B’s next high-score box no longer overlaps it and starts a new identity, 3. With BYTE, the low-score boxes keep updating B’s track, its filter learns that B slowed down, and B keeps identity 2 throughout. The low-score background box matches no track and is discarded in both runs. For brevity, the example omits the official code’s score-weighted cost, new-track threshold, and confirmation step.

Properties and Behavior

• Reported results. With a YOLOX-X detector trained on MOT17, CrowdHuman, CityPersons, and ETHZ, the authors reported 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on the MOT17 test set at 29.6 frames per second on a single V100 GPU, including detection, under the private-detection protocol. On the more crowded MOT20 test set, they reported 77.8 MOTA, 75.2 IDF1, and 61.3 HOTA at 17.5 frames per second. The authors reported that both entries ranked first on the respective leaderboards at the time. The test-set results also used linear interpolation across gaps of up to 20 frames as post-processing, which the authors found raised MOTA on validation from 76.6 to 78.3.
• Generic. BYTE is independent of the detector and the first-round cost. Added to nine existing trackers, including re-identification, motion-based, and attention-based ones, it improved MOTA, IDF1, and identity switches in almost all configurations on the authors’ MOT17 validation split, with a few drops of under one point; for CenterTrack, IDF1 rose from 64.2 to 74.0 and identity switches fell from 528 to 144.
• Motion over appearance in crowds. With the same detections, BYTE using IoU alone gave higher MOTA and fewer identity switches than DeepSORT on MOT17 validation. The authors note that appearance features degrade under severe occlusion.
• Recall from low scores. On MOT17 validation sequences, the authors found that the low-score boxes BYTE kept contained notably more true positives than false positives, even in sequences, such as MOT17-02, where false positives far outnumber true positives among all low-score boxes.

Limitations

• Linear motion and IoU. The constant-velocity model and overlap costs fail under nonlinear motion, low frame rates, and camera motion. On BDD100K driving video, the authors dropped the Kalman filter and used re-identification features in the first round instead.
• No long-term re-identification by default. A lost track is recovered only if its prediction still overlaps a high-score box within the buffer; after that, the object returns under a new identity.
• Coasting drift. As with SORT, a lost track drifts with its last velocity, which limits recovery after long occlusions.
• Detector dependence. Much of ByteTrack’s reported accuracy comes from its detector and training data. On MOT17 validation, the authors reported 76.6 MOTA with YOLOX-X but 64.4 with the much smaller YOLOX-Nano at a lower input resolution. Under the public-detection protocol, where new tracks may start only near the benchmark’s provided boxes, they reported 67.4 MOTA on the MOT17 test set, still ahead of the methods they compared.
• Threshold choices remain. $\tau_{\text{high}}$ is less sensitive than in SORT, but the low threshold, matching thresholds, and buffer still need tuning for new domains.

Variants

BoT-SORT (Aharon, Orfaig & Bobrovsky, 2022) extends ByteTrack with camera-motion compensation, warping Kalman predictions by a frame-to-frame affine estimate from tracked background keypoints, estimates box width and height directly in the filter, and, with re-identification, uses the element-wise minimum of the IoU distance and a gated appearance distance as the first-round cost.

OC-SORT (Cao et al., 2023) also builds on SORT’s structure. It re-updates a recovered track’s filter along a virtual trajectory across the occlusion gap, adds a motion-direction term to the cost, and matches unmatched tracks’ last observations to leftover detections. The authors report strong results on DanceTrack, whose motion is highly nonlinear.

Later trackers built on the two-stage, score-split association; BoT-SORT, for example, keeps it and adds appearance features to the first round, as the SORT and data association articles describe.