Computer Vision in Sports

Computer vision in sports turns video of a match or a training session into measurements: where every player and the ball are at each instant, what happened and when, and, for officials, whether a ball crossed a line or a player was offside. The field is also called sports video analysis, and its measurements feed sports analytics, broadcast graphics and officiating. Thomas et al. (2017) note in their survey that most sports make it impractical to fix sensors to players or equipment, which leaves cameras as the general-purpose sensor; their survey covers commercial applications, research topics and public datasets.

The building blocks have their own articles: multi-object tracking and human pose estimation, on which this article builds, and homography, camera calibration, object detection, action recognition and person re-identification. This article covers what is specific to sport: the camera set-ups and the data they produce, the vision tasks and how they combine, the public benchmarks with their protocols, officiating systems as their official documents describe them, and the documented limits. The public benchmarks discussed here are mostly in association football (soccer), and so are most examples; the word “football” below means association football, as SportsMOT and MultiSports use it.

Problem

Three groups use the output. Clubs and analysts want per-player and per-team statistics such as distance covered, top speed, ball possession and team formation; SoccerNet-Tracking’s authors motivate tracking with exactly these uses, from scouting a striker to studying a team’s strategy (Cioppa et al., 2022). Broadcasters want to understand the game well enough to choose cameras and select replays, which is the motivation of SoccerNet-v2 (Deliège et al., 2021), and camera calibration is what allows graphics to be inserted into the picture in augmented reality (Cioppa et al., 2021). Officials want support for decisions on goals, offside and line calls, which elite competitions now provide with dedicated multi-camera systems.

Two kinds of data come out of these systems, and the distinction runs through the rest of the article:

• Tracking data are positions over time: the pitch coordinates of each player and the ball in every frame, and for some systems body keypoints. FIFA’s semi-automated offside technology, for example, tracks the ball and up to 29 data points of each player 50 times per second (FIFA, SAOT page).
• Event data are a log of what happened and when: passes, shots, fouls, cards, substitutions. In the SoccerNet paper the chief executive of the data company Wyscout is reported as saying that providing up to 2,000 annotations for a game takes over eight hours of an analyst’s time (Giancola et al., 2018); automating that annotation is the motivation of the action spotting task described below. SoccerNet-v2’s authors cite a public data set of soccer events localized in time and on the pitch that was released without any video, and note that such data serve player and team statistics rather than video understanding (Deliège et al., 2021).

Vision supplies the first kind directly, through detection, tracking and registration to the pitch, and the second through temporal detection of events in video.

Visual Data

Broadcast video. A television broadcast is edited: it cuts between cameras, shows close-ups and replays, and inserts logo transitions. SoccerNet-v2 annotated the camera shots of 200 complete games and counted an average of 583 camera transitions per game, about one every 9 seconds, among 13 shot classes; 71.4% of transitions were abrupt cuts, 14.2% fades and 14.2% logo transitions, and 86.7% of shots were live, 10.9% replays of an action and 2.4% other replays (Deliège et al., 2021). The original SoccerNet videos came from online providers in several encodings, at 25 to 50 frames per second and resolutions from SD to Full HD (Giancola et al., 2018). Broadcast footage is the most widely available video of a match, but each cut breaks the continuity that tracking relies on, and the camera pans, tilts and zooms throughout.

A single main camera. Some data sets use the uncut feed of the main camera that follows play. SoccerNet-Tracking uses main-camera video of 12 games of the 2019 Swiss Super League at 1080p and 25 frames per second; its authors chose it because camera changes and replays in broadcast video break the continuity of tracks (Cioppa et al., 2022). SportsMOT keeps only overhead shots of its basketball, volleyball and football games, at 720p and 25 frames per second, and checks that no clip contains an abrupt change of viewpoint (Cui et al., 2023).

Fixed multi-camera installations. Officiating systems use cameras fixed in the stadium. FIFA describes semi-automated offside technology as using 12 dedicated tracking cameras mounted under the stadium roof (FIFA, SAOT page). Hawk-Eye states that its goal-line technology uses 14 tracking cameras permanently installed in each stadium (Hawk-Eye, 2026, GLC vs GLT). FIFA’s test manual for goal-line technology distinguishes camera-based systems from systems based on electromagnetic fields (FIFA, 2024), so not every certified system is a vision system.

Other sensors. FIFA’s offside system for the 2022 World Cup combined camera tracking with an inertial measurement unit at the centre of the match ball, sending data 500 times per second to detect the kick point (FIFA, SAOT page). Sensor data of this kind complement vision in elite settings; SoccerNet-Tracking’s authors argue that vision-based tracking, which needs no sensor on the players, applies to any team in any stadium (Cioppa et al., 2022).

Vision Tasks Involved

A full system chains several tasks. The SoccerNet game state reconstruction benchmark lists them for soccer: pitch localization and camera calibration; person detection, re-identification and tracking; jersey number recognition; and team affiliation (SoccerNet, sn-gamestate README). Event detection runs alongside, often on the same video.

Player detection and tracking

Players are detected with general object detectors fine-tuned to the sport. Detectors trained on generic person data also detect spectators, coaches and staff, which count as false positives when only the players on the pitch are annotated. In SoccerNet-Tracking’s baselines this was one of the two main reasons why fine-tuning a FairMOT model on soccer data raised its HOTA from 43.9 to 57.9 on the test set; the other was that person detectors never detect the ball (Cioppa et al., 2022, Table 3). SportsMOT’s authors likewise note that specialized detectors must be trained not to detect the audience and referees (Cui et al., 2023). Vandeghen et al. (2022) trained a Faster R-CNN with a feature pyramid network and a ResNet-50 backbone on the training set of the SoccerNet-v3 annotations (344,660 boxes of players, referees and staff and 26,939 of salient objects such as the ball in all) and reported 52.3% mAP (AP50:95) on its test set after semi-supervised training on the full labelled and unlabelled sets, about 1.6 million frames, against 51.0% for the supervised teacher.

Tracking then links detections into identities, usually by tracking-by-detection with a motion model and an appearance model. Sport stresses the parts of the problem that pedestrian benchmarks do not:

• Similar appearance. Players of one team wear the same kit, so team-mates differ only by jersey number and small details (Cioppa et al., 2022). SportsMOT’s authors call the appearance “similar yet distinguishable”: unlike the identically dressed dancers of DanceTrack, players carry different numbers and postures (Cui et al., 2023).
• Fast, variable-speed motion. SportsMOT’s objects have the lowest overlap between adjacent frames among SportsMOT, MOT17 and DanceTrack, and also the lowest overlap with a Kalman filter prediction made from the previous ground truth, which its authors read as motion that a constant-velocity model predicts poorly; football has the lowest values of its three sports (Cui et al., 2023, Figs. 2 and 3). In their ablation of a tracker built on ByteTrack on the SportsMOT test set, associating by plain IoU with each track’s last box gave 71.5 HOTA, and adding the Kalman filter’s prediction lowered it to 64.1 (Cui et al., 2023, Table 5). Motion Models for Tracking covers the models and why they fail on manoeuvres.
• Occlusion in clusters. Corners, free kicks, penalties and goal celebrations pack players together. In SoccerNet-Tracking’s per-scenario analysis of ByteTrack, corners had the lowest HOTA when ground-truth boxes were given (63.9, with the lowest association accuracy), while with the tracker’s own detections penalties (31.0) and goals (40.3) were hardest, because the detector failed in dense clusters such as walls of players and celebrations (Cioppa et al., 2022, Table 4).
• Leaving and re-entering the view. A player who leaves the main camera’s view and returns is the same player. SoccerNet-Tracking keeps the same identity in that case, unlike most tracking data sets, and its authors note that most trackers do not re-identify over such gaps (Cioppa et al., 2022).

Identity in the sporting sense, which player of which team, needs more than a track. Jersey numbers are read with optical character recognition on player crops; the SoccerNet jersey number task gives short tracklets, 2,853 in its data set and 1,211 in the challenge set, and asks for one number from 1 to 99 per tracklet, or a class for “not visible”, scored by accuracy; its organizers note that the thumbnails are small and blurred and the number may be visible in only a few frames of a tracklet (SoccerNet, task pages). Team affiliation is learned from appearance; SoccerNet-Tracking’s authors cite work that separates teams without supervision (Cioppa et al., 2022). Person re-identification covers the appearance embeddings that keep identities apart, and the SoccerNet game state baseline includes a re-identification model (SoccerNet, sn-gamestate README).

Ball detection and tracking

The ball is the hardest object in the frame. SoccerNet-Tracking’s authors list four reasons: it is small (their example is under 100 pixels), players occlude it, it moves fast enough to blur, and its shape in the image changes (Cioppa et al., 2022). Their data set annotates 297 ball tracklets with 215,156 boxes, against about 3 million for players (Cioppa et al., 2022, Table 2). TrackNet (Huang et al., 2019) is a deep network that its title describes as built for tracking high-speed and tiny objects in sports applications; its paper could not be read for this article.

A single camera gives the ball’s position only in the image. Because the ball leaves the ground, a pitch homography does not give its position on the field, unlike a player’s foot point (our reasoning from the planar model below); its 3D position needs triangulation from two or more calibrated cameras, or a physical model of its flight, which fits the many fixed cameras of the officiating systems described below.

Pitch registration and camera calibration

To turn image positions into metres, the system needs the mapping between the image and the pitch. The playing surface is close to a plane whose markings are fixed by the rules: the IFAB’s Law 1 sets the centre circle radius at 9.15 m, the penalty area at 16.5 m and the goal area at 5.5 m from the inside of each goalpost, 7.32 m apart, and lines no wider than 12 cm, while the length of the pitch varies between 100 and 110 m and its width between 64 and 75 m for international matches (IFAB, Law 1). Cioppa et al. (2021) describe the two parameterizations in use: a full perspective camera model with intrinsic and extrinsic parameters, or the homography between the pitch plane and the image, which is valid when the field is planar. Under the homography model, a player’s position on the pitch is the image of the foot point under the inverse homography.

Chen and Little (2019) note that most sports cameras are pan-tilt-zoom (PTZ) cameras on a fixed base, and decompose the camera matrix into a PTZ part and a fixed base,

$$P = K\,Q_\phi\,Q_\theta\,S_\rho\,S_{\phi'}\,[\,I \mid -C\,],$$

where $K$ holds the focal length, $Q_\theta$ and $Q_\phi$ are the pan and tilt, and the base rotation $S_\rho S_{\phi'}$ and camera centre $C$ stay fixed during a match; they argue that only three of the parameters are significant free parameters, so that sampling them generates diverse camera poses. Three families of methods appear in the literature:

• Geometric inference from markings. Homayounfar et al. (2017) segment the field surface, lines and circles with a network, estimate vanishing points from the lines, and search a grid of candidate field positions with branch and bound in a Markov random field. On their World Cup 2014 data set (395 annotated images from 20 games; 209 for training and validation and 186 from 10 other games for testing), they reported a mean IoU of 0.83 between the projected and true field (their Table 2). Cioppa et al. (2021) point out that the method needs at least two horizontal and two vertical lines to find the vanishing points, which restricts it in practice to views near the goal areas; Homayounfar et al. handle views of the centre, where vertical lines are scarce, by fitting an ellipse to the centre circle.
• Retrieval from a dictionary of views, then refinement. A dictionary pairs synthetic renderings of the field markings with their camera parameters; a network encodes the segmented markings of the input image, the nearest dictionary entry gives a rough camera, and a refinement aligns it. Chen and Little (2019) generated 100,000 camera poses from their PTZ model, learned 16-dimensional descriptors of edge images with a Siamese network, detected markings with a generative adversarial network, and refined the retrieved pose; on the World Cup data set they reported a mean IoU of 89.4% over the whole field template and 94.5% over its visible part, against 83 for Homayounfar et al. and 91.4 for an earlier dictionary method, respectively (their Table 1). Cioppa et al. (2021) note that such methods slow down as the dictionary grows.
• Tracking the camera through time. Some methods track the PTZ parameters from frame to frame with an extended Kalman filter or a particle filter; Cioppa et al. (2021) note that they need an uncut stream, which rules them out for edited broadcasts.

Public ground truth was scarce for a long time. Cioppa et al. (2021) generated 12,000 pseudo-ground-truth calibrations of SoccerNet frames with a commercial calibration tool, trained a dictionary-based network on them, and released calibrations for all 500 SoccerNet games. SoccerNet’s later challenges split the problem in two: a field localization task asks for the extremities of every visible line, circle and goal-post element in 20,028 annotated images (2,104 in the challenge set), scored as accuracy, precision and recall of extremities within a distance threshold in a 960 × 540 image, and a camera calibration task scores the camera parameters by the distance between each annotated point and the projection of the marking it belongs to, counting a marking as found only if all of its points are within the threshold, together with a completeness rate (SoccerNet, task pages). The example below shows why the image-space accuracy of a registration does not tell the whole story.

Actions, events and pose

Action spotting. Giancola et al. (2018) argued that sports events are better defined by a single instant than by start and end times, because the rules anchor them: a goal is the instant the ball crosses the goal line, a card the instant the referee shows it. They defined spotting as finding these anchors, with a predicted spot counted as correct if it lies within a tolerance $\delta$ of a ground-truth anchor of the same class. Average precision is computed per class and tolerance, averaged over classes, and the Average-mAP is the area under the mAP curve for $\delta$ from 5 to 60 seconds. Temporal action localization, by contrast, predicts segments and scores them by temporal IoU.

The first SoccerNet has 500 games from six European competitions between 2014 and 2017, 764 hours of video and 6,637 events of three classes (goal, card, substitution), on average one every 6.9 minutes, split 300/100/100 into training, validation and test games. The events were parsed from online match reports at one-minute resolution and refined by hand to one second; game time was aligned with video time by reading the on-screen clock with OCR and fitting a line of slope one with RANSAC, which found the start of both halves within two seconds in 90% of games. On the test set the best spotting baseline reached an Average-mAP of 49.7% (Giancola et al., 2018; the text of Section 5.3 rounds the same result to 50%). SoccerNet-v2 annotated 110,458 actions of 17 classes in the same games, about one every 25 seconds, and tagged 18% of them as unshown: not shown in the broadcast, for example because a replay was on screen, so that the viewer must infer them (Deliège et al., 2021). On its test set, the best re-implemented baseline, CALF with a context-aware loss, reached 40.7% Average-mAP, and the per-class results show how the class imbalance plays out: with only 8 red cards and 14 second yellow cards in the test set, every method scored under 1% on those classes (Deliège et al., 2021, Table 2). From 2022 the SoccerNet challenge also reports a tight Average-mAP with tolerances of 1 to 5 seconds, and its ball action spotting task, on seven English Football League games with 12 classes of ball actions such as passes, drives, headers and shots, uses mAP at a 1-second tolerance (SoccerNet, task pages).

Broadcast production tasks. SoccerNet-v2 also defined camera shot segmentation (13 classes, scored by mIoU), shot boundary detection (spotting mAP with a 1-second tolerance) and replay grounding, which finds the live moment that a replay shows (Deliège et al., 2021). On boundary detection, a simple intensity-histogram difference reached 78.5% mAP, above the learned CALF detector at 59.6%, although CALF did better on fades (Deliège et al., 2021, Table 3).

Spatio-temporal action detection. Some tasks need to know who did what as well as when. MultiSports annotates 66 fine-grained action classes in basketball, volleyball, football and aerobic gymnastics: 3,200 clips with 37,701 action instances and 902,000 boxes at 25 frames per second, with temporal labels from professional athletes and boxes from crowd annotators assisted by a tracker (Li et al., 2021). Its authors chose sports because many actions happen at once and the background says little about which action is under way; they note that recognizing whether a football kick is a pass, a long ball, a through ball or a cross may need up to 5 seconds of context around a half-second kick. Spatio-temporal action detection covers the task and its frame-mAP and video-mAP protocols. SportsMOT’s clips were drawn from MultiSports’ videos (Cui et al., 2023).

Pose. Body keypoints matter wherever a decision depends on a body part rather than a box, and offside is the clearest case. FIFA’s offside system tracks up to 29 data points per player, including “all limbs and extremities that are relevant for making offside calls”, and, following Law 11, draws the offside line at the closest part of the second-last defender’s body that can legitimately score a goal (FIFA, SAOT page). Hawk-Eye states that its SkeleTRACK system tracks 29 skeletal points per player in real time and powers its version of semi-automated offside (Hawk-Eye, officiating page). Human pose estimation and 3D human pose estimation cover the methods; skeleton-based action recognition classifies actions from keypoint sequences.

Game state reconstruction

Game state reconstruction combines the tasks above into the representation analysts want: a minimap of every person on the pitch with role, team and jersey number. The SoccerNet benchmark gives 30-second main-camera clips at 1080p (57 for training, 59 for validation and 50 for testing) and asks for each person’s pitch position, role (player, goalkeeper, referee or other), team (left or right as seen from the camera) and jersey number (SoccerNet, task pages). It is scored by GS-HOTA, which keeps the structure of HOTA but replaces box overlap with a similarity on the pitch,

$$\text{Sim}(P, G) = \exp\!\left(\ln(0.05)\,\frac{\lVert P - G\rVert_2^2}{\tau^2}\right) \cdot \text{IdSim}(P, G), \qquad \tau = 5\ \text{m},$$

where IdSim is 1 only if role, team and jersey number all match, so that a correctly placed detection with a wrong number counts as a false positive (SoccerNet, sn-gamestate README). At a distance of $\tau$ the localization similarity is 0.05. The metric therefore measures exactly the chain that ends in metric positions, including the registration error studied next.

Example: Pitch Registration and Speed Error

Statistics such as distance covered and top speed are computed from foot points mapped to the pitch. Pixel noise in a foot point becomes an error in metres that depends on where the player is, because a camera looks at the far side of the pitch at a grazing angle. The example below renders the straight markings of a 105 × 68 m pitch (the IFAB dimensions above; circles and arcs are left out) into a 3840 × 2160 image from a pinhole camera 20 m up and 30 m behind the near touchline on the halfway line, a geometry we chose for the example. It estimates the pitch-to-image homography from the 22 intersections of fitted lines with the normalized direct linear transform, then simulates players running at 7 m/s near the camera, in the middle and on the far side, parallel to the touchlines (along $x$) and towards the camera (along $y$), with 1 pixel of independent Gaussian noise on each foot point at 25 frames per second. Speed is computed in two ways, since the smoothing window is a design choice that the sources leave open: from consecutive frames, and from the displacement over one second. The line search starts from a deliberately perturbed camera, as a dictionary retrieval would supply, and is refined twice; which pixels belong to which line is taken from that estimate, so line labelling, which a segmentation network would do in practice, is assumed solved.

import numpy as np
from scipy.ndimage import gaussian_filter

# ---------- pitch model (IFAB Law 1 dimensions, 105 x 68 m; straight lines only) ----------
L, W, w_line = 105.0, 68.0, 0.12
pa, ga = 16.5, 5.5                       # penalty and goal area depth
pa_h, ga_h = 3.66 + 16.5, 3.66 + 5.5     # half-widths from the goal centre (posts 7.32 m apart)
c = W / 2
seg = {  # name: ((x0, y0), (x1, y1)) in metres; y = 0 is the touchline nearest the camera
    "touch_near": ((0, 0), (L, 0)), "touch_far": ((0, W), (L, W)),
    "goal_L": ((0, 0), (0, W)), "goal_R": ((L, 0), (L, W)), "halfway": ((L / 2, 0), (L / 2, W)),
}
for side, x0, s in (("L", 0.0, 1.0), ("R", L, -1.0)):
    for name, d, h in (("pa", pa, pa_h), ("ga", ga, ga_h)):
        seg[f"{name}_front_{side}"] = ((x0 + s * d, c - h), (x0 + s * d, c + h))
        seg[f"{name}_lo_{side}"] = ((x0, c - h), (x0 + s * d, c - h))
        seg[f"{name}_hi_{side}"] = ((x0, c + h), (x0 + s * d, c + h))
junctions = [("touch_near", "goal_L"), ("touch_near", "goal_R"), ("touch_far", "goal_L"),
             ("touch_far", "goal_R"), ("touch_near", "halfway"), ("touch_far", "halfway")]
for side in "LR":
    for name in ("pa", "ga"):
        for edge in ("lo", "hi"):
            junctions += [(f"{name}_{edge}_{side}", f"goal_{side}"),
                          (f"{name}_{edge}_{side}", f"{name}_front_{side}")]
names = list(seg)


def camera_H(C, target, f, pan=0.0, tilt=0.0):
    """Pitch-plane (x, y, 1) -> image homography K [r1 r2 t] for a pinhole camera."""
    z = target - C; z /= np.linalg.norm(z)
    x = np.cross(z, [0, 0, 1.0]); x /= np.linalg.norm(x)
    y = np.cross(z, x)
    R = np.stack([x, y, z])
    a, b = np.radians(pan), np.radians(tilt)
    Rp = np.array([[np.cos(a), 0, np.sin(a)], [0, 1, 0], [-np.sin(a), 0, np.cos(a)]])
    Rt = np.array([[1, 0, 0], [0, np.cos(b), -np.sin(b)], [0, np.sin(b), np.cos(b)]])
    R = Rt @ Rp @ R
    K = np.array([[f, 0, 1920], [0, f, 1080], [0, 0, 1.0]])
    t = -R @ C
    return K @ np.c_[R[:, 0], R[:, 1], t]


def apply(H, p):
    q = np.c_[p, np.ones(len(p))] @ H.T
    return q[:, :2] / q[:, 2:]


C, target = np.array([52.5, -30.0, 20.0]), np.array([52.5, 34.0, 0.0])
H_true = camera_H(C, target, 1200.0)
H0 = camera_H(C, target, 1200.0 * 1.002, pan=0.1, tilt=0.1)   # coarse initial estimate

# ---------- render a 3840 x 2160 image of the markings (2x2 supersampling) ----------
rng = np.random.default_rng(0)
img = np.full((2160, 3840), 0.3)
r0, r1 = 900, 1500                                    # rows that contain the pitch
Hinv = np.linalg.inv(H_true)
off = np.array([-0.25, 0.25])
for rs in range(r0, r1, 100):
    vv, uu = np.mgrid[rs:rs + 100, 0:3840].astype(float)
    cover = np.zeros(vv.shape)
    for dv in off:
        for du in off:
            P = apply(Hinv, np.c_[(uu + du).ravel(), (vv + dv).ravel()])
            hit = np.zeros(len(P), bool)
            for (a, b) in seg.values():
                a, b = np.array(a), np.array(b)
                t = np.clip((P - a) @ (b - a) / ((b - a) @ (b - a)), 0, 1)
                hit |= np.linalg.norm(P - (a + t[:, None] * (b - a)), axis=1) <= w_line / 2
            cover += hit.reshape(vv.shape) / 4
    img[rs:rs + 100] += 0.6 * cover
img = gaussian_filter(img, 0.7) + rng.normal(0, 0.03, img.shape)

# ---------- fit each line in the image, intersect, estimate H by normalized DLT ----------


def img_line(H, name):
    """Projected end points of a model segment."""
    return apply(H, np.array(seg[name], float))


def dist_to_segment(p, a, b):
    t = np.clip((p - a) @ (b - a) / ((b - a) @ (b - a)), 0, 1)
    return np.linalg.norm(p - (a + t[:, None] * (b - a)), axis=1)


def fit_lines(H, band=6, excl=6.0):
    """Sub-pixel line points along each predicted segment, then robust total least squares."""
    ends = {n: img_line(H, n) for n in names}
    lines = {}
    for n in names:
        a, b = ends[n]
        horiz = abs(b[0] - a[0]) >= abs(b[1] - a[1])
        k, j = (0, 1) if horiz else (1, 0)              # step along axis k, profile along axis j
        pts = []
        for s in np.arange(np.ceil(min(a[k], b[k])), np.floor(max(a[k], b[k])) + 1):
            c0 = a[j] + (s - a[k]) * (b[j] - a[j]) / (b[k] - a[k])   # predicted line position
            offs = np.arange(int(round(c0)) - band, int(round(c0)) + band + 1)
            q = np.zeros((len(offs), 2)); q[:, k] = s; q[:, j] = offs
            if any(np.min(dist_to_segment(q, *ends[m])) <= excl for m in names if m != n):
                continue                                # skip samples near other lines
            prof = img[offs, int(s)] if horiz else img[int(s), offs]
            prof = prof - np.median(prof)
            i0 = np.argmax(prof)
            win = np.abs(offs - offs[i0]) <= 1.5
            wpos = np.clip(prof[win], 0, None)
            if wpos.sum() <= 0:
                continue
            p = np.zeros(2); p[k] = s; p[j] = (wpos * offs[win]).sum() / wpos.sum()
            pts.append(p)
        pts = np.array(pts)
        keep = np.ones(len(pts), bool)
        for _ in range(3):                              # TLS with outlier rejection
            mu = pts[keep].mean(0)
            nrm = np.linalg.svd(pts[keep] - mu)[2][-1]
            r = np.abs((pts - mu) @ nrm)
            mad = 1.4826 * np.median(r[keep])
            keep = r <= max(3 * mad, 0.5)
        mu = pts[keep].mean(0)
        nrm = np.linalg.svd(pts[keep] - mu)[2][-1]
        lines[n] = np.r_[nrm, -nrm @ mu]                # line: n . p + c = 0
    return lines


def normalize(p):
    m = p.mean(0)
    s = np.sqrt(2) / np.linalg.norm(p - m, axis=1).mean()
    return np.array([[s, 0, -s * m[0]], [0, s, -s * m[1]], [0, 0, 1]])


def dlt(p, q):
    """Normalized DLT: q ~ H p."""
    T1, T2 = normalize(p), normalize(q)
    a, b = apply(T1, p), apply(T2, q)
    A = []
    for (x, y), (u, v) in zip(a, b):
        A += [[0, 0, 0, -x, -y, -1, v * x, v * y, v], [x, y, 1, 0, 0, 0, -u * x, -u * y, -u]]
    Hn = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
    H = np.linalg.inv(T2) @ Hn @ T1
    return H / H[2, 2]


def model_point(n1, n2):
    (a1, b1), (a2, b2) = np.array(seg[n1]), np.array(seg[n2])
    l1, l2 = np.cross(np.r_[a1, 1], np.r_[b1, 1]), np.cross(np.r_[a2, 1], np.r_[b2, 1])
    x = np.cross(l1, l2)
    return x[:2] / x[2]


world = np.array([model_point(*j) for j in junctions])
H = H0
for it in range(2):                                     # two coarse-to-fine passes
    lines = fit_lines(H)
    kp = []
    for n1, n2 in junctions:
        x = np.cross(lines[n1], lines[n2])
        kp.append(x[:2] / x[2])
    kp = np.array(kp)
    H = dlt(world, kp)
H_est = H
kp_err = np.linalg.norm(kp - apply(H_true, world), axis=1)
print(f"{len(junctions)} line intersections, keypoint error vs true projection: "
      f"RMS {np.sqrt(np.mean(kp_err ** 2)):.2f} px, max {kp_err.max():.2f} px")

# pitch-plane error of the estimated homography, by distance from the camera
gx, gy = np.meshgrid(np.linspace(0, L, 106), np.linspace(0, W, 69))
G = np.c_[gx.ravel(), gy.ravel()]
back = apply(np.linalg.inv(H_est), apply(H_true, G))
e = np.linalg.norm(back - G, axis=1)
for lo, hi, lab in ((0, 23, "near third "), (23, 46, "middle third"), (46, 68.1, "far third  ")):
    sel = (G[:, 1] >= lo) & (G[:, 1] < hi)
    print(f"  {lab} (y {lo:>2}-{min(hi, 68):.0f} m): H error RMS {100 * np.sqrt(np.mean(e[sel] ** 2)):5.1f} cm, "
          f"max {100 * e[sel].max():5.1f} cm")

# ---------- speed from noisy foot points ----------
fps, v, T, sigma, trials = 25, 7.0, 4.0, 1.0, 500
n = int(T * fps) + 1
rng = np.random.default_rng(1)


def jac(H, p):
    """2x2 Jacobian of pitch -> image at p."""
    h = 1e-4
    return np.stack([(apply(H, p + [h, 0]) - apply(H, p - [h, 0]))[0] / (2 * h),
                     (apply(H, p + [0, h]) - apply(H, p - [0, h]))[0] / (2 * h)], axis=1)


print(f"\nruns at {v} m/s for {T:.0f} s, {fps} fps, foot-point noise {sigma} px, {trials} trials")
print("where   direction | m/px along | pos RMS | speed RMS (m/s)  | speed bias | distance | 1-s mean speed")
print("                  |  motion    |   (m)   | 1 frame   1 s    | 1 frame    |  error   | est H    true H")
for y0, where in ((5.0, "near  "), (34.0, "middle"), (63.0, "far   ")):
    for dvec, dname in ((np.array([1.0, 0]), "along x"), (np.array([0, 1.0]), "along y")):
        tt = np.arange(n) / fps
        P = np.array([L / 2, y0]) + np.outer(tt - T / 2, v * dvec)
        img_pts = apply(H_true, P)
        gsd = np.linalg.norm(dvec @ np.linalg.inv(jac(H_true, np.array([[L / 2, y0]]))))
        res = {k: [] for k in ("pos", "s1", "s25", "b1", "dist", "m25e", "m25t")}
        for _ in range(trials):
            noisy = img_pts + rng.normal(0, sigma, img_pts.shape)
            for Hm, tag in ((H_est, "e"), (H_true, "t")):
                Q = apply(np.linalg.inv(Hm), noisy)
                s25 = np.linalg.norm(Q[25:] - Q[:-25], axis=1)       # displacement over 1 s
                res["m25" + tag].append(s25.mean())
                if tag == "e":
                    step = np.linalg.norm(np.diff(Q, axis=0), axis=1)
                    res["pos"].append(np.sqrt(np.mean(np.sum((Q - P) ** 2, 1))))
                    res["s1"].append(np.sqrt(np.mean((step * fps - v) ** 2)))
                    res["s25"].append(np.sqrt(np.mean((s25 - v) ** 2)))
                    res["b1"].append(np.mean(step * fps - v))
                    res["dist"].append(100 * (step.sum() - v * T) / (v * T))
        r = {k: np.mean(val) for k, val in res.items()}
        print(f"{where} {dname}  |  {gsd:6.3f}    | {r['pos']:6.3f}  | {r['s1']:6.2f}  {r['s25']:6.3f}  |"
              f"  {r['b1']:+6.2f}    | {r['dist']:+6.1f}%  | {r['m25e']:6.3f}  {r['m25t']:6.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3; about 10 seconds):

22 line intersections, keypoint error vs true projection: RMS 0.64 px, max 1.25 px
  near third  (y  0-23 m): H error RMS   0.4 cm, max   0.8 cm
  middle third (y 23-46 m): H error RMS   0.4 cm, max   0.7 cm
  far third   (y 46-68 m): H error RMS   1.2 cm, max   1.8 cm

runs at 7.0 m/s for 4 s, 25 fps, foot-point noise 1.0 px, 500 trials
where   direction | m/px along | pos RMS | speed RMS (m/s)  | speed bias | distance | 1-s mean speed
                  |  motion    |   (m)   | 1 frame   1 s    | 1 frame    |  error   | est H    true H
near   along x  |   0.033    |  0.073  |   1.31   0.048  |   +0.36    |   +5.2%  |  7.000   7.001
near   along y  |   0.065    |  0.079  |   2.41   0.099  |   +0.14    |   +2.0%  |  7.002   7.001
middle along x  |   0.056    |  0.196  |   3.90   0.083  |   +2.33    |  +33.3%  |  7.005   7.005
middle along y  |   0.187    |  0.202  |   5.50   0.271  |   +1.47    |  +20.9%  |  7.002   7.000
far    along x  |   0.079    |  0.384  |   9.49   0.122  |   +6.62    |  +94.5%  |  7.020   7.020
far    along y  |   0.374    |  0.389  |  10.50   0.531  |   +5.61    |  +80.2%  |  7.003   7.000

The registration itself is accurate: the intersections of the fitted lines are within about a pixel of their true projections, and the estimated homography places points on the pitch within about 1 cm near the camera and 2 cm on the far side. Using the estimated rather than the true homography changes the one-second mean speeds by at most 0.003 m/s. With line fits along the full length of the markings, the calibration is not the limiting factor in this set-up; the noise on the foot points is.

The second column shows why the far side is worse. One pixel along the direction of motion corresponds to 0.065 m near the camera and 0.374 m on the far side for running towards the camera, a factor of 5.8, but only 0.033 m and 0.079 m, a factor of 2.4, for running parallel to the touchlines. This follows from the geometry (our derivation, to first order): at range $r$ from a camera at height $h$ with focal length $f$ in pixels, a pixel spans about $r/f$ across the line of sight but about $r^2/(f h)$ along the ground in depth, because the ground is seen at a grazing angle whose sine is $h/r$. With $f = 1200$ px and $h = 20$ m these give 0.068 m and 0.377 m for the two runs towards the camera, close to the printed values. The total position error grows by a factor of about five from the near to the far side whichever way the player runs, because the depth component dominates it.

Speed from consecutive frames is poor near the camera and useless on the far side: a player at 7 m/s moves 0.28 m per frame, the position noise is already a quarter of that near the camera and larger than it on the far side, so the RMS speed error is 1.3 to 10.5 m/s, and since a length cannot be negative the noise inflates speed on average. Summing the frame-to-frame steps overstates the distance covered by 2 to 5% near the camera and by 80 to 95% on the far side. On the far side the overstatement is larger for running along the touchline than towards the camera: the large depth noise is then perpendicular to the motion and adds to each step in quadrature, while parallel noise partly cancels against the step (our reading). Differencing over one second cuts the speed error to 0.05 to 0.10 m/s near the camera, but on the far side it remains 0.53 m/s for running towards the camera, more than four times the error for running along the touchline at the same place. A real system smooths trajectories with a motion model; Motion Models for Tracking and the Kalman filter cover how, and the trade-off is that a longer window also flattens real accelerations, which matter for top speed and sprints (our reasoning). Larger random errors in the foot point, such as a box whose bottom edge sometimes cuts off a foot, enter the same way and grow with distance in the same proportion, to first order.

The first run of this script fitted each line by weighted second moments over the whole search band. The plan’s log records that every grass pixel in the band then added noise to the moment across the line, so that the line’s “normal direction is noise-dominated”; the estimator was replaced by the per-column peak location and robust total least squares shown here, with the rendering, noise, seeds, players and printed columns unchanged.

Common Methods

The methods in sports systems are mostly the general ones, adapted:

• Detection and tracking. Detectors fine-tuned on sport-specific boxes, tracking-by-detection with Kalman filters and association by IoU or appearance, as in ByteTrack and DeepSORT, which SoccerNet-Tracking used as baselines (Cioppa et al., 2022). SportsMOT’s MixSort adds a MixFormer-style appearance model, reduced from 12 to 4 mixed-attention modules, whose similarity heatmap is fused with IoU before assignment with the Hungarian algorithm; with OC-SORT it reported 74.1 HOTA on the SportsMOT test set, against 73.7 for OC-SORT alone with the same training data (Cui et al., 2023, Table 3). In its fusion study, pure motion-based association beat pure appearance-based association in all three sports, and appearance helped football most, the sport with the fastest motion (Cui et al., 2023, Table 4).
• Registration. Line and keypoint detection with a segmentation network, followed by a homography or a full camera model, initialized by retrieval or geometric search and refined, and propagated over time where the stream is uncut.
• Temporal event models. Frame features at a low rate (SoccerNet supplies ResNet, I3D and C3D features at 2 frames per second, reduced by PCA to 512 dimensions), pooled over windows with learned pooling such as NetVLAD, or modelled with temporal networks and losses that use the context around each event, such as CALF (Giancola et al., 2018; Deliège et al., 2021). Audio helps for events marked by a whistle: in SoccerNet-v2, adding audio features gave the best results on shown actions (Deliège et al., 2021).
• Calibration-based representations. Cioppa et al. (2021) turned calibrated player positions into a top-view image, a feature vector and a player graph, and fed them to an action spotting network; they reported improved spotting on SoccerNet-v2.

Datasets and Benchmarks

Data set	Sport, source	Contents	Main task and metric
SoccerNet (2018)	Soccer, broadcast	500 games, 764 h, 6,637 events, 3 classes	Action spotting, Average-mAP over 5–60 s
SoccerNet-v2 (2021)	Soccer, broadcast	Same games; 110,458 actions (17 classes), 158,493 camera-change timestamps, 32,932 replay shots	Spotting, shot segmentation (mIoU), boundary detection, replay grounding
SoccerNet-Tracking (2022)	Soccer, main camera, 1080p, 25 fps	200 clips of 30 s and a full 45-min half; 5,009 tracklets, 3.6 M boxes; players, goalkeepers, referees, ball, others	MOT with and without ground-truth boxes, HOTA
SoccerNet field localization and calibration	Soccer, broadcast frames	20,028 images (2,104 challenge)	Marking extremities (accuracy, precision, recall); camera parameters by reprojection
SoccerNet game state reconstruction	Soccer, main camera	30-s clips: 57 train, 59 validation, 50 test	GS-HOTA on the pitch
World Cup 2014 (Homayounfar et al., 2017)	Soccer, broadcast	395 images from 20 games	Field registration, IoU
SportsMOT (2023)	Basketball, volleyball, football, 720p, 25 fps	240 clips, 150,379 frames, 1.6 M boxes, players only	MOT, HOTA, IDF1 and AssA recommended
MultiSports (2021)	Four sports	3,200 clips, 37,701 actions, 66 classes	Spatio-temporal action detection, frame-mAP and video-mAP

Protocols matter when comparing numbers:

• SoccerNet-Tracking has two tasks: association given ground-truth boxes, and full tracking from raw video. The 12 games are split into four sets: 57 clips for training, 49 for testing with public ground truth, and two challenge sets of 58 and 37 clips, the second including the full half, with hidden ground truth and limited submissions; the files follow the MOT20 format so that the MOTChallenge tools apply (Cioppa et al., 2022). The paper’s baselines show how much the detector matters. With ground-truth boxes, ByteTrack reached 71.5 HOTA (84.3 DetA, 60.7 AssA); with its own detector pretrained on crowd data, 47.2; the fine-tuned FairMOT reached 57.9 with 83.6 MOTA, the best in both measures among trackers using their own detections (Cioppa et al., 2022, Table 3). The authors explain that DetA stays below 100 even with ground-truth boxes because the association step filters out some boxes during fast motion. The SoccerNet tracking page describes the public data as 100 clips, whereas the paper’s training and test sets add up to 106.
• SportsMOT splits its 240 clips into 45 for training, 45 for validation and 150 for testing, evaluated on a server; its licence is Creative Commons Attribution-NonCommercial 4.0 (Cui et al., 2023; MCG-NJU, SportsMOT README). Its authors argue from their benchmark that association, not detection, is the main difficulty: IDF1 and AssA were lower than on MOT17 while DetA was high (Cui et al., 2023). The abstract describes the frame count as almost 15 times MOT17’s, while Section 3.2 says 13.4 times, which matches 150,379 against 11,235 frames.
• SoccerNet action spotting results with the loose (5–60 s) and tight (1–5 s) Average-mAP are not comparable, and SoccerNet-v1 (3 classes) and SoccerNet-v2 (17 classes) results differ by task: the same NetVLAD baseline scored 49.7 on the first and 31.4 on the second (Deliège et al., 2021, Table 2).
• Calibration IoU has two variants: over the whole field template, or over its visible part, which gives higher numbers for the same method (Chen and Little, 2019).

Evaluation

Each output has its own measure, and they do not substitute for one another:

• Tracking is scored by HOTA, with its detection (DetA) and association (AssA) components, alongside MOTA and IDF1; Multi-Object Tracking defines them. SoccerNet-Tracking adopted HOTA as its main metric because MOTA weighs detection more heavily than association (Cioppa et al., 2022), and its results show the effect: without ground-truth boxes ByteTrack had the best HOTA of the three pretrained trackers but the lowest MOTA (31.7), and the authors attribute FairMOT’s higher MOTA to its better detection, which MOTA favours (Cioppa et al., 2022, Table 3).
• Events are scored by spotting mAP at one or more tolerances, from 5–60 s in the original SoccerNet protocol down to 1 s for ball actions.
• Registration is scored by IoU of the projected field, by reprojection error of the markings, or, at the end of the chain, by GS-HOTA’s localization on the pitch.
• Physical quantities such as speed and distance need errors in metres and metres per second, measured against reference systems. Image-space metrics do not show them: in the example, a registration whose keypoints have an RMS error under one pixel still gives a distance covered that is wrong by 80% or more on the far side when computed naively from noisy foot points.

Officiating requirements are set as tests. FIFA’s test manual for goal-line technology requires, among other tests, that shots against a target wall placed 120 to 240 mm behind the goal line be indicated correctly to an accuracy of ±1.5 cm, with a high-speed camera at 2,000 frames per second as the reference, and that the goal signal reach the referee within one second (FIFA, 2024, Tables 1, 3A and 3B).

Officiating Systems

Goal-line technology. The IFAB’s Law 1 allows goal-line technology to verify whether a goal has been scored, applies it solely to the goal line, and requires the indication to be immediate, automatically confirmed within one second and sent only to the match officials, by watch or earpiece, and possibly to the video operation room; systems must meet the requirements of the FIFA Quality Programme, and the referee must test the system before the match (IFAB, Law 1). FIFA’s test manual traces the history: the IFAB set four basic requirements on 20 October 2010; FIFA and the Swiss materials laboratory Empa ran test phases in 2011; and on 5 July 2012 the IFAB decided to introduce goal-line technology into the Laws, with certification (FIFA, 2024). Certification includes field tests at two stadiums chosen by the provider, with unrestricted shots, target-wall shots with and without dummies blocking the view, a slider test, and goal-signal delay (FIFA, 2024). Hawk-Eye describes its system as the first FIFA-certified goal-line system and states that it achieves “accuracy of under 1mm in FIFA installation tests” (Hawk-Eye, 2026, GLC vs GLT); this is the company’s claim, and the published certification threshold is ±1.5 cm.

Semi-automated offside technology. FIFA’s description, written for the 2022 World Cup and updated in 2023, is that the 12 roof cameras and the ball’s inertial sensor feed a system that alerts the video match officials when an attacker in an offside position receives the ball; the officials then check the automatically selected kick point and the automatically drawn offside line before informing the referee, a process FIFA says takes a few seconds. The same positional data generate a 3D animation for stadium screens and broadcasters. If the system malfunctions, the officials fall back on manual tools, and judgements such as whether an offside player interfered with an opponent remain with the referee. FIFA states that it is the sole owner of the tracking and ball data collected (FIFA, SAOT page). Hawk-Eye states that its own semi-automated offside system has been used in more than 900 matches (Hawk-Eye, officiating page).

Line calling in tennis and ball tracking in cricket. Hawk-Eye states that cricket has been its founding sport since 2001, and that its electronic line calling has operated in tennis since 2005 on more than 450 courts (Hawk-Eye, officiating page). Its account of its work with the ATP gives official approval in 2005, the first use in a player-challenge system at the 2006 Miami Masters, and the launch of fully automated calling as Hawk-Eye Live in 2017 (Hawk-Eye, 2026, ATP). These are the company’s statements; the tennis federation’s own page on approved electronic line calling systems could not be read for this article.

Failure Modes and Risks

• Dense clusters and occlusion. Detection fails in walls, corners and celebrations, and identities switch when players cross (Cioppa et al., 2022, Table 4).
• Motion that breaks the motion model. Accelerations and stops make constant-velocity prediction a liability in some sports, as SportsMOT’s ablation showed (Cui et al., 2023, Table 5).
• Broadcast discontinuities. Cuts, replays and graphics break tracks and calibration; events that happen during a replay are not shown at all, which is why SoccerNet-v2 annotates unshown actions (Deliège et al., 2021).
• Distance from the camera. The same pixel error means several times more metres on the far side of the pitch, and raw frame-to-frame speeds and distances inflate with noise (the example above).
• Model assumptions. A homography assumes a flat pitch and no lens distortion; the SoccerNet calibration task includes lens parameters in its camera model (SoccerNet, task pages). Objects off the ground, the ball above all, cannot be placed on the pitch from one view.
• Rare classes. Red cards and second yellow cards were near zero for every SoccerNet-v2 spotting baseline (Deliège et al., 2021, Table 2).
• Generic models in a new domain. Person detectors pretrained on pedestrian data detect spectators and miss the ball (Cioppa et al., 2022); results obtained on one league or broadcaster need checking on another (our caution: the benchmarks above each come from a limited set of competitions, such as 12 games of one league for SoccerNet-Tracking).
• Small or private evaluation sets. SoccerNet-Tracking’s authors note that most earlier work on soccer player tracking was evaluated on small proprietary data sets (Cioppa et al., 2022), which makes published numbers hard to compare.

Deployment Considerations

• Fixed installation or broadcast. Fixed, calibrated multi-camera systems give stable geometry and 3D, at the cost of installation in each stadium; broadcast-based systems work for any televised match but must recalibrate the moving camera and survive cuts (our summary of the sections above). Hawk-Eye’s distinction between its 14-camera goal-line technology and a two-camera goal-line review system for leagues that already run VAR illustrates the range (Hawk-Eye, 2026, GLC vs GLT).
• Latency. Goal-line indications must reach the referee within one second (IFAB, Law 1; FIFA, 2024); analytics for coaches can run after the match.
• Human review. In FIFA’s offside workflow the system proposes and the officials confirm, with manual fallback (FIFA, SAOT page).
• Data ownership and licences. Ownership of tracking data is set by whoever runs the system: FIFA states that it owns the data from its offside system. Public research data sets carry their own terms, such as SportsMOT’s non-commercial licence (FIFA, SAOT page; MCG-NJU, SportsMOT README).
• Calibration as an error budget. Metric statistics depend on the camera geometry as much as on the detector: in the example, the same pixel noise gives errors several times larger on the far side, so one accuracy figure for the whole pitch does not show where the statistics can be trusted (our reading of the example).