Ego4D

Ego4D is a dataset and benchmark suite of first-person (“egocentric”) video of daily life, released in 2022 by a consortium led by Kristen Grauman with Meta AI and university partners. Its CVPR 2022 paper describes 3,670 hours of video recorded by 931 camera wearers in 74 locations in 9 countries, with five benchmark families organized around the wearer’s past, present and future: episodic memory, hands and objects, audio-visual diarization, social interactions, and forecasting (Grauman et al., 2022). The name joins “Ego”, for egocentric, with “4D”, for 3D space plus time. The paper’s title says 3,000 hours, while its abstract and §3.1 give 3,670; §3.1 calls this first release Ego4D-3K. As of November 2026 the data are released under per-institution licence agreements executed through Lab22, and yearly challenges are still held at CVPR, in 2026 on Ego4D v2.0.

This article covers the dataset itself: the collection, the benchmark families with their annotations and measures, the versions, the licence regime, and the natural language query (NLQ) protocol as the official evaluation code computes it. Egocentric video as a setting and Ego4D’s place among video datasets belong to video understanding; temporal IoU and detection mAP belong to temporal action localization.

Purpose

The paper contrasts its data with Internet photos and clips, which capture “brief, isolated moments” from a third-person view and are framed by a human photographer. Robotics and augmented reality instead receive a long, continuous stream from a camera worn by an agent engaged with its surroundings, uncurated, in which hands, objects and other people must be interpreted from the wearer’s point of view (§1). Among earlier egocentric datasets of unscripted daily life, the paper names EPIC-KITCHENS, UT Ego, ADL and the Disney dataset; it notes that the largest ones were recorded only in kitchens and that existing datasets relied largely on graduate students as camera wearers. It compares 3,670 hours with 100 in the largest earlier dataset, and 931 camera wearers with 71 (§2).

To capture unscripted activity while covering common scenarios, the organizers consulted a U.S. Bureau of Labor Statistics survey of how people spend their time, recruited participants whose daily activities would span those scenarios, and asked them to wear the camera at least as long as its battery lasted. A typical raw clip lasts 8 minutes, against the 10-second clips the paper cites as common in third-person video understanding (§3.2). The exception is a set of multi-person scenarios at five sites, where participants who consented to share conversation audio and unblurred faces were asked to take part in social activities such as games; these feed the audio-visual and social benchmarks (§3.2).

Dataset Size

• Video. 3,670 hours from 931 unique camera wearers, each of whom wore a camera for 1 to 10 hours at a time, recruited by 14 teams in 9 countries (§3.1). Among the 64% of participants with self-reported demographics, 96 were over 50 and 45% were female (§3.1). Seven head-mounted cameras were used, among them GoPro, Vuzix Blade and Pupil Labs (§3.3).
• Modalities. Table 1 of the paper lists the hours per modality: RGB video, text narrations and precomputed SlowFast features 3,670 each; audio 2,535; unblurred faces 612; IMU 836; Matterport3D scans of the environment 491; multi-camera (several wearers at the same event) 224; stereo 80; gaze 45.
• Annotated subsets. The first release annotates 48 to 1,000 hours per benchmark on top of the 3,670 narrated hours (§5). The supplement’s Table 3 gives, in hours: visual queries 2D 432.9, moments 328.7, natural language queries 227.1, hands and objects 196.2, forecasting 110.5, audio-visual diarization and social 47.7 each, visual queries 3D 13.
• Download size. As of November 2026 the Start Here page gives about 7.1 TB for the full primary dataset, about 7 TB for the full-scale videos, about 2 GB of annotations, about 1 TB of benchmark clips, about 220 GB of features and 30+ TB for the entire dataset. The same page also says that a full download of the videos and annotations is “the full 5 TB” and advises against downloading the “full (> 5TB) dataset”; the 5 TB and 7 TB figures are not reconciled there.

Three counts differ between the sources:

• Camera wearers. 931 in the paper’s abstract and §3.1; 923 on the home page (“923 unique participants”); “more than 900 unique individuals” on the 2026 challenge page.
• Partners. “14 teams from universities and labs” and “14 institutions” in the paper (§1, §3.1); “an international consortium of 13 universities in partnership with Facebook AI” on the home page.
• Episodic-memory hours. “∼74K total queries spanning 1,000 hours of video” in the paper (§5.1); the same sentence with 800 hours on the Episodic Memory documentation page.

Annotation Types

Narrations. Before any benchmark annotation, every video was narrated: annotators watched a 5-minute clip, summarized it in a few sentences, then re-watched it, pausing to write a timestamped sentence for each thing the camera wearer did. Each video received two independent narrations, on average 13.2 sentences per minute, 3.85 million in total, using 1,772 unique verbs and 4,336 unique nouns. The narrations were mined to build the action and object taxonomies, to route videos to benchmarks and to seed annotation windows (§4).

Benchmark annotations differ by family (§5): natural-language and visual queries paired with a “response track” in the video; temporal segments of 110 moment categories; pre-condition, point-of-no-return and post-condition frames of object state changes, with boxes for hands, tools and objects; face boxes, active-speaker labels, voice-activity segments and English transcripts; “looking at me” and “talking to me” segments; and, for forecasting, contact frames with boxes on the next active objects and ego-trajectories from structure from motion.

NLQ annotations. Annotators watched clips of 8 or 20 minutes and wrote queries guided by 13 templates in three groups, objects, places and people (for example “Where is object X before/after event Y?”), paraphrasing rather than copying them; the slot values were also recorded. Queries are in the past tense, posed as if asked at the end of the clip, which picks the last relevant occurrence when there are several, and interruptions shorter than 3 seconds caused by a glance away still count as one contiguous window (supplement, §F). Queries average 8.3 ± 2.1 words and response windows 9.3 ± 21.5 seconds; the supplement’s metric section puts the average window at about 10 s, about 2% of a 500-second clip.

Splits

Within a family the splits are kept consistent: the episodic-memory tasks share one split, as do forecasting and hands and objects, so that a training video in one task is not a validation video in another; across families the annotated videos differ and so do the splits. Test annotations are withheld and reachable only through a submission server, and narrations that overlap any test set are withheld too (supplement, Appendix E). NLQ uses disjoint clips in a 60/20/20 split; the supplement’s Table 7, read from the page image, gives:

v1 NLQ split	Train	Val	Test
Video hours	136	45	46
Clips	1.0k	0.3k	0.3k
Queries	11.3k	3.9k	4.0k

The text gives 19.2K queries from 227 hours, consistent with the table’s sum. The Updates page, describing v2, says that train and validation now have “27k validated queries vs 17.3k in v1”, which does not match the 15.2k that Table 7 gives for v1’s train and validation; neither source explains the difference. MQ uses the same 6:2:2 ratio: 194.9, 68.5 and 62.9 hours, 13.6k, 4.3k and 4.3k instances (supplement, Table 8).

Versions (Updates page, as of November 2026):

• v1 (2022), the release described in the paper.
• v1.0.5, which corrected incorrectly rotated visual-query 2D boxes for the 2022 visual-queries challenge.
• v2 (February 2023): forecasting and hands-and-objects annotations roughly doubled (243 against 120 hours of annotated clips); NLQ extended as above; NLQ windows of zero length (start equal to end) re-annotated; a few erroneous videos removed (one with frozen frames, one with varying resolution, several too short); the visual-query box fix included. The Updates page names PACO (parts and attributes of common objects) and EgoTracks (long-term tracking) as the two related dataset enhancements of v2, while the 2026 challenge page describes v2.0’s two enhancements as “Ego Schema and Goal Step”.
• v2.1, adding Goal-Step annotations and “grouped” videos: 210 videos that metadata problems had kept apart are combined into the videos they belonged to.

The episodic-memory repository’s README still carries a note from the v1 period that about 14% of NLQ annotations have a near-zero window and “will result in under reported performance”; the example below shows why.

Tasks

The paper’s five benchmark families, with the measures it names (§5; Appendices F–J of the supplement give details and baselines):

Family	Tasks	Measures
Episodic memory (past)	natural language queries (NLQ); visual queries (VQ), 2D and optionally 3D; moments queries (MQ)	NLQ: recall@k at tIoU $m$; VQ: temporal and spatio-temporal AP, success, recovery and search efficiency; MQ: mAP at tIoU 0.1–0.5 and recall@kx
Hands and objects (present)	point-of-no-return temporal localization; state-change object detection; object state-change classification	absolute temporal error (s); AP; accuracy
Audio-visual diarization (present)	localization and tracking of speakers; active speaker detection; diarization; transcription	MOT metrics; speaker error rate; DER; WER
Social interactions (present)	looking at me (LAM); talking to me (TTM)	mAP; top-1 accuracy
Forecasting (future)	locomotion; hand movement; short-term object interaction anticipation; long-term action anticipation	L2 distance; top-5 mAP; edit distance

Episodic memory treats the video as the wearer’s memory, in Tulving’s sense of personal experiences as distinct from facts (§5.1). An NLQ (“What did I put in the drawer?”) is answered by a temporal window in which the answer can be seen or deduced; a VQ is an image of an object, answered by the last time the object was seen, as a temporal window with a box in each frame and optionally a 3D displacement from the current camera position; an MQ names one of 110 activities (“When did I read to my children?”) and is answered by every window in which it happens. The paper relates MQ to activity detection for the camera wearer, and the supplement adapts temporal-detection mAP to it, averaged over tIoU 0.1, 0.2, 0.3, 0.4 and 0.5, as the temporal action localization article summarizes. The VQ temporal and spatio-temporal AP use thresholds 0.25, 0.5, 0.75 and 0.95 (supplement).

Hands and objects asks when, where and how an object changes state: the frame of the point of no return, the box of the changing object in three frames, and whether a change happened at all (§5.2). Audio-visual diarization and social interactions use the multi-person subset with consenting, unblurred participants; the camera wearer, who never appears in view, is one of the speakers to be labelled (§5.3, §5.4). Forecasting predicts the wearer’s future ground-plane trajectory and hand positions, the next active objects with a verb and a time to contact, and the sequence of future actions; the paper’s top-5 mAP for short-term anticipation “discounts the Top-4 false negative predictions” (§5.5).

Evaluation

How the NLQ evaluator scores a submission

The supplement defines NLQ’s measure as recall@k, IoU = m for $k \in \{1, 5\}$ and $m \in \{0.3, 0.5\}$, “the percentage of times at least one of the top k predicted candidates have an intersection-over-union (IoU) of at least m”, and explains the low thresholds by the short windows (supplement, §F). This is recall at k with a temporal IoU test in place of a relevance label. The official evaluator in the Episodic Memory repository, NLQ/VSLNet/utils/evaluate_ego4d_nlq.py at commit 5275b95 (MIT License), was read line by line and its two scoring functions were ported unchanged into the example below. As written, it computes the following.

• Strict threshold. A query counts as recalled at $(k, m)$ when (overlap > threshold)[:K].any(): the IoU must be strictly greater than $m$, while the supplement’s NLQ definition says “at least m”. The same supplement, when it adapts the measure to MQ, describes NLQ’s recall as counting a prediction “with a tIoU larger than the threshold m”, so the paper uses both wordings.
• IoU of a pair of windows. compute_IoU divides the intersection by the hull of the two windows (from the earlier start to the later end). For windows that overlap this equals the usual union; for disjoint windows the intersection is 0 either way. A zero-length ground-truth window has intersection 0 with every prediction, so its IoU is 0, and if the prediction is zero-length at the same point the division is $0/0$ and gives NaN.
• Denominator. Recall is the mean over the predictions submitted, np.array(results).mean(axis=-1). The script counts the ground-truth queries but uses the count only in a printed line, Evaluated: n / N instances; a query left out of the submission is neither counted as a miss nor reported as an error. It also does not check that each query appears once.
• “mIoU”. For each query the script appends np.mean(np.sort(overlap[0])[-3:]). Because the ground truth is passed as a list of one window, overlap has one row per predicted window, overlap[0] is the first-ranked window’s row of length 1, and the term is that window’s IoU. The reported “mIoU” is therefore the mean IoU of the top-1 window. In our reading, the sort and the [-3:] suggest that a mean of the three largest IoUs was intended; the code does not compute it.
• Labels and format. The printed table labels the thresholds “mIoU@0.3” and “mIoU@0.5” although they are IoU thresholds, and the script’s entry point requires the submission’s version to be “1.0” and challenge to be “ego4d_nlq_challenge”.

Moments queries: the opposite discrepancy

The MQ evaluator, MQ/Evaluation/ego4d/eval_detection.py, states that it is taken from the official ActivityNet code with small modifications, and its matching follows the procedure that the temporal action localization article describes for ActivityNet’s code. It was read line by line, not run: it needs pandas and joblib and a package context. A detection is marked a false positive only when tiou_arr[jdx] < tiou_thr, so a tIoU equal to the threshold matches, a non-strict rule, while the supplement’s MQ text says a prediction is a true positive “if the tIoU is higher than a threshold”. In the official code, NLQ is stricter than its text and MQ more lenient than its text.

Example: the NLQ protocol on synthetic clips

The script ports compute_IoU and evaluate_nlq_performance unchanged, with the MIT notice the licence requires, and runs them on in-memory dictionaries in the layout of the official nlq_<set>.json files (videos, clips, annotations, language queries), since no Ego4D file was downloaded. Only display_results, which needs the terminaltables package, is replaced by our own table. Each call runs with its printed line captured and with NumPy’s warning for $0/0$ silenced (np.errstate(invalid="ignore")), so that the NaN case prints a value instead of a warning. An independent implementation computes IoU with the usual union and recall over the submitted queries under both > and >=.

The data are synthetic and contain no people. There are 50 clips of 480 one-second frames of $12 \times 12$ pixels, with 4 queries each, 200 in all. The query is a colour, our analogue of a language query: each query’s answer window, of 3, 6, 9, 12 or 15 seconds, shows a $4 \times 4$ square of that colour, and with probability 0.5 the square disappears for an interior gap of 1–2 seconds, in the spirit of the annotation rule that glances away of under 3 seconds do not break a window. Each query also has two distractor windows showing a square whose colour is 0.12 away from the query’s, and all twelve windows of a clip are disjoint. Pixel noise has standard deviation 0.08. Three predictors are scored:

• P0, an oracle, which submits the ground-truth window.
• P1, a classical colour predictor, our analogue of a grounding model: each second is scored by the best pixel’s similarity to the query colour, every integer window of 2–16 seconds by the sum of (score − 0.5) over its seconds, and the five best windows that do not overlap are submitted in order; the top score serves as the predictor’s confidence.
• P2, a boundary error model: rank 1 is the first half of the ground-truth window (IoU exactly 0.5), rank 2 extends the window by $7L/3$ seconds (IoU $L / (10L/3) = 0.3$ exactly, since $L$ is a multiple of 3), and ranks 3–5 lie clear of both (IoU 0).

P1 is also scored when only its 100 most confident queries are submitted, and against a copy of the ground truth in which 28 of the 200 windows (14%, the README’s figure) are replaced by zero-length windows at their midpoints, our analogue of the v1 annotations that v2 corrected. Before running P1 on the noisy clips we fixed eight checks, each derived from the code or the construction, and five hypotheses about P1; the planned base check, P1 on clips without noise, gaps or distractors, must find every window exactly. The checks are C1 (the oracle scores 1 everywhere), C2 (the port agrees with the independent > recall), C3 (the gap between >= and > recall equals the share of queries whose best top-$k$ IoU equals $m$ exactly), C4 (P2’s derived values), C5 (the printed mIoU equals the mean top-1 IoU), C6 (dropping queries changes only the denominator), C7 (zero-length windows contribute nothing) and C8 (recall cannot fall from $k = 1$ to 5 or rise from $m = 0.3$ to 0.5). The script runs in about 1 second with NumPy 2.3.5 and gives identical output on repeated runs.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"
import io
import contextlib
import numpy as np

# compute_IoU and evaluate_nlq_performance are re-typed, unchanged, from
# EGO4D/episodic-memory at commit 5275b95, NLQ/VSLNet/utils/evaluate_ego4d_nlq.py:
#
# MIT License
#
# Copyright (c) 2021 EGO4D Dataset and Benchmark Suite
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.


def compute_IoU(pred, gt):
    """Compute the IoU given predicted and ground truth windows."""
    assert isinstance(pred, list) and isinstance(gt, list)
    pred_is_list = isinstance(pred[0], list)
    gt_is_list = isinstance(gt[0], list)
    if not pred_is_list:
        pred = [pred]
    if not gt_is_list:
        gt = [gt]
    pred, gt = np.array(pred), np.array(gt)
    inter_left = np.maximum(pred[:, 0, None], gt[None, :, 0])
    inter_right = np.minimum(pred[:, 1, None], gt[None, :, 1])
    inter = np.maximum(0.0, inter_right - inter_left)
    union_left = np.minimum(pred[:, 0, None], gt[None, :, 0])
    union_right = np.maximum(pred[:, 1, None], gt[None, :, 1])
    union = np.maximum(0.0, union_right - union_left)
    overlap = 1.0 * inter / union
    if not gt_is_list:
        overlap = overlap[:, 0]
    if not pred_is_list:
        overlap = overlap[0]
    return overlap


def evaluate_nlq_performance(
    predictions, ground_truth, thresholds, topK, per_instance=False
):
    """Evalutes the performances."""
    gt_dict = {}
    num_gt_queries = 0

    for video_datum in ground_truth["videos"]:
        for clip_datum in video_datum["clips"]:
            clip_uid = clip_datum["clip_uid"]
            for ann_datum in clip_datum["annotations"]:
                key = (clip_uid, ann_datum["annotation_uid"])
                gt_dict[key] = ann_datum
                num_gt_queries += len(ann_datum["language_queries"])

    results = [[[] for _ in topK] for _ in thresholds]
    average_IoU = []
    num_instances = 0
    for pred_datum in predictions:
        key = (pred_datum["clip_uid"], pred_datum["annotation_uid"])
        assert key in gt_dict, "Instance not present!"
        query_id = pred_datum["query_idx"]
        gt_datum = gt_dict[key]
        gt_query_datum = gt_datum["language_queries"][query_id]

        # Compute overlap and recalls.
        overlap = compute_IoU(
            pred_datum["predicted_times"],
            [[gt_query_datum["clip_start_sec"], gt_query_datum["clip_end_sec"]]],
        )
        average_IoU.append(np.mean(np.sort(overlap[0])[-3:]))
        for tt, threshold in enumerate(thresholds):
            for rr, KK in enumerate(topK):
                results[tt][rr].append((overlap > threshold)[:KK].any())
        num_instances += 1

    mean_results = np.array(results).mean(axis=-1)
    mIoU = np.mean(average_IoU)
    print(f"Evaluated: {num_instances} / {num_gt_queries} instances")
    if per_instance:
        per_instance_results = {
            "overlap": overlap,
            "average_IoU": average_IoU,
            "results": results,
        }
        return mean_results, mIoU, per_instance_results
    else:
        return mean_results, mIoU

# ---------------------------------------------------------------------------
# Synthetic episodic-memory clips (our own code from here on).
T, H, W, SQ = 480, 12, 12, 4            # 480 one-second frames of 12 x 12 pixels; 4 x 4 squares
N_CLIPS, N_Q = 50, 4                     # 50 clips x 4 queries = 200 queries
LENS = (3, 6, 9, 12, 15)                 # ground-truth and distractor window lengths (s)
PALETTE = np.array([[a, b, c] for a in (0.2, 0.8) for b in (0.2, 0.8) for c in (0.2, 0.8)])
NOISE, DELTA, SIGMA_K, BIAS = 0.08, 0.12, 0.15, 0.5
THRESHOLDS, TOPK = [0.3, 0.5], [1, 5]
SETTINGS = [(k, m) for k in TOPK for m in THRESHOLDS]


def place(rng, lengths, his, gap=2):
    """Disjoint integer windows [s, s + L] at least `gap` s apart (rejection sampling)."""
    while True:
        starts = [int(rng.integers(0, hi + 1)) for hi in his]
        iv = sorted(zip(starts, lengths))
        if all(iv[i][0] + iv[i][1] + gap <= iv[i + 1][0] for i in range(len(iv) - 1)):
            return starts


def make_clip(rng, clean=False):
    cols = rng.choice(len(PALETTE), N_Q, replace=False)
    gl = [int(x) for x in rng.choice(LENS, N_Q)]
    dl = [int(x) for x in rng.choice(LENS, 2 * N_Q)]
    # ground-truth windows leave 7L/3 s after their end for the boundary predictor
    his = [T - L - 7 * L // 3 - 1 for L in gl] + [T - L for L in dl]
    st = place(rng, gl + dl, his)
    gs, ds = st[:N_Q], st[N_Q:]
    noise = 0.0 if clean else NOISE
    v = np.clip(0.5 + noise * rng.standard_normal((T, H, W, 3)), 0, 1)
    shown = np.zeros((N_Q, T), bool)      # frames in which query j's true square is drawn
    gaps = []
    for j in range(N_Q):
        y, x = rng.integers(0, H - SQ + 1, 2)
        frames = list(range(gs[j], gs[j] + gl[j]))
        g = int(rng.integers(1, 3)) if (rng.random() < 0.5 and gl[j] - 2 >= 1) else 0
        g = min(g, gl[j] - 2)
        if g and not clean:
            g0 = int(rng.integers(gs[j] + 1, gs[j] + gl[j] - g))   # interior gap, not at the ends
            frames = [t for t in frames if not (g0 <= t < g0 + g)]
        else:
            g = 0
        gaps.append(g)
        for t in frames:
            v[t, y:y + SQ, x:x + SQ] = np.clip(PALETTE[cols[j]] + noise * rng.standard_normal((SQ, SQ, 3)), 0, 1)
        shown[j, frames] = True
    dcol = []
    for i in range(2 * N_Q):
        j = i // 2
        u = rng.standard_normal(3)
        c = PALETTE[cols[j]] + DELTA * u / np.linalg.norm(u)
        dcol.append(c)
        if clean:
            continue
        y, x = rng.integers(0, H - SQ + 1, 2)
        for t in range(ds[i], ds[i] + dl[i]):
            v[t, y:y + SQ, x:x + SQ] = np.clip(c + noise * rng.standard_normal((SQ, SQ, 3)), 0, 1)
    return dict(video=v, cols=cols, gs=gs, gl=gl, ds=ds, dl=dl, gaps=gaps, shown=shown, dcol=np.array(dcol))


def frame_scores(video, colour):
    """Per-second colour match: best pixel similarity to the query colour."""
    d2 = ((video - colour) ** 2).sum(-1).reshape(len(video), -1)
    return np.exp(-d2.min(1) / (2 * SIGMA_K ** 2))


def sliding_top5(s):
    """All integer windows of 2-16 s scored by sum(s - BIAS); greedy top 5 that do not overlap."""
    c = np.concatenate([[0.0], np.cumsum(s - BIAS)])
    a, ln, sc = [], [], []
    for L in range(2, 17):
        st = np.arange(0, len(s) - L + 1)
        a.append(st); ln.append(np.full(len(st), L)); sc.append(c[st + L] - c[st])
    a, ln, sc = np.concatenate(a), np.concatenate(ln), np.concatenate(sc)
    order = np.lexsort((ln, a, -sc))           # score descending, then earlier start, then shorter
    picked = []
    for i in order:
        s0, e0 = a[i], a[i] + ln[i]
        if all(min(e0, e1) - max(s0, s1) <= 0 for s1, e1, _ in picked):
            picked.append((int(s0), int(e0), float(sc[i])))
            if len(picked) == 5:
                break
    return [[float(p[0]), float(p[1])] for p in picked], picked[0][2]


def ground_truth_json(clips, zero=()):
    vids = []
    for c, cl in enumerate(clips):
        lq = []
        for j in range(N_Q):
            s, e = float(cl["gs"][j]), float(cl["gs"][j] + cl["gl"][j])
            if (c, j) in zero:
                s = e = s + cl["gl"][j] / 2
            lq.append({"clip_start_sec": s, "clip_end_sec": e, "query": "colour %s" % PALETTE[cl["cols"][j]]})
        vids.append({"video_uid": f"v{c:02d}", "clips": [{"clip_uid": f"c{c:02d}",
                     "annotations": [{"annotation_uid": f"a{c:02d}", "language_queries": lq}]}]})
    return {"videos": vids}


def pred(c, j, windows):
    return {"clip_uid": f"c{c:02d}", "annotation_uid": f"a{c:02d}", "query_idx": j, "predicted_times": windows}


def oracle(clips):
    return [pred(c, j, [[float(cl["gs"][j]), float(cl["gs"][j] + cl["gl"][j])]])
            for c, cl in enumerate(clips) for j in range(N_Q)]


def boundary(clips):
    """Error model: rank 1 = first half of the window (IoU 0.5), rank 2 = the window extended by
    7L/3 s (IoU 0.3), ranks 3-5 = windows of length L clear of both (IoU 0)."""
    out = []
    for c, cl in enumerate(clips):
        for j in range(N_Q):
            s, L = cl["gs"][j], cl["gl"][j]
            w = [[float(s), s + L / 2], [float(s), float(s + L + 7 * L // 3)]]
            far = [x for x in range(0, T - L + 1, L + 1) if x + L <= s or x >= s + L + 7 * L // 3]
            w += [[float(x), float(x + L)] for x in far[:3]]
            out.append(pred(c, j, w))
    return out


def colour_predictor(clips):
    out, conf = [], []
    for c, cl in enumerate(clips):
        for j in range(N_Q):
            w, top = sliding_top5(frame_scores(cl["video"], PALETTE[cl["cols"][j]]))
            out.append(pred(c, j, w)); conf.append(top)
    return out, np.array(conf)


def official(preds, gt):
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf), np.errstate(invalid="ignore"):
        res, miou = evaluate_nlq_performance(preds, gt, THRESHOLDS, TOPK)
    return res, float(miou), buf.getvalue().strip()


def independent(preds, gt):
    """Our own reading of the paper: IoU = inter / (|p| + |g| - inter), recall over submitted queries."""
    lq = {(cl["clip_uid"], a["annotation_uid"]): a["language_queries"]
          for v in gt["videos"] for cl in v["clips"] for a in cl["annotations"]}
    ious = []
    for p in preds:
        g = lq[(p["clip_uid"], p["annotation_uid"])][p["query_idx"]]
        gs, ge = g["clip_start_sec"], g["clip_end_sec"]
        row = []
        for ps, pe in p["predicted_times"]:
            inter = max(0.0, min(pe, ge) - max(ps, gs))
            uni = (pe - ps) + (ge - gs) - inter
            row.append(inter / uni if uni > 0 else float("nan"))
        ious.append(row + [0.0] * (5 - len(row)))
    return np.array(ious)


def recall(ious, k, m, strict):
    top = ious[:, :k]
    hit = (top > m) if strict else (top >= m)
    return hit.any(1).mean(), int(hit.any(1).sum())


def report(name, preds, gt, n_total):
    res, miou, line = official(preds, gt)
    iou = independent(preds, gt)
    row = {"name": name, "line": line, "miou": miou}
    for (k, m) in SETTINGS:
        tt, rr = THRESHOLDS.index(m), TOPK.index(k)
        row[("code", k, m)] = float(res[tt][rr])
        row[("strict", k, m)], row[("hits", k, m)] = recall(iou, k, m, True)
        row[("atleast", k, m)], _ = recall(iou, k, m, False)
        best = np.nanmax(np.where(np.isnan(iou[:, :k]), -1, iou[:, :k]), 1)
        row[("ties", k, m)] = int((best == m).sum())
    with np.errstate(invalid="ignore"):
        row["top1"] = float(np.mean(iou[:, 0]))
        row["top3"] = float(np.mean(np.sort(np.nan_to_num(iou, nan=0.0), 1)[:, -3:].mean(1)))
    row["n"], row["N"] = len(preds), n_total
    return row


def show(rows):
    hdr = "condition            evaluated   " + "  ".join(f"R@{k},m={m}" for k, m in SETTINGS)
    print(hdr)
    for r in rows:
        print(f"{r['name']:<20} {r['line'].split(': ')[1].replace(' instances', ''):>9}   code (>):   "
              + "  ".join(f"{r[('code', k, m)]:9.4f}" for k, m in SETTINGS))
        print(f"{'':<34}paper (>=): " + "  ".join(f"{r[('atleast', k, m)]:9.4f}" for k, m in SETTINGS))
        print(f"{'':<34}ties = m:   " + "  ".join(f"{r[('ties', k, m)]:9d}" for k, m in SETTINGS))
        print(f"{'':<34}mIoU (code) {r['miou']:.6f}   mean top-1 IoU {r['top1']:.6f}   "
              f"mean of 3 largest IoUs {r['top3']:.6f}")


def data_checks(clips):
    ok_cols = all(len(set(cl["cols"].tolist())) == N_Q for cl in clips)
    lens = np.array([L for cl in clips for L in cl["gl"]])
    gapd = True
    for cl in clips:
        iv = sorted(zip(cl["gs"] + cl["ds"], cl["gl"] + cl["dl"]))
        gapd &= all(iv[i][0] + iv[i][1] + 2 <= iv[i + 1][0] for i in range(len(iv) - 1))
    gaps = np.array([g for cl in clips for g in cl["gaps"]])
    shown = sum(int(cl["shown"].sum()) for cl in clips)
    dev = max(abs(np.linalg.norm(cl["dcol"][i] - PALETTE[cl["cols"][i // 2]]) - DELTA)
              for cl in clips for i in range(2 * N_Q))
    bg = 0.0
    for cl in clips:
        for j in range(N_Q):
            s = frame_scores(cl["video"], PALETTE[cl["cols"][j]])
            busy = cl["shown"][j].copy()
            for i in (2 * j, 2 * j + 1):
                busy[cl["ds"][i]:cl["ds"][i] + cl["dl"][i]] = True
            bg = max(bg, s[~busy].max())
    print(f"queries {len(lens)}; distinct colours in every clip: {ok_cols}; window lengths "
          + ", ".join(f"{L} s: {(lens == L).sum()}" for L in LENS))
    print(f"all windows disjoint and >= 2 s apart: {gapd}; gaps of 1 s: {(gaps == 1).sum()}, of 2 s: {(gaps == 2).sum()}; "
          f"frames showing the true square: {shown} of {lens.sum()}")
    print(f"distractor colour distance - {DELTA}: max |dev| {dev:.1e}; "
          f"largest query score outside its own true and distractor windows: {bg:.4f}")


def hand_cases():
    print("hand cases (official compute_IoU):")
    a = float(compute_IoU([0.0, 5.0], [0.0, 10.0])); b = float(compute_IoU([0.0, 3.0], [0.0, 10.0]))
    print(f"  [0,5] vs [0,10]: IoU {a!r}; > 0.5: {a > 0.5}; >= 0.5: {a >= 0.5}")
    print(f"  [0,3] vs [0,10]: IoU {b!r}; equals 0.3: {b == 0.3}; > 0.3: {b > 0.3}")
    print(f"  [95,105] vs zero-length [100,100]: IoU {float(compute_IoU([95.0, 105.0], [100.0, 100.0]))!r}")
    with np.errstate(invalid="ignore"):
        z = float(compute_IoU([100.0, 100.0], [100.0, 100.0]))
    gt = {"videos": [{"video_uid": "v", "clips": [{"clip_uid": "c", "annotations": [{"annotation_uid": "a",
          "language_queries": [{"clip_start_sec": 0.0, "clip_end_sec": 10.0},
                               {"clip_start_sec": 100.0, "clip_end_sec": 100.0}]}]}]}]}
    pr = [{"clip_uid": "c", "annotation_uid": "a", "query_idx": 0, "predicted_times": [[0.0, 10.0]]},
          {"clip_uid": "c", "annotation_uid": "a", "query_idx": 1, "predicted_times": [[100.0, 100.0]]}]
    res, miou, line = official(pr, gt)
    print(f"  zero-length prediction on zero-length window: IoU {z!r}; two-query submission: "
          f"R@1,m=0.3 {res[0][0]:.4f}, mIoU {miou!r}")


def main():
    rng = np.random.default_rng(0)
    clips = [make_clip(rng) for _ in range(N_CLIPS)]
    N = N_CLIPS * N_Q
    print("== data")
    data_checks(clips)
    gt = ground_truth_json(clips)
    p2 = boundary(clips)
    i2 = independent(p2, gt)
    print(f"boundary predictor: rank-1 IoU all 0.5: {bool((i2[:, 0] == 0.5).all())}; rank-2 all 0.3: "
          f"{bool((i2[:, 1] == 0.3).all())}; ranks 3-5 all 0: {bool((i2[:, 2:] == 0).all())}")
    hand_cases()

    print("== base check: colour predictor on clean clips (no noise, gaps or distractors)")
    rngc = np.random.default_rng(0)
    clean = [make_clip(rngc, clean=True) for _ in range(N_CLIPS)]
    pc, _ = colour_predictor(clean)
    ic = independent(pc, ground_truth_json(clean))
    print(f"top-1 IoU = 1 for {(ic[:, 0] == 1).sum()} of {N} queries")

    print("== conditions")
    p0 = oracle(clips)
    p1, conf = colour_predictor(clips)
    keep = np.sort(np.argsort(-conf, kind="stable")[:N // 2])
    p1d = [p1[i] for i in keep]
    zero_idx = np.random.default_rng(1).choice(N, 28, replace=False)
    zero = {(int(i) // N_Q, int(i) % N_Q) for i in zero_idx}
    gt_z = ground_truth_json(clips, zero)
    rows = [report("P0 oracle", p0, gt, N), report("P1 colour", p1, gt, N),
            report("P1 top half", p1d, gt, N), report("P1, 28 zero-len", p1, gt_z, N),
            report("P2 boundary", p2, gt, N)]
    show(rows)
    R = {r["name"]: r for r in rows}

    i1 = independent(p1, gt)
    dis = 0
    for c, cl in enumerate(clips):
        for j in range(N_Q):
            a, b = p1[c * N_Q + j]["predicted_times"][0]
            dis += any(min(b, d + L) - max(a, d) > 0 for d, L in
                       ((cl["ds"][i], cl["dl"][i]) for i in (2 * j, 2 * j + 1)))
    print(f"P1: top-1 window overlaps one of the query's two near-colour distractors in {dis} of {N} queries")

    print("== checks")
    ok = lambda b: "held" if b else "FAILED"
    o = R["P0 oracle"]
    print("C1 oracle: every code recall = 1 and mIoU = 1:",
          ok(all(o[("code", k, m)] == 1.0 for k, m in SETTINGS) and o["miou"] == 1.0))
    d2 = max(abs(r[("code", k, m)] - r[("strict", k, m)]) for r in rows for k, m in SETTINGS)
    print(f"C2 port = independent '>' recall, all conditions: max |diff| {d2:.1e}", ok(d2 == 0))
    d3 = max(abs((r[("atleast", k, m)] - r[("code", k, m)]) * r["n"] - r[("ties", k, m)]) for r in rows for k, m in SETTINGS)
    print(f"C3 ('>=' - '>') x n = number of ties at m, all conditions: max |diff| {d3:.1e}", ok(d3 < 1e-9))
    b = R["P2 boundary"]
    exp = {(1, 0.3): 1.0, (1, 0.5): 0.0, (5, 0.3): 1.0, (5, 0.5): 0.0}
    c4 = (all(b[("code", k, m)] == exp[(k, m)] for k, m in SETTINGS)
          and all(b[("atleast", k, m)] == 1.0 for k, m in SETTINGS) and b["miou"] == 0.5
          and abs(b["top3"] - 0.8 / 3) < 1e-12)
    print(f"C4 boundary: code 1, 0, 1, 0; paper 1, 1, 1, 1; mIoU 0.5; 3-largest mean - 0.8/3 = {b['top3'] - 0.8 / 3:.1e}:", ok(c4))
    d5 = max(0.0 if (np.isnan(r["miou"]) and np.isnan(r["top1"])) else abs(r["miou"] - r["top1"]) for r in rows)
    print(f"C5 code mIoU = mean top-1 IoU, all conditions: max |diff| {d5:.1e}", ok(d5 == 0))
    dd = R["P1 top half"]
    c6 = dd["line"].endswith("100 / 200 instances") and all(
        abs(dd[("code", k, m)] * dd["n"] / N - dd[("hits", k, m)] / N) < 1e-12 for k, m in SETTINGS)
    print("C6 top half: '" + dd["line"] + "'; recall over all 200 = "
          + ", ".join(f"{dd[('hits', k, m)] / N:.4f}" for k, m in SETTINGS), ok(c6))
    zr = R["P1, 28 zero-len"]
    nz = np.array([i not in set(zero_idx.tolist()) for i in range(N)])
    iz = independent(p1, gt_z)
    zero_ok = bool((np.nan_to_num(iz[~nz], nan=0.0) == 0).all())
    sides = [(zr[("code", k, m)] * N, int(((i1[nz, :k] > m).any(1)).sum())) for k, m in SETTINGS]
    c7 = zero_ok and all(round(a) == b for a, b in sides)
    print(f"C7 zero-length windows: their IoUs all 0: {zero_ok}; code recall x 200 vs hits on the other 172: "
          + ", ".join(f"{a!r} vs {b}" for a, b in sides) + ";", ok(c7))
    c8 = all(r[("code", 5, m)] >= r[("code", 1, m)] and r[("code", k, 0.5)] <= r[("code", k, 0.3)]
             for r in rows for k, m in SETTINGS)
    print("C8 R@5 >= R@1 and R(m=0.5) <= R(m=0.3), all conditions:", ok(c8))

    print("== hypotheses")
    p = R["P1 colour"]
    print("H1 P1: '>' and '>=' differ in at least one setting:", ok(any(p[("ties", k, m)] > 0 for k, m in SETTINGS)))
    print("H2 P1 top half: reported recall above full submission in all four settings:",
          ok(all(dd[("code", k, m)] > p[("code", k, m)] for k, m in SETTINGS)))
    print(f"H3 P1: |code mIoU - mean of 3 largest IoUs| = {abs(p['miou'] - p['top3']):.4f} > 0.01:",
          ok(abs(p["miou"] - p["top3"]) > 0.01))
    print(f"H4 P1: 0.2 < R@1,m=0.5 = {p[('code', 1, 0.5)]:.4f} < 0.95 and R@5,m=0.3 > R@1,m=0.3:",
          ok(0.2 < p[("code", 1, 0.5)] < 0.95 and p[("code", 5, 0.3)] > p[("code", 1, 0.3)]))
    print("H5 P1: 28 zero-length windows lower recall in all four settings:",
          ok(all(zr[("code", k, m)] < p[("code", k, m)] for k, m in SETTINGS)))


if __name__ == "__main__":
    main()

== data
queries 200; distinct colours in every clip: True; window lengths 3 s: 37, 6 s: 43, 9 s: 45, 12 s: 41, 15 s: 34
all windows disjoint and >= 2 s apart: True; gaps of 1 s: 61, of 2 s: 36; frames showing the true square: 1643 of 1776
distractor colour distance - 0.12: max |dev| 6.9e-17; largest query score outside its own true and distractor windows: 0.7895
boundary predictor: rank-1 IoU all 0.5: True; rank-2 all 0.3: True; ranks 3-5 all 0: True
hand cases (official compute_IoU):
  [0,5] vs [0,10]: IoU 0.5; > 0.5: False; >= 0.5: True
  [0,3] vs [0,10]: IoU 0.3; equals 0.3: True; > 0.3: False
  [95,105] vs zero-length [100,100]: IoU 0.0
  zero-length prediction on zero-length window: IoU nan; two-query submission: R@1,m=0.3 0.5000, mIoU nan
== base check: colour predictor on clean clips (no noise, gaps or distractors)
top-1 IoU = 1 for 200 of 200 queries
== conditions
condition            evaluated   R@1,m=0.3  R@1,m=0.5  R@5,m=0.3  R@5,m=0.5
P0 oracle            200 / 200   code (>):      1.0000     1.0000     1.0000     1.0000
                                  paper (>=):    1.0000     1.0000     1.0000     1.0000
                                  ties = m:           0          0          0          0
                                  mIoU (code) 1.000000   mean top-1 IoU 1.000000   mean of 3 largest IoUs 0.333333
P1 colour            200 / 200   code (>):      0.3300     0.3300     1.0000     0.9850
                                  paper (>=):    0.3300     0.3300     1.0000     1.0000
                                  ties = m:           0          0          0          3
                                  mIoU (code) 0.329000   mean top-1 IoU 0.329000   mean of 3 largest IoUs 0.329516
P1 top half          100 / 200   code (>):      0.3200     0.3200     1.0000     1.0000
                                  paper (>=):    0.3200     0.3200     1.0000     1.0000
                                  ties = m:           0          0          0          0
                                  mIoU (code) 0.320000   mean top-1 IoU 0.320000   mean of 3 largest IoUs 0.330847
P1, 28 zero-len      200 / 200   code (>):      0.2800     0.2800     0.8600     0.8450
                                  paper (>=):    0.2800     0.2800     0.8600     0.8600
                                  ties = m:           0          0          0          3
                                  mIoU (code) 0.279000   mean top-1 IoU 0.279000   mean of 3 largest IoUs 0.282849
P2 boundary          200 / 200   code (>):      1.0000     0.0000     1.0000     0.0000
                                  paper (>=):    1.0000     1.0000     1.0000     1.0000
                                  ties = m:           0        200          0        200
                                  mIoU (code) 0.500000   mean top-1 IoU 0.500000   mean of 3 largest IoUs 0.266667
P1: top-1 window overlaps one of the query's two near-colour distractors in 134 of 200 queries
== checks
C1 oracle: every code recall = 1 and mIoU = 1: held
C2 port = independent '>' recall, all conditions: max |diff| 0.0e+00 held
C3 ('>=' - '>') x n = number of ties at m, all conditions: max |diff| 2.7e-15 held
C4 boundary: code 1, 0, 1, 0; paper 1, 1, 1, 1; mIoU 0.5; 3-largest mean - 0.8/3 = 0.0e+00: held
C5 code mIoU = mean top-1 IoU, all conditions: max |diff| 0.0e+00 held
C6 top half: 'Evaluated: 100 / 200 instances'; recall over all 200 = 0.1600, 0.1600, 0.5000, 0.5000 held
C7 zero-length windows: their IoUs all 0: True; code recall x 200 vs hits on the other 172: 56.00000000000001 vs 56, 56.00000000000001 vs 56, 172.0 vs 172, 169.0 vs 169; held
C8 R@5 >= R@1 and R(m=0.5) <= R(m=0.3), all conditions: held
== hypotheses
H1 P1: '>' and '>=' differ in at least one setting: held
H2 P1 top half: reported recall above full submission in all four settings: FAILED
H3 P1: |code mIoU - mean of 3 largest IoUs| = 0.0005 > 0.01: FAILED
H4 P1: 0.2 < R@1,m=0.5 = 0.3300 < 0.95 and R@5,m=0.3 > R@1,m=0.3: held
H5 P1: 28 zero-length windows lower recall in all four settings: held

The data checks show the generator as described: distinct colours in every clip, the five window lengths, disjoint windows at least 2 s apart, 97 gaps, distractors exactly 0.12 away, and P2’s three rank IoUs exactly as constructed. The hand cases show the evaluator’s edge cases directly: $[0, 5]$ against $[0, 10]$ has IoU 0.5 and is not counted at $m = 0.5$; $[0, 3]$ against $[0, 10]$ gives exactly the floating-point value 0.3 and is not counted at $m = 0.3$; a zero-length ground-truth window gives IoU 0, and a zero-length prediction on it gives NaN, which turns the whole submission’s mIoU into NaN while its recall is still computed. The base check held: on clean clips P1’s top window equals the ground truth for 200 of 200 queries.

All eight checks held as printed, C7 after a repair described below. C2 shows that the ported function computes recall exactly as the independent implementation with > does, and C3 that the whole difference between the code and the paper’s “at least m” comes from exact ties: for P2, built so that every top-1 IoU is exactly 0.5, the code gives R@1 = 0 at $m = 0.5$ and the paper’s wording gives 1, and its mIoU is 0.5, the top-1 IoU, against 0.266667 for the mean of the three largest IoUs (C4). C6 shows the denominator rule: submitting P1’s 100 most confident queries prints Evaluated: 100 / 200 instances and reports recall over those 100; over all 200 queries, with the missing ones counted as misses, R@1 would be 0.1600 instead of 0.3200.

C7 was repaired after the pilot run and before the run shown: as first written it compared the code’s recall multiplied by 200 with an integer count exactly, and the floating-point product 56.00000000000001 is not 56, so the check printed FAILED although the counts agreed. It now compares the rounded product and prints both sides, and its printed outcome changed from failed to held; nothing else in the script changed.

The hypotheses about P1, written before P1 had been run on the noisy clips:

• H1 held. The strict and non-strict rules differ for P1 in one setting: three queries have a best top-5 IoU of exactly 0.5, so R@5 at $m = 0.5$ is 0.9850 under the code and 1.0000 under the paper’s wording.
• H2 failed. Submitting only the most confident half did not raise the reported recall in all four settings: it was lower at R@1 (0.3200 against 0.3300 at both thresholds), equal at R@5, $m = 0.3$ (1.0000 against 1.0000) and higher at R@5, $m = 0.5$ (1.0000 against 0.9850). In our reading the reason is P1’s confidence itself: it is the score of the top-ranked window, and a window ranks first only by outscoring every other window, including the one on the true answer, so a top window on a distractor, which the printed line counts in 134 of 200 queries, carries at least the confidence the true window would have had. The denominator rule still holds, as C6 shows; selecting the most confident queries raises recall only if confidence tracks correctness.
• H3 failed. The code’s mIoU (0.329000) and the mean of the three largest IoUs (0.329516) differ by 0.0005, not by more than 0.01. The two quantities are different, as P2 shows (0.500000 against 0.266667); for P1 they happen to nearly coincide, and we give no reason beyond the printed values.
• H4 held. P1 is imperfect but useful: R@1 at $m = 0.5$ is 0.3300, between 0.2 and 0.95, and R@5 at $m = 0.3$ is 1.0000, above R@1’s 0.3300.
• H5 held. Replacing 28 windows by zero-length ones lowers every recall: 0.2800 against 0.3300 at R@1 for both thresholds, 0.8600 against 1.0000 and 0.8450 against 0.9850 at R@5. All 28 queries lost their hit at R@5, $m = 0.3$ (172/200 = 0.8600): a zero-length window cannot be recalled by any prediction, which is the under-reporting the README warned of for v1.

These outcomes test our analogues, a colour query and a colour predictor, not language grounding; what transfers to real submissions is the arithmetic of the evaluator. On real NLQ results the strict rule matters only for IoUs that hit 0.3 or 0.5 exactly, the denominator rule matters whenever a submission is incomplete, and the “mIoU” printed by this script is the mean top-1 IoU.

License

As of November 2026, Ego4D is distributed under licence agreements with the institutions that collected the video, not under a public licence. The Start Here page says that obtaining the dataset or any annotations requires reviewing and accepting the licence agreement on Lab22 (lab22.ai), that approved users receive AWS credentials by email after about 48 hours, and that the credentials expire after 14 days, since the data are meant to be downloaded rather than used in place; a licence can be executed by an individual or, by a signatory with authority, for an institution. A banner on each documentation page read for this article says that Ego4D and Ego-Exo4D access has moved to Lab22. The paper itself says only that the dataset is “publicly available under an Ego4D license” (§3). The paper thanks the Common Visual Data Foundation for hosting the dataset (acknowledgements).

The site links a public draft of the agreements, read for this article and not accepted; the agreements actually executed on Lab22 were not read. The draft is 36 pages long and contains twelve agreements, one for each licensor: the University of Bristol, Carnegie Mellon University, Facebook, Inc. and its affiliates, Georgia Tech Research Corporation, IIIT Hyderabad, Associazione IPLAB in Catania, Indiana University, KAUST, Universidad de los Andes, the National University of Singapore, the University of Minnesota and the University of Tokyo. Each grants “a non-exclusive, non-transferable license to use the Database” for a defined purpose, without the right to sublicense. In the Bristol agreement the purpose covers showing images or videos in research publications, using the data to research, develop, train or evaluate systems that recognize or understand objects, events, places or activities, model 3D objects or environments, or improve virtual or augmented reality applications, “for academic research, commercial or noncommercial product development”, and creating and distributing annotations under the agreement’s terms. The licensee keeps its rights in models and annotations derived from the data; the data may not otherwise appear in any dataset or product, or be provided to third parties; publications must cite the database in a prescribed form; the agreement ends automatically on any breach, after which all copies must be destroyed. Governing law differs between the agreements, for example the English courts for Bristol, Italy for Catania and Japan for Tokyo.

The evaluation code in EGO4D/episodic-memory and the facebookresearch/Ego4d tools are under the MIT License; the code’s licence does not extend to the data.

Limitations and Bias

• Coverage. The paper names its own biases (§3.5): 74 locations are far from global coverage; wearers live mostly in urban or college-town areas; the COVID-19 pandemic led to much stay-at-home footage (cooking, cleaning, crafts) and little from large public events; battery life means the recordings favour the more active parts of a day; and annotation by crowd workers at two sites in Africa may bias the narrations’ word choice.
• Privacy. Each partner set its own privacy policy, generally requiring informed consent of the wearers, who could review and redact their video, and blurring of faces and other identifying information when recording in public spaces (§3.4); some universities used commercial de-identification software (acknowledgements). The home page asks anyone who finds an insufficiently redacted clip to report it.
• Subsets per task. Each benchmark is annotated on a different, smaller subset (48–1,000 hours), chosen per task, so results on one benchmark describe its subset rather than all 3,670 hours (supplement, Appendix E).
• Annotation errors in v1. The near-zero NLQ windows, the rotated VQ boxes and the removed videos were corrected in v1.0.5 and v2; in our reading, results computed on v1 annotations are not directly comparable with v2 results, and, as the example shows, zero-length windows depress NLQ recall whatever the method.
• Low absolute scores. The supplement’s NLQ baselines recall the right window in the top five at IoU 0.3 for only 11–14% of test queries (below), which in our reading reflects the “needle in the haystack” setting the supplement describes: windows of about 2% of the clip.

Common Uses

• Benchmarking episodic-memory systems on NLQ, VQ and MQ, including through the yearly challenges.
• Hand–object interaction, state-change and anticipation research on the forecasting and hands-and-objects annotations.
• Building derived datasets on Ego4D video: EgoSchema, whose questions an LLM generated from Ego4D narrations before human curation (as the video-language model article tabulates), PACO and EgoTracks (Updates page), and Goal-Step (v2.1).
• Natural-language moment retrieval in long video, related to the query-based moment retrieval the video summarization article describes for QVHighlights.

Historical Milestones

• 2022. Release of v1 with the CVPR 2022 paper; the first challenges’ results were announced at the Joint 1st Ego4D Workshop, held with the 10th EPIC Workshop at CVPR 2022 (19 June 2022, home page and challenge page), followed by an ECCV 2022 workshop (24 October 2022). The supplement’s NLQ baselines (Table 11, read from the page image; recall in %, test split, r@1 and r@5 at IoU 0.3, then at IoU 0.5): 2D-TAN 5.80, 13.90, 2.34, 5.96; VSLNet 5.47, 11.21, 2.80, 6.57. On validation, 2D-TAN reached 5.04, 12.89, 2.02, 5.88 and VSLNet 5.45, 10.74, 3.12, 6.63. With randomized visual features VSLNet’s test r@1 at IoU 0.3 fell to 1.80, and with randomized text features to 3.05. The supplement does not name the code that computed them; the evaluator read for this article, at a 2023 commit, counts only IoUs strictly above the threshold. The MQ baseline, VSGN without its VSS component, reached an average test mAP of 5.68 over tIoU 0.1–0.5 (Table 12).
• February 2023. v2, with doubled forecasting and hands-and-objects annotations, 27k NLQ train and validation queries, the zero-length windows corrected, and the PACO and EgoTracks enhancements; CVPR 2023 workshop on 19 June 2023.
• 2024–2025. Challenges at the CVPR 2024 (17 June 2024) and CVPR 2025 workshops. The v2.1 release with Goal-Step is undated on the Updates page.
• 2026. At the Third Joint Egocentric Vision Workshop at CVPR 2026 the organizers ran three Ego4D challenges (NLQ, Goal Step and short-term object interaction anticipation) and two on Ego-Exo4D, from 15 March to 13 May 2026, still on Ego4D v2.0 with the same test set as earlier editions, and moved them from EvalAI to CodaBench, running a reduced set during the transition (challenge page, as of November 2026).

Related Datasets

• EPIC-KITCHENS records unscripted egocentric video in participants’ kitchens, labelled from narrations as verb–noun actions; its University of Bristol group is also an Ego4D partner.
• Ego-Exo4D centres on simultaneously captured egocentric and exocentric video of skilled activities such as sports, music, dance and bike repair; it shares the Lab22 access route and the yearly challenges with Ego4D (challenge page).
• EgoSchema, PACO, EgoTracks and Goal-Step are built on Ego4D video (above).
• ActivityNet, Kinetics and other third-person video datasets are the contrast the paper draws (§2); the video understanding article tabulates them with Ego4D.

Resources

• Home page and documentation: ego4d-data.org, with Start Here for access, Updates for versions and the challenge page for the current challenges and links to past workshops and winners.
• Paper and supplement: CVF open access.
• Code: EGO4D/episodic-memory (NLQ, VQ and MQ baselines and evaluators) and facebookresearch/Ego4d (download tool and loaders), both MIT.
• Current results: the CodaBench leaderboards linked from the challenge page.