Market-1501

Market-1501 is a person re-identification benchmark that Liang Zheng, Liyue Shen, Lu Tian and colleagues at Tsinghua University, Microsoft Research and the University of Texas at San Antonio published at ICCV 2015. Six cameras placed in front of a campus supermarket recorded pedestrians. A Deformable Part Model detector cropped 32,668 boxes of 1,501 people, each seen by at least two cameras, and the authors added a set of more than 500,000 further boxes to test search at scale (Zheng et al., 2015, §3). The paper argued that a benchmark with several correct matches per query needs mean average precision (mAP) as well as the cumulative matching characteristic (CMC), and proposed mAP for evaluation on it (§3.3).

The Person Re-identification article already summarises Market-1501’s counts, its junk rule, CMC and mAP, and the methods evaluated on it. This article covers the dataset as an object: how it was built, the codes its released files use, how the evaluation is computed in code, what can and cannot be said about its licence, and the withdrawal of its best-known companion benchmark, DukeMTMC-reID. The official page and the MATLAB evaluation code are on a site this project does not use, so they were not read. The evaluation is described through the paper and two MIT-licensed third-party implementations read at pinned commits: layumi’s evaluate_gpu.py (Person_reID_baseline_pytorch, commit 41f64e5) and torchreid’s rank.py (deep-person-reid, commit f8cd150). Both are labelled third-party throughout.

Purpose

The paper names three shortcomings of the re-identification datasets of its time (Abstract, §1). They were small, with identities “often confined in several hundred”. In most, the boxes were hand-drawn, whereas a deployed system crops people with a pedestrian detector, which misaligns boxes, cuts off body parts and fires on background. And a person was usually captured by two cameras with one image in each, so each query had a single correct match. Market-1501 answers each point. It has six cameras and up to six queries per identity. Its gallery consists of detector output, including detector false alarms labelled as distractors. Each identity may have several images in each camera, giving “multi-query, multi-ground truth per identity” (§1). The paper calls this collection setting an open system.

The paper’s second, “minor” contribution is an unsupervised bag-of-words descriptor that treats re-identification as image search (Abstract, §4). Its results are the dataset’s first baselines and are summarised under Historical Milestones.

Dataset Size

• Boxes: 32,668 annotated DPM boxes of 1,501 identities (§3.1, Table 1).
• Cameras: six, five at 1280×1080 HD and one at 720×576 SD, with overlapping views (§3.1). Each annotated identity was captured by at least two and at most six cameras.
• Queries: 3,368 hand-drawn query boxes, one per identity per camera, so at most six per identity (§3.3, Figure 4).
• Ground truths per query: on average 14.8 cross-camera matches (§3.3).
• Distractors: Table 1 lists “2,793 + 500K”. The 500K set has “over 500,000 bboxes” of background false alarms and of pedestrians outside the 1,501 identities (§3.2). Used together with the dataset, it is called Market-1501+500K.
• Image size: the paper’s figures show images normalized to 128×64 pixels for display (Figure 1). Its descriptor resizes them to the same size (§4.1).

As a third-party count, torchreid’s market1501.py docstring gives 12,936 training images, 3,368 query images and 15,913 gallery images. The Person Re-identification article gives 19,732 gallery images from later papers. In our reading, the difference of 3,819 is the number of junk boxes, because torchreid’s loader drops exactly those boxes (below); the official page, which would settle this, was not read.

Classes

Market-1501 has no object classes. Its labels are 1,501 identities, and it uses two extra codes for images that do not belong to any identity. Identities are anonymous numbers, and the paper describes no attribute or demographic labels.

Annotation Types

Good, distractor and junk. For every detected box to be annotated, the authors also drew a box by hand and computed the ratio of the overlap to the union of the two (§3.1). A detected box was marked “good” if the ratio was larger than 50%, “distractor” if it was smaller than 20%, and “junk” otherwise; some obvious false alarms were also marked as distractors. The paper says a junk image is “of zero influence to re-id accuracy”. The paper calls the 50% threshold “a routine in object detection”, citing Felzenszwalb et al.’s part-based detector, and cites Philbin et al. (2007), the Oxford Buildings paper, for the term “junk”.

Codes in the released files. The release can be read here only through third-party code, because the official page was not read:

• torchreid at f8cd150 names the release folder Market-1501-v15.09.15 and reads the folders bounding_box_train, query and bounding_box_test, plus images for the 500K distractors.
• layumi’s prepare.py at 41f64e5 also reads gt_bbox, which it copies to a multi-query folder, with the comment “for dukemtmc-reid, we do not need multi-query”.
• torchreid parses each file name with the pattern ([-\d]+)_c(\d): the name starts with the identity code, then _c and the camera number, 1 to 6.
• Identity code −1 marks junk boxes. torchreid’s loader skips them (“junk images are just ignored”).
• Identity code 0 is kept. torchreid’s loader asserts 0 <= pid <= 1501 and comments “pid == 0 means background”, so in our reading code 0 marks the distractors. The class attribute _junk_pids = [0, -1] lists both codes, but the loader drops only −1; the base class in dataset.py uses the attribute only in combine_all, which skips both codes when the query and gallery images are merged into the training set.

Queries. Query boxes are hand-drawn, not detected, because the paper argues that a user can easily draw a box interactively, and hand-drawn boxes give higher accuracy (§3.3).

Splits

The paper’s text says the dataset “is randomly divided into training and testing sets, containing 750 and 751 identities, respectively” (§3.3). The Person Re-identification article reports that later papers using the released split give the reverse, 751 training identities with 12,936 images and 750 test identities. The 12,936 training images agree with torchreid’s docstring. The two sources are reported here as they stand; the release itself was not read. The paper says it trained its codebook on the training set (§4.1). Its conclusion planned to treat the test data as a validation set and to annotate new test identities for a future challenge, a plan for which no readable source records a result.

Tasks

Market-1501 is used for image-based person re-identification: ranking the gallery for each query so that images of the same person from other cameras come first. The paper also notes that the hand-drawn boxes and identities make it usable for pedestrian detection (§3.2). With the 500K set it tests how retrieval degrades as the gallery grows, the effect Recall@K describes for retrieval in general. Embedding methods trained on it with the triplet loss report their results on it.

Evaluation

The protocol in the paper

Search is cross-camera: “relevant images captured in the same camera as the query are viewed as ‘junk’” (§3.3). For each query the paper computes “the area under the Precision-Recall curve”, its average precision, and reports the mean over queries. It also reports CMC. Figure 3 motivates the choice with three five-item ranked lists, whose CMC is 1 in every case. The lists have a single match at rank 1, two matches at ranks 1 and 2, and matches at ranks 1 and 5, and the figure gives AP = 1, 1 and 0.71. In single-query evaluation each query is searched alone. In multi-query evaluation the several query images of one identity in one camera are merged into a single query, by average pooling (“averaged sum”) or by max pooling (the maximum of each feature dimension) (§4.2). The general definitions of CMC and mAP are in Person Re-identification and Average Precision.

The protocol in third-party code

Two implementations are in common use; both are third-party, MIT-licensed and were read line by line at the pinned commits. They agree on the core rule and differ in details:

• Good and junk. layumi’s evaluate defines good matches as gallery images with the query’s identity and a different camera. Its junk is the same identity in the same camera, plus every image labelled −1. Junk images are deleted from the ranked list before anything is counted. torchreid’s eval_market1501 deletes images with the same identity and camera, and relies on its loader to have removed the −1 boxes. Distractors (code 0) are negatives in both.
• AP. layumi’s compute_mAP adds, for each good match, the average of the precision just before and just after it (old_precision = i/rows_good[i], or 1 at the top), a trapezoid under the precision–recall curve. torchreid averages the precision at each match, the non-interpolated sum. The Person Re-identification article attributes the trapezoid to compute_AP.m in Zhong et al.’s re-ranking repository. That repository has no licence file, so its code was not read for this article.
• Queries without a match. Both skip a query with no good match. layumi still divides the summed AP and the CMC by the number of all queries (ap/len(query_label)), whereas torchreid divides by the number of valid queries. In our reading, every Market-1501 query has a cross-camera match, since each identity was seen by at least two cameras, so the difference does not arise there. On another dataset loaded through the same script, the denominators differ.
• Multi-query. layumi averages the features of the gt_bbox images that share the query’s identity and camera; neither implementation has max pooling. torchreid has no multi-query mode in rank.py.
• Precision. torchreid’s default path is a Cython version (rank_cy.pyx), which follows the same logic in float32 and stores the CMC in float32 in its Python path as well (all_cmc.astype(np.float32)). layumi’s script also holds the features (torch.FloatTensor) and the summed CMC (CMC.float()) in float32.

Example: what the junk rule and the AP rule do

The script below ports layumi’s evaluate, compute_mAP and multi-query loop and torchreid’s loader filter and eval_market1501 (Python path) to NumPy, keeping both MIT notices in full. It runs them, together with our own implementation of each rule, on synthetic identities with no images. The only changes to the ported code are NumPy in place of torch, which also leaves layumi’s features and CMC in float64 where its script uses float32 (torchreid’s float32 cast is kept), the omission of torchreid’s warning for galleries smaller than max_rank, which no call here triggers, and two returned arrays used by checks C2 and C3, marked in comments.

Data. There are 300 identities, a size we chose so that the script would run in about a minute, where later papers give the released test split as 750 identities. There are six cameras. Each identity is seen by two to six cameras. For each camera in which it appears it has one to five gallery boxes and three query images: the first is the single query, and all three form the multi-query pool, our analogue of gt_bbox. A feature vector in 64 dimensions is $\mu_{\text{id}} + b\,c_{\text{cam}} + \sigma\varepsilon$, normalized to unit length, where $\mu$ and $c$ are random unit vectors for the identity and the camera, $\varepsilon \sim \mathcal{N}(0, I/64)$ and $\sigma = 1.25$. The camera bias $b \in \{0, 0.5, 1.0\}$ pulls every image of a camera towards the same direction, so that, for $b > 0$, images from the query’s own camera are easier to match. Junk boxes (−1), 20% as many as the good gallery boxes, are poor crops of a test person: $0.5\,\mu_{\text{id}} + b\,c + \sigma\varepsilon$ in one of that person’s cameras. Distractors (0), 15% as many, are random unit vectors plus the camera term and noise. These proportions are ours; the release’s counts are not reproduced. Scores are dot products, as in layumi’s code, and torchreid gets the distance $1 - \text{score}$.

What is compared. Rule 1 counts every image of the query’s identity as a hit and removes nothing. Rule 2 is the Market rule as layumi implements it. Rule 3 is torchreid as it runs, with −1 dropped by the loader and then same-identity, same-camera images removed. Because the drop happens in the loader and not in the evaluator, the script also runs torchreid’s evaluator on a gallery that still holds the −1 boxes (“−1 kept”). Every rule is scored with both AP rules, trapezoid (mAP_trap) and sum (mAP_sum), and with rank-1 accuracy (R1). Multi-query uses the Market rule with average pooling (layumi’s) and max pooling, which neither implementation has and which is our analogue of the paper’s description.

Checks (C), fixed before the pilot run with values a correct implementation reaches:

• C1: a ranking that puts every same-identity image first gives AP = 1 and R1 = 1 under every rule.
• C2: layumi’s rule and torchreid’s (with the drop) give identical hit positions for every query.
• C3: with no −1 boxes, torchreid’s evaluator without the drop agrees with layumi’s rule.
• C4: the ports agree with our implementation within $10^{-12}$ in mAP and R1.
• C5: no query has a trapezoid AP above its sum AP, and the two are equal exactly when the hits fill the top of the list. This follows by derivation: the $i$-th hit at rank $r_i \ge i$ adds $\tfrac{1}{2}\bigl(\tfrac{i-1}{r_i-1} + \tfrac{i}{r_i}\bigr)$ (with the first term taken as 1 when $r_i = 1$), which is at most $i/r_i$, with equality only when $r_i = i$. Average Precision states the same relation in prose.
• C6: the ports reproduce hand-computed values for five short lists, including the paper’s Figure 3 lists.
• C7: a three-query list in which one query has no cross-camera match shows the two denominators.
• C8: on noise-free data ($\sigma = 0$, $b = 0$) every pooling gives AP = 1 and R1 = 1 under rules 1 and 2.

Hypotheses (H), stated before any run:

• H1: the inflation of rule 1 over rule 2 grows with $b$ and is positive at $b = 1$, in R1 and in both mAPs.
• H3a: average pooling raises both mAPs over a single query.
• H3b: max pooling gives lower mAPs than average pooling. Our reason, written before the run: the element-wise maximum of noisy Gaussian vectors adds a positive offset to every dimension.
• H4: keeping −1 boxes as negatives lowers both mAPs.

A hypothesis H2, that the trapezoid never exceeds the sum, was planned and became check C5 once the derivation above showed it must hold. Each hypothesis is decided per seed and per $b$ from the printed means.

The noise level $\sigma = 1.25$ was chosen from a planning script that printed the rule-2 single-query mAP and R1 at $b = 0$ for four values of $\sigma$, then again with 1.25 added; we picked the value whose mAP was neither near 1 nor near 0. The values of $b$ and every check’s tolerance were fixed before the pilot. A pilot with seed 0 then ran the same script; it showed the C4 failure and the H3a and H3b failures at $b = 1.0$ reported below, and the editor kept the checks, the tolerance and the values of $b$ as planned. The script below is that script, run unchanged with seeds 0, 1 and 2 as command-line arguments. It uses NumPy 2.3.5, runs on one thread and took about a minute and a half on a laptop CPU. Two runs gave identical output.

# Market-1501 evaluation rules on synthetic identities (no images, no dataset files).
# Usage: python pilot.py SEED [SEED ...]
#
# Two functions below are ported from third-party MIT-licensed code:
#  - layumi_evaluate / layumi_compute_mAP / layumi_single / layumi_multi_avg:
#    evaluate_gpu.py of layumi/Person_reID_baseline_pytorch at commit 41f64e5
#    (torch.mm and torch tensors replaced by NumPy; logic unchanged).
#    MIT License
#
#    Copyright (c) 2018 Zhedong Zheng
#
#    Permission is hereby granted, free of charge, to any person obtaining a copy
#    of this software and associated documentation files (the "Software"), to deal
#    in the Software without restriction, including without limitation the rights
#    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#    copies of the Software, and to permit persons to whom the Software is
#    furnished to do so, subject to the following conditions:
#
#    The above copyright notice and this permission notice shall be included in all
#    copies or substantial portions of the Software.
#
#    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#    SOFTWARE.
#
#  - torchreid_drop_junk / torchreid_eval_market1501: market1501.py (process_dir)
#    and metrics/rank.py (eval_market1501, Python path) of KaiyangZhou/deep-person-reid
#    at commit f8cd150.
#    MIT License
#
#    Copyright (c) 2018 Kaiyang Zhou
#
#    Permission is hereby granted, free of charge, to any person obtaining a copy
#    of this software and associated documentation files (the "Software"), to deal
#    in the Software without restriction, including without limitation the rights
#    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#    copies of the Software, and to permit persons to whom the Software is
#    furnished to do so, subject to the following conditions:
#
#    The above copyright notice and this permission notice shall be included in all
#    copies or substantial portions of the Software.
#
#    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#    SOFTWARE.
import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"
import sys
import numpy as np

TOL = 1e-12

# ---------------- layumi evaluate_gpu.py (ported) ----------------
def layumi_evaluate(score, ql, qc, gl, gc):
    index = np.argsort(score)  # from small to large
    index = index[::-1]
    query_index = np.argwhere(gl == ql)
    camera_index = np.argwhere(gc == qc)
    good_index = np.setdiff1d(query_index, camera_index, assume_unique=True)
    junk_index1 = np.argwhere(gl == -1)
    junk_index2 = np.intersect1d(query_index, camera_index)
    junk_index = np.append(junk_index2, junk_index1)
    return layumi_compute_mAP(index, good_index, junk_index)

def layumi_compute_mAP(index, good_index, junk_index):
    ap = 0
    cmc = np.zeros(len(index), dtype=np.int32)
    if good_index.size == 0:  # if empty
        cmc[0] = -1
        return ap, cmc, None
    mask = np.isin(index, junk_index, invert=True)
    index = index[mask]
    ngood = len(good_index)
    mask = np.isin(index, good_index)
    rows_good = np.argwhere(mask == True)
    rows_good = rows_good.flatten()
    cmc[rows_good[0]:] = 1
    for i in range(ngood):
        d_recall = 1.0 / ngood
        precision = (i + 1) * 1.0 / (rows_good[i] + 1)
        if rows_good[i] != 0:
            old_precision = i * 1.0 / rows_good[i]
        else:
            old_precision = 1.0
        ap = ap + d_recall * (old_precision + precision) / 2
    return ap, cmc, rows_good  # rows_good returned in addition, for check C2

def layumi_single(qX, ql, qc, gX, gl, gc):
    CMC = np.zeros(len(gl), dtype=np.int64); ap = 0.0; rows = []
    for i in range(len(ql)):
        ap_tmp, CMC_tmp, rg = layumi_evaluate(gX @ qX[i], ql[i], qc[i], gl, gc)
        rows.append(rg)
        if CMC_tmp[0] == -1:
            continue
        CMC = CMC + CMC_tmp
        ap += ap_tmp
    CMC = CMC / len(ql)  # average CMC
    return CMC[0], ap / len(ql), rows

def layumi_multi_avg(ql, qc, mX, ml, mc, gX, gl, gc):
    CMC = np.zeros(len(gl), dtype=np.int64); ap = 0.0
    for i in range(len(ql)):
        mquery_index1 = np.argwhere(ml == ql[i])
        mquery_index2 = np.argwhere(mc == qc[i])
        mquery_index = np.intersect1d(mquery_index1, mquery_index2)
        mq = np.mean(mX[mquery_index, :], axis=0)
        ap_tmp, CMC_tmp, _ = layumi_evaluate(gX @ mq, ql[i], qc[i], gl, gc)
        if CMC_tmp[0] == -1:
            continue
        CMC = CMC + CMC_tmp
        ap += ap_tmp
    CMC = CMC / len(ql)
    return CMC[0], ap / len(ql)

# ---------------- torchreid (ported) ----------------
def torchreid_drop_junk(X, pids, camids):
    keep = pids != -1  # "junk images are just ignored" (process_dir)
    return X[keep], pids[keep], camids[keep]

def torchreid_eval_market1501(distmat, q_pids, g_pids, q_camids, g_camids, max_rank=50):
    num_q, num_g = distmat.shape
    indices = np.argsort(distmat, axis=1)
    matches = (g_pids[indices] == q_pids[:, np.newaxis]).astype(np.int32)
    all_cmc = []; all_AP = []; num_valid_q = 0.
    positions = []  # added: hit positions per query, for checks C2 and C3
    for q_idx in range(num_q):
        q_pid = q_pids[q_idx]; q_camid = q_camids[q_idx]
        order = indices[q_idx]
        remove = (g_pids[order] == q_pid) & (g_camids[order] == q_camid)
        keep = np.invert(remove)
        raw_cmc = matches[q_idx][keep]
        positions.append(np.flatnonzero(raw_cmc))
        if not np.any(raw_cmc):
            continue
        cmc = raw_cmc.cumsum()
        cmc[cmc > 1] = 1
        all_cmc.append(cmc[:max_rank])
        num_valid_q += 1.
        num_rel = raw_cmc.sum()
        tmp_cmc = raw_cmc.cumsum()
        tmp_cmc = [x / (i + 1.) for i, x in enumerate(tmp_cmc)]
        tmp_cmc = np.asarray(tmp_cmc) * raw_cmc
        AP = tmp_cmc.sum() / num_rel
        all_AP.append(AP)
    assert num_valid_q > 0, 'Error: all query identities do not appear in gallery'
    all_cmc = np.asarray(all_cmc).astype(np.float32)
    all_cmc = all_cmc.sum(0) / num_valid_q
    mAP = np.mean(all_AP)
    return all_cmc, mAP, positions

# ---------------- our own implementation of the three rules and two AP rules ----------------
def ap_trapezoid(hits):
    rows = np.flatnonzero(hits); n = len(rows)
    return float(np.sum(((np.arange(n) / np.maximum(rows, 1)) * (rows > 0) + (rows == 0)
                         + np.arange(1, n + 1) / (rows + 1)) / 2) / n)

def ap_sum(hits):
    rows = np.flatnonzero(hits); n = len(rows)
    return float(np.sum(np.arange(1, n + 1) / (rows + 1)) / n)

def hit_vector(score, ql, qc, gl, gc, rule):
    order = np.argsort(-score, kind="stable")
    L, C = gl[order], gc[order]
    same, samecam = L == ql, C == qc
    if rule == "none":          # 1. no removal: same-camera images of the person count as hits
        keep, good = np.ones(len(L), bool), same
    elif rule == "market":      # 2. layumi: remove same id & same camera, and label -1
        keep, good = ~((same & samecam) | (L == -1)), same & ~samecam
    elif rule == "keep_junk":   # torchreid's evaluator on a gallery that still holds -1 boxes
        keep, good = ~(same & samecam), same & ~samecam
    return good[keep].astype(int)

def evaluate(D, rule, pool=None):
    t, s, r1, top = [], [], [], []
    for i in range(len(D["ql"])):
        if pool is None:
            q = D["qX"][i]
        else:
            Q = D["mX"][(D["ml"] == D["ql"][i]) & (D["mc"] == D["qc"][i])]
            q = Q.mean(0) if pool == "avg" else Q.max(0)
        h = hit_vector(D["gX"] @ q, D["ql"][i], D["qc"][i], D["gl"], D["gc"], rule)
        t.append(ap_trapezoid(h)); s.append(ap_sum(h)); r1.append(h[0])
        n = h.sum(); top.append(bool(h[:n].all()))
    return np.array(t), np.array(s), np.array(r1), np.array(top)

# ---------------- synthetic identities ----------------
def make_data(seed, b, sigma, n_id=300, d=64, junk_frac=0.20, dist_frac=0.15, m_pool=3):
    rng = np.random.default_rng(seed)
    unit = lambda x: x / np.linalg.norm(x, axis=-1, keepdims=True)
    mu = unit(rng.standard_normal((n_id + 1, d)))   # identity directions (row 0 unused)
    cam = unit(rng.standard_normal((7, d)))         # camera directions, cameras 1..6
    eps = lambda n: rng.standard_normal((n, d)) / np.sqrt(d)
    gX, gl, gc, qX, ql, qc, mX, ml, mc = [], [], [], [], [], [], [], [], []
    for pid in range(1, n_id + 1):
        cams = rng.choice(np.arange(1, 7), size=rng.integers(2, 7), replace=False)
        for c in cams:
            n = rng.integers(1, 6)                  # 1-5 gallery boxes per camera
            gX.append(mu[pid] + b * cam[c] + sigma * eps(n)); gl += [pid] * n; gc += [c] * n
            Q = mu[pid] + b * cam[c] + sigma * eps(m_pool)   # query images of this camera
            qX.append(Q[:1]); ql.append(pid); qc.append(c)    # the first is the single query
            mX.append(Q); ml += [pid] * m_pool; mc += [c] * m_pool
    G = sum(len(x) for x in gX)
    nj, nd = int(round(junk_frac * G)), int(round(dist_frac * G))
    jp = rng.choice(gl, size=nj)                    # -1: poor crops of a test person
    jc = np.array([rng.choice(np.array(gc)[np.array(gl) == p]) for p in jp], dtype=int)
    jX = 0.5 * mu[jp] + b * cam[jc] + sigma * eps(nj)
    dc = rng.integers(1, 7, size=nd)                # 0: background and other people
    dX = unit(rng.standard_normal((nd, d))) + b * cam[dc] + sigma * eps(nd)
    gX = unit(np.vstack(gX + [jX.reshape(-1, d), dX]))
    return dict(gX=gX, gl=np.concatenate([gl, -np.ones(nj, int), np.zeros(nd, int)]),
                gc=np.concatenate([gc, jc, dc]), qX=unit(np.vstack(qX)), ql=np.array(ql),
                qc=np.array(qc), mX=unit(np.vstack(mX)), ml=np.array(ml), mc=np.array(mc))

def generator_check(D):
    gl, gc, ql, qc = D["gl"], D["gc"], D["ql"], D["qc"]
    ids = np.unique(ql)
    ncam = np.array([len(np.unique(qc[ql == p])) for p in ids])
    nogood = sum(not np.any((gl == ql[i]) & (gc != qc[i])) for i in range(len(ql)))
    pool = [int(np.sum((D["ml"] == ql[i]) & (D["mc"] == qc[i]))) for i in range(len(ql))]
    print("  identities %d, queries %d, gallery %d (good %d, junk -1 %d, distractors 0 %d)" % (
        len(ids), len(ql), len(gl), np.sum(gl > 0), np.sum(gl == -1), np.sum(gl == 0)))
    print("  cameras per identity %d-%d, queries per identity <= %d, queries without a "
          "cross-camera match %d, multi-query pool size %d-%d" % (
          ncam.min(), ncam.max(), np.bincount(ql).max(), nogood, min(pool), max(pool)))

SIGMA, BIASES = 1.25, (0.0, 0.5, 1.0)
seeds = [int(a) for a in sys.argv[1:]] or [0]

# ---- checks on hand-made rank lists (C6, C7) ----
print("C6 hand-made lists: AP by layumi port (trapezoid) and torchreid port (sum)")
expect = {"Fig.3a [1] of 5": ([1], 5, 1.0, 1.0), "Fig.3b [1,2] of 5": ([1, 2], 5, 1.0, 1.0),
          "Fig.3c [1,5] of 5": ([1, 5], 5, 53 / 80, 7 / 10),
          "[1,3,10] of 10": ([1, 3, 10], 10, 83 / 135, 59 / 90),
          "[5,6,7] of 10": ([5, 6, 7], 10, 157 / 630, 101 / 315)}
for name, (ranks, L, et, es) in expect.items():
    gl_ = np.full(L, 2); gl_[np.array(ranks) - 1] = 1; gc_ = np.full(L, 2)
    score = -np.arange(L, dtype=float)
    a_l, _, _ = layumi_evaluate(score, 1, 1, gl_, gc_)
    _, a_t, _ = torchreid_eval_market1501(-score[None], np.array([1]), gl_, np.array([1]), gc_, max_rank=L)
    print("  %-18s trapezoid %.4f (hand %.4f, diff %.1e)  sum %.4f (hand %.4f, diff %.1e)" % (
        name, a_l, et, abs(a_l - et), a_t, es, abs(a_t - es)))

print("C7 three queries, the third without a cross-camera match")
qid = np.array([1, 2, 3]); qcm = np.array([1, 1, 1])
glt = np.array([1, 9, 1, 2, 9, 9, 9, 9, 3, 3]); gct = np.array([2, 2, 2, 2, 2, 2, 2, 2, 1, 1])
dist = np.tile(np.arange(10, dtype=float), (3, 1)); dist[1] = [5, 0, 6, 1, 2, 3, 4, 7, 8, 9]
ap_l = []
for i in range(3):
    a, cm, _ = layumi_evaluate(-dist[i], qid[i], qcm[i], glt, gct)
    if cm[0] != -1:
        ap_l.append(a)
_, m_t, _ = torchreid_eval_market1501(dist, qid, glt, qcm, gct, max_rank=10)
print("  layumi: sum of AP / all 3 queries = %.4f (hand 25/72 = %.4f); torchreid: mean over "
      "valid queries = %.4f (hand %.4f)" % (sum(ap_l) / 3, 25 / 72, m_t, 2 / 3))

for seed in seeds:
    print("\n=== seed %d, sigma %.2f" % (seed, SIGMA))
    # ---- C8: noise-free data; C1: oracle ranking ----
    D0 = make_data(seed, 0.0, 0.0)
    for pool in (None, "avg", "max"):
        for rule in ("none", "market"):
            t, s, r1, _ = evaluate(D0, rule, pool)
            print("C8 noise-free  pool=%-4s rule=%-6s max|1-AP_trap| %.1e max|1-AP_sum| %.1e min R1 %d"
                  % (pool, rule, np.abs(1 - t).max(), np.abs(1 - s).max(), r1.min()))
    D = make_data(seed, 1.0, SIGMA)
    rng = np.random.default_rng(seed + 1000)
    worst = []
    for i in range(len(D["ql"])):
        same = D["gl"] == D["ql"][i]; cross = D["gc"] != D["qc"][i]
        sc = 3.0 * (same & cross) + 2.0 * (same & ~cross) + rng.random(len(same))
        for rule in ("none", "market", "keep_junk"):
            h = hit_vector(sc, D["ql"][i], D["qc"][i], D["gl"], D["gc"], rule)
            worst.append(max(abs(1 - ap_trapezoid(h)), abs(1 - ap_sum(h)), 1 - h[0]))
    print("C1 oracle ranking, every rule, b=1.0: max deviation from AP = 1 and R1 = 1: %.1e" % max(worst))
    # ---- C3: no -1 boxes ----
    Dn = make_data(seed, 1.0, SIGMA, junk_frac=0.0)
    _, _, rows_l = layumi_single(Dn["qX"], Dn["ql"], Dn["qc"], Dn["gX"], Dn["gl"], Dn["gc"])
    _, _, pos_t = torchreid_eval_market1501(1 - Dn["qX"] @ Dn["gX"].T, Dn["ql"], Dn["gl"], Dn["qc"], Dn["gc"])
    print("C3 no -1 boxes, b=1.0: queries whose hit positions differ, layumi vs torchreid evaluator "
          "without the loader's drop: %d" % sum(not np.array_equal(a, b) for a, b in zip(rows_l, pos_t)))

    for b in BIASES:
        D = make_data(seed, b, SIGMA)
        print("\n--- seed %d, camera bias b = %.1f" % (seed, b))
        generator_check(D)
        # ports
        r1_l, map_l, rows_l = layumi_single(D["qX"], D["ql"], D["qc"], D["gX"], D["gl"], D["gc"])
        gXd, gld, gcd = torchreid_drop_junk(D["gX"], D["gl"], D["gc"])
        cmc_t, map_t, pos_t = torchreid_eval_market1501(1 - D["qX"] @ gXd.T, D["ql"], gld, D["qc"], gcd)
        cmc_k, map_k, _ = torchreid_eval_market1501(1 - D["qX"] @ D["gX"].T, D["ql"], D["gl"], D["qc"], D["gc"])
        r1_lm, map_lm = layumi_multi_avg(D["ql"], D["qc"], D["mX"], D["ml"], D["mc"], D["gX"], D["gl"], D["gc"])
        # our implementation
        res = {}
        for key, rule, pool in (("none", "none", None), ("market", "market", None),
                                ("keep_junk", "keep_junk", None), ("market avg", "market", "avg"),
                                ("market max", "market", "max")):
            res[key] = evaluate(D, rule, pool)
        t, s, r1, top = res["market"]
        tk, sk, r1k, _ = res["keep_junk"]
        ta, sa, r1a, _ = res["market avg"]
        c4 = max(abs(map_l - t.mean()), abs(r1_l - r1.mean()), abs(map_t - s.mean()),
                 abs(cmc_t[0] - r1.mean()), abs(map_k - sk.mean()), abs(cmc_k[0] - r1k.mean()),
                 abs(map_lm - ta.mean()), abs(r1_lm - r1a.mean()))
        print("  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove "
              "same id and camera): %d" % sum(not np.array_equal(a, p) for a, p in zip(rows_l, pos_t)))
        print("  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and "
              "without drop): %.1e" % c4)
        viol = eq_bad = 0
        for k in ("none", "market", "keep_junk", "market avg", "market max"):
            tt, ss, _, tp = res[k]
            viol += int(np.sum(tt > ss + TOL)); eq_bad += int(np.sum((np.abs(tt - ss) <= TOL) != tp))
        print("  C5 queries with trapezoid AP > sum AP: %d; queries where (equal) != (hits on top): %d"
              % (viol, eq_bad))
        print("  %-26s %8s %10s %10s" % ("rule", "R1", "mAP_trap", "mAP_sum"))
        for k, lab in (("none", "1 no removal"), ("market", "2 Market rule (layumi)"),
                       ("market", "3 torchreid (drop -1)"), ("keep_junk", "  torchreid, -1 kept"),
                       ("market avg", "2 multi-query avg"), ("market max", "2 multi-query max")):
            tt, ss, rr, _ = res[k]
            print("  %-26s %8.5f %10.5f %10.5f" % (lab, rr.mean(), tt.mean(), ss.mean()))

C6 hand-made lists: AP by layumi port (trapezoid) and torchreid port (sum)
  Fig.3a [1] of 5    trapezoid 1.0000 (hand 1.0000, diff 0.0e+00)  sum 1.0000 (hand 1.0000, diff 0.0e+00)
  Fig.3b [1,2] of 5  trapezoid 1.0000 (hand 1.0000, diff 0.0e+00)  sum 1.0000 (hand 1.0000, diff 0.0e+00)
  Fig.3c [1,5] of 5  trapezoid 0.6625 (hand 0.6625, diff 0.0e+00)  sum 0.7000 (hand 0.7000, diff 0.0e+00)
  [1,3,10] of 10     trapezoid 0.6148 (hand 0.6148, diff 1.1e-16)  sum 0.6556 (hand 0.6556, diff 0.0e+00)
  [5,6,7] of 10      trapezoid 0.2492 (hand 0.2492, diff 2.8e-17)  sum 0.3206 (hand 0.3206, diff 5.6e-17)
C7 three queries, the third without a cross-camera match
  layumi: sum of AP / all 3 queries = 0.3472 (hand 25/72 = 0.3472); torchreid: mean over valid queries = 0.6667 (hand 0.6667)

=== seed 0, sigma 1.25
C8 noise-free  pool=None rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=None rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=avg  rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=avg  rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=max  rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=max  rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C1 oracle ranking, every rule, b=1.0: max deviation from AP = 1 and R1 = 1: 0.0e+00
C3 no -1 boxes, b=1.0: queries whose hit positions differ, layumi vs torchreid evaluator without the loader's drop: 0

--- seed 0, camera bias b = 0.0
  identities 300, queries 1204, gallery 4949 (good 3666, junk -1 733, distractors 0 550)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 1.9e-08
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.88455    0.59645    0.60874
  2 Market rule (layumi)      0.86130    0.59352    0.60916
  3 torchreid (drop -1)       0.86130    0.59352    0.60916
    torchreid, -1 kept        0.83555    0.56224    0.57899
  2 multi-query avg           0.99336    0.92857    0.93253
  2 multi-query max           0.72674    0.46350    0.48250

--- seed 0, camera bias b = 0.5
  identities 300, queries 1204, gallery 4949 (good 3666, junk -1 733, distractors 0 550)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 1.2e-08
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.81728    0.45709    0.47191
  2 Market rule (layumi)      0.61794    0.36381    0.38527
  3 torchreid (drop -1)       0.61794    0.36381    0.38527
    torchreid, -1 kept        0.57641    0.33157    0.35322
  2 multi-query avg           0.90365    0.67762    0.69250
  2 multi-query max           0.53073    0.29145    0.31134

--- seed 0, camera bias b = 1.0
  identities 300, queries 1204, gallery 4949 (good 3666, junk -1 733, distractors 0 550)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 8.2e-10
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.67940    0.17192    0.18261
  2 Market rule (layumi)      0.02907    0.02325    0.02783
  3 torchreid (drop -1)       0.02907    0.02325    0.02783
    torchreid, -1 kept        0.02409    0.01908    0.02284
  2 multi-query avg           0.01661    0.01900    0.02268
  2 multi-query max           0.02575    0.02357    0.02814

=== seed 1, sigma 1.25
C8 noise-free  pool=None rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=None rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=avg  rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=avg  rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=max  rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=max  rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C1 oracle ranking, every rule, b=1.0: max deviation from AP = 1 and R1 = 1: 0.0e+00
C3 no -1 boxes, b=1.0: queries whose hit positions differ, layumi vs torchreid evaluator without the loader's drop: 0

--- seed 1, camera bias b = 0.0
  identities 300, queries 1229, gallery 4975 (good 3685, junk -1 737, distractors 0 553)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 3.0e-08
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.89829    0.61660    0.62900
  2 Market rule (layumi)      0.86493    0.61309    0.62870
  3 torchreid (drop -1)       0.86493    0.61309    0.62870
    torchreid, -1 kept        0.85028    0.58368    0.60017
  2 multi-query avg           0.99756    0.94101    0.94376
  2 multi-query max           0.79333    0.50209    0.52028

--- seed 1, camera bias b = 0.5
  identities 300, queries 1229, gallery 4975 (good 3685, junk -1 737, distractors 0 553)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 1.1e-08
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.84215    0.47806    0.49282
  2 Market rule (layumi)      0.65826    0.39013    0.41076
  3 torchreid (drop -1)       0.65826    0.39013    0.41076
    torchreid, -1 kept        0.61351    0.35709    0.37837
  2 multi-query avg           0.92189    0.71096    0.72438
  2 multi-query max           0.56876    0.31414    0.33441

--- seed 1, camera bias b = 1.0
  identities 300, queries 1229, gallery 4975 (good 3685, junk -1 737, distractors 0 553)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 1.6e-09
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.71440    0.17847    0.18873
  2 Market rule (layumi)      0.03173    0.02501    0.02993
  3 torchreid (drop -1)       0.03173    0.02501    0.02993
    torchreid, -1 kept        0.02685    0.02054    0.02460
  2 multi-query avg           0.01871    0.02166    0.02603
  2 multi-query max           0.02929    0.02414    0.02868

=== seed 2, sigma 1.25
C8 noise-free  pool=None rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=None rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=avg  rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=avg  rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=max  rule=none   max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C8 noise-free  pool=max  rule=market max|1-AP_trap| 0.0e+00 max|1-AP_sum| 0.0e+00 min R1 1
C1 oracle ranking, every rule, b=1.0: max deviation from AP = 1 and R1 = 1: 0.0e+00
C3 no -1 boxes, b=1.0: queries whose hit positions differ, layumi vs torchreid evaluator without the loader's drop: 0

--- seed 2, camera bias b = 0.0
  identities 300, queries 1228, gallery 5015 (good 3715, junk -1 743, distractors 0 557)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 4.9e-09
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.88762    0.61259    0.62497
  2 Market rule (layumi)      0.86808    0.61035    0.62552
  3 torchreid (drop -1)       0.86808    0.61035    0.62552
    torchreid, -1 kept        0.83958    0.57887    0.59534
  2 multi-query avg           0.99186    0.93321    0.93706
  2 multi-query max           0.80293    0.50183    0.51944

--- seed 2, camera bias b = 0.5
  identities 300, queries 1228, gallery 5015 (good 3715, junk -1 743, distractors 0 557)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 2.6e-08
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.81433    0.47372    0.48820
  2 Market rule (layumi)      0.67345    0.39087    0.41083
  3 torchreid (drop -1)       0.67345    0.39087    0.41083
    torchreid, -1 kept        0.62622    0.35860    0.37892
  2 multi-query avg           0.92101    0.70658    0.71947
  2 multi-query max           0.58143    0.30896    0.32874

--- seed 2, camera bias b = 1.0
  identities 300, queries 1228, gallery 5015 (good 3715, junk -1 743, distractors 0 557)
  cameras per identity 2-6, queries per identity <= 6, queries without a cross-camera match 0, multi-query pool size 3-3
  C2 queries whose hit positions differ, layumi rule vs torchreid (drop -1, then remove same id and camera): 0
  C4 largest |port - ours| over mAP and R1 (layumi single and avg, torchreid with and without drop): 1.9e-10
  C5 queries with trapezoid AP > sum AP: 0; queries where (equal) != (hits on top): 0
  rule                             R1   mAP_trap    mAP_sum
  1 no removal                0.67997    0.18490    0.19574
  2 Market rule (layumi)      0.05782    0.03411    0.04061
  3 torchreid (drop -1)       0.05782    0.03411    0.04061
    torchreid, -1 kept        0.04642    0.02787    0.03342
  2 multi-query avg           0.05212    0.03265    0.03904
  2 multi-query max           0.07166    0.03611    0.04316

Generator. In every condition each identity is seen by 2–6 cameras, no identity has more than six queries, every query has a cross-camera match, and every pool has three images, as planned. The gallery holds about 3,700 good boxes, 740 junk boxes and 550 distractors.

Checks. C1, C2, C3, C5 and C8 held in all three seeds, with every printed count and deviation at 0, and C6 and C7, which run once before the seeds, held: C6’s ports are within $1.1 \times 10^{-16}$ of the hand values, and C7’s values equal them at the printed precision. C4 failed at every $b$ in every seed: the largest difference between the ports and our implementation was $1.9 \times 10^{-8}$, $1.2 \times 10^{-8}$ and $8.2 \times 10^{-10}$ for seed 0 ($b$ = 0, 0.5, 1.0), $3.0 \times 10^{-8}$, $1.1 \times 10^{-8}$ and $1.6 \times 10^{-9}$ for seed 1, and $4.9 \times 10^{-9}$, $2.6 \times 10^{-8}$ and $1.9 \times 10^{-10}$ for seed 2, all above the planned $10^{-12}$. In our reading the cause is that torchreid stores its CMC as float32 (all_cmc.astype(np.float32) at f8cd150), so the R1 it returns differs from our float64 mean at float32 resolution; the check compares the maximum over mAP and R1, and the script does not print them separately. C2 shows that the two implementations rank the same images as hits for every query, and the rows for rules 2 and 3 print identical values to five decimals.

C6 and the paper’s Figure 3. The ports give AP = 1 for lists (a) and (b) under both rules. For list (c), with matches at ranks 1 and 5, they give 0.6625 by the trapezoid (53/80) and 0.7000 by the sum (7/10). The paper prints 0.71, which matches neither computation. We could not find which calculation gives 0.71, and the question remains open; the Person Re-identification article quotes the paper’s value.

C7, the denominators. On the three-query list, layumi’s loop returns 0.3472, the two valid queries’ trapezoid APs divided by three, while torchreid returns 0.6667, the mean of the valid queries’ sum APs. A query with no cross-camera match lowers layumi’s mAP but leaves torchreid’s unchanged.

H1 held in all three seeds. The excess of rule 1 over rule 2 grows with $b$ in each measure: in seed 0 from 0.02325 to 0.19934 to 0.65033 in R1, and from 0.00293 to 0.09328 to 0.14867 in mAP_trap. At $b = 0$ rule 1 scores slightly lower than rule 2 in mAP_sum in seeds 0 and 2 (by 0.00042 and 0.00055); the hypothesis concerned only the growth and the sign at $b = 1$. By construction, a camera bias makes the query’s own-camera images the easiest matches, and rule 1 counts them as hits. At $b = 1$, R1 is 0.67940 under rule 1 against 0.02907 under the Market rule in seed 0: almost all of rule 1’s apparent success comes from same-camera images.

H3a held at $b = 0$ and $b = 0.5$ in every seed, under both AP rules: averaging three queries raised mAP_trap from 0.59352 to 0.92857 at $b = 0$ in seed 0. H3a failed at $b = 1.0$ in every seed, under both AP rules (seed 0: 0.01900 against 0.02325 for the trapezoid, 0.02268 against 0.02783 for the sum).

H3b held at $b = 0$ and $b = 0.5$ in every seed, under both AP rules (seed 0, $b = 0$: 0.46350 against 0.92857), and failed at $b = 1.0$ in every seed, under both AP rules (seed 0: 0.02357 against 0.01900 and 0.02814 against 0.02268). At $b = 1.0$ the Market rule leaves R1 between 0.02907 and 0.05782 across seeds, near the floor. By construction, at $b = 1$ the camera direction has the same length as the identity direction, so other people seen by the query’s camera compete with the true matches, and the pooled and single-query values compared there are all near zero. The paper found max pooling slightly better than average pooling on its bag-of-words histograms (Table 5); our Gaussian features are not a test of that finding.

H4 held at every $b$ in every seed, under both AP rules: keeping the −1 boxes lowered mAP_sum from 0.60916 to 0.57899 at $b = 0$ in seed 0. By construction the junk boxes point half-way along the query identity’s direction, so they score high and count as false matches when they are not deleted.

The example shows three things about the official protocol as third-party code computes it. First, the same-camera junk rule matters more as the camera bias grows. Second, the two common implementations rank the same matches but report different AP values, so under the Market rule mAP_trap is 0.015 to 0.021 below mAP_sum on the same features at $b = 0$ and $0.5$; in our reading, values from the two codes should not be compared across papers. Third, a −1 box is junk only if the loader or the evaluator removes it. This example differs from the junk comparison in the Oxford Buildings example in four ways: junk is defined here by the camera relation as well as by a label, a camera bias is varied, two third-party implementations with their denominators are compared, and queries are pooled.

License

As of November 2026 the licence or terms of use of Market-1501 could not be read for this article. The official page, which the paper gives as http://www.liangzheng.com.cn for both the dataset and the code, is on a site this project does not request. No other official host was found from a listing, and we do not know from a readable source whether registration is required. The third-party evaluation code cited here is under the MIT License (layumi: copyright 2018 Zhedong Zheng; torchreid: copyright 2018 Kaiyang Zhou). These licences cover only their code, not the images.

Limitations and Bias

• People in public. The images show pedestrians recorded outside a campus supermarket. The paper does not describe consent or notice. The Person Re-identification article’s section on privacy describes how the field’s datasets have been criticised.
• One site. All six cameras stand in front of one building (§3.1). The paper’s Figure 7 shows how strongly accuracy depends on the camera pair: the SD camera 6, with its distinct background, and the pairs 5-1 and 5-2 give low accuracy, while 1-4 and 3-5, which share more overlap, give relatively high accuracy (§5.3).
• Few identities for its size. The paper notes that its 1,501 identities are fewer than CUHK02’s and planned a version 2.0 with more (§3.2); no readable source records one.
• Split ambiguity. The paper’s 750/751 and the release’s 751/750, as later papers give it, disagree; see Splits.
• Implementation-dependent scores. As the example shows, the AP rule, the denominator and the treatment of −1 boxes depend on the evaluation code a paper used.
• Saturation. As the Person Re-identification article reports, Luo et al. (2019) reached 95.4% rank-1 and 94.2% mAP with re-ranking (their Table 4), which in our reading leaves little room on the original gallery.

Common Uses

• The standard image re-identification benchmark, reported as rank-1 and mAP in single-query mode, alongside DukeMTMC-reID, CUHK03 and MSMT17 (see Person Re-identification).
• Training data for re-identification embeddings and cross-dataset transfer experiments, with training on Market-1501 and testing on another dataset or the reverse.
• Testing metric learning and losses such as the triplet loss, and the effect of gallery size, as described in Recall@K.

Historical Milestones

• 2015: the baselines. On Market-1501, the paper’s bag-of-words descriptor with Color Names features reached 9.04% rank-1 and 3.26% mAP. Adding weak geometric constraints and a Gaussian background mask raised these to 34.38% and 14.10%; multi-query raised them to 41.21% and 17.63% with average pooling and 42.64% and 18.68% with max pooling; re-ranking with one expanded query raised mAP to 19.47% (Table 5). The paper’s text gives the baseline rank-1 accuracy on VIPeR and CUHK03 as 10.56% and 5.35%, where Table 5 prints 7.82% and 11.47%; the Market-1501 value, 9.04%, agrees between text and table.
• 2015: the 500K gallery. With approximate nearest-neighbour search, “BoW + MultiQ max” reached 10.92% mAP on Market-1501+500K, which the paper calls “a relative drop of 69.7%” from the original dataset (§5.3). In our arithmetic, a drop of 69.7% to 10.92% implies a starting value of about 36%, which does not appear in the paper. Table 5 has no row named “BoW + MultiQ max”; against its 18.68% for “BoW + Geo + Gauss + MultiQ max”, the configuration we take the paper to mean, the drop is 41.5%. The paper adds that its nearest-neighbour mAP in Figure 10 is slightly lower than in Table 5, because only the top 1,000 neighbours were returned. We report the discrepancy and do not resolve it.
• 2015: camera pairs. Averaged over camera pairs, cross-camera search reached 10.51% mAP and 13.72% rank-1, which the paper calls much lower than the results with images from all cameras in the gallery (Figure 7, §5.3).
• After 2015. The Person Re-identification article traces results on Market-1501 from hand-crafted features to the strong baseline of Luo et al. (2019) and TransReID (He et al., 2021).
• 2017–2019: DukeMTMC-reID and its parent’s withdrawal. Zheng, Zheng and Yang (2017) cut DukeMTMC-reID from the DukeMTMC tracking dataset, as the Person Re-identification article describes; layumi’s prepare.py notes that DukeMTMC-reID needs no multi-query folder. The records of the parent dataset’s withdrawal disagree on dates. Exposing.ai’s page on Duke MTMC carries an “Update May 2019” saying that Duke University had terminated the dataset in response to its report and an investigation by the Financial Times, and a note of 2 June 2019 saying that Duke University “seems to have shutdown” the project. The same page records on 2 June 2019 that the re-identification and multi-camera tracking challenge of a 2019 workshop using the data was cancelled. Peng et al. (2021), as the Person Re-identification article reports, give April 2019 as the month of retraction. We report both records and do not resolve the difference. Market-1501 itself has no recorded withdrawal in any source read for this article.

Related Datasets

• DukeMTMC-reID, from the DukeMTMC tracking recordings, has one query per test identity per camera, 702 training and 702 test identities and single-camera distractor identities, as the Person Re-identification article describes. Its parent was withdrawn in 2019 (see above).
• CUHK03 and MSMT17 are the other common image benchmarks; CUHK03 also used DPM boxes, which the paper calls “relatively good ones” compared with Market-1501’s (§3.1).
• MARS is a video re-identification dataset from six cameras, described in the Person Re-identification article.
• VIPeR, i-LIDS, CAVIAR and CUHK02 are among the older, mostly two-camera, hand-cropped datasets that Table 1 compares.
• Oxford Buildings is a landmark retrieval benchmark with a similar Junk label, deleted from the ranked list, and a trapezoidal AP.

Resources

• Paper: Scalable Person Re-identification: A Benchmark (ICCV 2015, CVF open access).
• Official page and MATLAB evaluation code: www.liangzheng.com.cn, as printed in the paper (not read for this article).
• Third-party evaluation code: Person_reID_baseline_pytorch (evaluate_gpu.py, prepare.py) and Torchreid (torchreid/metrics/rank.py, torchreid/data/datasets/image/market1501.py).
• Methods and results on Market-1501: Person Re-identification.