6-DoF Object Pose Estimation

6-DoF object pose estimation finds the rotation and translation of a rigid object relative to the camera: three degrees of freedom for orientation and three for position, hence the common names 6D pose, 6DoF pose and 6D object localization. The input is an RGB or RGB-D image and, in the classical setting, a 3D model of the object; the output is the rigid transform that carries the model into the camera frame. Robots need it to grasp and place parts (see robotic manipulation), and augmented reality needs it to attach virtual content to physical objects.

The problem is the camera pose problem seen from the object’s side, and when 2D–3D correspondences between image and model are available it reduces to Perspective-n-Point. What makes it a task of its own is everything around that geometric core: finding the object, establishing correspondences on texture-less, shiny or occluded surfaces, deciding what to do when several poses look identical, and agreeing on how to score an estimate. The BOP benchmark, started in 2017, has made the last point concrete, and most of this article’s evaluation material comes from it.

Problem Definition

A rigid object is described by a model $M$, usually a mesh with vertices $V_M$ in its own coordinate frame. Its pose is the matrix

$$\mathbf{P} = \begin{bmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^\top & 1 \end{bmatrix}, \qquad \mathbf{x}_c = \mathbf{P}\,\mathbf{x}_m ,$$

which maps a point $\mathbf{x}_m$ of the model to the point $\mathbf{x}_c$ in the camera frame (Hodaň et al., 2018, §2.2). In the vocabulary of the camera pose article, $\mathbf{P}$ is a model-to-camera transform $T_{cm}$, the analogue of world-to-camera extrinsics with the object’s frame in place of the world. BOP stores $\mathbf{R}$ row by row and $\mathbf{t}$ in millimetres, uses OpenCV’s camera axes ($x$ right, $y$ down, $z$ along the viewing direction) and projects with $s\,\mathbf{x}_i = \mathbf{K}[\mathbf{R}\,|\,\mathbf{t}]\,\mathbf{x}_m$, with the intrinsics $\mathbf{K}$ supplied per test image (BOP Challenge 2019 page, §6.1). This article and its example use the same conventions; the pinhole camera model article covers $\mathbf{K}$.

Two formulations differ in what is known at test time. Hodaň, Matas and Obdržálek (2016) define 6D localization, in which the image comes with a list of the objects present and how many instances of each, and 6D detection, in which nothing is known in advance and the method must also decide what is there; localization is scored by recall, detection by average precision. The first BOP paper used the simplest case of localization, a single instance of a single object per test target (SiSo), motivated by industrial bin picking, in which a robot needs any one instance of the requested part (Hodaň et al., 2018). From 2019 the challenge moved to a varying number of instances of a varying number of objects (ViVo), and from 2024 it also evaluates 6D detection (BOP Challenge 2019 and 2024 pages).

The training input differs as well. In the model-based, seen-object setting, the setting of BOP’s original task, a method may train on a 3D model of each test object and on images of it in known poses. In the unseen-object setting introduced in BOP 2023, the test objects are new: a method receives their 3D models only at an onboarding stage limited to 5 minutes on one GPU, and may render images of them but use no real images (Hodaň et al., 2024, §2.4). In the model-free setting added in 2024, no CAD model exists and a new object is onboarded from reference videos (BOP Challenge 2024 page). Category-level pose drops the specific model altogether and estimates the pose and size of any instance of a known category (Wang et al., 2019).

Input and Output

• Input: an RGB, grayscale or RGB-D image with known intrinsics; the identifiers of the target objects (for localization); and, depending on the setting, 3D models, training images, reference images or videos, or category-level training data.
• Output: for each target instance, a rotation $\mathbf{R} \in SO(3)$, a translation $\mathbf{t} \in \mathbb{R}^3$ and a confidence score; category-level methods also output a 3D size. BOP results are CSV lines scene_id, im_id, obj_id, score, R, t, time, with time the wall-clock time per image (BOP Challenge 2019 page, §6.1).

Variants

• RGB against RGB-D. Depth gives metric scale and surface shape directly; with color alone, the distance along the optical axis must be inferred from apparent size.
• Instance, unseen object, model-free and category level, as defined above.
• Single view against multiple views. CosyPose estimates object poses in each view and then jointly refines camera and object poses by object-level bundle adjustment (Labbé et al., 2020); BOP 2025 introduced multi-view industrial datasets so that single- and multi-view methods can be compared (BOP Challenge 2025 page).
• Tracking, in which the pose in the previous frame initializes the current one; FoundationPose handles estimation and tracking in one framework (Wen et al., 2024).

Challenges

• Symmetry and pose ambiguity. A box looks the same after half a turn about each of its axes, and a mug whose handle faces away from the camera looks the same at every rotation about its axis. The pose is then defined only up to a set of equivalent poses, which training losses and error functions must respect (Hodaň, Matas and Obdržálek, 2016).
• Occlusion. In the first BOP evaluation, every one of 15 methods scored at least 30% better on LM than on LM-O, which shows the same objects partly occluded, and recall fell quickly already at low levels of occlusion (Hodaň et al., 2018, §5.2).
• Texture-less and similar objects. Industrial parts give few features to match. The T-LESS dataset has 30 such objects with symmetries and mutual similarities; methods based on 3D local features and the learning-based methods of 2018 scored very low on it (Hodaň et al., 2018).
• Lighting and sensors. On TUD-L, recorded under eight lighting conditions, methods trained on color images rendered with fixed lighting struggled, while depth-only methods were more robust (Hodaň et al., 2018). Glossy surfaces leave holes in depth maps, which led BOP to change how visibility is estimated in its error function (below).
• Training data. Annotating real images with 6D poses takes significant effort, so the benchmark focuses on the case where only the models, from which synthetic training images can be rendered, are available (BOP Challenge 2019 page, §8.1); this leaves a gap between synthetic training images and real test images.

Approaches

The field moved from matching hand-designed templates and 3D features, through correspondence networks that feed a PnP solver, to render-and-compare refiners and, most recently, methods that onboard new objects in minutes. The BOP results show the turn clearly: methods based on point-pair features led until the learned CosyPose ended their dominance in 2020 (Hodaň et al., 2024, Fig. 1); by 2022 deep learning methods clearly outperformed them (Sundermeyer et al., 2023).

Template matching

Template methods render the model from many viewpoints and compare each rendering, or features of it, with the image. LINEMOD, by Hinterstoisser et al. (2012), is the best-known of these; in the first BOP evaluation, Hodaň et al.’s 2015 template method found candidate templates by hashing, verified them as LINEMOD does by matching surface normals, image gradients, depth and color at feature points, and refined the pose by particle swarm optimization, using 11,000 to 23,000 templates per object (Hodaň et al., 2018, §4.2). It reached an average recall of 67.2% under BOP’s 2018 criterion, fourth of 15 methods.

The Augmented Autoencoder (Sundermeyer et al., 2018) learns the templates instead. A denoising autoencoder is trained on rendered views with randomized backgrounds, lighting and occlusion to reconstruct the clean view; its latent codes for 2,562 viewpoints times 36 in-plane rotations, 92,232 entries, form a codebook of orientations. At test time a detected crop is encoded and its nearest codebook entries by cosine similarity give the rotation. Because the network learns to reconstruct views rather than regress a rotation, views that look identical map to similar codes, and the authors state that object and view symmetries are handled inherently.

Point-pair features

Point-pair features (PPF), introduced by Drost et al. (2010), work on the scene’s point cloud. BOP’s review describes the method: pairs of oriented model points are stored in a hash table; at test time reference points are fixed in the scene, pairs between a reference point and other scene points look up similar model pairs, and each match votes in a low-dimensional space of poses that align the reference point with the model; peaks become pose candidates, refined by ICP and rescored by the visible model surface. Color is not used (Hodaň et al., 2018, §4.3). An improved variant by Vidal et al. was the best method of the first BOP evaluation, with an average recall of 74.6%, and a method by Vidal et al. still set the state of the art of 2019, 56.9 $\text{AR}_\text{C}$ under the newer score (Sundermeyer et al., 2023).

Correspondences and PnP

The most widely used learned pipelines predict 2D–3D correspondences and pass them to PnP inside RANSAC, so the network solves the appearance problem and geometry solves the pose.

Sparse keypoints. Earlier networks regressed the image positions of the eight corners of the object’s 3D bounding box. PVNet (Peng et al., 2019) argued that such corners can lie far from the object’s pixels and that regressing coordinates or heatmaps directly is sensitive to occlusion and truncation. It selects $K = 8$ keypoints on the object surface by farthest-point sampling, starting from the object centre, predicts at every object pixel a unit vector pointing to each keypoint, and lets pixels vote for keypoint hypotheses in a RANSAC-like scheme; the spread of the votes gives each keypoint a covariance, which an uncertainty-driven PnP uses. On LINEMOD, with ADD-S for the symmetric eggbox and glue, it reported an average of 86.27% of poses within 10% of the object diameter by ADD(-S), and 40.77% on Occluded LINEMOD, without refinement (its Tables 3 and 5). The keypoint detection article covers keypoints as a recognition task in their own right.

Dense object coordinates. Here each object pixel predicts its 3D position in the model frame, its object coordinate, which gives a correspondence per pixel. Brachmann et al.’s 2014 method, as BOP describes it, used a random forest for this on RGB-D input and sampled triplets of 3D–3D correspondences in RANSAC (Hodaň et al., 2018, §4.1). Networks for color images followed. Pix2Pose (Park, Patten and Vincze, 2019) regresses coordinates and an expected error per pixel with an auto-encoder trained with adversarial losses to recover occluded parts, and keeps reliable pixels for PnP with RANSAC. DPOD (Zakharov, Shugurov and Ilic, 2019) textures each model with a two-channel correspondence map and predicts that map with an object mask, then refines the PnP pose with a learned refiner. CDPN (Li, Wang and Ji, 2019) argues that rotation and translation are affected by different factors: it solves rotation from predicted coordinates by PnP but regresses translation directly, in a scale-invariant form. EPOS (Hodaň, Baráth and Matas, 2020) represents the surface by compact fragments, predicts at each pixel the probability of each fragment and a 3D location on it, so that a pixel on a symmetric object can correspond to several fragments, and estimates poses with a robust PnP-RANSAC variant.

Direct regression

Direct methods output the pose from the network. PoseCNN (Xiang et al., 2018) labels pixels, lets them vote for the object centre in the image, predicts the centre’s depth for the translation, and regresses the rotation as a quaternion from features in the object’s box; its ShapeMatch-Loss compares each model point under the estimated rotation with the closest model point under the true one, so that symmetric rotations are not penalized. The robotic manipulation article gives its YCB-Video results and those of DenseFusion (Wang et al., 2019), which fuses per-pixel color features with PointNet-style features of the depth points (see PointNet).

In GDR-Net, Wang, Manhardt, Tombari and Ji (2021) argue that correspondence-plus-PnP pipelines are trained on a surrogate objective and are not differentiable with respect to the pose, while direct regression had been less accurate. Their network predicts dense correspondences and surface regions as intermediate maps and regresses the pose from them with a learned Patch-PnP module, end to end, with the rotation parameterized by its first two columns, the 6D representation of the rotation representations article. GDRNPP, which the BOP reports cite together with GDR-Net, set the 2022 state of the art at 83.7 $\text{AR}_\text{C}$ (Sundermeyer et al., 2023).

Refinement and render-and-compare

A coarse pose is usually refined. With depth, the classical refiner is iterative closest point against the observed points, which converges only from a nearby start. Without depth, a render-and-compare refiner renders the model at the current estimate and lets a network predict the pose update that aligns the rendering with the image. DeepIM (Li et al., 2018) proposed a deep network that iteratively matches renderings against the observed image, predicting a relative transformation with location and orientation untangled. CosyPose (Labbé et al., 2020) built its single-view estimator on the same idea with an EfficientNet-B3 backbone, the 6D rotation parameterization, a loss that disentangles depth and translation and handles symmetries explicitly, and a million rendered training images per dataset; giving a canonical pose as the input turns the same network into a coarse estimator. On YCB-Video it reported 89.8 AUC of ADD-S and 84.5 AUC of ADD(-S) against DeepIM’s 88.1 and 81.9 with the same detections and coarse estimates (its Table 1a). A variant of CosyPose with ICP set the 2020 state of the art at 69.8 $\text{AR}_\text{C}$ (Sundermeyer et al., 2023), ending the lead of point-pair methods (Hodaň et al., 2024, Fig. 1). The differentiable rendering article covers renderers that supply gradients for such fitting directly.

Handling symmetry in training

A network that regresses one rotation for an object with symmetries receives contradictory targets: the same image is labelled with several poses. The methods above respond in different ways. PoseCNN’s ShapeMatch-Loss and the closest-point distance avoid enumerating symmetries. Pix2Pose’s transformer loss takes the minimum of the coordinate loss over a predefined, finite pool of symmetric poses (Park, Patten and Vincze, 2019, Eq. 2). CosyPose defines the symmetries of an object as the transformations $\mathbf{S}$ that leave its rendered appearance unchanged and uses the symmetric distance

$$D_l(\mathbf{T}_1, \mathbf{T}_2) = \min_{\mathbf{S} \in \mathcal{S}(l)} \frac{1}{|X_l|} \sum_{\mathbf{x} \in X_l} \lVert \mathbf{T}_1\mathbf{S}\mathbf{x} - \mathbf{T}_2\mathbf{x} \rVert_2 ,$$

discretizing each continuous axis of symmetry into 64 rotations (Labbé et al., 2020, Eq. 2). EPOS avoids the question at the correspondence level by letting a pixel match several fragments.

Unseen and model-free objects

Training one network per object does not scale to objects that appear only at deployment. MegaPose (Labbé et al., 2023) estimates the pose of novel objects given only a detection and a CAD model at inference: the shape and coordinate frame enter the network through renderings of the model, a coarse network classifies whether a rendered hypothesis is close enough for the refiner to correct, and both networks are trained on a large synthetic dataset of thousands of objects. Its training data became BOP 2023’s training set for unseen objects, over 2 million images of more than 50,000 objects (Hodaň et al., 2024, §3.2). In that challenge GenFlow, the best method for unseen objects, reached 67.4 $\text{AR}_\text{C}$, about the level of CosyPose’s 2020 result for seen objects, at 34.6 s per image; given the default detections for seen objects instead of those for unseen objects, it reached 79.2, so the organizers see great potential in better detection of unseen objects (Hodaň et al., 2024, §4.4–4.5). As of November 2026 the BOP site states that accuracy in model-based localization of seen objects on the core datasets has risen from 56.9 to 86.0 AR, and that the best 2024 method for unseen objects reached 82.5 AR (BOP Challenge 2025 page).

FoundationPose (Wen et al., 2024) covers both the model-based and the model-free case with one model: with no CAD model, a neural implicit representation built from a small number of reference images synthesizes novel views, and the downstream estimation and tracking modules are the same in both cases. It is trained only on large-scale synthetic data, generated with help from a large language model. The robotic manipulation article summarizes its pipeline and BOP results.

Category level

Category-level methods cannot rely on an exact model. NOCS (Wang et al., 2019) defines a normalized object coordinate space, a unit cube in which all instances of a category are aligned consistently and scaled so that the diagonal of their tight bounding box is 1. A region-based network predicts class, mask and a NOCS map, the normalized coordinate of each visible pixel, from color; with the depth map, the NOCS points and the observed points are aligned by the Umeyama similarity transform inside RANSAC, which gives rotation, translation and metric size together (see the Kabsch algorithm for the closed form). The authors trained on CAMERA, synthetic objects composited onto real tabletop images in a context-aware way, plus a real dataset; on their real test set REAL275 they reported 76.4% mAP at 50% 3D IoU and 10.2% mAP at 5° and 5 cm. For evaluation they let bottles, bowls and cans rotate freely about their vertical axis, and treated a mug as symmetric when its handle is not visible.

Datasets and Benchmarks

BOP brings the main datasets into one format, with 3D models and RGB-D test images annotated with ground-truth poses, and runs an online evaluation system whose leaderboards stay open (Hodaň et al., 2018). The first release had eight datasets, 89 object models, 277,000 training images and 62,000 test images; the 2023 report lists twelve. Seven core datasets must all be evaluated to be considered for the main awards (Hodaň et al., 2024, Table 1):

Dataset	Objects	Test images used	Test instances used	Notes
LM-O	8	200	1,445	Occlusion annotations for one LINEMOD sequence
T-LESS	30	1,000	6,423	Texture-less industrial parts with symmetries and similarities
ITODD	28	721	3,041	Industrial objects; test annotations withheld
HB	33	300	1,630	Test annotations withheld; validation images public
YCB-V	21	900	4,123	YCB-Video household objects
TUD-L	3	600	600	Moving objects under eight lighting conditions
IC-BIN	2	150	1,786	Many instances in a bin, heavy occlusion

LM, the LINEMOD dataset of Hinterstoisser et al., was for years the most commonly used, although its authors did not intend it as a general benchmark: lighting is constant and its 15 texture-less household objects are easy to tell apart, unoccluded and near the image centre (Hodaň et al., 2018, §1 and §3.2). The robotic manipulation article describes LINEMOD, Occluded LINEMOD and YCB-Video, and their standard splits, in more detail.

Protocol. Several rules keep the comparison fair (BOP Challenge 2019 page; Hodaň et al., 2024, §4.1):

• A method uses one set of hyper-parameters for all objects and datasets.
• No pixel of the test images and no test annotation may be used for training; the only test-set information allowed is the range of camera–object distances and of azimuth and elevation angles.
• Only object instances with at least 10% of their projected surface visible are evaluated, on subsets of the test images defined in test_targets_bop19.json.
• From 2020 the organizers supplied 350,000 physically based rendered (PBR) training images for the core datasets, generated with BlenderProc; the 2023 report credits them with helping learned methods catch up with PPF methods. Most datasets also include OpenGL renderings on black backgrounds, and T-LESS, TUD-L and YCB-V include real training images (Hodaň et al., 2024, §1 and §3.1).
• Default 2D object detections are provided, so that methods can be compared with the detection stage held fixed (Hodaň et al., 2024).

Newer groups. As of November 2026 the datasets page lists BOP-Classic (the core datasets plus LM, HOPEv1, RU-APC, IC-MI and TYO-L), BOP-H3 (HOT3D, HOPEv2 and HANDAL, added in 2024 for model-based and model-free tasks with onboarding videos) and BOP-Industrial (IPD, XYZ-IBD and ITODD-MV, added in 2025 for bin picking, with multi-view images and untextured models). HOT3D, recorded with Meta’s Aria and Quest 3 headsets, has ground truth from a marker-based motion-capture system (BOP datasets and 2024 pages).

Category level. NOCS introduced CAMERA, with 275,000 training images and a validation set CAMERA25, and the real dataset REAL275, over six categories: bottle, bowl, camera, can, laptop and mug (Wang et al., 2019).

Evaluation Metrics

All the measures below compare an estimated pose $\hat{\mathbf{P}}$ with the ground truth $\bar{\mathbf{P}}$ using the object model; $d$ is the object’s diameter, the largest distance between two model vertices.

Rotation and translation errors. $e_\text{RE} = \arccos\big((\operatorname{tr}(\hat{\mathbf{R}}\bar{\mathbf{R}}^{-1}) - 1)/2\big)$ and $e_\text{TE} = \lVert \bar{\mathbf{t}} - \hat{\mathbf{t}} \rVert$, often combined as “within 5° and 5 cm”, as in NOCS. Hodaň, Matas and Obdržálek (2016) note that neither treats ambiguous poses as equivalent, and argue that the fit of the object’s surface is the main indicator of pose quality, so model-based errors are preferable.

ADD and ADD-S. The average distance of model points, which the BOP papers credit to Hinterstoisser et al. (2012), is

$$e_\text{ADD} = \operatorname*{avg}_{\mathbf{x} \in M} \lVert \bar{\mathbf{P}}\mathbf{x} - \hat{\mathbf{P}}\mathbf{x} \rVert_2, \qquad e_\text{ADI} = \operatorname*{avg}_{\mathbf{x}_1 \in M} \min_{\mathbf{x}_2 \in M} \lVert \bar{\mathbf{P}}\mathbf{x}_1 - \hat{\mathbf{P}}\mathbf{x}_2 \rVert_2 ,$$

the second for objects with indistinguishable views (Hodaň, Matas and Obdržálek, 2016, Eqs. 3–4). BOP calls the closest-point version ADI; PoseCNN calls it ADD-S. A pose counts as correct when the error is at most $0.1d$ (Hodaň et al., 2018, §2.2); PoseCNN also reports the area under the accuracy–threshold curve up to 10 cm on YCB-Video, and PVNet’s tables use “ADD(-S)” for ADD on asymmetric objects and ADD-S on symmetric ones. The criticisms are specific. ADI is a lower bound of ADD, so the two should not be compared directly; neither is ambiguity-invariant; and ADI yields small errors even for views that can be told apart (Hodaň, Matas and Obdržálek, 2016, §4.3), which the BOP paper puts down to its many-to-one matching (Hodaň et al., 2018, §2.2). The 2016 paper also questions the threshold of $0.1d$: larger objects cover more pixels and are easier, so a threshold tied to the object’s distance from the camera would make more sense (§4.5). The 2D projection error, the mean pixel distance between projected model points under a 5-pixel threshold, is common for color-only methods (Peng et al., 2019).

Visible Surface Discrepancy (VSD). The model is rendered in both poses, giving distance maps $\hat{D}$ and $\bar{D}$, in which each pixel stores the distance from the camera centre to the surface rather than the depth $z$. Comparing each with the test image’s distance map $D_I$ gives visibility masks $\hat{V}$ and $\bar{V}$: the pixels where the object is visible in this image. Then

$$e_\text{VSD} = \operatorname*{avg}_{p \in \hat{V} \cup \bar{V}} \begin{cases} 0 & \text{if } p \in \hat{V} \cap \bar{V} \,\wedge\, |\hat{D}(p) - \bar{D}(p)| < \tau, \\ 1 & \text{otherwise,} \end{cases}$$

with misalignment tolerance $\tau$ (Hodaň et al., 2018, Eq. 1). The visibility mask of the ground truth contains the pixels where the rendered surface lies in front of the scene or at most $\delta$ behind it; the estimate’s mask is computed the same way and, in addition, includes every pixel of the estimate that falls in the ground truth’s mask, so that the observed object cannot be said to occlude the estimate (Hodaň, Matas and Obdržálek, 2016, Eqs. 13–14). The 2016 version used a cost rising linearly to 1 at $\tau$; the 2018 step cost is easier to interpret and ignores small differences from sensor noise. Since 2019 a pixel with no depth measurement counts as visible, so that glossy ITODD objects can be evaluated (BOP Challenge 2019 page, §7.1). Because only the visible surface is compared, poses that cannot be told apart in the image are treated as equivalent without any list of symmetries, whether the ambiguity comes from global symmetry, self-occlusion or occlusion by other objects. VSD ignores color. The 2018 evaluation used $\tau = 20$ mm, $\delta = 15$ mm and a correctness threshold $\theta = 0.3$, which the authors suggest for robotic manipulation; for augmented reality, alignment along the optical axis matters less and $\tau$ can be relaxed (Hodaň et al., 2018).

MSSD and MSPD. From 2019, BOP added two maximum distances over the model vertices that account for global symmetries:

$$e_\text{MSSD} = \min_{\mathbf{S} \in S_M} \max_{\mathbf{x} \in V_M} \lVert \hat{\mathbf{P}}\mathbf{x} - \bar{\mathbf{P}}\mathbf{S}\mathbf{x} \rVert_2, \qquad e_\text{MSPD} = \min_{\mathbf{S} \in S_M} \max_{\mathbf{x} \in V_M} \lVert \operatorname{proj}(\hat{\mathbf{P}}\mathbf{x}) - \operatorname{proj}(\bar{\mathbf{P}}\mathbf{S}\mathbf{x}) \rVert_2 .$$

The organizers motivate MSSD by robotic grasping, where the maximum surface deviation strongly indicates the chance of a successful grasp, and choose the maximum because the average of ADD and ADI is dominated by finely detailed surface parts and depends on vertex sampling. MSPD, in pixels, ignores misalignment along the optical axis, which suits augmented reality and color-only methods. Both penalize misaligned parts that are invisible, which VSD does not (BOP Challenge 2019 page, §7.1 and §8.3).

Symmetries. The set $S_M$ is found in two steps: candidate transformations $\mathbf{S}$ whose Hausdorff distance $h(V_M, \mathbf{S}V_M)$ is below $\varepsilon = \max(15\ \text{mm}, 0.1d)$, the 15 mm floor keeping small details from breaking a symmetry; then the organizers remove, by their own judgment, those that the texture resolves. Continuous symmetries are discretized so that the vertex farthest from the axis moves at most 1% of the diameter between steps (BOP Challenge 2019 page, §7.2).

Score. A pose is correct for error $e$ if $e < \theta_e$; recall is the fraction of annotated instances with a correct pose, and $\text{AR}_e$ averages recall over a grid of settings. For VSD, $\tau$ runs from 5% to 50% of $d$ and $\theta$ from 0.05 to 0.5, both in steps of 5%; for MSSD, $\theta$ runs from 5% to 50% of $d$; for MSPD, from $5r$ to $50r$ pixels with $r = w/640$ for an image of width $w$. A dataset’s score is $\text{AR} = (\text{AR}_\text{VSD} + \text{AR}_\text{MSSD} + \text{AR}_\text{MSPD})/3$, and $\text{AR}_\text{C}$ averages it over the core datasets, so that large datasets do not dominate (BOP Challenge 2019 page, §7.3). Estimates are matched greedily to ground-truth instances in order of decreasing confidence, each used at most once (BOP toolkit, pose_matching.py). For 6D detection, evaluated from 2024, only MSSD and MSPD are used, which also allows RGB-only datasets (BOP Challenge 2024 page, §1.2). The BOP toolkit also records a lesson about evaluation code: in January 2020 all 2019 submissions were re-evaluated after two bugs were fixed, one of which had cast the discretization step for continuous symmetries to a Boolean, so that continuous symmetries were ignored (docs/bop_challenge_2019_re-evaluation_27-01-2020.md).

Example: one target, many errors

The script renders depth images of two synthetic objects and scores hand-made pose estimates with every error above. The objects are a block, a box of 120 × 70 × 50 mm, and a mug, a closed cylinder of radius 40 mm and height 100 mm with a handle reaching 18 mm beyond its side. Depth maps are rendered by ray casting the exact shapes at 640 × 480 pixels with a focal length of 600 pixels; the test image shows the object in its true pose 600 mm in front of the camera, before a wall at 900 mm, without noise; the block is tilted, and the mug is seen from the side with its axis along the image’s vertical and its handle pointing either away from the camera or to the side. The model points for ADD, ADD-S, MSSD, MSPD and the Hausdorff test are 4,000 surface samples per object, where BOP uses mesh vertices. Symmetries are searched among the 24 rotations that map the coordinate axes onto themselves, with BOP’s tolerance $\varepsilon$. The error functions follow the BOP toolkit’s pose_error.py and visibility.py (the ‘bop19’ visibility mode, step cost, $\delta = 15$ mm), and the threshold grids follow eval_bop19_pose.py; they were checked against that source by reading it, not by running it. With the conventions above, from_euler("xyz", ...) composes rotations about the fixed axes, and a symmetry acts in the model frame, $\bar{\mathbf{R}}\mathbf{S}$. ADD and ADD-S are counted correct below $0.1d$ (BOP’s 2018 text writes “at most”; no value here lies on the boundary). The VSD20 column uses the 2018 setting ($\tau = 20$ mm, correct below 0.3), and the AR columns give the fraction of BOP-19 threshold settings at which each single estimate counts as correct. It runs in a few seconds (about 15 on a heavily loaded machine) with NumPy 2.3.5 and SciPy 1.16.3, and prints the same output on every run.

import numpy as np
from itertools import permutations, product
from scipy.spatial import ConvexHull, cKDTree
from scipy.spatial.distance import directed_hausdorff, pdist
from scipy.spatial.transform import Rotation

W, H, F, CX, CY, WALL = 640, 480, 600.0, 320.0, 240.0, 900.0   # pixels, mm
K = np.array([[F, 0, CX], [0, F, CY], [0, 0, 1.0]])
u, v = np.meshgrid(np.arange(W), np.arange(H))
RAYS = np.stack([(u - CX) / F, (v - CY) / F, np.ones_like(u, float)], -1).reshape(-1, 3)

# --- Objects: lists of primitives in the model frame (mm) --------------------
BLOCK = [("box", np.array([-60, -35, -25.0]), np.array([60, 35, 25.0]))]
MUG = [("cyl", 40.0, -50.0, 50.0),                                   # body, axis = z
       ("box", np.array([40, -8, -30.0]), np.array([58, 8, 30.0]))]  # handle at +x

def hit(prim, o, d):
    """Ray parameter of the first hit (camera depth, since d_z = 1 in the camera), inf if none."""
    with np.errstate(divide="ignore", invalid="ignore"):
        if prim[0] == "box":
            t1, t2 = (prim[1] - o) / d, (prim[2] - o) / d
            near, far = np.minimum(t1, t2).max(1), np.maximum(t1, t2).min(1)
            return np.where((near <= far) & (near > 0), near, np.inf)
        r, z0, z1 = prim[1:]
        a = d[:, 0] ** 2 + d[:, 1] ** 2
        b = 2 * (o[:, 0] * d[:, 0] + o[:, 1] * d[:, 1])
        c = o[:, 0] ** 2 + o[:, 1] ** 2 - r ** 2
        s = (-b - np.sqrt(b ** 2 - 4 * a * c)) / (2 * a)              # nearer root
        z = o[:, 2] + s * d[:, 2]
        best = np.where((s > 0) & (z >= z0) & (z <= z1), s, np.inf)
        for zc in (z0, z1):                                           # end caps
            s = (zc - o[:, 2]) / d[:, 2]
            p = o + s[:, None] * d
            ok = (s > 0) & (p[:, 0] ** 2 + p[:, 1] ** 2 <= r ** 2)
            best = np.minimum(best, np.where(ok, s, np.inf))
        return best

def render(obj, R, t):
    """Depth image (0 = no object) and the index of the primitive seen at each pixel."""
    o = np.broadcast_to(-R.T @ t, RAYS.shape)                      # camera centre, model frame
    d = RAYS @ R                                                    # ray directions, model frame
    s = np.stack([hit(p, o, d) for p in obj])
    depth = s.min(0)
    prim = np.where(np.isfinite(depth), s.argmin(0), -1)
    return np.where(np.isfinite(depth), depth, 0).reshape(H, W), prim.reshape(H, W)

def dist_map(depth):                       # bop_toolkit misc.depth_im_to_dist_im_fast
    return np.sqrt(((u - CX) / F * depth) ** 2 + ((v - CY) / F * depth) ** 2 + depth ** 2)

def surface_points(obj, n, rng):
    """n points on the visible surface, by casting random rays from many directions."""
    pts = []
    while sum(len(p) for p in pts) < n:
        dirs = rng.normal(size=(20000, 3)); dirs /= np.linalg.norm(dirs, axis=1, keepdims=True)
        o = -300 * dirs + rng.uniform(-80, 80, (20000, 3))          # rays aimed through the object
        s = np.stack([hit(p, o, dirs) for p in obj]).min(0)
        pts.append((o + s[:, None] * dirs)[np.isfinite(s)])
    return np.vstack(pts)[:n]

# --- Pose errors, following bop_toolkit/pose_error.py ------------------------
def add(Re, te, Rg, tg, P):
    return np.linalg.norm((P @ Re.T + te) - (P @ Rg.T + tg), axis=1).mean()

def adi(Re, te, Rg, tg, P):                # ADD-S: GT points to the nearest estimated point
    return cKDTree(P @ Re.T + te).query(P @ Rg.T + tg)[0].mean()

def mssd(Re, te, Rg, tg, P, syms):
    return min(np.linalg.norm((P @ Re.T + te) - (P @ (Rg @ S).T + tg), axis=1).max() for S in syms)

def proj(X):
    x = X @ K.T
    return x[:, :2] / x[:, 2:]

def mspd(Re, te, Rg, tg, P, syms):
    return min(np.linalg.norm(proj(P @ Re.T + te) - proj(P @ (Rg @ S).T + tg), axis=1).max()
               for S in syms)

def vsd(dist_test, dist_est, dist_gt, delta, taus):
    """Visible Surface Discrepancy with the 'bop19' visibility masks and the step cost."""
    vis_gt = ((dist_gt - dist_test <= delta) | (dist_test == 0)) & (dist_gt > 0)
    vis_est = ((dist_est - dist_test <= delta) | (dist_test == 0)) & (dist_est > 0)
    vis_est |= vis_gt & (dist_est > 0)
    inter, union = vis_gt & vis_est, vis_gt | vis_est
    dists = np.abs(dist_gt[inter] - dist_est[inter])
    return [((dists >= tau).sum() + union.sum() - inter.sum()) / union.sum() for tau in taus]

# --- Global symmetries as in BOP: Hausdorff distance below max(15 mm, 0.1 d) --
def cube_rotations():
    out = []
    for perm in permutations(range(3)):
        for signs in product((1, -1), repeat=3):
            M = np.zeros((3, 3)); M[range(3), perm] = signs
            if np.linalg.det(M) > 0:
                out.append(M)
    return out                                                      # 24 rotations

def find_symmetries(P, d):
    eps = max(15.0, 0.1 * d)
    haus = lambda S: max(directed_hausdorff(P, P @ S.T)[0], directed_hausdorff(P @ S.T, P)[0])
    return [S for S in cube_rotations() if haus(S) < eps], eps, sorted(haus(S) for S in cube_rotations())

TH = np.arange(0.05, 0.51, 0.05)            # BOP-19 grids, as in scripts/eval_bop19_pose.py

def evaluate(name, obj, P, d, syms, Rg, tg, estimates, handle=None, syms_alt=None):
    depth_gt, _ = render(obj, Rg, tg)
    test = np.where(depth_gt > 0, depth_gt, WALL)                   # object in front of a wall
    dt, dg = dist_map(test), dist_map(depth_gt)
    print(f"\n{name}: diameter {d:.1f} mm, {len(syms)} symmetries, object covers "
          f"{(depth_gt > 0).mean() * 100:.1f}% of the image")
    head = "estimate                          ADD  ADD-S   MSSD   MSPD  VSD20  pass:ADD ADD-S VSD20   AR_VSD AR_MSSD AR_MSPD    AR"
    print(head + ("  handle px" if handle is not None else "")
          + ("   | MSSD4  MSPD4    AR4" if syms_alt is not None else ""))
    for label, (Re, te) in estimates.items():
        depth_e, prim_e = render(obj, Re, te)
        de = dist_map(depth_e)
        e_add, e_adi = add(Re, te, Rg, tg, P), adi(Re, te, Rg, tg, P)
        e_mssd, e_mspd = mssd(Re, te, Rg, tg, P, syms), mspd(Re, te, Rg, tg, P, syms)
        e_vsd20 = vsd(dt, de, dg, 15.0, [20.0])[0]                  # BOP 2018 setting
        e_vsd = vsd(dt / d, de / d, dg / d, 15.0 / d, TH)            # distances normalized by d
        ar_vsd = np.mean([[e < th for th in TH] for e in e_vsd])
        ar_mssd = np.mean(e_mssd / d < TH)
        ar_mspd = np.mean(e_mspd * 640 / W < 100 * TH)               # 5..50 px
        ok = lambda b: " yes" if b else "  no"
        line = (f"{label:31s} {e_add:6.1f} {e_adi:6.1f} {e_mssd:6.1f} {e_mspd:6.1f} {e_vsd20:6.3f}"
                f"     {ok(e_add < 0.1 * d)}  {ok(e_adi < 0.1 * d)}  {ok(e_vsd20 < 0.3)}"
                f"     {ar_vsd:5.2f}   {ar_mssd:5.2f}   {ar_mspd:5.2f}  {(ar_vsd + ar_mssd + ar_mspd) / 3:5.2f}")
        if handle is not None:
            line += f"  {(prim_e == handle).sum():8d}"
        if syms_alt is not None:                                    # exact symmetry group instead
            m4, p4 = mssd(Re, te, Rg, tg, P, syms_alt), mspd(Re, te, Rg, tg, P, syms_alt)
            a4 = (ar_vsd + np.mean(m4 / d < TH) + np.mean(p4 < 100 * TH)) / 3
            line += f"   | {m4:6.1f} {p4:6.1f}  {a4:5.2f}"
        print(line)

rng = np.random.default_rng(0)
Rx = lambda a: Rotation.from_euler("x", a, degrees=True).as_matrix()
Rz = lambda a: Rotation.from_euler("z", a, degrees=True).as_matrix()

# Block: a box with three different side lengths
P = surface_points(BLOCK, 4000, rng)
d = pdist(P[ConvexHull(P).vertices]).max()
syms, eps, haus = find_symmetries(P, d)
print(f"block: eps = {eps:.1f} mm; Hausdorff distances of the 24 cube rotations (mm):",
      np.round(haus, 1))
Rg, tg = Rotation.from_euler("xyz", [20, -30, 15], degrees=True).as_matrix(), np.array([0, 0, 600.0])
est = {"ground truth": (Rg, tg)}
for S in syms[1:]:
    rv = Rotation.from_matrix(S).as_rotvec(degrees=True)
    ang = np.linalg.norm(rv); ax = np.round(rv / ang, 2) + 0.0
    est[f"S: {ang:3.0f} deg about ({ax[0]:g},{ax[1]:g},{ax[2]:g})"] = (Rg @ S, tg)
est["sym. z + 3 deg, +2 mm"] = (Rg @ Rz(180) @ Rx(3), tg + [2, 0, 0])
est["90 deg about z"] = (Rg @ Rz(90), tg)
est["+20 mm along camera z"] = (Rg, tg + [0, 0, 20])
est["+20 mm along camera x"] = (Rg, tg + [20, 0, 0])
exact = [np.eye(3), Rx(180), Rotation.from_euler("y", 180, degrees=True).as_matrix(), Rz(180)]
evaluate("block", BLOCK, P, d, syms, Rg, tg, est, syms_alt=exact)

# Mug: seen from the side; the handle points away from the camera (hidden) or sideways
P = surface_points(MUG, 4000, rng)
d = pdist(P[ConvexHull(P).vertices]).max()
syms, eps, haus = find_symmetries(P, d)
print(f"\nmug: eps = {eps:.1f} mm; smallest Hausdorff distances (mm):", np.round(haus[:4], 1))
R_away = np.array([[0, -1, 0], [0, 0, -1], [1, 0, 0.0]])   # model x -> camera +z, model z -> camera -y
for gt_name, Rg in (("handle hidden", R_away), ("handle sideways", R_away @ Rz(90))):
    est = {f"turned {a:3d} deg about axis": (Rg @ Rz(a), tg) for a in (0, 20, 40, 90, 180)}
    evaluate(f"mug, {gt_name}", MUG, P, d, syms, Rg, tg, est, handle=1)

Output:

block: eps = 15.0 mm; Hausdorff distances of the 24 cube rotations (mm): [ 0.   5.   5.2  6.2 10.9 10.9 11.1 11.1 25.4 25.5 25.5 25.5 35.2 35.3
 35.3 35.3 35.3 35.3 35.3 35.3 35.3 35.3 35.3 35.3]

block: diameter 146.7 mm, 8 symmetries, object covers 3.4% of the image
estimate                          ADD  ADD-S   MSSD   MSPD  VSD20  pass:ADD ADD-S VSD20   AR_VSD AR_MSSD AR_MSPD    AR   | MSSD4  MSPD4    AR4
ground truth                       0.0    0.0    0.0    0.0  0.000      yes   yes   yes      1.00    1.00    1.00   1.00   |    0.0    0.0   1.00
S: 180 deg about (1,0,0)          64.4    1.5    0.0    0.0  0.000       no   yes   yes      1.00    1.00    1.00   1.00   |    0.0    0.0   1.00
S: 180 deg about (0,1,0)          83.5    1.5    0.0    0.0  0.000       no   yes   yes      1.00    1.00    1.00   1.00   |    0.0    0.0   1.00
S: 180 deg about (0,0,1)          89.8    1.5    0.0    0.0  0.000       no   yes   yes      1.00    1.00    1.00   1.00   |    0.0    0.0   1.00
S:  90 deg about (-1,0,0)         45.5    7.3    0.0    0.0  0.318       no   yes    no      0.48    1.00    1.00   0.83   |   60.8   58.8   0.23
S:  90 deg about (1,0,0)          45.5    7.4    0.0    0.0  0.318       no   yes    no      0.48    1.00    1.00   0.83   |   60.8   58.6   0.23
S: 180 deg about (0,0.71,0.71)    86.3    7.3    0.0    0.0  0.318       no   yes    no      0.48    1.00    1.00   0.83   |   60.8   58.4   0.23
S: 180 deg about (0,0.71,-0.71)   86.0    7.3    0.0    0.0  0.318       no   yes    no      0.48    1.00    1.00   0.83   |   60.8   58.8   0.23
sym. z + 3 deg, +2 mm             89.8    2.0    3.7    3.3  0.043       no   yes   yes      0.99    1.00    1.00   1.00   |    3.7    3.3   1.00
90 deg about z                    63.5   11.1   98.2   94.1  0.620       no   yes    no      0.03    0.00    0.00   0.01   |   98.2   94.1   0.01
+20 mm along camera z             20.0   11.0   20.0    2.4  0.768       no   yes    no      0.67    0.80    1.00   0.82   |   20.0    2.4   0.82
+20 mm along camera x             20.0    8.7   20.0   22.2  0.468       no   yes    no      0.27    0.80    0.60   0.56   |   20.0   22.2   0.56

mug: eps = 15.0 mm; smallest Hausdorff distances (mm): [ 0.   5.2 18.6 18.8]

mug, handle hidden: diameter 127.9 mm, 2 symmetries, object covers 2.8% of the image
estimate                          ADD  ADD-S   MSSD   MSPD  VSD20  pass:ADD ADD-S VSD20   AR_VSD AR_MSSD AR_MSPD    AR  handle px
turned   0 deg about axis          0.0    0.0    0.0    0.0  0.000      yes   yes   yes      1.00    1.00    1.00   1.00         0
turned  20 deg about axis         13.0    2.2   20.3   18.5  0.000       no   yes   yes      1.00    0.70    0.70   0.80         0
turned  40 deg about axis         25.7    2.5   40.0   36.1  0.000       no   yes   yes      1.00    0.40    0.30   0.57         0
turned  90 deg about axis         53.1    2.5   82.8   64.5  0.114       no   yes   yes      0.80    0.00    0.00   0.27      1098
turned 180 deg about axis         75.1    2.5  117.1   80.0  0.000       no   yes   yes      0.96    0.00    0.00   0.32      1139

mug, handle sideways: diameter 127.9 mm, 2 symmetries, object covers 3.1% of the image
estimate                          ADD  ADD-S   MSSD   MSPD  VSD20  pass:ADD ADD-S VSD20   AR_VSD AR_MSSD AR_MSPD    AR  handle px
turned   0 deg about axis          0.0    0.0    0.0    0.0  0.000      yes   yes   yes      1.00    1.00    1.00   1.00      1098
turned  20 deg about axis         13.0    2.2   20.3   14.9  0.003       no   yes   yes      0.96    0.70    0.80   0.82      1382
turned  40 deg about axis         25.7    2.5   40.0   29.2  0.144       no   yes   yes      0.89    0.40    0.50   0.60      1664
turned  90 deg about axis         53.1    2.5   82.8   67.4  0.114       no   yes   yes      0.76    0.00    0.00   0.25      1139
turned 180 deg about axis         75.1    2.5  117.1  116.0  0.204       no   yes   yes      0.60    0.00    0.00   0.20      1098

BOP’s symmetry test finds eight transformations for the block, not the four of its geometric symmetry group. Besides the identity, the printed Hausdorff distances fall into three groups. The three half-turns about the box’s axes give 5.0–6.2 mm, nonzero only because two point samples never coincide exactly. The quarter-turns about the long $x$ axis, which swap the 70 mm and 50 mm sides, and the half-turns about the diagonals of the $yz$ plane, which do the same, move each of those faces by 10 mm, and give 10.9–11.1 mm; all of these fall under the 15 mm floor of $\varepsilon$. Rotations that swap the 120 mm side with another move faces by at least 25 mm and fail. With that set, MSSD and MSPD are exactly zero for the quarter-turn, and its AR is 0.83, limited only by VSD, which sees the swapped 70 mm and 50 mm sides both in the outline and in depth (VSD20 = 0.318, just failing the 0.3 threshold, and failing at about half of the grid settings). With the exact group (the last three columns), the same estimate scores 0.23. BOP’s second step removes only candidates that the model texture resolves, as the organizers judge it; for an untextured box the written rule keeps all eight, so for an object whose sides differ by less than $\varepsilon$ the symmetry set is a modelling choice, and the code prints both. The second set of columns was added after a first trial run of the script showed the extra candidates; the planned set-up used only BOP’s procedure.

ADD and ADD-S fail in the two ways the robotic manipulation example shows after ICP: ADD rejects the true symmetries, for which every BOP error is zero, and ADD-S accepts every block estimate, including the quarter-turn about $z$, which swaps the 120 mm and 70 mm sides and which MSSD, VSD and AR all reject.

The two translation errors of 20 mm separate the measures cleanly. MSSD gives 20 mm for both, as it must for a pure translation. MSPD gives 2.4 pixels for the shift along the optical axis, a change of scale of about 3%, but 22.2 pixels for the sideways shift, so the depth error passes every MSPD threshold. VSD is the opposite: shifting the whole visible surface 20 mm back puts most pixels at or beyond the tolerance (VSD20 = 0.768), while the sideways shift costs only the pixels that no longer overlap and the tilted faces (0.468). A method that is accurate in the image but uncertain in depth is therefore rewarded by MSPD and penalized by VSD and MSSD; BOP averages the three because each suits a different application.

The mug reproduces the rotating-cup experiment of Hodaň, Matas and Obdržálek (2016). Its model has two global symmetries, the identity and a half-turn about the $x$ axis that turns it upside down with the handle in place; a real, open mug would have only the first. With the handle facing away, rotations of 20° and 40° about the mug’s axis leave the handle invisible (0 handle pixels in the estimate’s rendering) and the visible surface identical, so VSD is zero at every setting, as it should be for indistinguishable poses. MSSD and MSPD see only global symmetries and charge 20–40 mm and 18–36 pixels; ADD fails both; ADD-S passes everything, at no more than 2.5 mm. At 90° the handle appears at the side; by our arithmetic its 1,098 pixels are about 11% of the union of the two visibility masks (8,600 object pixels at 2.8% of the image, plus the handle), close to VSD20 = 0.114, which still passes $\theta = 0.3$. At 180° the handle points at the camera, yet VSD20 is 0 and $\text{AR}_\text{VSD}$ 0.96: its front lies only 18 mm in front of the cylinder, under $\tau = 20$ mm, and its silhouette lies inside the cylinder’s, so a depth comparison cannot distinguish it except at the smallest tolerances. VSD measures shape alignment in depth only, and a protrusion shallower than $\tau$ seen head-on is invisible to it. With the handle in view from the side, every rotation becomes distinguishable and $\text{AR}_\text{VSD}$ falls steadily from 0.96 at 20° to 0.60 at 180°.

After the first run, the label column was widened by three characters so that the two long symmetry labels align; no value or line changed.

Applications

• Robotic grasping and bin picking, the setting behind BOP’s task; the robotic manipulation article covers how poses become grasps, and robotic grasping and visual servoing cover neighbouring problems. The industrial machine vision article covers factory setups.
• Augmented and mixed reality, which needs perceptually accurate alignment and motivates MSPD; BOP-H3’s egocentric HOT3D data target this setting.
• Category-level perception for virtual and augmented reality, robotics and 3D scene understanding, the uses NOCS names (Wang et al., 2019).
• Multi-view scene reconstruction with known objects, as in CosyPose’s joint estimation of camera and object poses (Labbé et al., 2020).

Open Problems

• Occlusion, lighting, symmetries and similar objects, identified as open by the first BOP evaluation (Hodaň et al., 2018), remain hard: the 2023 report shows the best unseen-object method failing on heavily occluded objects (Hodaň et al., 2024, Fig. 3).
• Unseen objects at the speed of seen ones. In 2023 the best unseen-object method was more than ten times slower than the best seen-object pipeline, which needed 2.67 s per image, and detection of unseen objects lagged well behind detection of seen ones (Hodaň et al., 2024, §4.5).
• Model-free onboarding from video, which the 2024 challenge introduced for settings without CAD models, with onboarding from hand-held objects described as more challenging than from static ones (BOP Challenge 2024 page), and 3D models of transparent and highly specular objects, for which reconstruction approaches are only emerging (Hodaň et al., 2024, §1).
• Evaluation that matches the task. VSD, MSSD and MSPD each suit a different application, the symmetry set involves a subjective step, and a pose score does not measure grasp or insertion success directly.