Vision Systems & Engineering
Open3D
An open-source C++ and Python library for 3D data, with legacy and tensor geometry types, neighbour search, normals, registration and reconstruction pipelines and visualization, and a port of its colored ICP that separates what the geometric and photometric terms constrain on a textured plane.
intermediate
Open3D is an open-source library for 3D data that, in its README’s words, exposes “a set of carefully selected data structures and algorithms in both C++ and Python”, with a backend set up for parallel execution. It is developed in the GitHub organization isl-org under the MIT licence, with the copyright line “Copyright (c) 2018-2026 www.open3d.org”. Its README lists 3D data structures and processing, scene reconstruction, surface alignment, visualization and physically based rendering, machine-learning support with PyTorch and TensorFlow, and GPU acceleration of core operations, and asks users to cite Zhou, Park and Koltun (2018), which we have not read.
Several articles on this site already use parts of Open3D: Voxel Grid describes its voxel downsampling, Point Cloud Registration its RANSAC-based global registration, TSDF Fusion its volumes, Poisson Surface Reconstruction its Poisson wrapper, Tanks and Temples the benchmark’s evaluation script built on Open3D 0.9, and 3D Foundation Models the ICP defaults its evaluation code uses. This article covers how the library is organized, the conventions in its code that change results, and colored point cloud registration (Park, Zhou and Koltun, 2017), which no other article here covers. It was written from Open3D v0.20.0 (tag commit b6c5e19), read in the repository in November 2026; nothing is claimed about other versions beyond what the release notes say. Open3D could not be installed where this article was written, so the example is a NumPy and SciPy port of one mechanism.
Purpose
Open3D gives Python and C++ programs one set of types for point clouds, triangle meshes, RGB-D images and volumes, and the algorithms that connect them: neighbour search, normal estimation, downsampling, registration, odometry, volumetric integration, surface reconstruction and rendering. Its tutorials, kept in the repository as Jupyter notebooks, are written in Python, and the C++ classes are exposed to Python through pybind11 bindings in cpp/pybind, from which the Python defaults quoted below are taken.
The README presents Open3D-ML as “an extension of Open3D for 3D machine learning tasks”, built on the core library and used with PyTorch or TensorFlow. Learned point-cloud segmentation itself is the subject of Point Cloud Segmentation.
Core Capabilities
Two geometry APIs
Open3D has two parallel sets of types. The legacy API, open3d.geometry with open3d.pipelines, stores a point cloud as vectors of Eigen double-precision vectors and matrices (points_, normals_, colors_ and covariances_, PointCloud.h, v0.20.0, lines 554–560). The tensor API, open3d.t.geometry with open3d.t.pipelines, is built on open3d.core.Tensor, which the tensor tutorial describes as a “view” of a data blob with shape, stride and a data pointer; each tensor has a data type (Float32, Float64, signed and unsigned integers from 8 to 64 bits, Bool) and a device. The tensor point-cloud tutorial says a cloud lives on the device of its input tensor and can carry attributes given as a Python dictionary; to_legacy() and PointCloud.from_legacy(...) convert between the two. Creating a tensor from a NumPy array copies the data, while Tensor.from_numpy and .numpy() share memory.
The legacy colored-registration notebook now opens with a note that it uses the legacy pipeline and points new applications to the tensor pipelines, which “use explicit dtypes and devices”. The tensor API has its own colored ICP estimator, with the same default geometric weight of 0.968 (t/pipelines/registration/TransformationEstimation.h, v0.20.0, lines 293–298 and 347).
Neighbour search
KDTreeFlann offers k-nearest, radius and hybrid search; v0.20.0 runs them on nanoflann (KDTreeFlann.cpp, lines 106, 128 and 155). The hybrid search, KDTreeSearchParamHybrid(radius, max_nn), is the one the normal-estimation tutorials pass, colored ICP’s gradient fit uses and ICP’s correspondence search calls with max_nn = 1, and it is easy to misread. It does not return all points within the radius: it finds the max_nn nearest neighbours and then keeps those whose squared distance is strictly below the squared radius, by a lower_bound on the sorted distances (KDTreeFlann.cpp, v0.20.0, lines 140–164). A neighbourhood can therefore be cut by the count before the radius is reached, and a point exactly at the radius is excluded. The code treats the results as sorted by distance: the lower_bound needs it, and the colour-gradient fit below skips index 0 as the query point itself (nanoflann’s own code was not read).
Normals and their orientation
estimate_normals computes, for each point, the covariance of its neighbours and takes the eigenvector of the smallest eigenvalue, by a closed-form 3 × 3 solver by default (fast_normal_computation=True) or Eigen’s self-adjoint solver (EstimateNormals.cpp, v0.20.0, lines 209–217). Its default search is the 30 nearest neighbours (KDTreeSearchParam.h, line 50, and the Python default in cpp/pybind/geometry/pointcloud.cpp, line 151). A point with fewer than 3 neighbours gets an identity covariance (PointCloud.cpp, lines 693–725). Point Cloud explains why the fit gives an axis and not a direction.
Open3D orients a new estimate only against normals the cloud already has. When the cloud has no normals, the sign is whatever the eigen solver returns, and a zero vector is replaced by (0, 0, 1); when normals exist, each new normal is flipped to agree with the old one (EstimateNormals.cpp, lines 290–324). The point-cloud tutorial describes the first case as “a random guess”, though in the code it is the solver’s deterministic output. Orientation is a separate call: orient_normals_to_align_with_direction (default (0, 0, 1)), orient_normals_towards_camera_location (default the origin) or orient_normals_consistent_tangent_plane(k, lambda_penalty=0, cos_alpha_tol=1) (pointcloud.cpp bindings, lines 197–209). The Point Cloud Library differs here: its normal estimation flips each normal toward the viewpoint, as Point Cloud describes.
Downsampling
voxel_down_sample assigns points to a grid anchored at the cloud’s own minimum bound minus half a voxel and replaces the points of each voxel by their mean, with their normals, colours and covariances when present (PointCloud.cpp, lines 367–413). Voxel Grid describes the mechanism and the consequence of the per-cloud anchoring. The output comes in hash-map order, not in the input’s order.
Registration
The registration tutorials listed in docs/tutorial/pipelines/index.rst at v0.20.0 are ICP, NDT, generalized ICP, robust kernels, colored point cloud registration, global registration, multiway registration, RGB-D integration, RGB-D odometry and colour map optimization; the v0.20 release notes add symmetric ICP and 3D NDT as new methods. The tree at that commit holds no tutorial file for the generalized ICP entry, in docs/tutorial/pipelines or docs/jupyter/pipelines, but the method is in the library: a C++ implementation, GeneralizedICP.cpp, which we did not read, and the Python function registration_generalized_icp (cpp/pybind/pipelines/registration/registration.cpp, lines 851–859). Iterative Closest Point explains the ICP iteration and its point-to-point and point-to-plane metrics, and Point Cloud Registration the global methods.
Legacy ICP runs through one loop, RegistrationICP (Registration.cpp, v0.20.0, lines 144–193), which RegistrationColoredICP also calls (ColoredICP.cpp, lines 249–260). It applies the initial 4 × 4 transformation to the source, finds correspondences, and then, for up to max_iteration steps, computes an update from the current correspondences, left-multiplies it onto the transformation (update * transformation, line 179) and searches again. The returned transformation maps the source into the target’s frame. Three conventions in this loop change numbers:
- Correspondences are a hybrid search with
max_nn = 1: each source point’s nearest target point, kept only if strictly closer thanmax_correspondence_distance(lines 49–60). - Fitness and inlier RMSE are geometric for every variant: fitness is the number of correspondences divided by the number of source points, and
inlier_rmseis the root mean squared Euclidean distance over the correspondences (lines 96–103), even for colored ICP, whose objective is not that distance. - The stopping rule compares absolute changes despite the names: iteration stops when the absolute change in fitness is below
relative_fitnessand the absolute change ininlier_rmseis belowrelative_rmse(lines 185–190). The defaults areICPConvergenceCriteria(1e-6, 1e-6, 30)(Registration.h, lines 45–50), so with inlier RMSE in metres the second tolerance is one micrometre. Tanks and Temples shows how passing these arguments by position went wrong in the benchmark’s script.
Reconstruction, odometry and visualization
The pipelines also include RGB-D odometry, TSDF integration (ScalableTSDFVolume in the legacy API and VoxelBlockGrid in the tensor API, which TSDF Fusion describes, with KinectFusion for the tracking it builds on) and colour map optimization, and the geometry tutorials cover meshes, surface reconstruction, ray casting and distance queries. The release notes for v0.20 describe Gaussian splat rendering (3D Gaussian Splatting) in interactive and offscreen views, loading tensor point clouds from Gaussian PLY, SPLAT or SPZ files and mixing them with triangle meshes with depth occlusion.
Colored point cloud registration
Point-to-plane ICP cannot slide two flat surfaces into register, because sliding along the surface changes no point-to-plane distance; the colored-registration notebook shows misaligned textures on a wall for this reason. Park, Zhou and Koltun (2017) add a photometric term. In their formulation (§4.1, Eqs. 8–10), every target point with normal gets a linear colour model on its tangent plane, with , fitted by least squares to the neighbours’ colours. The joint objective (Eq. 12) is
where sums squared point-to-plane residuals and sums squared differences between each source point’s intensity and the target’s colour model evaluated at the source point projected onto the target’s tangent plane (Eqs. 13–19). With , the authors note, the method reduces to point-to-plane ICP (§4.2). They minimize by Gauss–Newton over a six-vector update (Eqs. 20–27) and run it coarse to fine on a voxel pyramid, since the objective is non-convex (§4.4). Their §5 states the joint term’s purpose: it locks the alignment along the tangent plane through the photometric term and along the normal through the geometric term. The optimal , they write, was found by grid search, detailed in the supplement, which we have not read.
Open3D’s implementation (ColoredICP.cpp, v0.20.0) fixes details the paper leaves open:
- The colour gradient of each target point (lines 31–93) is fitted over a hybrid neighbourhood of radius
max_correspondence_distanceand at most 30 points, set inside the library (lines 232–237). It needs at least 4 neighbours, otherwise the gradient stays zero. Each neighbour adds a row , skipping index 0, the point itself; the tangent-plane constraint is one more row, , weighted by the neighbour count (lines 77–81). Intensity is the mean of R, G and B (lines 50–52), not a luminance. - The residuals and Jacobians (lines 125–164), for a source point matched to target point with normal , gradient and , are
where is projected onto the target’s tangent plane and is lambda_geometric, Park et al.’s . The weight multiplies squared residuals; its default is 0.968, and a value outside is reset to 0.968 (ColoredICP.h, lines 36–43; also the Python default in registration.cpp, lines 840–847). 0.968 is Open3D’s default, not a value printed in the paper. The notebook writes the energy once as and once as ; both mean the same weighting.
- The solve accumulates and over both residuals of every correspondence, each weighted by the robust kernel (
Eigen.cpp, lines 252–261; the default kernel isL2Loss,ColoredICP.h, line 38, whose weight is 1 for every residual,RobustKernel.cpp, line 25), and solves (Eigen.cpp, lines 108–119, called atColoredICP.cpp, lines 175–176). The call at line 112 passes no option flags toSolveLinearSystemPSD, whose declaration givescheck_detand the other checks the defaultfalse(utility/Eigen.h, lines 58–64, repeated in comments atEigen.cpp, lines 27–30). With the checks off, the function runs Eigen’s LDLT solve and always reports success (lines 75–76), soComputeTransformation’s fall-back to the identity on failure (ColoredICP.cpp, line 178) is not reached. Withcheck_deton, a determinant of magnitude below , NaN or infinity returns failure (lines 38–43). The threshold is absolute, so in our reading whether it fires depends on the units and the number of points. - The update becomes with translation (
Eigen.cpp, lines 79–89); in our reading this is an exact rotation whose first-order form is the paper’s small-angle linearization (Eq. 20).
The notebook’s three-level scheme downsamples both clouds at voxel radii 0.04, 0.02 and 0.01 m, estimates normals with KDTreeSearchParamHybrid(radius=2 * r, max_nn=30) and runs colored ICP with max_correspondence_distance equal to the voxel radius and 50, 30 and 14 iterations, each level starting from the previous result.
Minimal Example
The example ports the colored ICP path above to NumPy 2.3.5 and SciPy 1.16.3, line by line from the files named in Core Capabilities, and asks which of the six pose components each weight recovers on a textured plane. Its outcomes test our port of Open3D v0.20.0, not Open3D. Neighbour search uses SciPy’s cKDTree with the hybrid rule; normals use NumPy’s eigh in place of Open3D’s closed-form solver; downsampled points come out sorted by voxel index rather than in hash-map order. Only the target’s normals are estimated, because ColoredICP.cpp (v0.20.0) reads only target normals (lines 133 and 196). Downsampled targets, their normals and gradients are computed once per voxel size and reused, since they do not depend on the run. Every residual gets weight 1, the default L2 kernel’s weight. Open3D’s ComputeTransformation calls utility::LogError when the correspondence set is empty (ColoredICP.cpp, lines 101–105); the port has no such check, so an empty set gives a zero and the identity update under both solves. The code carries Open3D’s MIT licence in full.
The scene. A 1 m × 1 m plane is sampled twice, independently, with 20,000 uniform points each, so the two samples share no points. Each point takes its grey colour (R = G = B) by bilinear lookup in a 1001 × 1001 texture image at 1 mm per pixel, the sum of four cosines: two of wavelength 0.80 m and two of 0.08 m, at different orientations. The second sample, the source, is then moved by a known motion : the small offset translates it by (6, −4, 3) mm and rotates it by 0.5° about the normal; the large offset by (60, −40, 3) mm and 3°; both tilt it by about 1°. In our arithmetic, the largest in-plane displacement is at most about 13 mm for the small offset (7.2 mm of translation plus 0.5° of rotation at a corner 0.71 m from the centre), below a quarter of the fine wavelength (20 mm), and at most about 109 mm for the large one (72.1 mm plus 3°), above half of it. A control scene puts a box 0.30 × 0.20 × 0.15 m on the plane, sampled at the same density. Registration starts from the identity. The error is the residual motion , where is the estimate, split into in-plane translation and normal translation (signed) at the plane’s centre, the frame’s origin, rotation about the normal (yaw, signed, from and as Open3D’s own TransformMatrix4dToVector6d extracts it) and tilt (the angle between and ).
Conditions. Three weights, (point-to-plane only), 0.968 (Open3D’s default) and 0 (photometric only); a single scale at the finest level (10 mm voxels, 94 iterations, the pyramid’s total) against the notebook’s three levels at its own radii, which already exceed the samples’ spacing of about 7 mm; and two solves:
- (a) NumPy’s minimum-norm
lstsqon , our analogue of Open3D’s default unchecked LDLT solve, which we did not read. Results under (a) test this analogue, not Open3D. - (b) Open3D’s own
check_detrule fromEigen.cpp(lines 38–43), not enabled on Open3D’s colored ICP path, followed by the samelstsq; a failed check gives the identity update.
The colour-gradient fits use lstsq in both. The box scene runs only at and the small offset.
What the Jacobians predict. The target lies exactly on , so its normals are . The sign cannot matter: and flip together, and , the projection and the gradient fit do not change. At , , so the , and columns are zero and . At , every neighbour has , so the fitted gradient is and : the column is zero, so again , and the tilt columns are proportional to the source point’s height . The colour residual drops when it projects the source point onto the tangent plane, so a tilt reaches it only as an in-plane compression by , which is second order. A Gauss–Newton step on that residual alone is , a halving, so in our derivation the tilt is weakly determined: it should shrink while the compression, about 0.08 mm at the plane’s edge for 1°, stands above the floor of the colour residuals, but not reach the geometric term’s accuracy. Under (b) both singular cases fail the check, so nothing should move. We also derived two bounds before any of the example’s scripts ran. With the initial tilt and = 3 mm the initial normal offset, the minimum-norm steps at have , so the in-plane translation should change by at most = 0.05236 mm and the yaw by at most = 0.01745°. At the normal translation changes only through tilt updates acting on the in-plane translation, so it should stay within = 3 ± 0.1259 mm for the small offset, assuming the tilt updates do not exceed in total.
Hypotheses and checks. The hypotheses were written before any of the example’s scripts ran:
- H1 (): under (a), on the plane at both offsets and scales, the in-plane and yaw changes stay within the two bounds and the normal and tilt errors fall below the H2 bars; under (b), every error keeps its initial value exactly; on the box, all four errors fall below the box bars under both solves.
- H2 (): at the small offset, all four errors fall below the bars at both scales and under both solves.
- H3 (, small offset): under (a), in-plane and yaw errors fall below the bars, the normal error stays within 3 ± 0.1259 mm, and the tilt ends below its initial value but above the 0.005° tilt bar; under (b), every error keeps its initial value exactly.
- H4 (large offset, (a)): at the pyramid meets all four bars and the single scale misses at least one; at the pyramid meets the in-plane and yaw bars and the single scale misses at least one of them.
The checks: C0 registers an exact copy of the target and must give exactly zero error everywhere; C-grad fits gradients to an exactly linear intensity field, , on a small separate cloud, where least squares is exact; C-update compares the ported update matrix with SciPy’s ; C-hybrid compares the ported search with brute force, including a point exactly at the radius; C-normal bounds the normals’ deviation from at each voxel size; C-data prints the generated data’s properties. The script also prints the determinant and eigenvalues of at the first iteration (M3).
Bars set after a planning script. A rule fixed before any of the example’s scripts set each error bar to twice the larger of the single-scale and pyramid errors reached when the registration is started at the true pose, plus 0.05 mm for translations or 0.005° for angles. One planning script then computed those floors on seed 1, the pilot’s data, and all ten bars, eight distinct values, were first written after it had printed them: for the plane at , 0.1522 mm in-plane, 0.0500 mm normal, 0.00829° yaw and 0.00500° tilt, the same in-plane and yaw values serving at ; for the box, 0.0835 mm, 0.0562 mm, 0.05129° and 0.00569°. The same script printed the values of , and used in the bounds. The check bars ( for C-grad, for C-update, degrees for C-normal, exact zero for C0) were fixed before it; C-hybrid’s kNN-rank condition and the data checks’ conditions (no shared point, R = G = B) were first written after it, though it ran neither check. The batch plan’s description of the example was extended, with the editor’s approval before the first run, by the four-cosine texture, the box scene’s restriction to , C0 under every weight, scale and solve, the data checks, the bounds and the second arm of H4. A pilot on seed 1 and the first run on seed 0, shown below, gave the same outcome for every check and hypothesis but one: H1 on the box held in the pilot, with an in-plane error of 0.01674 mm, and failed in the first run. The script runs in about 3 seconds, and two runs printed identical output.
import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
import numpy as np
from scipy.ndimage import map_coordinates
from scipy.spatial import cKDTree
# Ported (modified) from Open3D 0.20.0, commit b6c5e196384ad71e75b6e6f9c5da22d046221f1d:
# cpp/open3d/pipelines/registration/ColoredICP.cpp, Registration.cpp,
# cpp/open3d/utility/Eigen.cpp, cpp/open3d/geometry/KDTreeFlann.cpp,
# EstimateNormals.cpp and PointCloud.cpp. Licence of the original code:
#
# The MIT License (MIT)
#
# Open3D: www.open3d.org
# Copyright (c) 2018-2026 www.open3d.org
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
# SPDX-License-Identifier: MIT
def search_hybrid(tree, pts, query, radius, max_nn):
"""KDTreeFlann::SearchHybrid: kNN up to max_nn, then keep d2 < r2 (strict)."""
k = min(max_nn, len(pts))
_, idx = tree.query(query, k=k)
idx = idx.reshape(len(query), k)
d2 = np.sum((pts[idx] - query[:, None, :]) ** 2, axis=2)
keep = d2 < radius * radius # results are sorted, so this is lower_bound
return idx, d2, keep
def voxel_down_sample(points, colors, voxel):
"""PointCloud::VoxelDownSample: grid anchored at min bound - voxel/2; means per voxel.
Output order is sorted by voxel index (Open3D returns hash-map order)."""
vmin = points.min(axis=0) - 0.5 * voxel
key = np.floor((points - vmin) / voxel).astype(np.int64)
_, inv, cnt = np.unique(key, axis=0, return_inverse=True, return_counts=True)
inv = inv.ravel()
p = np.zeros((len(cnt), 3)); c = np.zeros((len(cnt), 3))
np.add.at(p, inv, points); np.add.at(c, inv, colors)
return p / cnt[:, None], c / cnt[:, None]
def estimate_normals(points, radius, max_nn=30):
"""EstimateNormals with KDTreeSearchParamHybrid: covariance of the neighbours
(identity if fewer than 3), normal = eigenvector of the smallest eigenvalue
(numpy eigh, our analogue of FastEigen3x3); no orientation."""
tree = cKDTree(points)
idx, d2, keep = search_hybrid(tree, points, points, radius, max_nn)
nnb = keep.sum(axis=1)
normals = np.zeros_like(points)
for i in range(len(points)):
if nnb[i] >= 3:
q = points[idx[i, :nnb[i]]]
cov = np.cov(q.T, bias=True)
else:
cov = np.eye(3)
w, v = np.linalg.eigh(cov)
n = v[:, 0]
if np.linalg.norm(n) == 0.0:
n = np.array([0.0, 0.0, 1.0])
normals[i] = n
return normals, nnb
def color_gradients(points, colors, normals, radius, max_nn=30):
"""InitializePointCloudForColoredICP (ColoredICP.cpp lines 31-93)."""
tree = cKDTree(points)
idx, d2, keep = search_hybrid(tree, points, points, radius, max_nn)
nnb = keep.sum(axis=1)
inten = colors.mean(axis=1)
grad = np.zeros_like(points)
for k in range(len(points)):
nn = nnb[k]
if nn >= 4:
nb = idx[k, :nn]
A = np.zeros((nn, 3)); b = np.zeros(nn)
A[:nn - 1] = points[nb[1:]] - points[k] # rows from neighbours 1..nn-1
b[:nn - 1] = inten[nb[1:]] - inten[k]
A[nn - 1] = (nn - 1) * normals[k] # orthogonality row
b[nn - 1] = 0.0
grad[k] = np.linalg.lstsq(A.T @ A, A.T @ b, rcond=None)[0]
return grad, nnb
def vec6_to_mat4(x):
"""TransformVector6dToMatrix4d (Eigen.cpp lines 79-89): R = Rz(x2) Ry(x1) Rx(x0)."""
a, b, g = x[0], x[1], x[2]
Rx = np.array([[1, 0, 0], [0, np.cos(a), -np.sin(a)], [0, np.sin(a), np.cos(a)]])
Ry = np.array([[np.cos(b), 0, np.sin(b)], [0, 1, 0], [-np.sin(b), 0, np.cos(b)]])
Rz = np.array([[np.cos(g), -np.sin(g), 0], [np.sin(g), np.cos(g), 0], [0, 0, 1]])
T = np.eye(4); T[:3, :3] = Rz @ Ry @ Rx; T[:3, 3] = x[3:6]
return T
def correspondences(tree, tpts, spts, max_dist):
"""GetRegistrationResultAndCorrespondences (Registration.cpp lines 74-105)."""
idx, d2, keep = search_hybrid(tree, tpts, spts, max_dist, 1)
keep = keep[:, 0]; d2 = d2[:, 0]; idx = idx[:, 0]
cs = np.nonzero(keep)[0]; ct = idx[keep]
if len(cs) == 0:
return cs, ct, 0.0, 0.0
fitness = len(cs) / len(spts)
rmse = np.sqrt(d2[keep].sum() / len(cs))
return cs, ct, fitness, rmse
def jtj_jtr(spts, scol, tpts, tcol, tnrm, tgrad, cs, ct, lam):
"""TransformationEstimationForColoredICP::ComputeTransformation (lines 119-171), L2 kernel."""
vs = spts[cs]; vt = tpts[ct]; nt = tnrm[ct]; dit = tgrad[ct]
sg = np.sqrt(lam); sp = np.sqrt(1.0 - lam)
J0 = sg * np.hstack([np.cross(vs, nt), nt])
r0 = sg * np.sum((vs - vt) * nt, axis=1)
vs_proj = vs - np.sum((vs - vt) * nt, axis=1)[:, None] * nt
i_s = scol[cs].mean(axis=1); i_t = tcol[ct].mean(axis=1)
is_proj = np.sum(dit * (vs_proj - vt), axis=1) + i_t
ditM = dit - np.sum(dit * nt, axis=1)[:, None] * nt # M d with M = I - n n^T
J1 = sp * np.hstack([np.cross(vs, ditM), ditM])
r1 = sp * (is_proj - i_s)
JTJ = J0.T @ J0 + J1.T @ J1
JTr = J0.T @ r0 + J1.T @ r1
return JTJ, JTr
def solve_update(JTJ, JTr, checked):
"""SolveJacobianSystemAndObtainExtrinsicMatrix (Eigen.cpp lines 108-119).
(a) unchecked: lstsq, our analogue of an always-successful LDLT solve;
(b) checked: Open3D's check_det option (lines 38-43), identity on failure."""
if checked:
det = np.linalg.det(JTJ)
if abs(det) < 1e-6 or np.isnan(det) or np.isinf(det):
return np.eye(4)
x = np.linalg.lstsq(JTJ, -JTr, rcond=None)[0]
return vec6_to_mat4(x)
def transform(T, p):
return p @ T[:3, :3].T + T[:3, 3]
def colored_icp(spts, scol, tgt, max_dist, init, lam, checked, max_iter, m3=None):
"""RegistrationICP (Registration.cpp lines 144-193) with colored-ICP estimation.
tgt = (points, colors, normals, gradients, tree) prepared at this level."""
tpts, tcol, tnrm, tgrad, tree = tgt
T = init.copy()
pcd = transform(init, spts) if not np.array_equal(init, np.eye(4)) else spts.copy()
cs, ct, fit, rmse = correspondences(tree, tpts, pcd, max_dist)
it = 0
for i in range(max_iter):
JTJ, JTr = jtj_jtr(pcd, scol, tpts, tcol, tnrm, tgrad, cs, ct, lam)
if m3 is not None and i == 0:
m3.append((np.linalg.det(JTJ), np.linalg.eigvalsh(JTJ)))
U = solve_update(JTJ, JTr, checked)
T = U @ T
pcd = transform(U, pcd)
it += 1
fit0, rmse0 = fit, rmse
cs, ct, fit, rmse = correspondences(tree, tpts, pcd, max_dist)
if abs(fit0 - fit) < 1e-6 and abs(rmse0 - rmse) < 1e-6:
break
return T, it, fit, rmse
# ---------------------------------------------------------------- scene
LC, LF = 0.80, 0.08
DIRS = np.deg2rad([20.0, 110.0, 65.0, 155.0])
PHASES = [0.3, 1.1, 2.0, 2.9]
def texture_value(x, y):
out = 0.5 * np.ones_like(x)
for k, (ang, ph) in enumerate(zip(DIRS, PHASES)):
L = LC if k < 2 else LF
out += 0.1 * np.cos(2 * np.pi * (np.cos(ang) * x + np.sin(ang) * y) / L + ph)
return out
def texture_image():
g = -0.5 + np.arange(1001) / 1000.0
X, Y = np.meshgrid(g, g) # row i <-> y, column j <-> x
return texture_value(X, Y)
def lookup(img, x, y):
cols = (x + 0.5) * 1000.0; rows = (y + 0.5) * 1000.0
v = map_coordinates(img, [rows, cols], order=1, mode="nearest")
return np.repeat(v[:, None], 3, axis=1) # grey: R = G = B
BOX = (-0.05, 0.25, -0.15, 0.05, 0.15) # x0, x1, y0, y1, height
def sample_plane(rng, n, box=False):
p = np.column_stack([rng.uniform(-0.5, 0.5, n), rng.uniform(-0.5, 0.5, n), np.zeros(n)])
if not box:
return p
x0, x1, y0, y1, h = BOX
inside = (p[:, 0] >= x0) & (p[:, 0] <= x1) & (p[:, 1] >= y0) & (p[:, 1] <= y1)
p = p[~inside]
dens = n / 1.0
parts = [p]
top = int(round(dens * (x1 - x0) * (y1 - y0)))
parts.append(np.column_stack([rng.uniform(x0, x1, top), rng.uniform(y0, y1, top), np.full(top, h)]))
for (ax, val, lo, hi) in [(0, x0, y0, y1), (0, x1, y0, y1), (1, y0, x0, x1), (1, y1, x0, x1)]:
m = int(round(dens * (hi - lo) * h))
s = np.empty((m, 3)); s[:, ax] = val; s[:, 1 - ax] = rng.uniform(lo, hi, m)
s[:, 2] = rng.uniform(0.0, h, m)
parts.append(s)
return np.vstack(parts)
def make_G(t_mm, yaw, alpha, beta):
G = vec6_to_mat4(np.array([np.deg2rad(alpha), np.deg2rad(beta), np.deg2rad(yaw), 0, 0, 0]))
G[:3, 3] = np.array(t_mm) / 1000.0
return G
OFFSETS = {"small": make_G((6, -4, 3), 0.5, 0.8, -0.6),
"large": make_G((60, -40, 3), 3.0, 0.8, -0.6)}
LEVELS = {"single": [(0.01, 94)], "pyramid": [(0.04, 50), (0.02, 30), (0.01, 14)]}
def m1(E):
R, t = E[:3, :3], E[:3, 3]
return (np.hypot(t[0], t[1]) * 1000, t[2] * 1000,
np.degrees(np.arctan2(R[1, 0], R[0, 0])), np.degrees(np.arccos(np.clip(R[2, 2], -1, 1))))
class Prepared:
"""Per (scene, voxel): downsampled target with normals and gradients (deterministic, so
computed once and reused; Open3D recomputes them in every call with the same result)."""
def __init__(self, tpts, tcol):
self.tpts, self.tcol, self.cache = tpts, tcol, {}
def get(self, voxel):
if voxel not in self.cache:
p, c = voxel_down_sample(self.tpts, self.tcol, voxel)
n, nnb = estimate_normals(p, 2 * voxel, 30)
g, gnb = color_gradients(p, c, n, 2 * voxel, 30)
self.cache[voxel] = ((p, c, n, g, cKDTree(p)), nnb, gnb)
return self.cache[voxel]
def register(spts, scol, prep, scheme, lam, checked, init=None, m3=None):
T = np.eye(4) if init is None else init.copy()
iters = []
for voxel, n_it in LEVELS[scheme]:
sp, sc = voxel_down_sample(spts, scol, voxel)
tgt = prep.get(voxel)[0]
T, it, fit, rmse = colored_icp(sp, sc, tgt, voxel, T, lam, checked, n_it, m3)
iters.append(it)
m3 = None
return T, iters, fit, rmse
# ================================================================ example
import sys
from scipy.spatial.transform import Rotation
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
rng = np.random.default_rng(seed)
img = texture_image()
A = sample_plane(rng, 20000); B = sample_plane(rng, 20000) # target, source samples
cA, cB = lookup(img, A[:, 0], A[:, 1]), lookup(img, B[:, 0], B[:, 1])
rngb = np.random.default_rng(seed + 100)
Ab = sample_plane(rngb, 20000, box=True); Bb = sample_plane(rngb, 20000, box=True)
cAb, cBb = lookup(img, Ab[:, 0], Ab[:, 1]), lookup(img, Bb[:, 0], Bb[:, 1])
prep, prepb = Prepared(A, cA), Prepared(Ab, cAb)
print(f"seed {seed}")
# ---- C-data: the generated data have the planned properties
dmin = cKDTree(A).query(B, k=1)[0].min()
print(f"C-data: plane samples {len(A)} and {len(B)}, box samples {len(Ab)} and {len(Bb)}; "
f"max |z| {np.abs(A[:, 2]).max():.1e}, {np.abs(B[:, 2]).max():.1e}")
print(f"C-data: texture image {img.shape}, range [{img.min():.4f}, {img.max():.4f}] within [0.1, 0.9]; "
f"bilinear lookup vs analytic texture, max |diff| {np.abs(cA[:, 0] - texture_value(A[:, 0], A[:, 1])).max():.2e}; "
f"R = G = B: {np.array_equal(cA[:, 0], cA[:, 1]) and np.array_equal(cA[:, 0], cA[:, 2])}")
print(f"C-data: smallest distance between the two plane samples {dmin * 1000:.4f} mm (> 0: no shared points)")
for off, G in OFFSETS.items():
back = transform(np.linalg.inv(G), transform(G, B))
print(f"C-data: {off} offset, initial M1 (in-plane mm, normal mm, yaw deg, tilt deg) = " +
", ".join(f"{v:.5f}" for v in m1(G)) + f"; max |z| of G^-1 G B {np.abs(back[:, 2]).max():.1e}")
# ---- C-update
x = np.array([0.3, -0.2, 0.5, 1.0, 2.0, 3.0])
Ri = Rotation.from_euler("ZYX", [x[2], x[1], x[0]]).as_matrix()
du = np.abs(vec6_to_mat4(x)[:3, :3] - Ri).max()
print(f"C-update: max |port - Rz Ry Rx (SciPy)| = {du:.3e} (bar 1e-14): {'held' if du <= 1e-14 else 'FAILED'}")
# ---- C-grad: exactly linear intensity on a small plane cloud
lin = np.random.default_rng(seed + 200)
P = np.column_stack([lin.uniform(-0.1, 0.1, 400), lin.uniform(-0.1, 0.1, 400), np.zeros(400)])
Cl = np.repeat((0.7 * P[:, 0] - 0.4 * P[:, 1] + 0.5)[:, None], 3, axis=1)
nl, _ = estimate_normals(P, 0.04, 30)
gl, gnb = color_gradients(P, Cl, nl, 0.04, 30)
ok = gnb >= 4
dg = np.abs(gl[ok] - [0.7, -0.4, 0.0]).max()
print(f"C-grad: {ok.sum()} of 400 points with >= 4 neighbours, max |d - (0.7, -0.4, 0)| = {dg:.3e} "
f"(bar 1e-9): {'held' if ok.sum() == 400 and dg <= 1e-9 else 'FAILED'}")
# ---- C-hybrid: port against brute force, with a point exactly at the radius
hq = np.random.default_rng(seed + 300)
H = np.vstack([hq.uniform(-1, 1, (300, 3)), [[0.25, 0.0, 0.0], [0.1, 0.0, 0.0], [0.0, 0.12, 0.0]]])
Q = np.vstack([H, [[0.0, 0.0, 0.0]]])
r, k = 0.25, 10
idx, d2, keep = search_hybrid(cKDTree(H), H, Q, r, k)
mism = 0
for i in range(len(Q)):
dd = np.sum((H - Q[i]) ** 2, axis=1)
order = np.argsort(dd, kind="stable")[:k]
brute = [j for j in order if dd[j] < r * r]
mism += list(idx[i][keep[i]]) != brute
rank = list(idx[-1]).index(300)
excl = 300 not in list(idx[-1][keep[-1]])
print(f"C-hybrid: {len(Q)} queries, mismatches {mism}; boundary point (d2 = {np.sum(H[300] ** 2):.17g} = r2) "
f"has kNN rank {rank} of {k} and is excluded: {excl}: {'held' if mism == 0 and excl and rank < k else 'FAILED'}")
# ---- C-normal: estimated target normals against +-e_z (plane scene)
for v in (0.04, 0.02, 0.01):
(p, c, n, g, tr), nnb, gnb = prep.get(v)
dev = np.degrees(np.arccos(np.clip(np.abs(n[:, 2]), -1, 1))).max()
print(f"C-normal: voxel {v:.2f} m, {len(p)} target points, max deviation {dev:.3e} deg (bar 1e-6): "
f"{'held' if dev <= 1e-6 else 'FAILED'}; min normal neighbours {nnb.min()}, "
f"< 4 gradient neighbours {(gnb < 4).sum()}, n_z < 0: {(n[:, 2] < 0).sum()}")
# ---- C0: source = exact copy of the target, identity motion
c0 = []
for lam in (1.0, 0.968, 0.0):
for scheme in ("single", "pyramid"):
for checked in (False, True):
T, its, fit, rmse = register(A, cA, prep, scheme, lam, checked)
c0.append((max(abs(v) for v in m1(T)), its))
c0max = max(v for v, _ in c0)
print(f"C0: 12 runs, max |M1 component| = {c0max!r}, iterations per level {sorted(set(tuple(i) for _, i in c0))}: "
f"{'held' if c0max == 0.0 else 'FAILED'}")
# ---- main runs
hdr = f"{'scene':5s} {'lam':>5s} {'scheme':7s} {'off':5s} {'solve':5s} {'iters':>8s} {'fitness':>8s} {'rmse_mm':>9s} {'inpl_mm':>9s} {'norm_mm':>8s} {'yaw_deg':>9s} {'tilt_deg':>8s}"
print(hdr)
res, m3s = {}, {}
for scene, lam_list, offs in [("plane", (1.0, 0.968, 0.0), ("small", "large")), ("box", (1.0,), ("small",))]:
for lam in lam_list:
for scheme in ("single", "pyramid"):
for off in offs:
for checked in (False, True):
G = OFFSETS[off]
if scene == "plane":
s, c, pp = transform(G, B), cB, prep
else:
s, c, pp = transform(G, Bb), cBb, prepb
m3 = [] if (scheme == "single" and off == "small" and not checked) else None
T, its, fit, rmse = register(s, c, pp, scheme, lam, checked, m3=m3)
e = m1(T @ G)
res[(scene, lam, scheme, off, checked)] = (T @ G, e, its)
if m3:
m3s[(scene, lam)] = m3[0]
print(f"{scene:5s} {lam:5.3f} {scheme:7s} {off:5s} {'(b)' if checked else '(a)':5s} "
f"{'+'.join(map(str, its)):>8s} {fit:8.6f} {rmse * 1000:9.6f} {e[0]:9.4f} {e[1]:8.4f} {e[2]:9.5f} {e[3]:8.5f}")
# ---- M3: JTJ at the first iteration (single scale, small offset)
for key, (det, ev) in m3s.items():
print(f"M3 {key[0]} lam={key[1]:.3f}: det {det:.3e}, eigenvalues " + " ".join(f"{v:.3e}" for v in ev))
# ---- hypotheses (bars from the reachable-value script, seed 1)
BAR = {0.968: (0.1522, 0.0500, 0.00829, 0.00500), 0.0: (0.1522, 0.0500, 0.00829, 0.00500),
"box": (0.0835, 0.0562, 0.05129, 0.00569)}
def below(e, bar, comps=(0, 1, 2, 3)):
return all(abs(e[k]) < bar[k] for k in comps)
def same_as_initial(key):
E, e, its = res[key]
return e == m1(OFFSETS[key[3]])
th0 = np.radians(m1(OFFSETS["small"])[3])
# H1
out = []
for scheme in ("single", "pyramid"):
for off in ("small", "large"):
E, e, its = res[("plane", 1.0, scheme, off, False)]
G = OFFSETS[off]
dxy = np.hypot(*(E[:2, 3] - G[:2, 3])) * 1000
dyaw = abs(e[2] - m1(G)[2])
ok = dxy <= th0 * 3 and dyaw <= np.degrees(th0 ** 2) and abs(e[1]) < BAR[0.968][1] and e[3] < BAR[0.968][3]
out.append(ok)
print(f"H1 plane (a) {scheme} {off}: |dt_xy| {dxy:.5f} mm (<= {th0 * 3:.5f}), |dyaw| {dyaw:.6f} deg "
f"(<= {np.degrees(th0 ** 2):.5f}), normal {e[1]:.5f} mm, tilt {e[3]:.6f} deg: {'held' if ok else 'failed'}")
ok = same_as_initial(("plane", 1.0, scheme, off, True))
out.append(ok)
print(f"H1 plane (b) {scheme} {off}: M1 equals initial exactly: {'held' if ok else 'failed'}")
for scheme in ("single", "pyramid"):
for checked in (False, True):
e = res[("box", 1.0, scheme, "small", checked)][1]
ok = below(e, BAR["box"])
print(f"H1 box {'(b)' if checked else '(a)'} {scheme}: |M1| " + ", ".join(f"{abs(v):.5f}" for v in e) +
f" below box bars: {'held' if ok else 'failed'}")
# H2
for scheme in ("single", "pyramid"):
for checked in (False, True):
e = res[("plane", 0.968, scheme, "small", checked)][1]
ok = below(e, BAR[0.968])
print(f"H2 {'(b)' if checked else '(a)'} {scheme} small: |M1| " + ", ".join(f"{abs(v):.5f}" for v in e) +
f" below bars: {'held' if ok else 'failed'}")
# H3
for scheme in ("single", "pyramid"):
e = res[("plane", 0.0, scheme, "small", False)][1]
band = th0 * np.hypot(*OFFSETS["small"][:2, 3]) * 1000
ok_ip = below(e, BAR[0.0], (0, 2))
ok_n = abs(e[1] - 3.0) <= band
ok_t = BAR[0.968][3] < e[3] < m1(OFFSETS["small"])[3]
print(f"H3 (a) {scheme} small: in-plane {e[0]:.5f} mm, |yaw| {abs(e[2]):.6f} deg below bars: {'held' if ok_ip else 'failed'}; "
f"normal {e[1]:.5f} mm within 3 +- {band:.4f}: {'held' if ok_n else 'failed'}; "
f"tilt {e[3]:.6f} deg in ({BAR[0.968][3]}, {m1(OFFSETS['small'])[3]:.6f}): {'held' if ok_t else 'failed'}")
ok = same_as_initial(("plane", 0.0, scheme, "small", True))
print(f"H3 (b) {scheme} small: M1 equals initial exactly: {'held' if ok else 'failed'}")
# H4
ep = res[("plane", 0.968, "pyramid", "large", False)][1]; es = res[("plane", 0.968, "single", "large", False)][1]
ok = below(ep, BAR[0.968]) and not below(es, BAR[0.968])
print(f"H4 lam=0.968 large (a): pyramid below all bars {below(ep, BAR[0.968])}, single misses a bar {not below(es, BAR[0.968])}: {'held' if ok else 'failed'}")
ep = res[("plane", 0.0, "pyramid", "large", False)][1]; es = res[("plane", 0.0, "single", "large", False)][1]
ok = below(ep, BAR[0.0], (0, 2)) and not below(es, BAR[0.0], (0, 2))
print(f"H4 lam=0 large (a): pyramid below in-plane and yaw bars {below(ep, BAR[0.0], (0, 2))}, single misses one {not below(es, BAR[0.0], (0, 2))}: {'held' if ok else 'failed'}")
# (a) and (b) agree where det passes
same = all(res[("plane", 0.968, s, o, False)][1] == res[("plane", 0.968, s, o, True)][1] and
res[("plane", 0.968, s, o, False)][2] == res[("plane", 0.968, s, o, True)][2]
for s in ("single", "pyramid") for o in ("small", "large"))
sameb = all(res[("box", 1.0, s, "small", False)][1] == res[("box", 1.0, s, "small", True)][1] for s in ("single", "pyramid"))
print(f"(a) = (b) for lam = 0.968: {same}; for the box at lam = 1: {sameb}")
seed 0
C-data: plane samples 20000 and 20000, box samples 23032 and 23023; max |z| 0.0e+00, 0.0e+00
C-data: texture image (1001, 1001), range [0.1016, 0.8949] within [0.1, 0.9]; bilinear lookup vs analytic texture, max |diff| 1.53e-04; R = G = B: True
C-data: smallest distance between the two plane samples 0.0273 mm (> 0: no shared points)
C-data: small offset, initial M1 (in-plane mm, normal mm, yaw deg, tilt deg) = 7.21110, 3.00000, 0.50000, 0.99999; max |z| of G^-1 G B 4.3e-18
C-data: large offset, initial M1 (in-plane mm, normal mm, yaw deg, tilt deg) = 72.11103, 3.00000, 3.00000, 0.99999; max |z| of G^-1 G B 4.8e-18
C-update: max |port - Rz Ry Rx (SciPy)| = 2.220e-16 (bar 1e-14): held
C-grad: 400 of 400 points with >= 4 neighbours, max |d - (0.7, -0.4, 0)| = 4.052e-15 (bar 1e-9): held
C-hybrid: 304 queries, mismatches 0; boundary point (d2 = 0.0625 = r2) has kNN rank 7 of 10 and is excluded: True: held
C-normal: voxel 0.04 m, 676 target points, max deviation 0.000e+00 deg (bar 1e-6): held; min normal neighbours 7, < 4 gradient neighbours 0, n_z < 0: 0
C-normal: voxel 0.02 m, 2597 target points, max deviation 0.000e+00 deg (bar 1e-6): held; min normal neighbours 6, < 4 gradient neighbours 0, n_z < 0: 0
C-normal: voxel 0.01 m, 8687 target points, max deviation 0.000e+00 deg (bar 1e-6): held; min normal neighbours 3, < 4 gradient neighbours 2, n_z < 0: 0
C0: 12 runs, max |M1 component| = np.float64(0.0), iterations per level [(1,), (1, 1, 1)]: held
scene lam scheme off solve iters fitness rmse_mm inpl_mm norm_mm yaw_deg tilt_deg
plane 1.000 single small (a) 2 0.987160 5.013339 7.2137 0.0000 0.50836 0.00000
plane 1.000 single small (b) 1 0.823341 6.523796 7.2111 3.0000 0.50000 0.99999
plane 1.000 single large (a) 2 0.904274 5.025554 72.1112 0.0000 3.00823 0.00000
plane 1.000 single large (b) 1 0.762165 6.498731 72.1110 3.0000 3.00000 0.99999
plane 1.000 pyramid small (a) 2+1+1 0.987160 5.013339 7.2137 0.0000 0.50836 0.00000
plane 1.000 pyramid small (b) 1+1+1 0.823341 6.523796 7.2111 3.0000 0.50000 0.99999
plane 1.000 pyramid large (a) 2+1+1 0.904274 5.025554 72.1112 0.0000 3.00823 0.00000
plane 1.000 pyramid large (b) 1+1+1 0.762165 6.498731 72.1110 3.0000 3.00000 0.99999
plane 0.968 single small (a) 3 0.990485 4.916782 0.0207 0.0000 0.00096 0.00000
plane 0.968 single small (b) 3 0.990485 4.916782 0.0207 0.0000 0.00096 0.00000
plane 0.968 single large (a) 5 0.898917 4.974361 78.9217 0.0000 -0.00388 0.00000
plane 0.968 single large (b) 5 0.898917 4.974361 78.9217 0.0000 -0.00388 0.00000
plane 0.968 pyramid small (a) 3+30+3 0.990485 4.916870 0.0227 0.0000 0.00141 0.00000
plane 0.968 pyramid small (b) 3+30+3 0.990485 4.916870 0.0227 0.0000 0.00141 0.00000
plane 0.968 pyramid large (a) 50+30+1 0.991225 4.915934 0.0394 0.0000 -0.00104 0.00000
plane 0.968 pyramid large (b) 50+30+1 0.991225 4.915934 0.0394 0.0000 -0.00104 0.00000
plane 0.000 single small (a) 94 0.933394 6.418801 0.0153 3.6936 -0.00577 0.55595
plane 0.000 single small (b) 1 0.823341 6.523796 7.2111 3.0000 0.50000 0.99999
plane 0.000 single large (a) 94 0.018689 6.824096 54.9599 127.1803 139.82922 77.74818
plane 0.000 single large (b) 1 0.762165 6.498731 72.1110 3.0000 3.00000 0.99999
plane 0.000 pyramid small (a) 26+1+1 0.000000 0.000000 672.4996 727.1707 -160.81876 42.59203
plane 0.000 pyramid small (b) 1+1+1 0.823341 6.523796 7.2111 3.0000 0.50000 0.99999
plane 0.000 pyramid large (a) 50+30+14 0.030769 6.759272 31.2424 13.9876 51.26014 33.32134
plane 0.000 pyramid large (b) 1+1+1 0.762165 6.498731 72.1110 3.0000 3.00000 0.99999
box 1.000 single small (a) 4 0.990520 4.893962 0.0984 0.0036 0.00805 0.00037
box 1.000 single small (b) 4 0.990520 4.893962 0.0984 0.0036 0.00805 0.00037
box 1.000 pyramid small (a) 4+4+3 0.990520 4.893962 0.0984 0.0036 0.00805 0.00037
box 1.000 pyramid small (b) 4+4+3 0.990520 4.893962 0.0984 0.0036 0.00805 0.00037
M3 plane lam=1.000: det 0.000e+00, eigenvalues 0.000e+00 0.000e+00 0.000e+00 3.922e+02 6.686e+02 7.225e+03
M3 plane lam=0.968: det 3.160e+19, eigenvalues 3.797e+02 6.470e+02 7.370e+02 4.961e+03 5.029e+03 6.994e+03
M3 plane lam=0.000: det 0.000e+00, eigenvalues 0.000e+00 1.619e+00 2.628e+00 2.302e+04 1.550e+05 1.572e+05
M3 box lam=1.000: det 4.263e+15, eigenvalues 5.911e+00 3.920e+02 4.797e+02 6.738e+02 7.835e+02 7.265e+03
H1 plane (a) single small: |dt_xy| 0.05283 mm (<= 0.05236), |dyaw| 0.008356 deg (<= 0.01745), normal 0.00000 mm, tilt 0.000000 deg: failed
H1 plane (b) single small: M1 equals initial exactly: held
H1 plane (a) single large: |dt_xy| 0.05741 mm (<= 0.05236), |dyaw| 0.008227 deg (<= 0.01745), normal 0.00000 mm, tilt 0.000000 deg: failed
H1 plane (b) single large: M1 equals initial exactly: held
H1 plane (a) pyramid small: |dt_xy| 0.05283 mm (<= 0.05236), |dyaw| 0.008356 deg (<= 0.01745), normal 0.00000 mm, tilt 0.000000 deg: failed
H1 plane (b) pyramid small: M1 equals initial exactly: held
H1 plane (a) pyramid large: |dt_xy| 0.05741 mm (<= 0.05236), |dyaw| 0.008227 deg (<= 0.01745), normal 0.00000 mm, tilt 0.000000 deg: failed
H1 plane (b) pyramid large: M1 equals initial exactly: held
H1 box (a) single: |M1| 0.09839, 0.00361, 0.00805, 0.00037 below box bars: failed
H1 box (b) single: |M1| 0.09839, 0.00361, 0.00805, 0.00037 below box bars: failed
H1 box (a) pyramid: |M1| 0.09839, 0.00361, 0.00805, 0.00037 below box bars: failed
H1 box (b) pyramid: |M1| 0.09839, 0.00361, 0.00805, 0.00037 below box bars: failed
H2 (a) single small: |M1| 0.02074, 0.00000, 0.00096, 0.00000 below bars: held
H2 (b) single small: |M1| 0.02074, 0.00000, 0.00096, 0.00000 below bars: held
H2 (a) pyramid small: |M1| 0.02269, 0.00000, 0.00141, 0.00000 below bars: held
H2 (b) pyramid small: |M1| 0.02269, 0.00000, 0.00141, 0.00000 below bars: held
H3 (a) single small: in-plane 0.01533 mm, |yaw| 0.005770 deg below bars: held; normal 3.69360 mm within 3 +- 0.1259: failed; tilt 0.555951 deg in (0.005, 0.999988): held
H3 (b) single small: M1 equals initial exactly: held
H3 (a) pyramid small: in-plane 672.49955 mm, |yaw| 160.818759 deg below bars: failed; normal 727.17072 mm within 3 +- 0.1259: failed; tilt 42.592030 deg in (0.005, 0.999988): failed
H3 (b) pyramid small: M1 equals initial exactly: held
H4 lam=0.968 large (a): pyramid below all bars True, single misses a bar True: held
H4 lam=0 large (a): pyramid below in-plane and yaw bars False, single misses one True: failed
(a) = (b) for lam = 0.968: True; for the box at lam = 1: True
Checks. Every check held: the data had the planned properties (texture values within [0.1, 0.9], R = G = B, no point shared by the two samples, and the source lying back on under to rounding); C0 returned exactly zero error in all twelve runs, after one iteration per level; the gradient fit recovered the linear field to rounding; the update matrix agreed with SciPy’s; the hybrid search agreed with brute force at every query and excluded the point at exactly the radius, which was within the ten nearest; and the estimated normals were exactly at all three voxel sizes. All of them came out with positive sign here, so the sign argument above was not exercised by the data.
The structure of . The M3 lines confirm the derivation: at and at the plane’s has determinant exactly 0, with three zero eigenvalues at and one at , while and the box at have every eigenvalue well above zero. At the next two eigenvalues, 1.619 and 2.628, lie nearly four orders of magnitude below the other three. Because of the zero determinants, every row under solve (b) at and on the plane reprints the initial errors after one iteration per level, and the (a) and (b) rows agree exactly wherever the determinant passed.
Outcomes.
- H1 failed under solve (a) on the plane in all four conditions, on the in-plane bound: the in-plane translation changed by 0.05283 mm at the small offset and 0.05741 mm at the large one, against the planned 0.05236 mm. Its other parts were met: the yaw changed by 0.008356° and 0.008227°, within 0.01745°, and the normal and tilt errors reached 0.0000 mm and 0.00000°. In every case the in-plane error stayed near its initial 7.2 or 72.1 mm. Under (b), H1 held: nothing moved. On the box, H1 failed under both solves and both schemes: the in-plane error of 0.09839 mm exceeded the box bar of 0.0835 mm, while the normal, yaw and tilt errors were below theirs. In the pilot (seed 1), H1 on the box held, with an in-plane error of 0.01674 mm.
- H2 held: at the small offset, reached in-plane errors of 0.0207 mm (single scale) and 0.0227 mm (pyramid), with normal and tilt errors of 0.0000 and yaw below 0.0015°, under both solves.
- H3 failed under solve (a). At single scale, the in-plane and yaw errors were met (0.0153 mm and −0.00577°) and the tilt fell from about 1° to 0.556°, inside its predicted range, but the normal error ended at 3.6936 mm, outside 3 ± 0.1259 mm. The pyramid run diverged, ending 672 mm in-plane and 727 mm along the normal from the truth, so all three parts failed there. Under (b), H3 held.
- H4 held at : from the large offset the pyramid reached 0.0394 mm in-plane and −0.00104° yaw, while the single scale stopped after five iterations 78.9 mm from the truth. H4 failed at : neither the pyramid (31.2 mm in-plane) nor the single scale (55.0 mm) met the bars.
Read together, the printed rows show the division of labour Park et al. describe in §5, on our port and scene: under solve (a), the geometric term alone fixed the normal offset and tilt and left the in-plane pose near where it started, and the default weight brought all four error components to within a few hundredths of a millimetre and a few thousandths of a degree from the small offset, and from the large offset only with the pyramid.
Ecosystem and Integrations
The legacy API exchanges data with NumPy through open3d.utility.Vector3dVector and numpy.asarray, and the tensor API through Tensor.from_numpy (shared memory) and DLPack, which the tensor tutorial describes for PyTorch. NumPy arrays are therefore a bridge to scikit-image, OpenCV and learning frameworks. The README links Open3D-ML for 3D machine learning and a standalone Open3D-Viewer application for Debian and Ubuntu, macOS and Windows, and gives two CMake example projects for using the C++ library from another project. The v0.20 release notes add USD and USDZ import (experimental), a vcpkg build option and WebRTC remote visualization improvements. Evaluation code uses it too: the Tanks and Temples script and the 3D foundation model evaluations described in those articles.
Alternatives
- The Point Cloud Library (PCL), a C++ library whose PCD format Point Cloud describes, whose FPFH features Point Cloud Registration ports and whose prerejective RANSAC it describes. Its normals are oriented toward a viewpoint, unlike Open3D’s.
- COLMAP (COLMAP) reconstructs camera poses and dense points from images; Open3D starts from point clouds, meshes or RGB-D frames rather than unordered photographs.
- ROS (ROS) is robotics middleware that passes data between components as messages; REP 2000, as ROS describes, lists PCL among each distribution’s dependencies, so it sits beside a point-cloud library rather than replacing one.
- Implementations written for one method, such as Kazhdan’s PoissonRecon, whose code Open3D wraps according to Poisson Surface Reconstruction, or PCL’s KinectFusion implementation, KinFu, for which TUM RGB-D gives a result.
Versions and Compatibility
As of November 2026, v0.20.0, released on 16 September 2026, is the newest release in the repository’s release list, after v0.19.0 (8 January 2025) and v0.18.0 (5 January 2024). Its getting-started page lists Python 3.10 to 3.14 on Ubuntu 22.04 or later, macOS 10.15 or later and 64-bit Windows 10 or later, with three pip packages: open3d (with CUDA on x86_64 Linux), open3d-cpu (a smaller CPU-only wheel on x86_64 Linux, since v0.17) and open3d-xpu (Intel GPUs on x86_64 Windows and Linux, since v0.20), plus a preview Windows CUDA wheel. The README at the same tag gives Ubuntu 20.04 or later instead of 22.04. NumPy 2 needs Open3D newer than 0.18.0, according to the same page.
The v0.20 release notes list these changes, among others: Gaussian splat rendering; many more tensor pipelines on Intel GPUs through SYCL, registration and odometry among them; a wheel about 335 MB smaller, with Open3D-ML’s Python dependencies moved to the open3d[ml] extra; Python type information; symmetric ICP, 3D NDT and point-cloud smoothing filters; and new Poisson parameters, which Poisson Surface Reconstruction covers. The breaking changes: OpenMP was removed in favour of oneTBB, so OMP_NUM_THREADS no longer controls Open3D’s threads and open3d.utility.set_max_threads() does; macOS builds are arm64 only; and Linux headless rendering uses EGL instead of OSMesa. Tanks and Temples documents how an older pinned version, 0.9, constrains the benchmark’s Python versions.
Limitations
- Normals are not oriented beyond agreeing with normals the cloud already has. Any method that needs consistent signs, such as Poisson reconstruction, needs an explicit orientation call.
- Some defaults sit inside the code. The colour-gradient neighbourhood of colored ICP (
max_correspondence_distance, 30 points) cannot be set from the API, and the hybrid search cuts atmax_nnbefore the radius. - Reported registration quality is geometric. Fitness and inlier RMSE measure point distances for every estimator; in our reading they therefore cannot reveal an in-plane slide on a flat surface, where every source point keeps a close neighbour. In the example, the rows at the small offset under solve (a) end with a fitness of 0.987 and an inlier RMSE of 5.0 mm while 7.2 mm of in-plane offset remain.
- The default solve reports no failure. With the default flags,
SolveLinearSystemPSDreturns success for a singular system, so a degenerate scene does not stop ICP or warn; whether Eigen’s LDLT then returns a minimum-norm or another solution was not read here, and the example’s (a) rows use NumPy’s minimum-norm solution as a stand-in. - Two APIs. Legacy and tensor types are separate classes with conversions between them, and the tutorials mark the legacy pipelines as legacy, so code written against one does not run on the other unchanged.
Related
- Point Cloud Registration
Estimating the transformation that aligns two or more point clouds of the same object or scene, by local refinement such as ICP, by global methods built on 3D feature correspondences, by globally optimal search, or by probabilistic and non-rigid models, with the benchmarks and metrics used to compare them.
- Voxel Grid
A regular 3D grid of small cubes that stores occupancy, signed distance or other values per cell, and the dense, octree, hashed and VDB structures that make such grids affordable at high resolution.
- KinectFusion
A real-time dense RGB-D SLAM system that fuses every depth frame into a TSDF volume on the GPU and tracks the camera by point-to-plane ICP against a surface ray cast from that volume.
- TSDF Fusion
Fusing many depth maps with known camera poses into one voxel grid of truncated signed distances by a per-voxel weighted average, then extracting the surface at the zero level set.
- Poisson Surface Reconstruction
A method that turns an oriented point cloud into a watertight mesh by finding the indicator function whose gradient best matches the point normals, which reduces to solving a Poisson equation.
- Tanks and Temples
A benchmark of 14 large indoor and outdoor scenes, recorded as 4K video and scanned with an industrial laser scanner, that scores complete image-based 3D reconstruction pipelines by precision, recall and F-score at a per-scene distance threshold.
References
- Open3D contributors (2026). README.md and LICENSE (MIT), Open3D v0.20.0 source code (commit b6c5e19).
- Open3D contributors (2026). Getting started, docs/getting_started.in.rst, Open3D v0.20.0 source code.
- Open3D (2026). Open3D v0.20 release notes, GitHub release v0.20.0, published 16 September 2026 (read through the GitHub releases API).
- Open3D contributors (2026). Colored point cloud registration, docs/jupyter/pipelines/colored_pointcloud_registration.ipynb, Open3D v0.20.0 source code.
- Open3D contributors (2026). Point cloud tutorials (legacy and tensor), docs/jupyter/geometry/pointcloud.ipynb and docs/jupyter/t_geometry/pointcloud.ipynb; Tensor, docs/jupyter/core/tensor.ipynb; tutorial indexes docs/tutorial/geometry/index.rst and docs/tutorial/pipelines/index.rst, Open3D v0.20.0 source code.
- Open3D contributors (2026). Colored ICP and robust kernels, cpp/open3d/pipelines/registration/ColoredICP.cpp, ColoredICP.h, RobustKernel.cpp and RobustKernel.h, Open3D v0.20.0 source code.
- Open3D contributors (2026). ICP loop and convergence criteria, cpp/open3d/pipelines/registration/Registration.cpp and Registration.h, Open3D v0.20.0 source code.
- Open3D contributors (2026). Linear solves and pose updates, cpp/open3d/utility/Eigen.cpp and Eigen.h, Open3D v0.20.0 source code.
- Open3D contributors (2026). Neighbour search, cpp/open3d/geometry/KDTreeFlann.cpp and KDTreeSearchParam.h, Open3D v0.20.0 source code.
- Open3D contributors (2026). Normals, downsampling and covariances, cpp/open3d/geometry/EstimateNormals.cpp, PointCloud.cpp and PointCloud.h, Open3D v0.20.0 source code.
- Open3D contributors (2026). Python bindings, cpp/pybind/geometry/pointcloud.cpp, cpp/pybind/pipelines/registration/registration.cpp, and tensor colored ICP, cpp/open3d/t/pipelines/registration/TransformationEstimation.h, Open3D v0.20.0 source code.
- Park, J., Zhou, Q.-Y. & Koltun, V. (2017). Colored Point Cloud Registration Revisited. IEEE International Conference on Computer Vision (ICCV).
- Zhou, Q.-Y., Park, J. & Koltun, V. (2018). Open3D: A Modern Library for 3D Data Processing. arXiv:1801.09847. (not read)