NYU Depth v2

NYU Depth v2 is a dataset of indoor scenes recorded with the RGB and depth cameras of a Microsoft Kinect. Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus released it with their ECCV 2012 paper on indoor segmentation and support inference. It has two parts: 1,449 frames with aligned color, depth maps, and dense per-pixel object labels, and the raw video they were chosen from. It is now best known as the standard indoor benchmark for single-image depth estimation, and as an RGB-D benchmark for semantic segmentation.

Purpose

Silberman et al. (2012) wanted a physical parse of cluttered, lived-in rooms: floor, walls, supporting surfaces, and objects, with the support relations between them (a cup on a table, a picture on a wall). The paper found existing Kinect datasets too limited: an earlier NYU indoor dataset (Silberman and Fergus, 2011) had only 67 scenes, and another labeled only a few objects per scene.

Dataset Size

The official page lists, as of October 2026:

• 1,449 densely labeled pairs of aligned RGB and depth images, at 640 × 480 (per the paper).
• 464 scenes from 3 cities. The paper adds: commercial and residential buildings in three US cities, in 26 scene classes.
• 407,024 unlabeled frames of raw video. The paper gives a different count: it says the labeled frames were hand-selected from 435,103 video frames.

The labeled subset is one MATLAB file of about 2.8 GB; the raw dataset is about 428 GB (as of October 2026).

Classes

Labels were collected per pixel on Amazon Mechanical Turk, with an instance number for each object (cup 1, cup 2). The paper counts 35,064 objects in 894 classes, so work on the dataset uses smaller sets:

• 4 structure classes, defined in the paper for support reasoning: ground, furniture (large objects that cannot be carried), props (small movable objects), and structure (walls, ceiling, columns).
• 40 classes, the most common ones, chosen by Gupta, Arbeláez, and Malik (2013): structure such as walls, floors, and doors; furniture such as beds, chairs, and tables; and objects such as lamps, bags, and boxes.
• 13 classes, a set from Couprie et al., also used in later work (Eigen and Fergus, 2015).

Annotation Types

The labeled MATLAB file holds, per frame:

• images: the RGB image.
• rawDepths: the Kinect depth projected onto the RGB image plane, converted to meters, with missing values left as missing.
• depths: the same depth with holes filled by the colorization method of Levin et al. (2004), according to the official page. The toolbox version solves a sparse linear system that pulls each pixel’s depth toward a weighted average of its eight neighbors’, with larger weights between pixels of similar gray level, and pulls measured pixels toward their measurements.
• labels (0 means unlabeled) and instances, with names and a name-to-ID map.
• Scene names and scene types, accelerometer readings (roll, yaw, pitch, and tilt), and the file names of the raw frames used.

The paper adds manual support annotations for all 1,449 images: triples of supported region, supporting region, and type (from below or from behind).

The raw dataset holds the unprocessed RGB (.ppm), depth (.pgm), and accelerometer dumps, with timestamps in the file names. RGB and depth run at 20 to 30 frames per second, not synchronized, and the raw depth is not yet projected onto the RGB image.

The official MATLAB toolbox aligns and processes these. Its camera_params.m holds the Kinect calibration: RGB focal lengths of about 518.9 and 519.5 pixels with principal point (325.6, 253.7), the depth camera’s intrinsics, distortion, and the transform between the cameras. depth_rel2depth_abs.m converts raw Kinect values $d$ to meters as $Z = 351.3 / (1092.5 - d)$, clipped at 10 m. Two functions fill holes (by colorization or a multi-scale cross-bilateral filter), and crop_image.m keeps the 427 × 561 rectangle where projected depth exists.

Splits

The standard split, distributed as splits.mat from the paper’s project page, has 795 training and 654 test frames. Eigen, Puhrsch, and Fergus (2014) describe it as 249 training and 215 test scenes, together the 464 scenes. There is no official validation set.

Tasks

• Support inference and structure classes, the paper’s own tasks.
• Monocular depth estimation: depth from the RGB image alone.
• RGB-D semantic segmentation: labeling pixels with 4, 13, or 40 classes from color and depth.
• Surface normal estimation: the orientation of the surface at every pixel.

Evaluation

The paper. Silberman et al. scored the fraction of labeled regions whose support is correct; with the support type required to match, their linear-programming solution reached 72.6% on ground-truth regions and 54.5% on their own segmentation. The official page notes that a bug inflated the paper’s segmentation scores by about 2%, and lists corrected values.

Monocular depth. Eigen et al. (2014) set the protocol much later work follows: train on raw frames from the training scenes, mask pixels without depth in the training loss, and test on the official test images against the filled-in depth. Their text states a “694-image” test set; the official split has 654 test images, and later papers report results on 654 (Bhat et al., 2021). Eigen and Fergus (2015) restrict evaluation to the axis-aligned rectangle where the projected depth is valid, as the toolbox’s crop does; later papers call it the center crop of Eigen et al. and treat 10 m as the upper bound of depth (Bhat et al., 2021). The metrics are the threshold accuracies $\delta < 1.25^k$, absolute relative error, and RMSE, defined in depth estimation; many papers add the mean $\log_{10}$ error (Bhat et al., 2021).

Segmentation. Gupta et al. (2013) scored the 40-class task with the per-class Jaccard index (intersection over union), aggregated mainly by its pixel-frequency-weighted average, and also reported the plain mean and pixel accuracy.

Surface normals. Ground-truth normals come from depth; Eigen and Fergus (2015) used two versions: Silberman et al.’s method, which fits least-squares planes to neighboring 3D points, and the smoother ground truth of Ladicky et al. They scored mean and median angular error and the fraction of pixels within 11.25°, 22.5°, and 30°.

Simulating the evaluation

The code ray-casts a synthetic room with a table, a cabinet, and a stool through the toolbox’s RGB intrinsics and imitates Kinect artifacts with a fixed seed: noise in the toolbox’s raw units, rounded to whole units; projector shadows beside depth edges (7.5 cm projector offset); and no return from a dark TV screen, a window onto a building 8 m away, and random pixels. The room, noise level, offset, dropout rate, and border width are our choices, not Kinect specifications. A simplified colorization-style solve fills the holes; two predictions are scored: a perfect one (oracle) and one with smoothed edges (blurry).

import numpy as np
from scipy.ndimage import gaussian_filter
from scipy.sparse import coo_matrix
from scipy.sparse.linalg import spsolve

H, W = 480, 640
fx, fy, cx, cy = 518.86, 519.47, 325.58, 253.74   # RGB intrinsics, toolbox camera_params.m (rounded)
A_RAW, B_RAW, MAX_DEPTH = 351.3, 1092.5, 10.0     # depth_rel2depth_abs.m: Z = 351.3 / (1092.5 - raw)

# Rays through every pixel, camera looking along +Z, y pointing down.
u, v = np.meshgrid(np.arange(W), np.arange(H))
x_hat, y_hat = (u - cx) / fx, (v - cy) / fy        # ray (x_hat, y_hat, 1): ray parameter = depth Z

def box_hit(lo, hi):
    """Depth at which each pixel's ray enters an axis-aligned box (inf if missed)."""
    d = np.stack([x_hat, y_hat, np.ones_like(x_hat)])
    with np.errstate(divide="ignore", invalid="ignore"):
        t1 = (np.array(lo)[:, None, None]) / d
        t2 = (np.array(hi)[:, None, None]) / d
    t_near = np.nanmax(np.minimum(t1, t2), axis=0)
    t_far = np.nanmin(np.maximum(t1, t2), axis=0)
    return np.where((t_far >= t_near) & (t_near > 0), t_near, np.inf)

# A 4 m x 2.6 m room, back wall 4.5 m away, floor 1.2 m below the camera.
with np.errstate(divide="ignore"):
    walls = [np.where(x_hat > 0, 2.0 / x_hat, np.inf), np.where(x_hat < 0, -2.0 / x_hat, np.inf),
             np.where(y_hat > 0, 1.2 / y_hat, np.inf), np.where(y_hat < 0, -1.4 / y_hat, np.inf),
             np.full((H, W), 4.5)]
furniture = [box_hit((-1.6, 0.45, 2.2), (-0.4, 1.2, 3.0)),   # table
             box_hit((0.6, -0.6, 3.3), (1.6, 1.2, 4.0)),     # cabinet
             box_hit((-0.3, 0.7, 1.6), (0.2, 1.2, 2.0))]     # low stool
Z_true = np.min(np.stack(walls + furniture), axis=0)
on_back_wall = Z_true == 4.5
window = on_back_wall & (np.abs(x_hat * 4.5 + 1.1) < 0.4) & (np.abs(y_hat * 4.5 + 0.5) < 0.4)
Z_true[window] = 8.0                               # glass: the true surface is a building 8 m away

# Grey image: flat shading, a bright window, and a dark, glossy TV screen on the back wall.
gray = 0.35 + 0.4 * np.minimum(Z_true, 4.5) / 4.5
screen = on_back_wall & (np.abs(x_hat * 4.5 - 0.0) < 0.4) & (np.abs(y_hat * 4.5 + 0.2) < 0.3)
gray[window], gray[screen] = 0.95, 0.05

def kinect(Z, seed, baseline=0.075, sigma_raw=0.5, dropout=0.01):
    rng = np.random.default_rng(seed)
    # Shadows: the projector sits beside the IR camera; a pixel is dark when a nearer
    # surface on its row hides it from the projector (projector column u - f*b/Z).
    u_proj = u - fx * baseline / Z
    hidden = np.minimum.accumulate(u_proj[:, ::-1], axis=1)[:, ::-1]
    shadow = np.zeros_like(Z, dtype=bool)
    shadow[:, :-1] = hidden[:, 1:] < u_proj[:, :-1] - 0.5
    # Raw disparity-like units: noise and integer rounding, then back to metres.
    raw = np.round(B_RAW - A_RAW / Z + rng.normal(0, sigma_raw, Z.shape))
    D = np.clip(A_RAW / (B_RAW - raw), 0, MAX_DEPTH)
    missing = shadow | screen | window | (rng.random(Z.shape) < dropout)
    missing[:, :30] = True                       # border the projected depth does not cover
    D[missing] = 0.0                             # 0 marks "no measurement"
    return D

def fill(D, gray, beta=200.0):
    """Fill zeros by intensity-weighted diffusion (simplified colorization-style solve)."""
    hole = D == 0
    idx = -np.ones(D.shape, int); idx[hole] = np.arange(hole.sum())
    rows, cols, vals, rhs = [], [], [], np.zeros(hole.sum())
    hv, hu = np.nonzero(hole)
    diag = np.zeros(hole.sum())
    for dv, du in [(0, 1), (0, -1), (1, 0), (-1, 0)]:
        nv, nu = hv + dv, hu + du
        ok = (nv >= 0) & (nv < H) & (nu >= 0) & (nu < W)
        p, nv, nu = np.nonzero(ok)[0], nv[ok], nu[ok]
        w = np.exp(-beta * (gray[hv[p], hu[p]] - gray[nv, nu]) ** 2) + 1e-3
        diag[p] += w
        q = idx[nv, nu]
        unk = q >= 0
        rows += [p[unk]]; cols += [q[unk]]; vals += [-w[unk]]
        np.add.at(rhs, p[~unk], w[~unk] * D[nv[~unk], nu[~unk]])
    A = coo_matrix((np.concatenate(vals + [diag]),
                    (np.concatenate(rows + [np.arange(hole.sum())]),
                     np.concatenate(cols + [np.arange(hole.sum())])))).tocsr()
    F = D.copy(); F[hole] = spsolve(A, rhs)
    return F

def metrics(pred, gt, mask):
    p, g = np.clip(pred[mask], 1e-3, MAX_DEPTH), gt[mask]
    r = np.maximum(p / g, g / p)
    return np.mean(np.abs(p - g) / g), np.sqrt(np.mean((p - g) ** 2)), np.mean(r < 1.25)

crop = np.zeros((H, W), bool); crop[44:471, 40:601] = True   # toolbox get_projection_mask (1-based 45:471, 41:601)
oracle = Z_true                                  # a perfect predictor
blurry = gaussian_filter(Z_true, 6)              # a stand-in for a prediction with smoothed edges

def run(seed, verbose=False):
    D = kinect(Z_true, seed)
    F = fill(D, gray)
    valid = D > 0
    masks = {"raw, valid in crop": (D, valid & crop), "filled, whole crop": (F, crop),
             "filled, holes only": (F, crop & ~valid)}
    out = {(p, k): metrics(pred, gt, m) for p, pred in [("oracle", oracle), ("blurry", blurry)]
           for k, (gt, m) in masks.items()}
    if verbose:
        print(f"crop {crop[:, 40].sum()} x {crop[44].sum()}, valid pixels in crop {valid[crop].mean():.1%}")
        for lo, hi in [(1.5, 2.0), (4.0, 4.5)]:
            sel = valid & (Z_true >= lo) & (Z_true <= hi)
            print(f"raw depth RMS error, true depth {lo}-{hi} m: {np.sqrt(np.mean((D - Z_true)[sel] ** 2)) * 1000:.1f} mm")
        e = np.abs(F - Z_true)[crop & ~valid]
        print(f"filled depth in holes: mean |error| {e.mean() * 100:.1f} cm, max {e.max():.2f} m")
        for (p, k), (a, r, d1) in out.items():
            print(f"{p} vs {k:19s} AbsRel {a:.4f}  RMSE {r:.3f} m  d1 {d1:.3f}")
    return out

run(0, verbose=True)
s = np.array([[run(seed)[(p, "filled, holes only")][0] for p in ("oracle", "blurry")] for seed in range(10)])
print("seeds 0-9, AbsRel in holes: oracle %.3f-%.3f, blurry %.3f-%.3f"
      % (s[:, 0].min(), s[:, 0].max(), s[:, 1].min(), s[:, 1].max()))

Output:

crop 427 x 561, valid pixels in crop 92.8%
raw depth RMS error, true depth 1.5-2.0 m: 5.3 mm
raw depth RMS error, true depth 4.0-4.5 m: 32.7 mm
filled depth in holes: mean |error| 173.3 cm, max 3.50 m
oracle vs raw, valid in crop  AbsRel 0.0050  RMSE 0.026 m  d1 1.000
oracle vs filled, whole crop  AbsRel 0.0325  RMSE 0.661 m  d1 0.964
oracle vs filled, holes only  AbsRel 0.3855  RMSE 2.459 m  d1 0.505
blurry vs raw, valid in crop  AbsRel 0.0154  RMSE 0.150 m  d1 0.994
blurry vs filled, whole crop  AbsRel 0.0394  RMSE 0.620 m  d1 0.958
blurry vs filled, holes only  AbsRel 0.3475  RMSE 2.242 m  d1 0.505
seeds 0-9, AbsRel in holes: oracle 0.383-0.388, blurry 0.346-0.350

Against raw depth masked to valid pixels in the crop, the perfect prediction scores only the sensor noise: AbsRel 0.0050 and RMSE 2.6 cm. Against the filled depth over the whole crop, the same prediction scores an RMSE of 0.661 m, because the fill continued the wall across the window and the scorer now counts the true 8 m as an error. The blurry prediction even beats it on RMSE (0.620 m), since blurring pulls its window values toward the wall. Over seeds 0 to 9, the perfect prediction’s AbsRel in the holes stays between 0.383 and 0.388. The noise grows with distance, from 5.3 mm at 1.5 to 2 m to 32.7 mm at 4 to 4.5 m. By our own reasoning from the conversion formula, one raw unit spans $Z^2 / 351.3$ meters: about 3 mm at 1 m and 28 cm at 10 m. On the real dataset, the size of the effect depends on how many test pixels are filled.

License

As of October 2026, the official page asks users to cite the ECCV 2012 paper but states no license or terms of use. The license is therefore unclear, so check with the authors before commercial use.

Limitations and Bias

• Kinect v1 depth. The first Kinect is a structured-light sensor (depth estimation). The official page attributes missing depth to shadows from the offset between the infrared emitter and camera, and to specular or low-albedo surfaces; Eigen et al. (2014) add windows and open doorways. Song et al. (2015) note visible quantization in Kinect v1 depth.
• Filled depth is not a measurement. The depths field is interpolated in every hole, and the standard benchmark scores against it. Silberman et al. themselves weighted inpainted pixels at 0.25 of measured ones in their plane segmentation. Evaluating on rawDepths with a validity mask avoids this, at the cost of comparability with published numbers.
• Indoor scale. Depth is capped at 10 m, so the dataset says little about outdoor or long-range depth. A model trained only here sees one camera’s focal length and indoor distances, which, by our reasoning, ties its metric scale to that setting.
• Label noise. Crowd-sourced labels over 894 raw classes make names, boundaries, and the 40-class mapping imperfect (our reasoning).
• A small test set. 654 test frames from 215 scenes are few, so small differences between methods may not be meaningful (our reasoning).
• Narrow capture. Three US cities, one sensor, and a single calibration file.

Common Uses

• Training and testing monocular depth networks, from Eigen et al. (2014) onward, and, with KITTI, fine-tuning metric depth models such as ZoeDepth (depth estimation).
• RGB-D semantic segmentation on the 40-class set, and surface normal prediction.
• Dense 3D reconstruction of the raw sequences (3D reconstruction); ScanNet provides reconstructions and annotations of all NYU v2 sequences (Dai et al., 2017).

Historical Milestones

• 2012. The dataset and the support inference task (Silberman et al., 2012).
• 2013. Gupta et al. defined the 40-class task and used the 795/654 split for contours and segmentation.
• 2014. Eigen et al. trained a two-scale network on the raw data and reported $\delta < 1.25$ of 0.611, AbsRel 0.215, and RMSE 0.907 m for the full model (Table 1).
• 2015. Eigen and Fergus predicted depth, normals, and labels with one architecture; their VGG model reported $\delta < 1.25$ of 0.769, AbsRel 0.158, and RMSE 0.641 m. Their Table 1 lists the 2014 model with slightly different values (0.614, 0.214, 0.877).

Related Datasets

SUN RGB-D (Song et al., 2015) gathers 10,335 RGB-D images from four sensors, including the 1,449 NYU frames, whose 2D segmentation it reuses while redoing the 3D annotation. ScanNet (Dai et al., 2017) moved to whole-room 3D: 2.5 million RGB-D views in 1,513 scans of 707 spaces, captured with a Kinect-v1-like Structure sensor, with labels mapped to the NYU v2 categories among others. KITTI is the outdoor counterpart for depth benchmarks.

Resources

• NYU Depth Dataset V2: the official page, with the labeled file, the raw parts, the toolbox, and the corrected segmentation results.
• Project page (archived): the train/test split, the support labels, and the paper’s code.