Surgical Computer Vision

Surgical computer vision analyses the video recorded during operations: the view of the endoscope in laparoscopic and robot-assisted surgery, and of the microscope in eye surgery. It recognizes which instruments are in view and where, which phase or step of the procedure is under way, which actions are performed, and how skilfully, and it locates anatomy relevant to safety. The work is also called surgical workflow analysis or surgical video analysis, and it forms the video-based core of the wider field its community calls surgical data science, which covers the capture, organization, analysis and modelling of data from interventional care (Maier-Hein et al., 2022).

Medical Computer Vision places surgical video among the medical image types, and Action Recognition describes how models classify what happens in a video clip. This article covers what is specific to surgery: the data, the tasks, the public datasets and challenges with their protocols, temporal models in outline, how results are scored, and what the evidence on use in the operating room consists of. It describes methods, evidence and limits for practitioners and researchers, gives no clinical guidance, and attributes every reported result to its authors with its set-up.

Problem

In minimally invasive surgery the surgeon operates through small incisions and sees the field only on a monitor fed by an endoscope, so the whole visible procedure is already a video stream. Recognizing the surgical workflow is motivated by its authors as a basis for context-aware systems that monitor the procedure, schedule the operating room and staff, assist the team, index video archives, and, in the long run, warn of complications (Twinanda et al., 2017). Most of these aims are stated as motivation; the evidence section below shows how few have been tested in operations.

Two settings define the problem. An online system sees only the frames up to the present, as an assistant in the operating room must; an offline system sees the whole recording, as a tool for documentation, indexing or post-operative feedback can (Wagner et al., 2023; Özkurt et al., 2026). The same model family often comes in both forms, and the results differ: EndoNet’s phase recognition on Cholec80 reached 92.0% accuracy offline and 81.7% online with the same features (Twinanda et al., 2017, Table IV).

Visual Data

Laparoscopic video. The laparoscope moves throughout the operation, which causes motion blur and large changes in the scene, and blood can stain the lens and blur or hide the view (Twinanda et al., 2017). CholecTrack20’s annotators labelled eight kinds of visual challenge: blurring, bleeding, lens fouling, crowded scenes, occlusion, smoke, specular reflection and views near the trocar (Nwoye et al., 2025; the paper’s Fig. 1 also shows blood stains as a ninth). Recording conditions vary between hospitals even for one operation: the 33 HeiChole videos came from two camera systems with 30° optics, at 960 × 540 pixels and 25 frames per second in one centre and mostly 1920 × 1080 at 50 frames per second in the others, with three videos at 720 × 576 (Wagner et al., 2023). The instruments differ too: in HeiChole an argon beamer appears only in videos from one hospital and a crocodile grasper only in those from the other two.

Frames outside the patient. When the camera is withdrawn, for cleaning for example, it can film the operating room, the staff and the patient. HeiChole’s organizers replaced such frames with white frames before release (Wagner et al., 2023), and CholecTrack20’s checked and anonymized them (Nwoye et al., 2025).

Frame rate in practice. Recordings are often 25 frames per second, but most work processes 1 frame per second: Cholec80’s videos were downsampled to 1 fps for processing (Twinanda et al., 2017), and CholecT50 and CholecTrack20 are annotated at 1 fps (CAMMA datasets page; Nwoye et al., 2025).

Robot-assisted surgery. A surgical robot can record the motion of its instruments as well as the video. The JIGSAWS dataset, recorded on a da Vinci system at Johns Hopkins University, provides kinematic data read from the robot’s interface together with video, gesture labels and skill ratings (Lefor et al., 2020). It shows bench-top training tasks, not patients (Wagner et al., 2023).

Microscope video. In cataract surgery the camera of the operating microscope is static and the instruments are not articulated, unlike in cholecystectomy, which is also longer and has smoke and rapid camera motion (Twinanda et al., 2017).

Flexible endoscopy of the gastrointestinal tract, where vision systems detect lesions such as polyps, is a related application with its own literature and is not treated here.

Vision Tasks Involved

Instrument presence, detection and segmentation

Presence. The simplest instrument task asks which tools are visible in a frame, without locating them. Cholec80 counts a tool as present when at least half of its tip is visible (Twinanda et al., 2017). HeiChole marks an instrument as visible as soon as its tip appears and keeps the label while only its shaft remains in view; a shaft that enters without a visible tip is labelled an “undefined instrument shaft”, except for instruments with characteristic shafts (Wagner et al., 2023). Presence labels are cheap to produce, and EndoNet’s authors found them sufficient to train a network: on the 40 evaluation videos of Cholec80, EndoNet reached a mean average precision of 81.0 over seven tools, against 58.4 for a deformable part model trained on bounding boxes, with scissors the hardest tool (58.6), which the authors suggest may be because it has the least training data, appearing only briefly, and resembles the grasper (Twinanda et al., 2017, Table II).

Segmentation. Pixel masks of instruments are needed for overlays, for robot control and for measuring tool–tissue interaction. The task is semantic segmentation when only the instrument class matters, and instance segmentation when two graspers must be told apart. Alabi et al. (2025) note that most standard datasets for tool segmentation come from porcine surgery and that few provide instance labels; their comparison is summarized in the dataset table below.

Instrument tracking

Tracking links instrument detections over time (Multi-Object Tracking covers the general problem and its metrics). Surgery adds a problem of definition: a tool leaves the camera’s field of view while staying in the body, or is withdrawn and later reinserted. CholecTrack20 therefore annotates three kinds of trajectory for each tool: a visibility trajectory that ends when the tool leaves the field of view, an intracorporeal trajectory that lasts from entry into the body to exit through a trocar, even off camera, and an intraoperative trajectory that follows a tool from its first appearance to its last in the whole procedure, which requires re-identification after reinsertion (Nwoye et al., 2025). Identity is assigned with clinical context, such as the hand of the operator and the trocar port that holds the tool.

Phase, step and action recognition

Surgical activity is described at several levels of detail. Phases are the major stages of an operation, such as clipping and cutting; steps are finer processes within a phase, such as clipping the cystic duct; and activities are the smallest units, such as placing the clip applier on the duct (Özkurt et al., 2026, following the literature they review). Phase recognition assigns one phase to every frame of a full-length video, which in general video terms is temporal action segmentation, a task that Video Understanding lists alongside the instance-level search of Temporal Action Localization. Phases last minutes, follow a largely fixed order, and cover the whole video, so models exploit order and duration as well as appearance.

Cholec80’s phases. The seven phases of laparoscopic cholecystectomy (removal of the gallbladder) in Cholec80 were defined by a senior surgeon at the partner hospital (Twinanda et al., 2017, Table I-a):

Phase	Mean ± SD duration (s)
P1 Preparation	125 ± 95
P2 Calot triangle dissection	954 ± 538
P3 Clipping and cutting	168 ± 152
P4 Gallbladder dissection	857 ± 551
P5 Gallbladder packaging	98 ± 53
P6 Cleaning and coagulation	178 ± 166
P7 Gallbladder retraction	83 ± 56

The order is linear except that P5 and P6 can alternate before P7 (Twinanda et al., 2017, Fig. 7). Two phases take most of the time, and several short phases are hard to separate at their boundaries.

Actions. HeiChole annotated four actions frame by frame, grasp, hold, cut and clip, with the hand that performed them (Wagner et al., 2023). CholecT50 describes fine-grained activity as triplets of instrument, verb and target: 50 videos annotated at 1 fps with 100 triplet classes built from 6 instruments, 10 verbs and 15 targets, scored mainly by the average precision of the full triplet (Nwoye et al., 2022).

Skill assessment

Video-based skill assessment predicts the ratings that expert observers give with structured scales. JIGSAWS rates each trial with a global rating scale, a modified form of the Objective Structured Assessment of Technical Skills with six items (respect for tissue, suture and needle handling, time and motion, flow of operation, overall performance and quality of the final product) scored from 1 to 5; its eight participants also classified themselves as novice (under 10 hours of robotic experience), intermediate (10–100 hours) or expert (over 100 hours) (Lefor et al., 2020). The two measures do not agree well: in Lefor et al.’s analysis, self-declared skill correlated with the rating only in knot-tying (Spearman $r = 0.55$), not in suturing or needle passing, so a model that predicts “expert” and one that predicts the rating answer different questions.

HeiChole rated each operation, and its Calot triangle and gallbladder dissection phases, with a modified Global Operative Assessment of Laparoscopic Skills (GOALS): depth perception, bimanual dexterity, efficiency and tissue handling, each 1 to 5, with operative difficulty as a fifth item. The organizers left out the autonomy item because the abdominal video alone cannot show how much help a senior surgeon gave (Wagner et al., 2023).

Anatomy and safety

A further group of tasks locates anatomy. In cholecystectomy, work has focused on the critical view of safety, a set of visual criteria on the hepatocystic anatomy: DeepCVS segments the anatomy and assesses the criteria, and other models mark zones of safe and dangerous dissection (as summarized by Maier-Hein et al., 2022). Tokuyasu et al. (2021) trained a YOLOv3 detector on about 2,000 images from 76 videos to outline four landmarks; on 23 unseen videos the average precision per landmark ranged from 0.074 (cystic duct) to 0.320 (common bile duct), while the two expert surgeons who annotated the data judged the indications valid in 22 of the 23 videos. These tasks use the segmentation and detection methods of other articles; the evidence section returns to how they were tested.

Datasets and Challenge Protocols

Public surgical datasets are few, small by the standards of general video, and concentrated on one operation. In a 2019 survey of 77 participants of a surgical data science workshop, EndoVis (28%), Cholec80 (21%) and JIGSAWS (17%) were named most often as the most useful public datasets, and the small size and limited representativeness of the data were named as a core issue (45%) (Maier-Hein et al., 2022).

Dataset	Content	Size	Labels
Cholec80	Laparoscopic cholecystectomy, Strasbourg	80 videos, 13 surgeons	7 phases, 7 tool presence
EndoVis 2015 workflow	Laparoscopic cholecystectomy, Munich	7 videos	Phases with other definitions
m2cai16-workflow	Laparoscopic cholecystectomy, Strasbourg and Munich	41 videos	Phases
HeiChole (EndoVis 2019)	Laparoscopic cholecystectomy, 3 German centres	33 videos (24 train, 9 test), 22 h	7 phases, 4 actions, 21 instruments in 7 categories, skill
CholecT50	Laparoscopic cholecystectomy	50 videos at 1 fps	100 instrument–verb–target triplets
CholecTrack20	Laparoscopic cholecystectomy	20 videos, over 35K frames, 65K tool boxes	Tracks in 3 perspectives
CholecInstanceSeg	Laparoscopic cholecystectomy	85 sequences, 41.9k frames, 64.4k instances	Instance masks
JIGSAWS	Bench-top robotic tasks	8 participants, 3 tasks	Kinematics, video, gestures, ratings

Sources: Twinanda et al. (2017); CAMMA datasets page; Wagner et al. (2023); Nwoye et al. (2022, 2025); Alabi et al. (2025); Lefor et al. (2020).

Cholec80. The 80 videos were recorded at the University Hospital of Strasbourg at 25 fps and labelled with the phase of every frame and the presence of seven tools at 1 fps (Twinanda et al., 2017; CAMMA datasets page). Its authors split it into a fine-tuning half and an evaluation half of 40 videos each, and evaluated phase recognition by 4-fold cross-validation on the evaluation half, averaging five runs because parts of the pipeline were randomly initialized. Later papers use other splits: Özkurt et al. (2026, Table 1) list 40/40, 32/8/40 and 40/8/32 (training/validation/test) among the 50 Cholec80 studies they reviewed. As of November 2026, the dataset is distributed under CC BY-NC-SA 4.0 (CAMMA datasets page). Several later datasets reuse its videos: CholecT50 includes 45 of them, and CholecInstanceSeg draws frames from Cholec80, CholecT50 and CholecSeg8k, all subsets of the group’s in-house collection of 120 videos (Alabi et al., 2025), so a model trained on one and tested on another may see the same operations; the group publishes the overlaps (CAMMA datasets page).

EndoVis 2015 workflow and m2cai16. The seven videos of the 2015 Endoscopic Vision (EndoVis) workflow challenge at MICCAI were recorded at Klinikum rechts der Isar in Munich; their phases are defined differently from Cholec80’s, with a trocar-placement phase that is not always visible from the laparoscope and a preparation phase that includes Calot triangle dissection (Twinanda et al., 2017, Table I-b). The m2cai16-workflow dataset of the M2CAI 2016 challenge has 41 cholecystectomy videos from Strasbourg and Munich, and access is by request form (CAMMA datasets page).

HeiChole, the EndoVis 2019 workflow and skill sub-challenge. Its protocol shows how a surgical challenge is run (Wagner et al., 2023):

• Data. 33 videos from three centres, split randomly with stratification by centre into 24 training videos (12 from each of two centres) and 9 test videos (3 from each centre, including the third centre, absent from training). The training set was released in two parts in 2019 and the test set was not published.
• Labels. Phases were annotated by three medical students and skill by two, independently, following written rules, with disagreements resolved by consensus. Agreement on phases before consensus (Fleiss’ kappa per video) was 0.92 ± 0.06 on the training videos and 0.95 ± 0.04 on the test videos, with one video at 0.71.
• Submission. Teams submitted Docker containers that read a video and wrote predictions; the organizers ran them, which kept the test data private.
• Online only, on trust. Phase, action and instrument methods were not allowed to use future frames, but the rule was not enforced technically; the organizers relied on the teams’ honour.
• Metrics and ranking. F1 score per class and video, averaged over classes and videos so that each class and each video has equal weight; mean absolute error for skill; ranking stability by 1,000 bootstrap samples of the test cases and Kendall’s tau; pairwise Wilcoxon signed-rank tests with Holm correction.

The published results: instrument presence F1 from 38.5% to 63.8% across eight teams, action F1 from 21.8% to 23.3% across five, and skill mean absolute error 0.78 from the single team that entered (Wagner et al., 2023, abstract and Table 2). For phase the abstract gives F1 from 23.9% to 67.7% across nine teams, but the leaderboard (Table 2) does not contain 67.7%: the challenge entries range from 23.93% to 65.38%, and a submission received after the deadline (CAMMA, with pre-training) reached 68.78%. Two details matter more than the numbers. All action methods recognized hold, the action present during 73.3% of an operation, with F1 from 0.84 to 0.91, and averaged below 0.03 on every other action. And the skill entry gave every test video the same scores: 4 for depth perception, bimanual dexterity and efficiency, 3 for tissue handling and 2 for difficulty. Both are outcomes that a mean error or an averaged F1 alone would not reveal.

Instrument segmentation datasets. Alabi et al. (2025, Table 1) compare the main tool segmentation datasets. EndoVis 2015 is human ex vivo and in vivo, 8 procedures with 10k annotated frames and 3 classes; EndoVis 2017 and 2018 are porcine in vivo, 10 and 16 procedures with 2.4k frames each and 10 classes; ROBUST-MIS 2019 is human in vivo, 30 procedures with 10k frames, 2 classes and instance labels; CholecSeg8k, AutoLaparo and SAR-RARP50 are human data with semantic labels only; and CholecInstanceSeg is human in vivo, with instance labels; the table gives it 8 classes, while the text lists seven instrument classes, among them a snare, which is not one of Cholec80’s tools. ROBUST-MIS 2019 used the Heidelberg colorectal dataset, 30 videos of three kinds of laparoscopic colorectal surgery (Wagner et al., 2023). The official challenge pages and the EndoVis 2017 challenge report could not be read for this article; the figures are those of Alabi et al.

CholecTrack20. Its 20 videos are split by procedure into training, validation and test sets in the ratio 5:1:4, balanced by duration (Nwoye et al., 2025). Trackers were scored with HOTA, CLEAR MOT and identity metrics. On visibility trajectories the best tracker, Bot-SORT, reached 44.7% HOTA; on intracorporeal trajectories the best was 27.0% and on intraoperative ones 17.4% (Table 3). The text calls the intracorporeal perspective the most challenging and places the intraoperative one in the middle, which the table does not show: in Table 3 every tracker scores lowest on intraoperative trajectories. Tools with few instances, such as scissors and irrigator, were tracked worst.

JIGSAWS. Eight participants performed up to five trials each of suturing, knot-tying and needle passing on a da Vinci system; some trials were corrupted and are missing, leaving 39, 36 and 28 trials for the three tasks (Lefor et al., 2020, Table 1).

Common Methods

Temporal models are where surgical work departs most from frame-level image analysis. In outline, the families follow those of Temporal Modeling, applied to whole operations rather than short clips.

Per-frame features with a state model. EndoNet fine-tuned an ImageNet-pretrained AlexNet to predict phase and tool presence at once, passed its features to a support vector machine, and enforced the order of the workflow with a two-level hierarchical hidden Markov model, decoded by the Viterbi algorithm offline and by the forward algorithm online (Twinanda et al., 2017). On Cholec80 the hidden Markov model raised accuracy from 75.2% before it to 92.0% offline and 81.7% online (Tables III and IV). Multi-task training helped: features trained for phase and tools beat features trained for phase alone (PhaseNet, 89.1% offline).

Recurrent networks. In the 2019 HeiChole challenge the three best phase methods all combined a convolutional network with an LSTM. The organizers suggest that recurrence can carry information from any earlier frame, whereas a 3D convolution sees only as far back as its kernels and depth reach; the leading method, CAMMA’s submission after the deadline, merged both, with an I3D and an Inception-ResNet followed by LSTMs on each and on their combined features (Wagner et al., 2023). In their statistical models across methods only a temporal component showed a difference, and only among results with non-zero F1; given the wide uncertainty of the model, the organizers did not conclude that it improves phase recognition. The apparent effects of multi-task learning, optimizer and augmentation in descriptive plots were not confirmed.

Temporal convolutions. MS-TCN, introduced for action segmentation in general video, stacks stages of dilated temporal convolutions, each refining the previous stage’s frame-wise predictions, with a smoothing loss against over-segmentation (Abu Farha & Gall, 2019). TeCNO applied a causal version to phase recognition, which allows online inference with smooth predictions, and reported that it outperformed LSTM approaches on two cholecystectomy datasets (Czempiel et al., 2020, abstract). Golany et al. (2022) used a frozen ResNet-50 for per-frame features and an MS-TCN over the whole video; in this two-stage design, image features come first and a temporal model is trained on top, which in our reading keeps sequences as long as whole operations tractable.

Attention. Rendezvous recognizes action triplets with two levels of attention: a spatial attention guided by instrument activations to find verbs and targets, and a Transformer-inspired mix of self- and cross-attention to associate instrument, verb and target (Nwoye et al., 2022). The general designs are those of Video Transformers.

Instruments. Segmentation and tracking reuse the general architectures: encoder–decoder and instance segmentation networks, and tracking-by-detection trackers such as ByteTrack, which CholecTrack20 benchmarks (Nwoye et al., 2025).

Example: Phase Recognition, Smoothing and Event Metrics

The example builds a synthetic surgical-video benchmark. Each video is a sequence of 24 × 24 RGB frames at 1 frame per second, following Cholec80’s seven phases with durations drawn from gamma distributions that have Cholec80’s mean and standard deviation per phase; in 30% of videos packaging and cleaning alternate (P5, P6, P5, P6). A frame shows tissue whose colour depends weakly on the phase and drifts slowly, seven instruments drawn as bars whose presence follows a Markov chain with a phase-dependent probability (hook in the dissection phases, clipper and scissors in P3, specimen bag in P5 and P7, irrigator and bipolar in P6), smoke in the dissection phases, and a gain and colour cast per video, standing in for different cameras. By construction, Calot triangle dissection (P2) and gallbladder dissection (P4) use the same instruments and differ in tissue colour by 0.04 in one channel, less than the spread of the colour casts, much as the two phases look alike in real video.

A logistic regression on 12 per-frame features (mean colour, contrast, edge energy and the fraction of pixels near each instrument’s colour) is trained on the frames of 24 videos and tested on 16 others. Its per-frame probabilities are then decoded five ways: frame by frame; with a causal exponential moving average (online); with a centred 61-frame moving average (offline); and with a hidden Markov model whose transitions are counted from the training labels, by forward filtering (online) and by Viterbi decoding (offline). The two hidden Markov decoders are run with two estimates of the transition matrix: one adds a pseudo-count of one to every pair of phases (add-one), the other allows only the transitions seen in the training videos (seen-only). Each test video is scored by frame accuracy, by accuracy with the relaxed boundaries of the widely used Cholec80 evaluation script (described under Evaluation), by the number of predicted segments, by the edit score and by segmental F1 at overlaps of 10%, 25% and 50%, with the matching rules of the MS-TCN evaluation script (Abu Farha, 2019) but averaged per video, whereas that script pools segment matches over all test videos. The script runs five seeds and prints the mean and range over seeds of the per-video averages.

import numpy as np
from scipy.ndimage import uniform_filter1d
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Cholec80 phase durations in seconds, mean and std (Twinanda et al., Table I-a)
DUR = [(125, 95), (954, 538), (168, 152), (857, 551), (98, 53), (178, 166), (83, 56)]
TOOLS = ["grasper", "bipolar", "hook", "scissors", "clipper", "irrigator", "bag"]
COLOR = np.array([[.55, .55, .58], [.25, .35, .75], [.90, .90, .85], [.15, .15, .15],
                  [.75, .70, .40], [.42, .42, .45], [.30, .70, .35]])
# probability that each tool is in view, per phase (rows P1..P7)
P_TOOL = np.array([[.50, .02, .20, .02, .02, .02, .02], [.95, .05, .85, .02, .02, .03, .02],
                   [.90, .02, .10, .35, .60, .02, .02], [.95, .10, .85, .02, .02, .05, .02],
                   [.80, .02, .02, .02, .02, .02, .85], [.50, .50, .02, .02, .02, .70, .02],
                   [.70, .02, .02, .02, .02, .02, .60]])
TISSUE = np.array([.70, .28, .27]) + .04 * np.array(
    [[0, 0, 0], [-1, 0, 0], [0, 1, 0], [-1, 1, 0], [1, 1, -1], [0, -1, 1], [1, 0, 1]])
S = 24  # frame size in pixels

def mask_bank(rng, n=40):
    yy, xx = np.mgrid[:S, :S]
    bank = np.zeros((len(TOOLS), n, S, S), bool)
    for k, width in enumerate([2, 2, 1, 1.5, 2, 3, 3]):
        for j in range(n):
            x0, y0 = rng.choice([(0, rng.uniform(0, S)), (S, rng.uniform(0, S)), (rng.uniform(0, S), S)])
            ang, length = rng.uniform(0, 2 * np.pi), rng.uniform(10, 20)
            dx, dy = np.cos(ang), np.sin(ang)
            t = np.clip((xx - x0) * dx + (yy - y0) * dy, 0, length)
            bank[k, j] = np.hypot(xx - x0 - t * dx, yy - y0 - t * dy) < width
    return bank

def phase_sequence(rng):
    d = [max(10, int(rng.gamma((m / s) ** 2, s ** 2 / m))) for m, s in DUR]
    order = [0, 1, 2, 3, 4, 5, 6]
    if rng.random() < 0.3:  # cleaning sometimes interrupts packaging: P5, P6, P5, P6
        order = [0, 1, 2, 3, 4, 5, 4, 5, 6]
    return np.concatenate([np.full(d[p] // order.count(p), p) for p in order])

def render_video(rng, bank):
    y = phase_sequence(rng)
    T = len(y)
    drift = np.cumsum(rng.normal(0, .004, (T, 3)), 0) * .3
    frames = (TISSUE[y] + drift)[:, None, None, :] + rng.normal(0, .04, (T, S, S, 3))
    for k in range(len(TOOLS)):  # each tool: two-state Markov chain, mean visit 30 s
        p = P_TOOL[y, k]
        on = np.zeros(T, bool)
        for t in range(1, T):
            on[t] = rng.random() < (1 - 1 / 30 if on[t - 1] else p[t] / 30 / (1 - p[t]))
        m = bank[k, rng.integers(0, bank.shape[1], T)] & on[:, None, None]
        frames[m] = COLOR[k] + rng.normal(0, .03, (m.sum(), 3))
    smoke = np.zeros(T)
    for t in np.flatnonzero(np.isin(y, [1, 3]) & (rng.random(T) < .01)):
        smoke[t:t + rng.integers(8, 20)] = rng.uniform(.5, .8)
    frames = (1 - smoke)[:, None, None, None] * frames + smoke[:, None, None, None] * .85
    gain, cast = rng.uniform(.8, 1.2), rng.normal(0, .03, 3)  # camera differences
    return np.clip(frames * gain + cast, 0, 1), y

def features(frames):
    grey = frames.mean(-1)
    grad = np.abs(np.diff(grey, axis=1)).mean((1, 2)) + np.abs(np.diff(grey, axis=2)).mean((1, 2))
    near = [(np.linalg.norm(frames - c, axis=-1) < .12).mean((1, 2)) for c in COLOR]
    return np.column_stack([frames.mean((1, 2)), grey.std((1, 2)), grad] + near)

def segments(y):
    cut = np.flatnonzero(np.diff(y)) + 1
    starts, ends = np.r_[0, cut], np.r_[cut, len(y)]
    return y[starts], starts, ends

def edit_score(p, g):
    a, b = segments(p)[0], segments(g)[0]
    D = np.zeros((len(a) + 1, len(b) + 1))
    D[:, 0], D[0, :] = np.arange(len(a) + 1), np.arange(len(b) + 1)
    for i in range(1, len(a) + 1):
        for j in range(1, len(b) + 1):
            D[i, j] = min(D[i - 1, j] + 1, D[i, j - 1] + 1, D[i - 1, j - 1] + (a[i - 1] != b[j - 1]))
    return 100 * (1 - D[-1, -1] / max(len(a), len(b)))

def f1_at(p, g, thr):  # segmental F1 as in MS-TCN (Lea et al.)
    pl, ps, pe = segments(p)
    gl, gs, ge = segments(g)
    hit, tp = np.zeros(len(gl), bool), 0
    for l, s, e in zip(pl, ps, pe):
        inter = np.clip(np.minimum(e, ge) - np.maximum(s, gs), 0, None)
        iou = np.where(gl == l, inter / (np.maximum(e, ge) - np.minimum(s, gs)), 0)
        j = iou.argmax()
        if iou[j] >= thr and not hit[j]:
            hit[j], tp = True, tp + 1
    fp, fn = len(pl) - tp, len(gl) - tp
    return 100 * 2 * tp / (2 * tp + fp + fn)

def relaxed_acc(p, g, fps=1):  # 10 s rule of the Cholec80 evaluation script, as its comments intend
    diff, upd = p - g, (p - g).copy()
    for l, s, e in zip(*segments(g)):
        t, d = min(10 * fps, e - s), diff[s:e].copy()
        late = [-1, -2] if l in (5, 6) else [-1]   # phases 6, 7 (0-based 5, 6)
        early = [1, 2] if l in (3, 4) else [1]     # phases 4, 5 (0-based 3, 4)
        d[:t][np.isin(d[:t], late)] = 0
        d[-t:][np.isin(d[-t:], early)] = 0
        upd[s:e] = d
    return 100 * np.mean(upd == 0)

def decode(prob, A, prior, method):
    if method == "per-frame":
        return prob.argmax(1)
    if method == "EMA (online)":
        out = prob.copy()
        for t in range(1, len(out)):
            out[t] = .9 * out[t - 1] + .1 * prob[t]
        return out.argmax(1)
    if method == "moving avg (offline)":
        return uniform_filter1d(prob, 61, axis=0, mode="nearest").argmax(1)
    e = prob / prior
    if method == "HMM forward (online)":
        a, out = e[0] * prior, []
        for t in range(len(e)):
            a = (a @ A if t else a) * (e[t] if t else 1)
            a /= a.sum()
            out.append(a.argmax())
        return np.array(out)
    logA, le = np.log(A, where=A > 0, out=np.full_like(A, -np.inf)), np.log(e)  # Viterbi
    score, back = np.log(prior) + le[0], np.zeros(e.shape, int)
    for t in range(1, len(e)):
        cand = score[:, None] + logA
        back[t], score = cand.argmax(0), cand.max(0) + le[t]
    path = [score.argmax()]
    for t in range(len(e) - 1, 0, -1):
        path.append(back[t, path[-1]])
    return np.array(path[::-1])

HMM = ["HMM forward (online)", "HMM Viterbi (offline)"]
RUNS = ([(m, m, None) for m in ["per-frame", "EMA (online)", "moving avg (offline)"]]
        + [(f"{m}, {k}", m, k) for k in ["add-one", "seen-only"] for m in HMM])
COLS = ["acc", "relaxed", "segments", "edit", "F1@10", "F1@25", "F1@50"]
results = {name: [] for name, _, _ in RUNS}
for seed in range(5):
    rng = np.random.default_rng(seed)
    bank = mask_bank(rng)
    videos = [render_video(rng, bank) for _ in range(40)]
    data = [(features(f), y) for f, y in videos]
    train, test = data[:24], data[24:]
    Xtr, ytr = np.vstack([x for x, _ in train]), np.concatenate([y for _, y in train])
    scaler = StandardScaler().fit(Xtr)
    clf = LogisticRegression(max_iter=2000).fit(scaler.transform(Xtr), ytr)
    prior = np.bincount(ytr, minlength=7) / len(ytr)
    counts = np.zeros((7, 7))
    for _, y in train:
        np.add.at(counts, (y[:-1], y[1:]), 1)
    trans = {"add-one": counts + 1,  # pseudo-count on every pair: unseen transitions allowed
             "seen-only": counts}    # only transitions seen in training are allowed
    trans = {k: c / c.sum(1, keepdims=True) for k, c in trans.items()}
    for name, m, k in RUNS:
        rows = []
        for x, y in test:
            p = decode(clf.predict_proba(scaler.transform(x)), trans.get(k), prior, m)
            rows.append([100 * np.mean(p == y), relaxed_acc(p, y), len(segments(p)[0]),
                         edit_score(p, y), f1_at(p, y, .1), f1_at(p, y, .25), f1_at(p, y, .5)])
        results[name].append(np.mean(rows, 0))

true_segments = np.mean([len(segments(y)[0]) for _, y in test])
print(f"test videos per seed: 16, true segments per video (seed 4): {true_segments:.1f}")
print("mean over seeds 0-4 [min-max]")
print(f"{'method':33s}" + "".join(f"{c:>16s}" for c in COLS))
for name, _, _ in RUNS:
    r = np.array(results[name])
    print(f"{name:33s}" + "".join(f"{a:6.1f} [{lo:4.0f}-{hi:4.0f}]".rjust(16)
                               for a, lo, hi in zip(r.mean(0), r.min(0), r.max(0))))

test videos per seed: 16, true segments per video (seed 4): 7.5
mean over seeds 0-4 [min-max]
method                                        acc         relaxed        segments            edit           F1@10           F1@25           F1@50
per-frame                          46.5 [  41-  54]  47.7 [  42-  55] 644.4 [ 492- 725]   1.5 [   1-   2]   1.1 [   1-   1]   0.6 [   0-   1]   0.3 [   0-   1]
EMA (online)                       49.9 [  41-  60]  51.5 [  42-  62]  69.5 [  58-  76]  10.3 [   7-  12]  13.1 [  11-  16]  10.1 [   8-  12]   5.6 [   3-   8]
moving avg (offline)               50.4 [  40-  61]  52.1 [  42-  63]  18.6 [  16-  20]  23.9 [  19-  30]  31.6 [  23-  39]  26.1 [  17-  35]  17.9 [   9-  26]
HMM forward (online), add-one      48.3 [  45-  54]  49.8 [  46-  55]  77.5 [  57-  92]  11.6 [  11-  14]  16.3 [  15-  19]  13.6 [  11-  16]   9.3 [   7-  12]
HMM Viterbi (offline), add-one     47.8 [  44-  54]  49.2 [  45-  56]  22.0 [  15-  29]  31.0 [  27-  39]  37.8 [  34-  46]  34.3 [  29-  43]  25.3 [  21-  33]
HMM forward (online), seen-only    50.3 [  47-  54]  51.9 [  49-  56]  36.3 [  28-  42]  20.9 [  19-  24]  24.9 [  23-  28]  21.7 [  19-  25]  15.8 [  14-  18]
HMM Viterbi (offline), seen-only   60.2 [  56-  63]  61.8 [  57-  65]   7.4 [   6-  10]  80.9 [  78-  87]  65.2 [  60-  69]  59.0 [  52-  64]  48.5 [  41-  55]

The frame classifier gets under half of the frames right, and its errors are scattered: each video, which has seven or eight true segments, is cut into hundreds of predicted ones, so the edit score and segmental F1 are near zero. A report of frame accuracy alone would describe this output as a mediocre recognizer; the event metrics show it is useless as a timeline.

The two moving averages hardly change frame accuracy, by three to four points, but remove most of the flicker, and the centred window, which can look ahead, does so far better than the causal one. Event metrics separate the decoders that frame accuracy ranks together, and for the hidden Markov model they depend on how the transition matrix was estimated. With add-one pseudo-counts, every transition is possible, including those never seen in training, such as a jump from gallbladder retraction back to dissection. Each such transition gets a pseudo-count of one, while a real forward transition occurs about once per training video, so an unseen transition is only about an order of magnitude less probable than a real one (our reading of the counts). The decoders can then follow the classifier’s confusions back and forth: forward filtering behaves much like the causal moving average, and Viterbi fragments each video into about three times the true number of segments, with a somewhat higher edit score and segmental F1 than the centred moving average but no better frame accuracy. With seen-only transitions, a backward jump has probability zero. Forward filtering then falls between the two moving averages, because choosing the most probable phase at each instant still does not produce a path that respects the allowed transitions. Viterbi decoding, which chooses the best path through the allowed transitions, predicts about the true number of segments, more than doubles the edit score of any other decoder, and is the only decoder that raises frame accuracy appreciably, by about 14 points. The same model with the same classifier thus looks like a minor improvement or a large one depending on a choice that a description such as “transitions estimated from the training labels” leaves open.

Even seen-only Viterbi stays near 60% frame accuracy across the seeds, because the dominant errors are not flicker. Gallbladder dissection frames look like Calot triangle dissection frames by construction, so the classifier calls long stretches of P4 P2, and no smoothing can correct a confusion that lasts minutes; only the order constraint, through the short clipping phase between them, recovers part of it (our reading of the design). EndoNet’s hidden Markov model raised Cholec80 accuracy by 17 points offline (Twinanda et al., 2017); how much smoothing helps depends on how much of the error is short-lived. The relaxed boundaries add one to two points to every decoder; they forgive errors only within ten seconds of a true transition, which is small against phases lasting minutes.

The example was planned before it was first run, and the script was changed three times after results had been seen. The plan said only that the transition matrix was estimated from the training labels. The first version added a pseudo-count to every pair of phases. After its results had been seen, the second version removed the pseudo-counts but kept a floor of $10^{-12}$ on the transition probabilities, which still allowed unseen transitions at a finite cost, and the third set their log-probability to minus infinity, so that only transitions seen in training are allowed, as in EndoNet’s model, whose top-level transitions are those defined in the dataset (Twinanda et al., 2017, Fig. 7). The same edit dropped a $10^{-12}$ that the first version added to the emission terms before taking their logarithm. Because both the first and the third version are reasonable readings of the plan and the result depends on the choice, the editor then decided that the shown script runs the two hidden Markov decoders with both estimates, the first version’s pseudo-counts and the seen-only transitions, and prints both sets of rows; the emission term is the third version’s in both. Nothing else changed: the data generation, features, classifier, other decoders, metrics and seeds are those of the first run. The script runs in about 50 seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2, and two runs gave identical output.

Evaluation

Frame-level metrics. Phase recognition is usually scored per frame. In EndoNet’s evaluation, recall and precision per phase divide the correctly labelled frames by the length of the true phase and of the predicted phase, and are averaged over phases; accuracy is the fraction of correctly labelled frames in the whole operation (Twinanda et al., 2017). The Jaccard index per phase, the intersection over union of predicted and true frames, is also common. In their review of 50 Cholec80 studies, Özkurt et al. (2026) found accuracy, precision and recall the most frequent metrics, followed by the Jaccard index, with segment-based metrics increasingly used since about 2023. Few studies explained their choice of metrics: 46% referred to earlier work and 38% gave no explanation.

How averages are formed. The MATLAB script for Cholec80 in the SV-RCNet repository, the public code of one of the methods Özkurt et al. review, computes each metric per phase and video, averages each phase over the 40 test videos while skipping phases absent from a video, and then averages over phases. Its reported standard deviation of the Jaccard index is taken across the seven phase means, whereas that of accuracy is taken across videos, and per-phase values above 100, which the relaxed rule below makes possible for precision and recall, are clipped to 100 (Jin, 2018). A “± SD” from this script therefore measures different spreads for different metrics. HeiChole averaged F1 over classes and videos so that each has equal weight regardless of length (Wagner et al., 2023); a pooled average over all frames would let the long dissection phases dominate.

Relaxed boundaries. Because annotators disagree about exactly when one phase ends, many studies forgive errors near transitions. In the SV-RCNet script, within the first ten seconds of each true phase a prediction of the previous phase is not counted as an error, and within the last ten seconds a prediction of the next phase is not, with wider rules for phases that can alternate (Jin, 2018). Özkurt et al. (2026) report that the public implementations they found all use this ten-second window, that none of the code or papers explains its choice, that relaxed boundaries were applied in nine of 26 online and one of eight offline settings, and that results with relaxed boundaries tend to have larger standard deviations. As we read the MATLAB code, the line meant to forgive early transitions builds its mask from the last ten seconds of the phase, but MATLAB applies a logical mask shorter than the array to its first elements, so the mask acts on the first ten seconds: early transitions are not forgiven as such, and a frame near the start of a phase can be counted as correct whatever its error when the matching frame near the end was an early transition. The example implements the rule as the comments describe it. Relaxed and strict numbers are not comparable.

Segment-level metrics. Frame accuracy is dominated by long phases and barely registers over-segmentation, the many short false segments that flickering predictions produce. Action segmentation research therefore adds the segmental edit score, a normalized edit distance between the predicted and true sequences of segment labels, and the segmental F1 score at IoU thresholds of 10%, 25% and 50% (Abu Farha & Gall, 2019). The paper names these metrics; the definitions are in the authors’ evaluation script, adapted from earlier action segmentation code: the edit score is 100 times one minus the edit distance divided by the longer of the two label sequences, and a predicted segment is a true positive if the true segment of the same class that it overlaps most has an IoU of at least the threshold and has not already been matched (Abu Farha, 2019). The gap can be large: on the 50Salads dataset a single-stage model and a four-stage MS-TCN had similar frame accuracy (78.2% and 80.7%) but segmental F1@10 of 27.0 and 76.3 (Abu Farha & Gall, 2019, Table 1). The example above shows the same pattern.

Numbers that do not compare. Splits, relaxed boundaries, online or offline mode, and metric all change a reported phase result. The HeiChole paper, for instance, contrasts its best F1 of 65% with “average precision of 91%” on Cholec80 attributed to EndoNet (Wagner et al., 2023), but EndoNet’s Cholec80 tables give 84.8% average precision and 92.0% accuracy offline, and 91.0% is its average precision on the seven EndoVis 2015 videos (Twinanda et al., 2017, Table IV); the challenge was also online, EndoNet’s figure offline. The difference between the two datasets is real (HeiChole is multicentre with a more variable phase order), but its size is not the difference between those two numbers.

Instruments and skill. Instrument presence is scored with average precision or F1, segmentation with the Dice coefficient and IoU, and tracking with the multi-object tracking metrics (HOTA, MOTA, IDF1) of Multi-Object Tracking. Skill predictions are scored by mean absolute error or correlation with expert ratings. The HeiChole skill entry shows why an error alone is not enough: constant predictions near the upper middle of the scale reached a mean absolute error of 0.78, and in that dataset the four skill items were never graded 1 or 2 (Wagner et al., 2023).

Evidence on Use in the Operating Room

The evidence for surgical computer vision in operations is thin, and it is mostly of designs that cannot show an effect on patients. Clinical Validation of Medical AI defines the designs used below and the reporting guidelines for early live evaluation.

A field without success stories in 2019. According to an international poll in 2019, no commonly recognized surgical data science success story existed (Maier-Hein et al., 2022). In the authors’ table of selected clinical studies of intraoperative systems with a machine-learning component, from a June 2021 search, every study of surgical video is cross-sectional: detection of events and case severity in 1,051 cholecystectomy videos, segmentation of safe and dangerous zones of dissection in 290 videos, assessment of the critical view of safety in 201 videos, landmark detection evaluated in one patient (with 99 videos), and classification of the uterine artery and ureter in images from 35 patients in gynaecological surgery. The one randomized controlled trial in their table, with 68 patients, tested an early warning of low blood pressure from invasive monitoring, not a vision system. They describe convenience samples as the norm in studies that recognize objects or anatomy in the surgical field, which makes the findings susceptible to selection bias, and they name small, single-centre data, unaccounted variability among human comparators, missing error analysis and failure to separate training and test data as prevalent methodological shortcomings, citing an earlier scoping review.

A scoping review to June 2025. Buchanan et al. (2026) reviewed 113 studies of vision systems in laparoscopic or robot-assisted general surgery published between June 2015 and June 2025. All were observational, and 93 (82%) were retrospective feasibility analyses. Thirteen (12%) described real-time use in the operating room, predominantly small single-centre feasibility or pilot studies of 1 to 51 procedures: in the reviewers’ table of these studies, five are prospective real-time feasibility studies and eight real-time experimental pilots after retrospective training (their table of primary study stages, which assigns each study one stage, counts 12 studies in its two real-time categories). Five blinded the operating surgeons to the system’s output, and few had comparators or prespecified clinical endpoints; reported endpoints included the rate of malfunctions (the primary endpoint of one study) and the appropriateness of the timing of landmark detection. External or multicentre validation appeared in 12% of studies, latency was reported in 28%, compatibility across endoscopic systems in one study, and reference to an AI-specific reporting guideline in 8%. In one study the rate of attaining the critical view of safety rose although surgeons could not see the algorithm’s output, which the reviewers attribute, presumably, to a Hawthorne effect: being studied changed behaviour.

What retrospective studies show. Retrospective studies can still measure how a model generalizes. Golany et al. (2022) trained a ResNet-50 with an MS-TCN on cholecystectomy videos from four Israeli hospitals and Cholec80, split by video into 294 for training and 77 for testing, stratified by hospital, complexity and adverse events. Test accuracy was 78% for the frame model and 89% with the temporal model, against 90% agreement between two surgeon annotators, and fell from 92% at the lowest complexity level to 81% at the highest. The dataset excluded 77 videos whose two annotators agreed on fewer than 80% of frames, so the hardest-to-label operations were left out of both training and test. In HeiChole, the best phase methods scored best on videos from Salem Hospital, one of the two centres with more standard cases, and worst on those from the university hospital, whose cases were more difficult; videos from the third centre, absent from training, fell between (Wagner et al., 2023). The organizers conclude that case difficulty and anatomy matter at least as much as the number of centres.

What live tests have shown. Tokuyasu et al. (2021) ran their landmark detector during one operation on a patient with gallstones and report that it made the team more aware of the landmarks; with average precisions of 0.074 to 0.320 on the retrospective test videos, the live test was a demonstration of feasibility, not of accuracy or benefit.

As of the sources read for this article (November 2026), no randomized trial of a surgical video system appears in either review, and no study they summarize shows an effect on patient outcomes; the reviews’ scopes are limited to their search dates and, for Buchanan et al., to general surgery.

Failure Modes and Risks

• Systematic confusion between similar phases. Smoothing and temporal models remove flicker but not long confusions between phases that look alike, as the example shows; order constraints help only when a distinctive phase lies between them.
• Imbalance. Rare classes are learned poorly: all HeiChole action methods predicted hold almost everywhere (Wagner et al., 2023), and in CholecTrack20 rarely used tools were tracked worst (Nwoye et al., 2025).
• Degenerate outputs hidden by averages. A skill model that outputs the same scores for every video, or an action model that outputs one class, can reach unremarkable average errors.
• Shift between centres and cases. Instruments, cameras, resolution, frame rate and case difficulty differ between hospitals; Dataset Bias describes the general problem. Single-centre benchmarks overstate performance elsewhere, and selecting easy-to-annotate videos overstates it further.
• Different label definitions. Phase definitions differ between datasets, as EndoVis 2015 and Cholec80 show, and HeiChole’s authors cite the heterogeneity of phase definitions across groups as a problem for comparison (Wagner et al., 2023).
• Overlapping datasets. Datasets derived from the same video collection share operations; training on one and testing on another can leak.
• Unverifiable online claims. A method reported as online may use future frames unless the evaluation enforces causality, which HeiChole did not (Wagner et al., 2023).
• Visual degradation. Smoke, blood on the lens, reflections and motion blur are frequent and correlated with surgical events, such as coagulation; CholecTrack20’s analysis reports detection hardest in blurred scenes, near trocars and with specular reflection (Nwoye et al., 2025).

Deployment Considerations

A system for the operating room must run online, on the live video signal, with a latency suited to its purpose, and its output must reach the team in a usable form; in the real-time studies Buchanan et al. (2026) reviewed, outputs were mostly overlays on auxiliary monitors, often requiring manual activation, and surgeon feedback, where collected, was mostly gathered with Likert scales. Video capture itself is a barrier: Maier-Hein et al. (2022) note that digitalization has not reached the operating room to the extent it has other areas of medicine, and that the lack of standardized procedures hampers standardized annotation and multicentre studies. Recordings contain frames from outside the patient that can identify people and must be removed or anonymized before data leave the hospital. Systems whose output influences care are medical devices, and the path from retrospective validation to prospective study, reporting and regulation is described in Clinical Validation of Medical AI. Offline uses, such as indexing archives or post-operative feedback, avoid the latency constraint but not the questions of shift and evaluation above.