Temporal Action Segmentation

Temporal action segmentation assigns an action label to every frame of a long, untrimmed video. Its typical input is a recording of a procedure, such as making coffee or a salad, that lasts minutes and consists of a few to a few dozen steps, and its output is the sequence of those steps with their boundaries: background, take cup, pour coffee, pour milk, stir. The task is also called action segmentation, and Lea et al. (2017) note that the same problem appears in the literature as action detection, with different methods and metrics. Among the tasks surveyed in video understanding, it is the dense counterpart of temporal action localization, which returns a sparse set of scored segments, and it relies on the ways of combining frames described in temporal modeling.

Problem Definition

Given the frames $x_{1:T} = (x_1, \dots, x_T)$ of a video, the goal is to infer a class label for each frame, $c_{1:T} = (c_1, \dots, c_T)$, where $T$ is the length of the video (Abu Farha & Gall, 2019, §3). The classes are a fixed set of actions, usually including a background or idle class. Consecutive frames with the same label form a segment, so a frame labelling is equivalent to an ordered list of segments, each with a label, a start and an end.

Relation to temporal action localization. Lea et al. (2017, §2) distinguish the two by their output: segmentation predicts the action at every frame, detection outputs a sparse set of segments defined by start, end and class, and one can be converted to the other by adding or removing background segments. In practice the tasks differ in their data and evaluation. Localization benchmarks such as THUMOS14 and ActivityNet annotate a few instances per video, which in some datasets overlap, and detections are ranked by confidence and scored by average precision (see temporal action localization). Segmentation benchmarks are densely annotated procedures in which every frame carries exactly one label, background included, and the output has no confidence ranking; it is scored by frame accuracy and by segment-level measures that need no confidence (Lea et al., 2017, §4.1).

Over-segmentation. A frame classifier that is right most of the time can still flip between labels for a few frames inside a long action. Each flip creates two spurious boundaries, so a prediction can match most frames while describing a sequence of steps that never happened. Abu Farha and Gall (2019) call these over-segmentation errors and designed both their architecture and their loss to reduce them; much of the later work on the task is about the same problem.

Input and Output

Precomputed features. Most methods do not read pixels. A video network trained for action recognition runs over the video once and produces one feature vector per frame or snippet, and the segmentation model is trained on the resulting $T \times D$ sequence. MS-TCN used I3D features without fine-tuning, at 15 frames per second for all three standard datasets, downsampling 50Salads from 30 to 15 fps (Abu Farha & Gall, 2019, §4). The authors released these features with the ground-truth labels and splits, a download of about 30 GB (ms-tcn README), and later work such as ASFormer points to the same files (ASFormer README); DiffAct describes the I3D features as those used in most prior work, with 2048 dimensions (Liu et al., 2023, §5.1). In our reading, the shared release means that most published comparisons on these benchmarks differ in the temporal model rather than in the frame features. Earlier work used other inputs: improved dense trajectories with Fisher vectors on Breakfast (Abu Farha & Gall, 2019, §2), and Lea et al. (2017, §4.3) used small VGG-style spatial CNNs trained on each dataset and sampled 50Salads at about 1 fps and GTEA at 3 fps.

Output. One label per frame, written out as a frame-level label file; segments are read off as runs of equal labels. Most methods are offline: they see the whole video before labelling it, and Lea et al. (2017, §3.4) contrast these acausal models with causal ones that use only past frames, which robotics needs.

Variants

• Supervision. Full supervision uses a label for every training frame. Cheaper settings use an ordered list of the actions in each training video (transcripts), an unordered set of the actions present, or one labelled frame per action segment (timestamps). The Breakfast page lists a weakly supervised alignment task, in which the transcript is also given at test time, and weakly supervised segmentation without transcripts (Breakfast page). These settings are outlined under Approaches.
• Online segmentation. Causal models label each frame from past frames only (Lea et al., 2017); surgical phase recognition, described in surgical computer vision, is one setting where this matters.
• Granularity. The same video can be labelled at several levels. Breakfast was annotated with 48 coarse action units such as “pour milk” and, by a second group of annotators, with finer units such as “grab milk”, “twist cap” and “open cap” (Kuehne et al., 2014, §3); 50Salads is evaluated at a mid level of 17 classes or a higher level of 9 classes plus background (Lea et al., 2017, §4.3).

Challenges

• Long sequences. A five-minute video at 15 fps has 4,500 frames, and, in our reading, the identity of a step can depend on context far away in time. Earlier methods that solved a maximization over the whole sequence were slow (Abu Farha & Gall, 2019, §1), and recurrent models are slow because each step waits for the previous one (§4.9).
• Similar appearance. Steps of one procedure share a scene, hands and objects. On Breakfast, MS-TCN reached a higher F1 score with motion-only dense-trajectory features than with I3D features, which the authors explain by appearance giving little evidence about the action when frames look alike (Abu Farha & Gall, 2019, §4.9). On 50Salads, Lea et al. (2017, §4.4) traced many errors to the vinegar and olive-oil bottles, cutting different vegetables and placing different ingredients.
• Ambiguous boundaries. Transitions between actions are gradual, so the features around a boundary are ambiguous (Liu et al., 2023, §1); evaluation therefore tolerates small boundary shifts.
• Variable order and duration. Breakfast’s participants were given only a recipe, with no script or rehearsal, so the choice and order of steps vary between videos (Kuehne et al., 2014, §3), and a short step can sit between two long ones.
• Over-segmentation. Frame-level errors fragment segments, as above, and a measure based on frames alone barely registers it.

Approaches

Sliding windows and probabilistic temporal models

Before temporal convolutions, Abu Farha and Gall (2019, §§1–2) group methods into sliding-window detectors with non-maximum suppression and hybrids that run a frame-wise classifier and then a coarse temporal model, usually a Markov model. Kuehne et al. (2014) introduced Breakfast with such a system: each coarse action unit is a hidden Markov model built with the speech recognition toolkit HTK, and a grammar of the allowed orders of units, built from the annotations, combines them into activities, by analogy with phonemes and words. Richard and Gall (CVPR 2016) added an explicit model of segment lengths and a language model over action sequences, decoded by dynamic programming (as summarized by Abu Farha & Gall, 2019, §2). Singh et al. (CVPR 2016) passed two-stream features of short chunks to a bidirectional LSTM, which Abu Farha and Gall describe as slow because of its sequential prediction.

Temporal convolutional networks

Lea et al. (2017) replaced these pipelines with temporal convolutional networks (TCNs) over frame features, in which every time step is computed in parallel and each prediction depends on a fixed window, the receptive field. Their encoder–decoder TCN (ED-TCN) alternates temporal convolutions, a non-linearity and max pooling by 2 in the encoder, and upsampling by repetition, convolution and the non-linearity in the decoder, with a softmax per frame; with $L$ layers and filters spanning $d$ frames, its receptive field is $d(2^L - 1) + 1$ frames. A second model, the Dilated TCN, adapted WaveNet’s stacks of dilated convolutions. On both 50Salads granularities ED-TCN scored best on all metrics, the TCNs trained more than an order of magnitude faster than a bidirectional LSTM baseline, and it produced many fewer over-segmentation errors than the other models (Lea et al., 2017, §1 and Table 2). Lei and Todorovic (CVPR 2018) added deformable convolutions and a residual stream to the encoder–decoder (as summarized by Abu Farha & Gall, 2019). Both models ran on video sampled at 1–3 frames per second (Abu Farha & Gall, 2019, §2).

MS-TCN

The multi-stage temporal convolutional network (MS-TCN) of Abu Farha and Gall (2019) keeps the full temporal resolution and drops pooling and fully connected layers. A single stage (SS-TCN) is a $1 \times 1$ convolution that maps the input features to $D = 64$ channels, followed by $L = 10$ dilated residual layers and a $1 \times 1$ convolution with a softmax over the $C$ classes. Layer $l$ applies an acausal convolution of kernel size 3 with dilation $2^{l-1}$ (1, 2, 4, …, 512), a ReLU, a $1 \times 1$ convolution and a residual connection (Eqs. 1–2):

$$\hat H_l = \operatorname{ReLU}(W_1 * H_{l-1} + b_1), \qquad H_l = H_{l-1} + W_2 * \hat H_l + b_2 .$$

Doubling the dilation makes the receptive field grow exponentially with depth, $2^{l+1} - 1$ frames after layer $l$ for kernel size 3 (Eq. 3), so ten layers see 2,047 frames, about 136 seconds at 15 fps by our arithmetic, with few parameters.

Stages. Four such stages are stacked. The first takes the features, $Y^0 = x_{1:T}$, and each later stage refines the previous stage’s output, $Y^s = \mathcal{F}(Y^{s-1})$, taking as input only the frame-wise class probabilities, without features (Eqs. 5–6). The idea comes from stacked predictors in human pose estimation, such as stacked hourglass networks and convolutional pose machines, where each stage refines the last; the authors argue that a stage which sees only predictions can learn plausible action sequences and so reduce over-segmentation. Feeding features to the later stages, as pose models do, lowered F1 and edit by about 20 points on 50Salads (Table 5), which the authors attribute to classes with similar appearance and motion producing small spurious segments.

Loss. Each stage is trained with cross-entropy over frames plus a smoothing loss, a truncated mean squared error over the frame-wise log-probabilities (Eqs. 7–13):

$$\mathcal{L}_{\text{T-MSE}} = \frac{1}{TC} \sum_{t,c} \tilde\Delta_{t,c}^2, \qquad \tilde\Delta_{t,c} = \min(\Delta_{t,c}, \tau), \qquad \Delta_{t,c} = \lvert \log y_{t,c} - \log y_{t-1,c} \rvert,$$

where $y_{t-1,c}$ is treated as a constant, so gradients flow only through $y_{t,c}$. A stage’s loss is $\mathcal{L}_s = \mathcal{L}_{\text{cls}} + \lambda \mathcal{L}_{\text{T-MSE}}$ and the network minimizes $\sum_s \mathcal{L}_s$, with $\tau = 4$ and $\lambda = 0.15$; training used Adam with learning rate 0.0005 and dropout after each layer (§3.4). The released code clamps the squared difference at 16, which is $\tau^2$, and runs 50 epochs (model.py and main.py). The truncation matters: with $\tau = 5$ the loss also penalized confident changes of class at true boundaries, and F1@10 on 50Salads fell from 76.3 to 66.6; the authors found the symmetric truncated loss better than a Kullback–Leibler smoothing term, which barely penalizes small differences (Tables 3–4, Fig. 5).

What the ablations show. All results are on 50Salads (mid level, five-fold cross-validation, I3D features at 15 fps) unless noted:

Configuration (Abu Farha & Gall, 2019)	F1@{10, 25, 50}	Edit	Acc
SS-TCN, 1 stage (Table 1)	27.0 / 25.3 / 21.5	20.5	78.2
MS-TCN, 2 stages (Table 1)	55.5 / 52.9 / 47.3	47.9	79.8
MS-TCN, 4 stages (Table 1)	76.3 / 74.0 / 64.5	67.9	80.7
SS-TCN with 48 layers, same parameter count (Table 2)	49.0 / 46.4 / 40.2	40.7	78.0
MS-TCN, cross-entropy only (Table 3)	71.3 / 69.7 / 60.7	64.2	79.9
MS-TCN at 1 fps instead of 15 fps (Table 6)	77.8 / 74.9 / 64.0	70.7	78.6

Frame accuracy hardly moves between one and four stages, while the segment measures roughly triple, and a single stage as deep as four stages recovers only part of the gain. With five stages (Table 1) F1@10 and the edit score rose slightly (76.4 and 69.2) while F1@25, F1@50 and accuracy fell; the text describes performance as starting to degrade and suspects over-fitting. At 1 fps the segment measures improved and accuracy fell: the coarser sequence is less prone to over-segmentation but loses boundary precision and short segments (§4.6). On GTEA, fine-tuning the I3D features raised MS-TCN’s F1@50 from 69.8 to 74.6, which the authors note is a smaller effect than fine-tuning has for recognition (Table 9).

Later fully supervised models

MS-TCN++, the journal extension by Li, Abu Farha, Liu, Cheng and Gall, appeared online in TPAMI in 2020 and in print in 2023; its text could not be read for this article. Later papers list its results beside MS-TCN’s on the same features (Li et al., 2021, Tables 6–8; Liu et al., 2023, Table 1).

Boundary-aware refinement. ASRF (Ishikawa et al., 2021) adds to an MS-TCN-style action segmentation branch a boundary regression branch that predicts, for every frame, the probability of an action boundary. At inference, local maxima above a threshold cut the video into segments, and each segment takes the majority class of the frame-wise predictions inside it. On 50Salads they compared this with simpler post-processing of the same branch’s output (Table 3):

Post-processing (Ishikawa et al., 2021, Table 3, 50Salads)	F1@10	F1@50	Edit	Acc
none	76.1	66.7	68.5	82.6
Gaussian smoothing of probabilities, kernel 15	80.8	70.9	73.6	82.6
relabel segments shorter than a span $\theta_t = 15$ with the previous label	82.4	72.5	76.2	82.7
boundary branch (ASRF)	84.9	77.3	79.3	84.5

Relabelling depends on its threshold: a large one removes over-segmentation but also drops short actions, which the boundary branch avoids (§4.2). The paper’s ablation tables on 50Salads (Tables 3–7) label the middle F1 threshold 20%, while its comparison tables (Tables 1–2) and the other papers use 25%, and Table 1 prints for ASRF on 50Salads the same value at 25% (83.5) that Table 3 prints at 20%; that column is left out here.

Transformers. ASFormer (Yi et al., 2021) replaces the dilated convolutions with transformer layers; the paper could not be read, and this description follows the official code. An encoder of ten layers each applies a dilated temporal convolution (dilation $2^i$ in layer $i$, counting from 0) followed by self-attention restricted to a local window: frames are grouped in blocks of $2^i$, and each block attends to a window of roughly twice that length around it, so windows grow with depth as receptive fields do in MS-TCN. The README notes a small error in the window mask that the authors kept so that their released models still load (as of November 2026). Three decoders then refine the encoder’s predictions, as MS-TCN’s stages do: each takes the previous softmax output and attends with queries and keys from its own features and values from the features of the encoder (for the first decoder) or of the previous decoder, and the weight of the attention branch is multiplied by $e^{-3}$ from one decoder to the next (model.py).

Diffusion. DiffAct (Liu et al., 2023) treats segmentation as generating the frame-wise label sequence with a diffusion model conditioned on the video. In training, the ground-truth sequence is corrupted with noise and a decoder learns to restore it, with cross-entropy, temporal smoothness and boundary alignment losses; at inference, the decoder starts from random noise and denoises in 25 steps. The authors present the iterative denoising as sharing the spirit of multi-stage refinement. During training they also mask the conditioning features at random: all of them, so that only the video length and frame positions remain; those near action boundaries; or the segments of one random class. The three masks are meant to make the model learn where actions tend to occur in a video, what surrounds a boundary, and how actions follow each other (§1, §4). The encoder and decoder are re-implementations of ASFormer’s (§5.1).

Weakly supervised settings

Frame-level labels are expensive, and several weaker forms of supervision have been studied; this outline follows the papers’ own descriptions.

• Transcripts. Training videos come with the ordered list of their actions but no boundaries. Kuehne, Richard and Gall (2017) learned an HMM per action with a Gaussian mixture model for the observations; Richard, Kuehne and Gall (2017) replaced the mixture with a recurrent network (a GRU) over subactions whose number per action is re-estimated; both start from a uniform alignment of the transcript to the frames and alternate realignment and retraining (as summarized by Abu Farha & Gall, 2019; Richard et al., 2017, §1). Ding and Xu (2018) trained a temporal convolutional feature pyramid network the same way, with soft labels at the boundaries. NeuralNetwork-Viterbi (Richard et al., 2018) replaced the separate alignment step with a Viterbi-based loss that allows online learning, and added a class-dependent Poisson length model and a grammar estimated from the training transcripts. CDFL (Li, Lei & Todorovic, 2019) trains the HMM and GRU with a loss that contrasts the energy of all labellings consistent with the transcript against inconsistent ones, rather than relying on a single inferred segmentation. Earlier transcript-based work includes ordering constraints (Bojanowski et al., ECCV 2014) and a connectionist temporal classification approach (Huang et al., ECCV 2016), both named in MS-TCN’s review.
• Action sets. Only the set of actions in each video is known, without order; Li et al. (2021, Table 7) list Action Sets, SCV and SCT among such methods.
• Timestamps. One frame per action segment is labelled. Li, Abu Farha and Gall (2021) generate frame-wise labels by estimating where the action changes between consecutive timestamps from the model’s own output, and add a confidence loss that makes predicted probabilities decrease monotonically with distance from each timestamp. In their experiments the timestamp was a random frame of each segment (§4.1), and they cite an estimate that marking start and end frames takes annotators six times longer than marking a single timestamp (§3.1, citing Ma et al., ECCV 2020).

Li et al. (2021, Table 7) gathered frame accuracies on Breakfast for these settings. Fully supervised MS-TCN reached 66.3%; their timestamp-supervised model, an MS-TCN variant, 64.1%; the transcript-based methods ranged from 27.7% (ECTC) through 43.0% (NN-Viterbi) to 50.2% (CDFL); and the set-supervised ones 23.3–30.4%. The NN-Viterbi and CDFL values match those papers’ own tables (Richard et al., 2018, Table 3; Li et al., 2019, Table 1, where the measure is called mean-over-frames, Mof).

Datasets and Benchmarks

Three datasets, all of food preparation, have served as the standard benchmarks since Lea et al. (2017) and MS-TCN; none has its own article on this site yet, so their protocols are described here.

Breakfast (Kuehne et al., 2014). 52 participants each performed 10 breakfast activities (coffee, orange juice, chocolate milk, tea, cereals, fried eggs, pancakes, fruit salad, sandwich, scrambled eggs) in 18 different kitchens, filmed by 3–5 uncalibrated cameras per location: about 77 hours and more than 4 million frames, down-sampled to $320 \times 240$ pixels at 15 fps. Videos from laterally placed cameras were also mirrored. The coarse annotation has 48 action units with 11,267 samples, including about 3,600 “silence” samples, and a grammar defines the allowed orders of units for each activity (§3). The participants are divided into four groups that serve in turn as test sets; the dataset page lists them as P03–P15, P16–P28, P29–P41 and P42–P54 (Breakfast page, November 2026). MS-TCN counts 1,712 videos, with 6 action instances per video on average, and reports the average over the four splits (§4); DiffAct gives 7 instances and an average length of about two minutes (§5.1). The original paper reported unit accuracy, an analogue of word accuracy in speech recognition, alongside frame-based accuracy (§5.3). As of November 2026 the page points to a maintained download of features and annotations and notes that the raw-video mirror is incomplete.

50Salads (Stein & McKenna, 2013). 25 people each prepared two mixed salads, giving 50 videos recorded with depth maps and accelerometer data; the paper that introduced the dataset could not be read for this article, and this description follows the papers that use it. Videos last 5–10 minutes (Lea et al., 2017) or 6.4 minutes on average (Abu Farha & Gall, 2019), with 9,000 to 18,000 RGB frames each (Ishikawa et al., 2021) recorded at 30 fps (Abu Farha & Gall, 2019), and DiffAct describes them as top-view. Lea et al. (2017) evaluated a higher level of 9 actions plus background and a mid level of 17 actions that, for example, separates cutting a tomato from cutting a cucumber; MS-TCN and later work use the 17 classes with five-fold cross-validation. The papers disagree on density: about 30 instances per video in Lea et al. (2017), 20 in MS-TCN and DiffAct.

GTEA (Fathi, Ren & Rehg, 2011). Georgia Tech Egocentric Activities was recorded with a GoPro camera on a cap: 4 subjects performing 7 activities (hot dog sandwich, instant coffee, peanut butter sandwich, jam and peanut butter sandwich, sweet tea, coffee and honey, cheese sandwich), captured at $1280 \times 720$ and 30 fps and extracted at 15 fps, 31,222 frames in all (§5.1). The dataset was introduced for weakly supervised object recognition; the 11 action classes used for segmentation come from a later paper of the same group, Fathi, Farhadi and Rehg (ICCV 2011), as Lea et al. (2017, §4.3) note. MS-TCN counts 28 videos with 11 classes including background and 20 instances per video on average, and uses leave-one-subject-out cross-validation (§4), four splits in the ms-tcn README; Lea et al. (2017) instead used leave-one-user-out cross-validation over users 1–3, following EgoNet, at 3 fps.

Comparing numbers. Splits, frame rates and features differ between papers, and tables often copy earlier rows. MS-TCN’s 50Salads comparison (Table 10) lists ED-TCN at 68.0 / 63.9 / 52.6, edit 59.8 and accuracy 64.7, the values Lea et al. (2017, Table 2) obtained at the mid level with their own spatial CNN features at about 1 fps, against MS-TCN’s I3D features at 15 fps; by our reading, that row compares temporal models and features at once. Its Breakfast rows for earlier methods report accuracy only. ASRF’s Breakfast comparison (Ishikawa et al., 2021, Table 2) lists MS-TCN with the values MS-TCN printed for dense-trajectory features (F1@50 40.8, accuracy 65.1), although ASRF itself uses I3D features, with which MS-TCN printed 37.9 and 66.3.

Evaluation Metrics

Three measures are reported together: frame-wise accuracy, the segmental edit score and the segmental F1 score at overlap thresholds of 10%, 25% and 50% (Abu Farha & Gall, 2019, §4). The paper names them; their exact definitions are in the official evaluation script, eval.py, whose header says it was adapted from Lea et al.’s metrics code.

Frame accuracy. The percentage of correctly labelled frames, counted over all frames of all test videos together (eval.py), in our reading a frame-weighted rather than video-weighted mean. It is dominated by long actions and barely registers over-segmentation (Abu Farha & Gall, 2019, §4); the weakly supervised literature reports the same quantity as mean-over-frames (Mof in Li et al., 2019). It is the accuracy of a frame classifier.

Edit score. Let $a$ and $b$ be the sequences of segment labels of the prediction and the ground truth, after merging runs of equal labels and dropping background segments. With $D(a, b)$ the Levenshtein distance (insertions, deletions and substitutions, each of cost 1),

$$\text{Edit} = 100 \left( 1 - \frac{D(a, b)}{\max(\lvert a \rvert, \lvert b \rvert)} \right),$$

computed for each video and averaged over videos (eval.py). It ignores where boundaries lie and penalizes out-of-order predictions and over-segmentation; Lea et al. (2017) cite two of their own 2016 papers for the segmental edit score, the ECCV paper on segmental spatiotemporal CNNs and a paper on learning convolutional action primitives (§4.1), and the second alone in the caption of Table 2.

Segmental F1@k. Lea et al. (2017, §4.1) proposed it to penalize over-segmentation, tolerate small temporal shifts that may reflect annotator variability, and depend on the number of actions rather than their durations, without needing confidence scores. In eval.py, each predicted segment, in temporal order, is compared with every ground-truth segment of the same class by intersection over union and takes the one it overlaps most. It is a true positive if that IoU is at least $k$ and the ground-truth segment is not yet matched; otherwise it is a false positive. Unmatched ground-truth segments are false negatives, and

$$F1@k = 100 \cdot \frac{2\,TP}{2\,TP + FP + FN},$$

the F1 score of segment precision and recall. Three details follow from the code. A predicted segment whose best match is already taken counts as a false positive even if another unmatched segment of its class overlaps it enough. The union is computed as the span from the earlier start to the later end, which equals the true union whenever the segments overlap and gives a negative IoU otherwise. And the script drops only segments whose label is the string “background”, so whether a dataset’s idle frames are excluded depends on how its label files name them.

Aggregation. The scripts differ in how they pool videos. MS-TCN’s eval.py sums true positives, false positives and false negatives over all test videos and computes one F1, while Lea et al.’s metrics.py computes F1, and also accuracy, per video and averages them; both average the edit score over videos. Results are then averaged over the splits (four for Breakfast and GTEA, five for 50Salads). The eval.py used today also contains a fix of July 2020 (commit 0e418c0) that makes a segment running to the last frame end at the last frame rather than one frame earlier; results computed with the script before that date used the earlier version, which shortens such segments by one frame.

Representative Results

The table gives results on the three benchmarks with I3D features, as printed in MS-TCN’s own Table 10 and as listed in DiffAct’s comparison (Liu et al., 2023, Table 1; MS-TCN++ and ASFormer are listed there, as their papers could not be read). Values are F1@50 / edit / accuracy, averaged over the standard splits. DiffAct’s ASRF values match ASRF’s own tables for 50Salads and Breakfast (Ishikawa et al., 2021, Tables 2–3).

Method	GTEA	50Salads	Breakfast
MS-TCN (Abu Farha & Gall, 2019, Table 10)	69.8 / 79.0 / 76.3	64.5 / 67.9 / 80.7	37.9 / 61.7 / 66.3
MS-TCN++ (as listed by Liu et al., 2023)	76.0 / 83.5 / 80.1	70.1 / 74.3 / 83.7	45.9 / 65.6 / 67.6
ASRF (Ishikawa et al., 2021, as listed by Liu et al., 2023)	79.8 / 83.7 / 77.3	77.3 / 79.3 / 84.5	56.1 / 72.4 / 67.6
ASFormer (as listed by Liu et al., 2023)	79.2 / 84.6 / 79.7	76.0 / 79.6 / 85.6	57.4 / 75.0 / 73.5
DiffAct (Liu et al., 2023, Table 1)	84.7 / 89.6 / 82.2	83.7 / 85.0 / 88.9	64.6 / 78.4 / 76.4

The rows trace the field’s emphasis: accuracy rose by about 10 points on Breakfast and 8 on 50Salads from MS-TCN to DiffAct, while F1@50 rose by 27 and 19 points. On Breakfast, MS-TCN with dense-trajectory features printed a higher F1@50 (40.8) than with I3D (Table 10). DiffAct’s table also lists methods that use extra multimodal or hand-pose features, which its authors mark as not directly comparable.

Example: Frame-Wise Labelling and Temporal Smoothing on Synthetic Procedures

The example renders synthetic “make coffee” videos as $32 \times 32$ RGB frames: a table, a cup, two jugs and a hand whose position follows the current step, with camera gain and colour differences between videos, per-frame lighting flicker, pixel noise and an occasional second hand passing through. The recipe is background, take cup, pour coffee, optionally pour milk and add sugar (each followed by stirring, so “stir” recurs), drink, background, with step lengths of 15–150 frames. Pouring coffee and pouring milk differ only in the colour of the lifted jug, the hardest pair by design. Each frame is described by a $4 \times 4$ grid of mean colours and of frame-difference energy, and a logistic regression trained on the frames of 40 videos labels each frame of 20 test videos. The labels are then decoded in three ways: frame by frame; after a centred moving average of the class probabilities over 9, 31 or 91 frames; and by two refinement stages in the spirit of MS-TCN’s probabilities-only stages, each a logistic regression on the previous stage’s probabilities at dilated offsets of 0, ±1, ±2, …, ±64 frames. The refinement stages are trained on out-of-fold predictions for the training videos, so that they see the kind of errors they meet at test time; they are an analogue of a multi-stage network, not MS-TCN. The metrics follow eval.py at commit 33ed91c, and the script first compares them with a line-by-line Python 3 port of that file’s functions on random label sequences.

import numpy as np
from scipy.ndimage import uniform_filter1d
from sklearn.linear_model import LogisticRegression

CLASSES = ["background", "take_cup", "pour_coffee", "pour_milk", "add_sugar", "stir", "drink"]
S = 32                                                    # frames are 32x32 RGB images
yy, xx = np.mgrid[0:S, 0:S]

def recipe(rng):
    """Ordered action list of one 'make coffee' video: optional milk and sugar, each followed by stirring."""
    extras = [a for a, p in ((3, 0.7), (4, 0.6)) if rng.random() < p]
    if len(extras) == 2 and rng.random() < 0.3:
        extras = extras[::-1]
    steps = [0, 1, 2] + [s for a in extras for s in (a, 5)] + [6, 0]
    dur = np.clip(np.exp(rng.normal(np.log(45), 0.5, len(steps))), 15, 150).astype(int)
    return np.repeat(steps, dur)

def disc(img, cy, cx, r, color):
    img[(yy - cy) ** 2 + (xx - cx) ** 2 <= r * r] = color

def render(labels, rng):
    """Render one frame per label: a table, a cup, two jugs and a hand whose pose depends on the action."""
    T = len(labels)
    table = np.array([0.55, 0.50, 0.45]) + rng.normal(0, 0.03, 3)
    cast, gain = rng.normal(0, 0.05, 3), rng.uniform(0.8, 1.2)       # per-video camera differences
    coffee, milk = np.array([0.42, 0.28, 0.18]), np.array([0.52, 0.38, 0.26])
    frames = np.empty((T, S, S, 3))
    for t, a in enumerate(labels):
        img = np.broadcast_to(table, (S, S, 3)).copy()
        ph = 0.35 * t + rng.normal(0, 0.3)                          # motion phase with jitter
        cup = (20, 14) if a not in (1, 6) else ((20, 14 + 4 * np.sin(ph) ** 2) if a == 1 else (20 - 6 * np.sin(ph) ** 2, 14))
        img[int(cup[0]) - 3:int(cup[0]) + 4, int(cup[1]) - 3:int(cup[1]) + 4] = (0.9, 0.9, 0.88)
        img[4:10, 3:7], img[4:10, 25:29] = coffee, milk              # jugs at rest
        if a in (2, 3):                                              # pouring: a jug is lifted over the cup
            jug = coffee if a == 2 else milk
            img[8:14, 11:17] = jug
            img[14:17, 13:15] = jug * 0.8                            # the stream
            hand = (8 + rng.normal(0, 1.5), 18 + rng.normal(0, 1.5))
        elif a == 4:
            img[13:17, 16:19] = (0.95, 0.95, 0.95)                   # sugar spoon
            hand = (14 + 2 * np.sin(ph), 20 + rng.normal(0, 1.5))
        elif a == 5:
            hand = (18 + 3 * np.sin(ph), 14 + 3 * np.cos(ph))        # circular stirring over the cup
        elif a in (1, 6):
            hand = (cup[0] + rng.normal(0, 1), cup[1] + 4 + rng.normal(0, 1))
        else:
            hand = (28 + rng.normal(0, 1), 28 + rng.normal(0, 1))    # hand resting
        disc(img, hand[0], hand[1], 4, (0.85, 0.65, 0.55))
        if rng.random() < 0.15:                                      # occasional distractor: a second hand passes
            disc(img, rng.uniform(0, S), rng.uniform(0, S), 5, (0.85, 0.65, 0.55))
        frames[t] = img
    frames = (frames * (gain + rng.normal(0, 0.04, (T, 1, 1, 1))) + cast) + rng.normal(0, 0.08, frames.shape)
    return np.clip(frames, 0, 1)

def descriptors(frames):
    """Per-frame descriptor from the images: 4x4 grid of mean colours and of frame-difference energy."""
    T = len(frames)
    col = frames.reshape(T, 4, 8, 4, 8, 3).mean((2, 4)).reshape(T, -1)
    diff = np.abs(np.diff(frames, axis=0, prepend=frames[:1])).mean(-1)
    mot = diff.reshape(T, 4, 8, 4, 8).mean((2, 4)).reshape(T, -1)
    return np.hstack([col, mot])

def make_video(rng):
    y = recipe(rng)
    return descriptors(render(y, rng)), y

# --- experiment
def segments(y, bg=(0,)):
    """Labels, starts and ends (exclusive) of the runs of y, background runs dropped (as in ms-tcn eval.py)."""
    cut = np.flatnonzero(np.diff(y)) + 1
    s, e = np.r_[0, cut], np.r_[cut, len(y)]
    keep = ~np.isin(y[s], bg)
    return y[s][keep], s[keep], e[keep]

def edit_score(p, g):
    """100 * (1 - Levenshtein distance between segment label sequences / the longer sequence)."""
    a, b = segments(p)[0], segments(g)[0]
    row = np.arange(len(b) + 1.0)
    for i in range(1, len(a) + 1):                    # one DP row at a time
        new = np.empty_like(row); new[0] = i
        for j in range(1, len(b) + 1):
            new[j] = min(row[j] + 1, new[j - 1] + 1, row[j - 1] + (a[i - 1] != b[j - 1]))
        row = new
    return 100 * (1 - row[-1] / max(len(a), len(b), 1))

def f1_counts(p, g, thr):
    """TP, FP, FN at IoU threshold thr: each predicted segment, in temporal order, is matched to the same-class
    true segment it overlaps most; a hit counts only if IoU >= thr and that true segment is still unmatched."""
    pl, ps, pe = segments(p)
    gl, gs, ge = segments(g)
    inter = np.minimum(pe[:, None], ge) - np.maximum(ps[:, None], gs)
    iou = inter / (np.maximum(pe[:, None], ge) - np.minimum(ps[:, None], gs)) * (pl[:, None] == gl)
    used, tp = np.zeros(len(gl), bool), 0
    for row in iou:
        j = row.argmax()
        if row[j] >= thr and not used[j]:
            used[j], tp = True, tp + 1
    return tp, len(pl) - tp, len(gl) - tp

def evaluate(preds, gts, thrs=(0.10, 0.25, 0.50)):
    """ms-tcn eval.py aggregation: accuracy over all frames, edit averaged over videos, F1 from pooled counts.
    Also returns F1@50 averaged per video (the aggregation of Lea et al.'s metrics.py)."""
    acc = 100 * np.mean(np.concatenate([p == g for p, g in zip(preds, gts)]))
    edit = np.mean([edit_score(p, g) for p, g in zip(preds, gts)])
    f1 = []
    for t in thrs:
        tp, fp, fn = np.sum([f1_counts(p, g, t) for p, g in zip(preds, gts)], axis=0)
        f1.append(200 * tp / (2 * tp + fp + fn))
    per_video = np.mean([200 * c[0] / max(2 * c[0] + c[1] + c[2], 1) for c in (f1_counts(p, g, 0.5) for p, g in zip(preds, gts))])
    nseg = np.mean([len(segments(p)[0]) for p in preds])
    return [acc, edit, *f1, per_video, nseg]

# --- check against a literal Python 3 port of ms-tcn eval.py (commit 33ed91c)
def ref_segments(fl, bg=(0,)):
    labels, starts, ends = [], [], []
    last = fl[0]
    if fl[0] not in bg:
        labels.append(fl[0]); starts.append(0)
    for i in range(len(fl)):
        if fl[i] != last:
            if fl[i] not in bg:
                labels.append(fl[i]); starts.append(i)
            if last not in bg:
                ends.append(i)
            last = fl[i]
    if last not in bg:
        ends.append(i + 1)
    return labels, starts, ends

def ref_edit(p, y):
    P, Y = ref_segments(p)[0], ref_segments(y)[0]
    D = np.zeros([len(P) + 1, len(Y) + 1])
    D[:, 0], D[0, :] = np.arange(len(P) + 1), np.arange(len(Y) + 1)
    for j in range(1, len(Y) + 1):
        for i in range(1, len(P) + 1):
            D[i, j] = D[i - 1, j - 1] if Y[j - 1] == P[i - 1] else min(D[i - 1, j], D[i, j - 1], D[i - 1, j - 1]) + 1
    return (1 - D[-1, -1] / max(len(P), len(Y))) * 100

def ref_f(p, y, overlap):
    pl, ps, pe = ref_segments(p)
    yl, ys, ye = ref_segments(y)
    tp, fp, hits = 0, 0, np.zeros(len(yl))
    for j in range(len(pl)):
        inter = np.minimum(pe[j], ye) - np.maximum(ps[j], ys)
        union = np.maximum(pe[j], ye) - np.minimum(ps[j], ys)
        IoU = (1.0 * inter / union) * np.array([pl[j] == yl[x] for x in range(len(yl))])
        idx = IoU.argmax()
        if IoU[idx] >= overlap and not hits[idx]:
            tp += 1; hits[idx] = 1
        else:
            fp += 1
    return tp, fp, len(yl) - hits.sum()

rng = np.random.default_rng(123)
worst = 0.0
for _ in range(300):
    g = np.repeat(rng.integers(0, 4, 12), rng.integers(1, 30, 12))           # includes background runs and 1-frame runs
    p = np.where(rng.random(len(g)) < 0.1, rng.integers(0, 4, len(g)), np.roll(g, int(rng.integers(-5, 6))))
    if len(segments(g)[0]) == 0:
        continue
    worst = max(worst, abs(edit_score(p, g) - ref_edit(list(p), list(g))),
                *(np.abs(np.subtract(f1_counts(p, g, t), ref_f(list(p), list(g), t))).max() for t in (0.1, 0.25, 0.5)))
print(f"largest difference from the eval.py port over 300 random pairs: {worst}")

g = np.repeat([0, 1, 2, 3, 0], [20, 100, 100, 100, 20])
flip = g.copy(); flip[170] = 3                                                  # one wrong frame inside a segment
late = np.repeat([0, 1, 2, 3, 0], [20, 130, 100, 70, 20])                         # both inner boundaries 30 frames late
for name, p in (("one wrong frame", flip), ("boundaries shifted by 30", late)):
    r = evaluate([p], [g])
    print(f"{name:25s} acc {r[0]:5.1f}  edit {r[1]:5.1f}  F1@10/25/50 {r[2]:5.1f} {r[3]:5.1f} {r[4]:5.1f}")

# --- experiment: frame-wise labelling with and without temporal smoothing
def fit_lr(X, y):
    return LogisticRegression(C=1.0, max_iter=3000).fit(X, y)          # lbfgs: deterministic

def stack(P, offsets=(0, 1, 2, 4, 8, 16, 32, 64)):
    """Input of a refinement stage: the previous stage's probabilities at dilated offsets (edges repeated)."""
    T = len(P)
    cols = [P[np.clip(np.arange(T) + s * d, 0, T - 1)] for d in offsets for s in ((1,) if d == 0 else (-1, 1))]
    return np.hstack(cols)

def out_of_fold(Xs, ys):
    """Probabilities for each training video from a model fitted on the other half of the training videos."""
    half = len(Xs) // 2
    out = [None] * len(Xs)
    for fit_idx, pred_idx in ((range(half), range(half, len(Xs))), (range(half, len(Xs)), range(half))):
        m = fit_lr(np.vstack([Xs[i] for i in fit_idx]), np.concatenate([ys[i] for i in fit_idx]))
        for i in pred_idx:
            out[i] = m.predict_proba(Xs[i])
    return out

def run(seed, n_train=40, n_test=20):
    rng = np.random.default_rng(seed)
    train = [make_video(rng) for _ in range(n_train)]
    test = [make_video(rng) for _ in range(n_test)]
    mu = np.vstack([x for x, _ in train]).mean(0); sd = np.vstack([x for x, _ in train]).std(0) + 1e-6
    Xtr, ytr = [(x - mu) / sd for x, _ in train], [y for _, y in train]
    Xte, yte = [(x - mu) / sd for x, _ in test], [y for _, y in test]
    m1 = fit_lr(np.vstack(Xtr), np.concatenate(ytr))
    P1 = [m1.predict_proba(x) for x in Xte]
    res = {"frame-wise": [p.argmax(1) for p in P1]}
    for w in (9, 31, 91):
        res[f"moving average {w}"] = [uniform_filter1d(p, w, axis=0, mode="nearest").argmax(1) for p in P1]
    Ptr = out_of_fold(Xtr, ytr)                                          # stage-1 outputs as seen at test time
    P = P1
    for stage in (2, 3):
        Str = [stack(p) for p in Ptr]
        m = fit_lr(np.vstack(Str), np.concatenate(ytr))
        P = [m.predict_proba(stack(p)) for p in P]
        res[f"stage {stage} (dilated, probabilities only)"] = [p.argmax(1) for p in P]
        if stage == 2:
            Ptr = out_of_fold(Str, ytr)
    true_seg = np.mean([len(segments(y)[0]) for y in yte])
    return {k: evaluate(v, yte) for k, v in res.items()}, true_seg

results, true_seg = run(0)
print(f"\nseed 0: 20 test videos, {true_seg:.1f} true action segments per video (background excluded)")
print(f"{'decoder':38s}  acc   edit  F1@10 F1@25 F1@50 | F1@50 per-video  segments")
for k, r in results.items():
    print(f"{k:38s}" + "".join(f"{v:6.1f}" for v in r[:5]) + f" | {r[5]:9.1f} {r[6]:13.1f}")

allr = [run(s)[0] for s in range(5)]
print("\nseeds 0-4, mean (min-max):  acc | edit | F1@50")
for k in results:
    a = np.array([r[k] for r in allr])
    print(f"{k:38s}" + " | ".join(f"{a[:, i].mean():5.1f} ({a[:, i].min():.1f}-{a[:, i].max():.1f})" for i in (0, 1, 4)))

largest difference from the eval.py port over 300 random pairs: 0.0
one wrong frame           acc  99.7  edit  60.0  F1@10/25/50  75.0  75.0  75.0
boundaries shifted by 30  acc  82.4  edit 100.0  F1@10/25/50 100.0 100.0 100.0

seed 0: 20 test videos, 6.1 true action segments per video (background excluded)
decoder                                 acc   edit  F1@10 F1@25 F1@50 | F1@50 per-video  segments
frame-wise                              91.3  10.5  17.5  13.2   7.1 |       7.5          61.9
moving average 9                        98.3  84.8  90.0  89.3  88.6 |      89.7           7.5
moving average 31                       96.6  92.4  95.2  94.4  94.4 |      94.7           6.5
moving average 91                       82.7  76.4  79.8  70.9  61.0 |      61.8           5.0
stage 2 (dilated, probabilities only)   98.8  82.7  84.7  84.7  81.9 |      86.6           8.3
stage 3 (dilated, probabilities only)   99.0  96.3  97.6  97.6  96.0 |      96.5           6.4

seeds 0-4, mean (min-max):  acc | edit | F1@50
frame-wise                             91.8 (91.3-92.4) |  11.3 (10.5-11.9) |   6.7 (6.0-7.1)
moving average 9                       98.7 (98.3-99.3) |  92.2 (84.8-98.2) |  92.4 (88.6-98.4)
moving average 31                      97.1 (96.6-97.6) |  96.9 (92.4-100.0) |  96.7 (94.3-100.0)
moving average 91                      82.9 (82.3-83.3) |  76.7 (74.3-78.6) |  67.3 (61.0-72.2)
stage 2 (dilated, probabilities only)  99.4 (98.8-99.7) |  89.6 (82.7-96.0) |  88.9 (81.9-97.3)
stage 3 (dilated, probabilities only)  99.3 (98.7-99.7) |  96.9 (91.8-98.9) |  97.1 (92.0-99.2)

The port and the implementation agree exactly on every random pair, including background runs, one-frame segments and segments that touch the ends of the sequence. The two toy cases show what each measure is blind to: a single wrong frame inside a 100-frame segment costs 0.3% of accuracy but splits one segment into three, so the edit score falls to 60 and F1 to 75; moving both inner boundaries by 30 frames costs 18% of accuracy, while every segment keeps its label and order and still overlaps its true segment with an IoU above 0.5, so edit and F1 stay at 100.

The frame classifier is right on about 92% of test frames over seeds 0–4, more than real features give on the benchmarks, because the synthetic task is easier, yet in seed 0 its scattered errors cut each test video, which has about six true action segments, into about 62 predicted ones, and over the five seeds the edit score of about 11 and F1@50 of about 7 describe that output as useless as a timeline, which accuracy alone would not. A moving average over 9 frames removes most of the flicker and raises accuracy as well. Over 31 frames it gives the best segment measures of the three windows at a slightly lower accuracy, a trade-off like the one in MS-TCN’s 1 fps experiment. Over 91 frames the window is longer than the shortest steps (15 frames), and in our reading it absorbs such steps into their neighbours and moves boundaries: in seed 0 fewer segments are predicted than exist and F1@50 falls much more than F1@10, and over all seeds accuracy drops below the frame-wise classifier’s.

The refinement stages see only probabilities, as MS-TCN’s later stages do. The second stage gives the highest mean frame accuracy over the five seeds but still leaves about two extra segments per video in seed 0, so its edit score is below the 31-frame moving average’s; a third stage removes most of them, and over the five seeds it matches the best moving average on edit score and F1@50 with higher accuracy, which we read as sharper boundaries. This does not reproduce MS-TCN’s stage ablation, which trained deep convolutional stages with a smoothing loss on real features; it shows the same direction, an extra stage that reads only predictions removing fragments that the stage before left. In seed 0, F1@50 averaged per video, as in Lea et al.’s code, differs from the pooled value of eval.py by up to about five points (second stage), so the two aggregations should not be mixed in one table. The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2, gave identical output in two runs, and takes about half a minute; the only change after its first run corrected a code comment on the shifted toy sequence, which had described one boundary as late and one as early; the code and its output are those of the first run.

Applications

• Instructional and cooking video. All three standard benchmarks are recordings of food preparation, and the Breakfast authors motivated their dataset by the monitoring and analysis of daily activities (Kuehne et al., 2014).
• Surgery. Surgical phase recognition labels every frame of an operation with its phase, which is temporal action segmentation in a medical domain; MS-TCN-style temporal models have been applied there, as described in surgical computer vision.
• Robotics and human–robot interaction. Lea et al. (2017, §3.4, §4.1) argue that over-segmentation errors must be avoided for human–robot interaction and video summarization, and that causal models matter for robotics.
• Skill assessment and video summarization. DiffAct’s authors list both among the task’s uses, with video surveillance (Liu et al., 2023, §1).

Open Problems

In our assessment, the main open questions as of November 2026 are these.

• Annotation cost. Frame-level labels for minutes-long videos are expensive. Timestamp supervision came within a few points of fully supervised MS-TCN in frame accuracy on Breakfast in Li et al.’s comparison, while transcript- and set-based methods remained far behind.
• Small, saturated benchmarks. GTEA has 28 videos and 50Salads 50; the gaps between recent methods on them are a few points, and splits, aggregation conventions and copied table rows make small differences hard to interpret.
• Frozen features. Most models still read I3D features extracted once and never fine-tuned; MS-TCN’s GTEA experiment suggests that fine-tuning helps, at a cost the standard pipeline avoids.
• Long-range structure. Receptive fields of a few thousand frames, local attention windows and the authors’ own observation that MS-TCN does slightly worse on longer GTEA videos (Table 8) point to the difficulty of modelling the order of steps over whole procedures.
• Online use. The benchmarks and most methods are offline; labelling a procedure as it happens, from past frames only, is the setting robotics and surgical assistance need.