Semi-Supervised Learning

Semi-supervised learning trains a model on a small set of labelled examples together with a larger set of unlabelled ones drawn from the same source. The labelled images teach the task; the unlabelled images, which are cheap to collect, are used to learn the structure of the data, such as where its clusters lie and which images resemble each other, so that the few labels go further. Murphy (2022, §19.3) states the goal as learning the high-level structure of the data distribution from unlabelled data while relying on the labels only for the fine-grained details of the task. The machine learning article places the setting between supervised and unsupervised learning. This article covers the assumptions that make unlabelled data useful, the main families of methods (pseudo-labelling and self-training, consistency regularization, teacher–student training, confidence-thresholded weak-to-strong training and graph methods), how semi-supervised learning relates to self-supervised pretraining, and the ways it fails.

Purpose

Labels are the expensive part of most vision datasets. A pixel-accurate segmentation mask or an expert’s reading of a medical image costs far more than the image itself, and the semantic segmentation article names that cost as the motivation for weak, semi-supervised and synthetic-data training. Semi-supervised learning assumes two samples: labelled pairs $(\mathbf{x}, y) \sim p(\mathbf{x}, y)$ and many more inputs $\mathbf{x} \sim p(\mathbf{x})$ without labels (Murphy, §19.3). It asks how the second sample can improve an estimate of $p(y \mid \mathbf{x})$.

On its own, $p(\mathbf{x})$ says nothing about $p(y \mid \mathbf{x})$. Zhu’s survey (2008, §1) answers the question “can we really learn anything from unlabeled data?” with yes, under certain assumptions: the gain comes from a good match between the structure of the problem and the assumptions of the method, and a bad match can make the classifier worse. Every method below therefore rests on an assumption that ties the labels to the shape of $p(\mathbf{x})$.

When to Use It

Semi-supervised learning fits when unlabelled images of the task’s own distribution are plentiful, labels are scarce, and the unlabelled images can be assumed to contain the same classes as the labelled ones. Large-scale self-training of ImageNet classifiers with about 300 million unlabelled images (Xie et al., 2020), and semi-supervised segmentation benchmarks with 92 to 1,464 labelled images out of 10,582 (Yang et al., 2023), are examples.

It is a weaker fit in three situations:

• Labels are not scarce. With enough labels, the supervised model is already close to its ceiling. In the example below, every method is within about seven points of the others at 100 labels per class, and none beats the supervised fit by more than a tenth of a point.
• The unlabelled images come from another distribution. If they contain classes absent from the labelled set, or come from another domain, the assumptions below fail. Guo et al. (2020) report that several deep semi-supervised methods fell below the supervised baseline when 40% of the unlabelled images came from unseen classes (see Pitfalls). Adapting to a shifted but related domain is the problem of domain adaptation.
• A pretrained representation is available. Transfer learning from a model pretrained on other data, or self-supervised pretraining on the unlabelled images, followed by fine-tuning, can give the same benefit more simply, and the two are often combined (see “Relation to self-supervised learning” below).

How It Works

Assumptions

Three assumptions recur. The sources group them slightly differently.

• Smoothness. Points that are close are likely to share a label. Chen et al. (2021, §2) list smoothness, consistency, low-density and clustered data as the assumptions behind most classical solutions, and gloss them as “neighboring data have a high probability of belonging to the same class” or a decision boundary in low-density regions.
• Cluster or low-density assumption. The decision boundary should pass through regions where data are sparse, not through clusters. Murphy (§19.3.2.1) calls this the “cluster assumption”: unlabelled data estimate the shape of the data manifold, and the method moves the decision boundary away from it.
• Manifold assumption. Murphy (§19.3.4) uses the term for the idea that two data points that are “similar” in a meaningful way are likely to share a label, and builds label propagation on it. Zhu (§6) says graph methods usually assume label smoothness over the graph.

Zhu (§1) gives a counterexample to the low-density assumption: if two classes are heavily overlapping Gaussians, the best boundary goes through the densest region, and methods that avoid dense regions perform badly there.

Self-training and pseudo-labelling

Self-training trains a classifier on the labelled data, uses it to predict labels for unlabelled data, adds some of these predictions to the training set as if they were true labels, and repeats. Zhu (§3) describes adding the most confident unlabelled points with their predicted labels, and notes that the classifier uses its own predictions to teach itself, so that a mistake can reinforce itself. Murphy (§19.3.1) cites work from 1965 to 1975 for it and notes that the predicted labels are now often called pseudo-labels, a name he credits to Lee (2013). Yarowsky (1995) is an early example: starting from a few seed collocations for each sense of a word, a decision-list classifier tags the rest of the corpus, the examples tagged with probability above a threshold join the seed sets, and the process repeats until the untagged residual stops changing (his Steps 3b–4).

Murphy distinguishes two procedures. The offline procedure pseudo-labels the whole unlabelled set, retrains a model to convergence, and relabels; it has worked well with very large unlabelled collections. The online procedure pseudo-labels random batches and trains on them immediately, and is often one component of a larger method. Murphy writes that neither is fundamentally better: offline labels go stale between rounds, and online relabelling costs more computation.

Two choices control self-training. A selection metric decides which pseudo-labels are kept. The common one keeps a pseudo-label only if the largest class probability exceeds a threshold, which keeps mostly correct labels if the probabilities are well calibrated (Murphy, §19.3.1); the expected calibration error article measures how far they are from that. Hard or soft labels decide what is kept: the arg max class, or the full predicted distribution. scikit-learn’s SelfTrainingClassifier implements a hard-label version that refits the classifier between rounds, with either a probability threshold or the $k$ most confident predictions per round, adds the selected images to the labelled set permanently (unlike Murphy’s offline procedure, it never relabels them), and stops after a set number of rounds or when no image is added; its documentation states that the threshold criterion should be used with a well-calibrated classifier (scikit-learn 1.7.2 source).

Self-training also lowers the entropy of the model’s predictions. In the online setting with a cross-entropy loss, training on hard pseudo-labels minimizes $-\max_c \log p_\theta(y = c \mid \mathbf{x})$ on each unlabelled image (Murphy, Eq. 19.16). Entropy minimization minimizes the entropy $-\sum_c p_\theta(y = c \mid \mathbf{x}) \log p_\theta(y = c \mid \mathbf{x})$ instead, the same loss with a soft target. Murphy explains that a smooth decision boundary through a dense region must produce high-entropy predictions for some images, so both losses push the boundary into low-density regions, the cluster assumption in practice (§19.3.2). Raising the target probabilities to the power $1/T$ and renormalizing interpolates between the two: $T = 1$ gives entropy minimization, and $T \to 0$ gives hard self-training. Murphy names this sharpening as the basis of MixMatch (Berthelot et al., 2019).

Consistency regularization

Consistency regularization uses the idea that a perturbed input, or a perturbed model, should not change the prediction much. Since the loss needs only the model’s own outputs, it applies to unlabelled images. In Murphy’s basic form (§19.3.5, Eq. 19.27), with $M$ labelled and $N$ unlabelled images,

$$\mathcal{L}(\theta) = -\sum_{i=1}^{M} \log p_\theta(y = y_i \mid \mathbf{x}_i) + \lambda \sum_{j=1}^{N} \big\lVert p_\theta(y \mid \mathbf{x}_j) - p_\theta(y \mid \mathbf{x}'_j) \big\rVert^2,$$

where $\mathbf{x}'_j$ is a random transformation of $\mathbf{x}_j$ (in vision, a data augmentation such as a rotation or added noise), the model may itself be stochastic (for example through dropout), and the first term $p_\theta(y \mid \mathbf{x}_j)$ is usually treated as a fixed target, with no gradient through it. Murphy lists the design choices:

• The weight $\lambda$. If it is too large, the model reinforces its own bad predictions, as in self-training. Because the model is poor at the start, $\lambda$ is often started at zero and increased during training.
• The transformations. They should not change the label, and strong augmentations that corrupt the input heavily (but arguably keep the label) have given particularly strong results.
• Adversarial perturbations. Virtual adversarial training (VAT; Miyato et al., 2019), as Murphy describes it, replaces random transformations with a perturbation chosen to change the model’s output the most, approximated by one gradient step from a random direction.
• The distance. Squared error and KL divergence are both used. The squared error stops updating when predictions are very unstable, while the KL divergence has the same scale as the cross-entropy on labelled data, which makes $\lambda$ easier to tune.

Guo et al. (2020, §4.1) describe two early variants they compare against: the Π-model, which penalizes the distance between the predictions for an unlabelled image and for a stochastic perturbation of it, and Temporal Ensembling (Laine & Aila, 2017), which uses an ensemble of predictions accumulated during training as the target.

Teacher–student methods

Several methods separate the model that produces targets (the teacher) from the model that learns from them (the student).

Mean Teacher (Tarvainen & Valpola, 2017), as Chen et al. (2021, §4) and Guo et al. (2020) describe it, feeds two differently augmented versions of an unlabelled image to a student network and to a teacher of the same architecture whose parameters $\bar\theta$ are an exponential moving average of the student’s parameters $\theta$. The student is trained to match the teacher’s predictions, with no gradient through the teacher. Chen et al. call Mean Teacher an extension of Temporal Ensembling. The same weight averaging appears without labels in DINO: Caron et al. (2021, §3.1) update their teacher as $\theta_t \leftarrow \lambda \theta_t + (1 - \lambda)\theta_s$ and write that its role may be closer to that of the mean teacher used in self-training than to the momentum encoder of contrastive learning.

Noisy Student (Xie et al., 2020) is offline self-training at scale (Algorithm 1). A teacher is trained on labelled images; the teacher, without noise, produces soft or hard pseudo-labels for unlabelled images; an equal-or-larger student is trained on labelled and pseudo-labelled images together with noise injected (RandAugment data augmentation, dropout and stochastic depth); and the student becomes the next teacher. The authors filter out images on which the teacher has low confidence, since these are usually out-of-domain, and balance the number of pseudo-labelled images per class. They hypothesize that noising the student is what lets it surpass the teacher: with soft pseudo-labels, a student identical to the teacher would get zero loss on unlabelled images (§4). The EfficientNet article reports the method’s ImageNet results and its EfficientNet-L2 model. Xie et al. contrast their method with knowledge distillation, where a small student is often used to be faster than the teacher and adding noise is not the core concern; they call theirs “knowledge expansion”.

Meta Pseudo Labels (Pham et al., 2021) keeps the teacher training during the student’s training: the student learns from the teacher’s pseudo-labels on unlabelled images, and the teacher is updated from how well the student then performs on labelled images, a mechanism the authors design for the teacher to correct its bias by observing how its pseudo-labels affect the student (§1–2).

Depth Anything applies the teacher–student recipe to monocular depth: a teacher pseudo-labels about 62 million unlabelled images, and the student is trained on them under strong colour distortions and CutMix.

Weak-to-strong consistency with a confidence threshold

FixMatch (Sohn et al., 2020) combines pseudo-labelling and consistency. As Yang et al. (2023, §3.1) restate it, each unlabelled image $x^u$ is perturbed weakly by $A_w$ (for example cropping) and strongly by $A_s$ (for example colour jitter); the prediction on the weak view supervises the prediction on the strong view, but only where the weak prediction is confident:

$$\mathcal{L}_u = \frac{1}{B_u} \sum \mathbb{1}\big(\max(p^w) \ge \tau\big)\, H(p^w, p^s), \qquad p^w = \hat F\big(A_w(x^u)\big), \quad p^s = F\big(A_s(A_w(x^u))\big),$$

where $B_u$ is the unlabelled batch size, $\tau$ is a fixed confidence threshold, $H$ is the cross-entropy, and the teacher $\hat F$ is the same network as the student $F$ (their Eqs. 2–3, “following FixMatch”). Yang et al. add this to the supervised loss with equal weights (their Eq. 1) and set $\tau = 0.95$ for every dataset except Cityscapes. Their reasoning is that the model is more likely to predict well on the weak view, while the strong view gives more to learn from and mitigates confirmation bias. Without the strong perturbations, they write, FixMatch degenerates to naive online self-training.

Graph methods

Graph methods build a graph whose nodes are all the labelled and unlabelled examples and whose edge weights $w_{ij}$ measure similarity, and spread the known labels along the edges. Zhu (§6) describes them as nonparametric, discriminative and transductive: they label the given unlabelled points rather than learn a function for new ones. In Zhu’s terms, each method estimates a function $f$ on the nodes that should be close to the given labels and smooth on the graph, and the methods differ in the loss and the regularizer.

Label propagation (the harmonic function of Zhu, Ghahramani & Lafferty, 2003) clamps the labelled nodes to their labels and makes the value at each unlabelled node the weighted average of its neighbours (their Eq. 3). With the weight matrix $W$ and degree matrix $D$ split into labelled ($l$) and unlabelled ($u$) blocks, the solution is

$$f_u = (D_{uu} - W_{uu})^{-1} W_{ul} f_l$$

(their Eq. 5), which minimizes $\sum_{i,j} w_{ij}(f_i - f_j)^2$ with the labels held fixed. Murphy (§19.3.4) describes an iterative version, which Zhu (§6.1.3) calls equivalent to harmonic functions: multiply the label matrix by the normalized transition matrix, renormalize, reset the labelled rows to their one-hot labels, and repeat until nothing changes.

Label spreading, the local and global consistency method of Zhou et al. (2004) as Zhu (§6.1.4) describes it, does not clamp the labels. It uses a squared loss to the initial labels $Y$ and the normalized Laplacian $I - S$, with $S = D^{-1/2} W D^{-1/2}$, as the regularizer. Writing the objective with a weight $\mu$ and differentiating (our derivation),

$$\lVert F - Y \rVert^2 + \mu\, \operatorname{tr}\!\big(F^\top (I - S) F\big) \;\Rightarrow\; F = \tfrac{1}{1+\mu} (I - \alpha S)^{-1} Y, \qquad \alpha = \tfrac{\mu}{1+\mu},$$

and each unlabelled image takes the class of the largest entry in its row. scikit-learn’s LabelSpreading iterates $F \leftarrow \alpha S F + (1 - \alpha) Y$ (at most 30 times by default), whose fixed point $(1 - \alpha)(I - \alpha S)^{-1} Y$ has the same row-wise arg max; its documentation calls $\alpha$ the clamping factor, the relative amount by which an example adopts its neighbours’ information instead of its initial label, with a default of 0.2. A labelled node may change its label, which Zhu’s loss allows, and scikit-learn’s documentation describes label spreading as more robust to noise than its LabelPropagation.

The graph matters more than the method. Zhu (§6.1) believes it is more important to construct a good graph than to choose among the methods, and Murphy notes that Euclidean distance may not reflect class membership for high-dimensional data. On images, graphs are built on features rather than raw pixels. The same propagation of seed labels over a pixel graph underlies the random walker of interactive segmentation, and a graph neural network spreads features rather than labels along a graph’s edges.

Relation to self-supervised learning

The two settings use the same data but differ in what they produce. Self-supervised learning derives a training signal from unlabelled images alone to learn a representation for later tasks; semi-supervised learning uses the unlabelled images and the labels together for one task. In the context of deep learning, the Deep Learning book (§7.6) writes that semi-supervised learning usually refers to learning a representation in which examples of the same class are close, in which a linear classifier may generalize better in many cases.

The two are often combined. Zhai et al. (2019) train one network with a supervised loss on labelled images and a self-supervised loss (predicting rotations, or an exemplar loss) on the unlabelled images, optionally also on the labelled ones, which they call S4L. In their Table 1 (ResNet50v2 on ILSVRC-2012, top-5 accuracy), S4L-Rotation reached 83.82% with 10% of the labels and 53.37% with 1%, against 80.43% and 48.43% for the supervised baseline and 82.41% and 51.56% for pseudo-labels. Fine-tuning the rotation-pretrained network instead gave 78.53% and 45.11%. Murphy (§19.3.7) describes a pipeline that pretrains with SimCLR on the unlabelled data, fine-tunes on the labels, and distils the result into a student on the unlabelled data.

How to Apply It

In our reading, a reasonable starting point is a pretrained backbone, a supervised baseline tuned on the same labels, and one well-tested semi-supervised recipe for the task: weak-to-strong consistency with a confidence threshold for classification and segmentation (Yang et al. build their segmentation method on FixMatch), or offline teacher–student pseudo-labelling when the unlabelled set is very large (Xie et al., 2020). The supervised baseline is essential: the gain from semi-supervised learning is the difference between the two, and Pham et al. (2021, §3) tune their baselines with the same resources as their own method for that reason.

Example: thresholded self-training and label spreading

The example below compares two convex or closed-form methods on synthetic images with few labels. Each image is $24 \times 24$ pixels and shows one shape (disk, square, triangle or cross) with a random centre (within 3 pixels of the middle), circumradius (4 to 9 pixels), rotation and contrast on a grey background with Gaussian noise. The pool has 500 images per class whose labels are hidden from the methods, apart from the few that are revealed; 1,000 further images serve only for the oracle’s test score.

Instead of raw pixels, each image is described by six fixed, hand-designed features of its largest bright region after blurring and Otsu thresholding: solidity, circularity, eccentricity and the logarithms of the first three Hu moments, all unchanged by shifts, rotations and scaling, so that a linear classifier sees the shape rather than its position or orientation. They stand in for the frozen features of a pretrained backbone. A scaler fitted on the pool, which uses no labels, standardizes them.

With $m$ labels per class revealed ($m$ = 1, 3, 10 or 100, drawn ten times with seeds 0–9), the code compares:

• supervised: logistic regression ($C = 1$) on the labelled images only;
• self-training: scikit-learn’s SelfTrainingClassifier around the same logistic regression, at thresholds $\tau$ = 0, 0.6, 0.9 and 0.99 and its defaults otherwise (at most 10 rounds; an image is added when its largest class probability exceeds $\tau$, and is never removed);
• label spreading: the closed form $F = (I - \alpha S)^{-1} Y$ on a symmetric 7-nearest-neighbour graph of the pool ($w_{ij} = 1$ if either image is among the other’s 7 nearest neighbours in feature space; 7 is scikit-learn’s default n_neighbors for its knn kernel), at $\alpha = 0.2$ (scikit-learn’s default) and $\alpha = 0.99$ (our choice of a value near 1). The sources leave $\alpha$ open, so both are printed. scikit-learn’s own knn graph differs: it uses the directed graph from kneighbors_graph, which lists each image among its own 7 neighbours, normalizes it with SciPy’s normalized Laplacian and then zeroes the diagonal (scikit-learn 1.7.2 source).

The score is accuracy on the hidden pool images, a transductive measure that every method can be given. For self-training the code also prints the share of hidden images that received a pseudo-label, the share of those pseudo-labels that are wrong (counted as 0 for a draw that pseudo-labels nothing), and the share of wrong pseudo-labels that the final model still predicts (“wrong kept”), a direct measure of confirmation bias. “Wrong kept” is averaged over the seeds on which at least one pseudo-label was wrong; it is undefined, and would print as nan, if no seed had one. Three checks come first: an oracle fitted with all 2,000 pool labels shows that the labels can be learned from the features and that the model cannot fit them perfectly; the share of each image’s 7 neighbours with its own class shows how well the graph follows the classes; and the closed form is compared with 20,000 iterations of $F \leftarrow \alpha S F + Y$, whose fixed point $(I - \alpha S)^{-1}Y$ has the same row-wise arg max as scikit-learn’s iteration.

The code uses NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2, run with OMP_NUM_THREADS=1. It took between about 6 and 30 seconds on one CPU thread, depending on the load from other jobs on the machine, and two runs printed identical output.

import sys
import numpy as np
from scipy import ndimage as ndi, sparse
from scipy.sparse.linalg import spsolve
from skimage.filters import threshold_otsu
from skimage.measure import label, regionprops
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import StandardScaler
from sklearn.semi_supervised import SelfTrainingClassifier

SIZE, SS = 24, 4                                  # image size, supersampling for anti-aliasing
CLASSES = ["disk", "square", "triangle", "cross"]
SEEDS = [int(s) for s in sys.argv[1:]] or list(range(10))


def shape_mask(kind, cx, cy, r, theta):
    """Fraction of each pixel covered by one shape of circumradius r, rotated by theta."""
    yy, xx = (np.mgrid[0:SIZE * SS, 0:SIZE * SS] + 0.5) / SS
    u, v = xx - cx, yy - cy
    a, b = np.cos(theta) * u + np.sin(theta) * v, -np.sin(theta) * u + np.cos(theta) * v
    if kind == "disk":
        m = u ** 2 + v ** 2 <= r ** 2
    elif kind == "square":
        m = (np.abs(a) <= r / np.sqrt(2)) & (np.abs(b) <= r / np.sqrt(2))
    elif kind == "triangle":
        m = np.ones_like(a, bool)
        for k in range(3):
            ang = theta + np.pi / 2 + 2 * np.pi * k / 3
            m &= u * np.cos(ang) + v * np.sin(ang) >= -r / 2
    else:                                         # cross: two bars of half-length r, width r/2.5
        w = r / 5
        m = ((np.abs(a) <= r) & (np.abs(b) <= w)) | ((np.abs(b) <= r) & (np.abs(a) <= w))
    return m.reshape(SIZE, SS, SIZE, SS).mean(axis=(1, 3))


def render(n_per_class, rng):
    X, y, area = [], [], []
    for k, kind in enumerate(CLASSES):
        for _ in range(n_per_class):
            cx, cy = SIZE / 2 + rng.uniform(-3, 3, 2)
            m = shape_mask(kind, cx, cy, rng.uniform(4, 9), rng.uniform(0, 2 * np.pi))
            img = 0.2 + rng.uniform(0.3, 0.7) * m + rng.normal(0, 0.1, (SIZE, SIZE))
            X.append(np.clip(img, 0, 1))
            y.append(k)
            area.append(m.sum())
    return np.array(X), np.array(y), np.array(area)


def features(X):
    """Rotation-, scale- and shift-invariant descriptors of the largest bright region."""
    out = []
    for img in X:
        img = ndi.gaussian_filter(img, 1.0)
        reg = max(regionprops(label(img > threshold_otsu(img))), key=lambda p: p.area)
        out.append([reg.solidity, 4 * np.pi * reg.area / reg.perimeter ** 2, reg.eccentricity,
                    *np.log(np.abs(reg.moments_hu[:3]) + 1e-12)])
    return np.array(out)


rng = np.random.default_rng(0)
Xpool, ypool, area = render(500, rng)             # 2,000 images whose labels are hidden...
Xtest, ytest, _ = render(250, rng)                # ...and 1,000 for the oracle's test score
print(f"pool per class {np.bincount(ypool)}, pixels in [{Xpool.min():.2f}, {Xpool.max():.2f}], "
      f"smallest shape {area.min():.1f} px")
scaler = StandardScaler().fit(features(Xpool))    # uses unlabeled images only
F, Ftest = scaler.transform(features(Xpool)), scaler.transform(features(Xtest))
n = len(ypool)

LR = lambda: LogisticRegression(C=1.0, max_iter=5000)
oracle = LR().fit(F, ypool)
print(f"oracle, all {n} labels: pool accuracy {oracle.score(F, ypool):.3f}, "
      f"test accuracy {oracle.score(Ftest, ytest):.3f}")

# symmetric 7-nearest-neighbour graph on the features, S = D^-1/2 W D^-1/2
idx = NearestNeighbors(n_neighbors=8).fit(F).kneighbors(F)[1][:, 1:]
W = sparse.csr_matrix((np.ones(idx.size), (np.repeat(np.arange(n), 7), idx.ravel())), shape=(n, n))
W = ((W + W.T) > 0).astype(float)
d = np.asarray(W.sum(1)).ravel()
S = (sparse.diags(d ** -0.5) @ W @ sparse.diags(d ** -0.5)).tocsc()
vote = np.array([np.bincount(ypool[i], minlength=4).argmax() for i in idx])
print(f"7-NN graph: same-class neighbours {(ypool[idx] == ypool[:, None]).mean():.3f}, "
      f"leave-one-out 7-NN vote accuracy {(vote == ypool).mean():.3f}")


def spread(lab, alpha):
    """Label spreading in closed form: F = (I - alpha S)^-1 Y."""
    Y = np.zeros((n, 4))
    Y[lab, ypool[lab]] = 1
    return spsolve(sparse.identity(n, format="csc") - alpha * S, Y), Y


TAUS, ALPHAS = (0.0, 0.6, 0.9, 0.99), (0.2, 0.99)
for m in (1, 3, 10, 100):                         # labels per class
    rows = {}
    for seed in SEEDS:
        r = np.random.default_rng([seed, m])
        lab = np.concatenate([r.choice(np.flatnonzero(ypool == k), m, replace=False) for k in range(4)])
        unl = np.setdiff1d(np.arange(n), lab)
        acc = lambda pred: (pred[unl] == ypool[unl]).mean()
        rows.setdefault("supervised", []).append([acc(LR().fit(F[lab], ypool[lab]).predict(F))])
        for tau in TAUS:
            y_in = np.full(n, -1)
            y_in[lab] = ypool[lab]
            st = SelfTrainingClassifier(LR(), threshold=tau).fit(F, y_in)
            pred = st.predict(F)
            took = unl[st.labeled_iter_[unl] > 0]              # pseudo-labelled images
            wrong = took[st.transduction_[took] != ypool[took]]
            kept = (pred[wrong] == st.transduction_[wrong]).mean() if len(wrong) else np.nan
            rows.setdefault(f"self-training tau={tau}", []).append(
                [acc(pred), len(took) / len(unl), len(wrong) / max(len(took), 1), kept])
        for alpha in ALPHAS:
            Fs, Y = spread(lab, alpha)
            rows.setdefault(f"label spreading alpha={alpha}", []).append([acc(Fs.argmax(1))])
            if m == 3 and seed == SEEDS[0]:
                Fi = np.zeros_like(Y)
                for _ in range(20000):
                    Fi = alpha * (S @ Fi) + Y
                print(f"[check] alpha={alpha}: max |closed form - iteration| = {np.abs(Fs - Fi).max():.1e}")
    print(f"\n{m} label(s) per class, seeds {SEEDS[0]}-{SEEDS[-1]}: mean (min-max) accuracy on unlabeled images")
    print(f"{'method':30s} {'accuracy':>20s} {'pseudo-labelled':>16s} {'of them wrong':>14s} {'wrong kept':>11s}")
    for name, v in rows.items():
        v = np.array(v)
        line = f"{name:30s} {v[:, 0].mean():.3f} ({v[:, 0].min():.3f}-{v[:, 0].max():.3f})"
        if v.shape[1] > 1:
            line += f" {np.mean(v[:, 1]):16.3f} {np.mean(v[:, 2]):14.3f} {np.nanmean(v[:, 3]):11.3f}"
        print(line)

pool per class [500 500 500 500], pixels in [0.00, 1.00], smallest shape 20.7 px
oracle, all 2000 labels: pool accuracy 0.968, test accuracy 0.966
7-NN graph: same-class neighbours 0.888, leave-one-out 7-NN vote accuracy 0.918

1 label(s) per class, seeds 0-9: mean (min-max) accuracy on unlabeled images
method                                     accuracy  pseudo-labelled  of them wrong  wrong kept
supervised                     0.613 (0.474-0.736)
self-training tau=0.0          0.613 (0.467-0.741)            1.000          0.387       0.980
self-training tau=0.6          0.590 (0.416-0.706)            0.976          0.405       0.996
self-training tau=0.9          0.418 (0.251-0.510)            0.923          0.566       1.000
self-training tau=0.99         0.610 (0.474-0.736)            0.001          0.183       1.000
label spreading alpha=0.2      0.590 (0.455-0.737)
label spreading alpha=0.99     0.703 (0.476-0.808)
[check] alpha=0.2: max |closed form - iteration| = 1.1e-15
[check] alpha=0.99: max |closed form - iteration| = 3.3e-15

3 label(s) per class, seeds 0-9: mean (min-max) accuracy on unlabeled images
method                                     accuracy  pseudo-labelled  of them wrong  wrong kept
supervised                     0.754 (0.706-0.791)
self-training tau=0.0          0.752 (0.697-0.793)            1.000          0.246       0.976
self-training tau=0.6          0.743 (0.684-0.774)            0.982          0.252       0.989
self-training tau=0.9          0.544 (0.469-0.739)            0.763          0.359       1.000
self-training tau=0.99         0.746 (0.659-0.790)            0.083          0.252       1.000
label spreading alpha=0.2      0.746 (0.681-0.797)
label spreading alpha=0.99     0.788 (0.735-0.856)

10 label(s) per class, seeds 0-9: mean (min-max) accuracy on unlabeled images
method                                     accuracy  pseudo-labelled  of them wrong  wrong kept
supervised                     0.846 (0.810-0.872)
self-training tau=0.0          0.847 (0.807-0.874)            1.000          0.154       0.963
self-training tau=0.6          0.820 (0.744-0.864)            0.974          0.171       0.986
self-training tau=0.9          0.769 (0.736-0.847)            0.701          0.110       1.000
self-training tau=0.99         0.853 (0.814-0.879)            0.152          0.019       1.000
label spreading alpha=0.2      0.798 (0.769-0.826)
label spreading alpha=0.99     0.820 (0.786-0.864)

100 label(s) per class, seeds 0-9: mean (min-max) accuracy on unlabeled images
method                                     accuracy  pseudo-labelled  of them wrong  wrong kept
supervised                     0.938 (0.927-0.950)
self-training tau=0.0          0.935 (0.921-0.948)            1.000          0.062       0.977
self-training tau=0.6          0.928 (0.916-0.936)            0.970          0.061       0.999
self-training tau=0.9          0.929 (0.910-0.944)            0.538          0.014       1.000
self-training tau=0.99         0.939 (0.929-0.950)            0.190          0.001       1.000
label spreading alpha=0.2      0.880 (0.871-0.892)
label spreading alpha=0.99     0.869 (0.843-0.887)

The checks. The oracle’s accuracy on the pool and on the test images is about 0.97, so the features carry the labels but a linear model cannot interpolate them. In the graph, 89% of neighbours share an image’s class, and a vote among the 7 neighbours with every other label known is right 92% of the time. The closed form agrees with the iteration to rounding error.

Label spreading helps most with the fewest labels. With one label per class, the supervised fit reaches 0.61 on average, and the range over draws (0.47–0.74) shows how much depends on which four images are labelled. Spreading with $\alpha = 0.99$ adds about nine points at one label per class and three at three, and has the best mean of all methods in both cases. With 10 and 100 labels per class it falls behind the supervised fit, and at 100 it is about seven points below it and below the leave-one-out neighbour vote. In our reading, the graph links 11% of neighbours across classes, and spreading follows those links whatever the number of labels, while a linear boundary on these features does better once enough labels fix it. With $\alpha = 0.2$, spreading is never better than the supervised fit here; in our reading, each further step of the series $(I - \alpha S)^{-1} = \sum_k \alpha^k S^k$ is weighted five times less than the one before, so a hidden image takes roughly the class of the labelled images fewest steps away, whatever lies beyond them.

Self-training never helps by much and can fail badly. At $\tau = 0$ every hidden image is pseudo-labelled in the first round (share 1.000), so the share of wrong pseudo-labels is simply the supervised fit’s error, and refitting on the supervised fit’s own predictions returns almost the same model: the accuracy matches the supervised fit to within 0.003 at every $m$. At $\tau = 0.99$ only a small share is ever accepted (0.1% at one label per class, 15% at ten), and the result stays within about a point of the supervised fit, slightly above it at 10 and 100 labels per class. At $\tau = 0.6$ almost everything is accepted over the rounds and accuracy drops by one to three points at 1–10 labels per class. At $\tau = 0.9$ self-training collapses with few labels: at one label per class it pseudo-labels 92% of the hidden images, more than half of those labels are wrong, and its mean accuracy falls to 0.42, with the worst draw at 0.25, the level of guessing among four classes. At ten labels per class it is still eight points below the supervised fit, and only at 100 does it recover.

Wrong pseudo-labels are kept. The final model still predicts between 96% and 100% of the wrong pseudo-labels it accepted, at every threshold and every $m$. Once an error enters the training set, the refitted model agrees with it: the confirmation bias that Murphy describes. The output does not show why $\tau = 0.9$ fails where $\tau = 0$ and $\tau = 0.99$ do not. In our reading, $\tau = 0$ commits all of its errors at once to a model that already makes them, $\tau = 0.99$ accepts too few images to move the boundary, while $\tau = 0.9$ accepts a confident subset in each of ten rounds, and each refit on a subset that the previous model chose can move the boundary further from the truth before the next round’s selection.

Against the expectations in the plan. Before the run we expected that spreading with $\alpha = 0.99$ would beat the supervised fit at one and three labels per class, which held. We expected that self-training at $\tau = 0.9$ or 0.99 would match or beat the supervised fit at 3 and 10 labels per class; this failed at $\tau = 0.9$ at both, and at $\tau = 0.99$ with three labels per class, where it ends just below the supervised fit. We expected the share of wrong pseudo-labels to rise as the threshold falls, and most wrong pseudo-labels to be kept; this held only in part: they are kept, but with 1, 3 and 10 labels per class the wrong share does not rise steadily as the threshold falls, and with 1 and 3 it is highest at $\tau = 0.9$. We expected all methods to be within a few points of each other and of the oracle at 100 labels per class; this held only in part, since spreading stays six to seven points behind the supervised fit. A higher threshold is not automatically safer: what a threshold means depends on how the classifier’s probabilities are calibrated, as scikit-learn’s documentation warns, and on how many rounds the procedure runs.

The graph neural network article’s example is the feature-propagation counterpart of the label spreading here: with two labels per class on a superpixel graph, it propagates features rather than labels and shows accuracy first rising and then falling with the number of propagation steps.

Variants

• Offline self-training relabels the whole unlabelled set between training runs, as Noisy Student does; online pseudo-labelling labels each batch as it is drawn (Murphy, §19.3.1).
• Co-training trains two models on two views of the data (two independent sets of features) and lets each add the other’s confident pseudo-labels; tri-training uses three models trained on bootstrap samples and adds a pseudo-label to one model’s training set when the other two agree (Murphy, §19.3.3).
• Entropy minimization and sharpened targets, as in MixMatch, sit between soft and hard self-training (Murphy, §19.3.2).
• Consistency regularization with random augmentation, dropout or adversarial perturbation (VAT), and its averaged-target versions, Temporal Ensembling and Mean Teacher, as Guo et al. (2020, §4.1) describe them.
• Thresholded weak-to-strong consistency in the manner of FixMatch, and per-class thresholds, which Yang et al. (2023, §2) attribute to FlexMatch and FreeMatch.
• Cross pseudo supervision (Chen et al., 2021) trains two differently initialized segmentation networks, each supervised by the other’s hard pseudo-labels.
• Graph methods: mincut, label propagation (harmonic functions), label spreading and manifold regularization (Zhu, §6).
• Generative methods that model $p(\mathbf{x})$ with a VAE, a GAN or a flow and share parameters with the classifier (Murphy, §19.3.6; Deep Learning book, §7.6).
• Semi-supervised segmentation and detection apply the same ideas per pixel or per box; Yang et al. (2023) describe the move from GAN-based training to consistency regularization and self-training in segmentation.

Best Practices

• Compare with a strong supervised baseline on the same labels, architecture, augmentation and tuning budget. Pham et al. (2021, §3) do this deliberately, and Zhai et al. (2019) compare their methods with carefully tuned baselines.
• Keep the teacher clean and the student challenged. Xie et al. (2020, Table 6) found that removing noise from the student, or adding it to the teacher, lowered accuracy. Yang et al. (2023, Table 1) found the same for FixMatch’s strong perturbations in segmentation.
• Choose thresholds on held-out labels of realistic size, not by intuition. In the example above, the threshold that failed was neither the lowest nor the highest tried.
• Check that the unlabelled images match the labelled classes and domain; filter out-of-distribution images, as Noisy Student filters low-confidence ones.
• Build the graph on good features. For graph methods, the quality of the graph limits what propagation can achieve (Zhu, §6.1); in our suggestion, the share of same-class neighbours among labelled images is a cheap check of it.

Pitfalls

• Confirmation bias. A model trained on its own wrong predictions confirms them. Murphy (§19.3.1) credits the name to Tarvainen and Valpola (2017), while Pham et al. (2021, §1) cite Arazo et al. for “the problem of confirmation bias in pseudo-labeling”. The example shows the effect directly: the final model keeps almost every wrong pseudo-label it accepted, and at one threshold the worst draw falls to chance level.
• Class distribution mismatch. Guo et al. (2020, §4) labelled 10 MNIST images per class from six classes and varied the share of unlabelled images from four other classes. All the deep semi-supervised methods they compared (Pseudo-Labeling, the Π-model, Temporal Ensembling, Mean Teacher and VAT) were clearly better than the supervised baseline with no mismatch, but their accuracy fell rapidly as the mismatch grew, and many fell below the baseline at 40% unseen-class images; they report the same pattern on a six-class CIFAR-10 task with a Wide ResNet-28-10. Guo et al. cite Oliver et al. (2018) among the reports of such degradation.
• Unrealistic evaluation. Many results tune hyperparameters on a validation set much larger than the labelled training set, which a practitioner with few labels would not have. Zhai et al. (2019, §4) note that a validation set of the usual size does not correspond to a realistic scenario and run their 1% experiments with 5,000 validation images. Pham et al. (2021, Table 2) mark which numbers come from the same controlled set-up and which were copied from other papers, and warn that the latter are not directly comparable; they follow Oliver et al.’s splits.
• Imbalanced pseudo-labels. In our reasoning, a single threshold admits more images from the classes the model is already confident about. Noisy Student balances the pseudo-labelled set per class so that it matches the class balance of the labelled set (Xie et al., 2020, §2), and FlexMatch and FreeMatch, as Yang et al. (2023, §2) describe them, use class-wise thresholds that take each class’s learning status into account. The class imbalance article covers long-tailed label distributions in general.
• Transductive results reported as inductive. Graph methods label the given unlabelled images; Zhu (§1) calls such a learner transductive, one that cannot handle unseen data, so their accuracy on those images is not a test score on new images.

Evidence and Impact

At ImageNet scale, self-training with unlabelled images set new accuracy records. Noisy Student reached 88.4% top-1 accuracy on the ImageNet validation set with an EfficientNet-L2 and 300M unlabelled JFT images (Xie et al., 2020), and Meta Pseudo Labels reported 90.2% with a pair of EfficientNet-L2 networks (Pham et al., 2021, Table 1). In Noisy Student’s ablation (Table 6, EfficientNet-B5 teacher), the student reached 83.9% and 84.9% top-1 with 1.3M and 130M unlabelled images, against 83.3% and 84.0% for the supervised B5; with augmentation, stochastic depth and dropout removed from the student on the unlabelled images it reached 83.2% and 84.3%.

On small benchmarks the field moved quickly. In Pham et al.’s Table 2 (WideResNet-28-2, CIFAR-10 with 4,000 labels), the reported mean accuracies include 84.13% for Mean Teacher, 93.76% for MixMatch, 95.74% for FixMatch and 96.11% for Meta Pseudo Labels, against 94.92% for supervised training on all 50,000 labels; the authors note that, apart from UDA and their own method, these numbers come from other papers and do not share a controlled set-up. On ImageNet with 10% of the labels and a ResNet-50, the same table gives Meta Pseudo Labels 73.89% top-1 against 71.5% for FixMatch and 76.89% with all labels.

In segmentation, Yang et al. (2023, Table 1) report that FixMatch with 92 labelled Pascal VOC 2012 images reaches 63.9 mIoU with their strong perturbations, 56.7 with CutMix alone, and 39.5 with no strong perturbation. Table 1’s caption does not name the backbone; the text says the Pascal ablations use ResNet-101. With the same 92 labelled images and a ResNet-50, their full method, which adds a feature-perturbation stream and two strong views, reached 71.9 mIoU against 44.0 for the supervised baseline (Table 2).

Pseudo-labelled images also feed large vision models outside classification: Depth Anything trains its students on tens of millions of them. And the weight-averaged teacher of Mean Teacher reappears without labels in DINO, whose authors draw the comparison themselves.