Domain Adaptation

Domain adaptation trains a model for a target domain whose images look different from those of the source domain where the labels are. A classifier trained on product photos is used on webcam frames; a segmentation network trained on frames rendered from a video game is used on street photographs; a detector trained on one hospital’s scanner is used on another’s. The task and the label set stay the same, but the input distribution shifts, and accuracy falls with it. In the most studied setting, unsupervised domain adaptation, the target images come without labels: the method sees labelled source images and unlabelled target images, and is judged by its accuracy on the target. The term covers a family of answers: re-weighting or re-normalizing features, aligning feature distributions by matching their statistics or by fooling a domain classifier, translating source images into the target’s style, training on the model’s own confident target predictions, and adapting at test time. A related problem, domain generalization, has no target data at all and must prepare for domains it has not seen.

Dataset bias describes how datasets differ and how much accuracy is lost when a model crosses from one to another; transfer learning covers reuse of pretrained models in general, of which domain adaptation is the case where the task is fixed and the domain moves. This article covers the settings, the theory of what feature alignment can and cannot guarantee, the main method families and their evidence, and the benchmarks, with a worked example of correlation alignment and of its failure when class frequencies differ between domains.

Purpose

A model learns whatever statistics of its training images help it predict the labels, including ones that belong to the camera, the renderer or the hospital rather than to the object. When those statistics change, the learned decision rule is applied to inputs it was never fitted to. The sim-to-real and medical imaging articles give examples: a localization network trained on clean renders that fails on textured images, and a lesion classifier whose operating point collapses on another vendor’s scanner.

Labelling the target domain would solve the problem, and a target-trained model serves as the upper reference in several of the papers below; Ganin et al. (2016) call it an upper bound on adaptation performance. Domain adaptation exists because target labels are often expensive or unavailable while target images are cheap: unlabelled street photographs, frames from a new camera, scans from a new site. It asks how much of the gap the unlabelled images alone can close. Ganin et al. (2016, Table 2) express results this way: on four digit and traffic-sign shifts, their adapted networks covered between 42.6% and 79.7% of the gap between the source-only network and one trained on target labels.

When to Use It

Settings. The methods below differ in what they assume is available:

• Unsupervised domain adaptation (UDA): labelled source data and unlabelled target data, both available during training. Most of the methods below were designed for it.
• Semi-supervised domain adaptation: a few labelled target images as well. Saito et al. (2019) report that UDA methods applied to this setting need strong base networks to beat training on source and labelled target images together, and in some cases do not improve on it: on seven shifts among four DomainNet domains with a ResNet-34, that baseline averaged 56.9% and 60.0% accuracy with one and three labelled target images per class, DANN 58.4% and 60.7%, CDAN 62.5% and 66.5%, and their minimax-entropy method (below) 66.4% and 68.9% (Table 1). Ganin et al. (2016, Figure 7) also show validation curves in which 430 labelled real traffic signs plus about 31,000 unlabelled ones gave the lowest error, and leave a thorough study of that setting to future work.
• Source-free adaptation: only a trained source model and unlabelled target data, for example when the source images are private (Liang et al., 2020).
• Test-time adaptation: the model adapts on the test images as they arrive, one image or one batch at a time, with no separate adaptation phase (Sun et al., 2020; the Tent repository).
• Multi-source adaptation: several labelled source domains and one unlabelled target, as in DomainNet’s benchmark (Peng et al., 2019).
• Domain generalization: several source domains and no target data; the model is tested on a domain held out entirely (Li et al., 2017).
• Partial-set and open-set adaptation: the target lacks some source classes or contains classes the source does not; Liang et al. (2020) evaluate these variants alongside the closed-set case.

When it helps. Adaptation is meant for covariate shift, in the vocabulary of dataset bias: the appearance of the images changes ($P(x)$), while what an image of a given content should be labelled ($P(y \mid x)$) does not. Changes of camera, rendering, style, weather or scanner are examples. It needs enough unlabelled target images to estimate the target distribution, and its gains depend on the source model already having features that carry the label in both domains.

When it does not. If the class frequencies differ between domains, a prior probability shift (label shift in Lipton et al., 2018), distribution-aligning methods can make matters worse; Zhao et al. (2019) prove this, and the example below shows it. Correcting a known or estimated change of class prior is a different technique, covered in class imbalance, including Lipton et al.’s estimation of the target prior from unlabelled images. If the labelling rule itself changes between domains, no amount of unlabelled target data reveals it. And if a modest number of target images can be labelled, fine-tuning on them is, in our reading, the baseline an adaptation method has to beat.

How It Works

What theory says alignment buys

Notation: a source distribution $\mathcal{D}_S$ with labelling function $f_S$, a target distribution $\mathcal{D}_T$ with $f_T$, a hypothesis $h$, and source and target errors $\varepsilon_S(h)$ and $\varepsilon_T(h)$.

The bound of Ben-David et al. As Ganin et al. (2016, §3) state it, Ben-David et al. measured the difference between two domains with the $\mathcal{H}$-divergence, twice the largest difference, over classifiers $\eta$ in a hypothesis class $\mathcal{H}$, between the probabilities that $\eta$ outputs 1 on source and on target inputs. It is the ability of classifiers from $\mathcal{H}$ to tell the domains apart. For a symmetric class it can be estimated from unlabelled samples by training a classifier to separate source from target: with its held-out error $\epsilon$, the proxy $\mathcal{A}$-distance is $\hat{d}_\mathcal{A} = 2(1 - 2\epsilon)$. Ganin et al.’s Theorem 2, attributed to Ben-David et al. (2006), bounds the target error of every $\eta \in \mathcal{H}$, with probability $1 - \delta$, by

$$\varepsilon_T(\eta) \le \hat{\varepsilon}_S(\eta) + \hat{d}_\mathcal{H}(S, T) + \text{(complexity terms in the VC dimension and sample size)} + \beta, \qquad \beta \ge \inf_{\eta^* \in \mathcal{H}} \left[\varepsilon_S(\eta^*) + \varepsilon_T(\eta^*)\right].$$

The bound has three parts: the empirical source error, how well $\mathcal{H}$ can tell the domains apart, and $\beta$, the best joint error any single classifier achieves on both domains. Ganin et al. read it as a recipe: find a representation in which the domains are as indistinguishable as possible while the source error stays low. Zhao et al. (2019, Theorem 2.1) quote a version with the $\mathcal{H}\Delta\mathcal{H}$-divergence, which they attribute to the same NIPS paper of Ben-David et al. (dated 2007 in their reference list), and call this bound the foundation of many methods that learn invariant representations. The original papers were not read for this article.

Zhao et al.’s counterexample. The recipe ignores the last term, and Zhao et al. (2019, §4.1) show what that costs. Let the source be uniform on $[-1, 0]$ with label 1 for $x > -1/2$, and the target uniform on $[1, 2]$ with label 1 for $x < 3/2$. The interval classifier “1 if $-1/2 < x < 3/2$” is perfect on both. Now map both domains with $g(x) = x + 1$ for $x \le 0$ and $x - 1$ for $x > 0$: both become uniform on $[0, 1]$, perfectly aligned, but the source’s positives land on $(1/2, 1]$ and the target’s on $[0, 1/2)$. Every classifier on the aligned feature has $\varepsilon_S + \varepsilon_T = 1$. Alignment destroyed the structure that made the problem easy, and the smaller the source error, the larger the target error.

An upper bound with the labelling functions. Their Theorem 4.1 replaces $\beta$ by a term that does not depend on $\mathcal{H}$:

$$\varepsilon_T(h) \le \varepsilon_S(h) + d_{\tilde{\mathcal{H}}}(\mathcal{D}_S, \mathcal{D}_T) + \min\left\{\mathbb{E}_{\mathcal{D}_S}\left[|f_S - f_T|\right], \mathbb{E}_{\mathcal{D}_T}\left[|f_S - f_T|\right]\right\},$$

with $\tilde{\mathcal{H}}$ a class derived from $\mathcal{H}$. The last term is how differently the two domains label the same inputs. Under covariate shift it vanishes, and matching the marginal distributions while keeping the source error small suffices; otherwise it does not.

A lower bound when class frequencies differ. For binary classification, with $d_{JS}$ the Jensen–Shannon distance (the square root of the Jensen–Shannon divergence, in bits), any representation $Z = g(X)$ followed by a classifier $h$ obeys (their Theorem 4.3; Corollary 4.1 allows different maps $g_S$ and $g_T$ for the two domains)

$$\varepsilon_S(h \circ g) + \varepsilon_T(h \circ g) \ge \tfrac{1}{2}\left(d_{JS}(\mathcal{D}^Y_S, \mathcal{D}^Y_T) - d_{JS}(\mathcal{D}^Z_S, \mathcal{D}^Z_T)\right)^2$$

whenever the bracket is non-negative. If the label distributions $\mathcal{D}^Y$ of the two domains differ and the representation is made invariant ($d_{JS}(\mathcal{D}^Z_S, \mathcal{D}^Z_T) = 0$), the joint error cannot be small: an invariant representation passed through one classifier predicts the same class frequencies in both domains, and at least one domain must then be wrong in proportion to how much its true frequencies differ. Zhao et al. note that the bound is not necessarily tight; in their own counterexample it is zero although the joint error is 1. In digit experiments with DANN between MNIST, USPS and SVHN, whose label distributions differ, they observe target accuracy rising quickly and then declining as adversarial training continued, while source accuracy kept improving (their Figure 2), and they recommend aligning label distributions as well as features.

Discrepancy-based alignment

The simplest methods choose a statistic of the feature distribution and make it equal across domains.

Maximum mean discrepancy (MMD). Gretton et al. (2012) define the MMD between distributions $p$ and $q$ as the largest difference in expectation over a class of functions; for the unit ball of a reproducing kernel Hilbert space with kernel $k$ it is the distance between the two mean embeddings, and (their Lemma 6)

$$\mathrm{MMD}^2(p, q) = \mathbb{E}_{x,x'}\left[k(x, x')\right] - 2\,\mathbb{E}_{x,y}\left[k(x, y)\right] + \mathbb{E}_{y,y'}\left[k(y, y')\right],$$

with $x, x' \sim p$ and $y, y' \sim q$ independent. For a characteristic kernel such as the Gaussian it is zero only when $p = q$, and it is estimated directly from samples. Long et al.’s Deep Adaptation Network (DAN, 2015) adds a multi-kernel MMD between source and target activations of an ImageNet-pretrained AlexNet’s fully connected layers fc6–fc8 to the classification loss, $\min_\Theta \frac{1}{n_a}\sum_i J(\theta(x_i^a), y_i^a) + \lambda \sum_{\ell=6}^{8} d_k^2(\mathcal{D}_s^\ell, \mathcal{D}_t^\ell)$, freezing conv1–conv3 and fine-tuning conv4–conv5, with a linear-time unbiased estimate of the MMD so that it can be computed on minibatches.

Correlation alignment (CORAL). Sun, Feng and Saenko (2016) align only second-order statistics, and in closed form. After normalizing each domain’s features to zero mean and unit variance, they look for a linear map $A$ of the source features that makes their covariance $C_S$ equal to the target’s $C_T$,

$$\min_A \left\| A^\top C_S A - C_T \right\|_F^2,$$

whose solution, when both covariances have full rank, whitens the source and re-colours it with the target covariance, $A = C_S^{-1/2} C_T^{1/2}$ (their Eq. 2 gives the general low-rank solution). In practice they add the identity to both covariances, $C \leftarrow C + \lambda I$ with $\lambda = 1$, which their Algorithm 1 implements in four lines of MATLAB, and report that accuracy was similar to the exact solution and stable in $\lambda$ (Figure 3). A classifier is then trained on the transformed source features and applied to the target features unchanged. They argue that per-feature normalization is not enough, because domains with the same per-feature variances can still differ in correlation, and that whitening both domains fails because they may lie on different subspaces. They also relate CORAL to MMD with a second-degree polynomial kernel, and contrast its asymmetric map, which moves the source towards the target, with methods that transform both domains alike. Deep CORAL (Sun & Saenko, 2016, not read), as WILDS describes it (Koh et al., 2021), instead trains the network with a penalty on the differences in the means and covariances of the last-layer features across domains. The same whiten-and-colour algebra transfers style statistics in neural style transfer.

Adversarial alignment

Domain-adversarial training (DANN). Ganin and Lempitsky (2015; journal version Ganin et al., 2016) estimate the divergence with a learned domain classifier and train the features to defeat it, using a gradient reversal layer that is the identity in the forward pass and negates (and scales) the gradient in the backward pass. The objective, its saddle point, the schedule for the adversarial weight and their synthetic-to-real digit and traffic-sign results are given in Sim-to-Real Transfer, whose example also shows DANN failing on a localization task. On the Office benchmark (Saenko et al., 2010, as Ganin et al. describe it: Amazon, DSLR and Webcam domains, 31 categories, 2,817 images in the largest domain), fine-tuning an ImageNet-pretrained AlexNet with DANN raised Amazon→Webcam accuracy from 0.642 for the source-only network to 0.730, the largest gain of their three tasks, under a “fully transductive” protocol that uses all labelled source and all unlabelled target images (Ganin et al., 2016, Table 3).

Adversarial Discriminative Domain Adaptation (ADDA). Tzeng et al. (2017) cast earlier methods in one framework, choosing whether the model is generative or discriminative, whether the source and target encoders share weights, and which adversarial loss is used. ADDA trains a source encoder and classifier on labelled source images, then freezes them and learns a separate target encoder, initialized from the source one, with the “inverted label” GAN loss: a discriminator separates encoded source from encoded target images, and the target encoder is trained to make target images classified as source. At test time target images pass through the target encoder and the source classifier. They report SVHN→MNIST accuracy of 0.760 against 0.601 for the source-only model (Table 2).

Output-space alignment. For semantic segmentation, Tsai et al. (2018) apply the discriminator to the network’s softmax output maps, arguing that segmentation outputs share spatial layout across domains even when appearance does not, and add a second discriminator at an intermediate layer. Training on GTA5 game frames and adapting to unlabelled Cityscapes training images, a ResNet-101-based DeepLab-v2 model scored 36.6 mean IoU on the 500 Cityscapes validation images without adaptation, 39.3 with feature-level adaptation, 41.4 with single-level output adaptation and 42.4 with two levels, against 65.1 for a model trained on Cityscapes labels (Tables 1 and 2).

Pixel-level adaptation

Instead of aligning features, these methods change the source images to look like target images and keep their labels, a form of unpaired image-to-image translation with a generative adversarial network.

• PixelDA (Bousmalis et al., 2017) trains a generator that maps a source image and noise to a target-style image, a discriminator on real and generated target images, and a task classifier on the generated images. On MNIST to MNIST-M it reported 98.2% accuracy, above the 96.4% of the same classifier trained on target labels, against 63.6% for source only; CORAL, by their implementation with the same task network, scored 57.7%, below source only (Table 1). They tuned hyperparameters on the labels of 1,000 MNIST-M training images, so the setting is not entirely unsupervised.
• CyCADA (Hoffman et al., 2018) combines translation in both directions with a cycle-consistency loss, a semantic consistency loss that asks the source classifier to give the same label before and after translation, and feature-level adversarial alignment. With a VGG16-FCN8s network on GTA5→Cityscapes, mean IoU rose from 17.9 for source only to 35.4, against 60.3 for training on Cityscapes labels; pixel adaptation alone gave 34.8, and without cycle consistency 19.8 (Table 5). On SVHN→MNIST it reported 90.4% against 67.1% source only (Table 2). The authors observed that semantic consistency prevented label flipping on SVHN→MNIST but was not needed for the segmentation task, where cycle consistency was critical.

Self-training and entropy minimization

A third family uses the model’s own predictions on target images. SHOT (Liang et al., 2020) adapts without source data: it freezes the source classifier head and trains the target feature extractor with an information maximization loss, which makes each target prediction confident (low entropy) while keeping the average prediction over the target set diverse, so that not every image collapses to one class, plus pseudo-labels from target class centroids computed by a weighted k-means step and nearest-centroid assignment by cosine distance. On Office-Home with ResNet-50, its average over twelve shifts was 71.8%, against 60.2% for its own source model; the table’s other source-only row, labelled “ResNet-50 (He et al., 2016)”, averages 46.1%, and DANN 57.6% (Table 4). The difference between the two source-only rows shows how much a baseline depends on how it is trained, which in our reading makes cross-paper comparisons hard.

Minimax entropy (Saito et al., 2019), for semi-supervised adaptation, uses a cosine-similarity classifier whose weights act as class prototypes; the classifier is trained to maximize the entropy of unlabelled target predictions, moving the prototypes towards target features, and the feature extractor to minimize it, clustering target features around the prototypes. Self-training for semi-supervised learning in general, without a domain shift, is covered in semi-supervised learning.

Test-time adaptation

Test-time training (Sun et al., 2020) trains a network with two heads on a shared feature extractor: the main classifier and a self-supervised head that predicts which of four rotations (0, 90, 180 or 270 degrees) was applied to the image. At test time, the shared extractor is fine-tuned on the rotation loss of the single test image (with augmented copies) before the prediction is made; the online version keeps the updated weights from one test image to the next. They replaced batch normalization by group normalization because a batch of copies of one image gives poor batch statistics. On CIFAR-10-C, across all fifteen corruptions and five levels, both versions improved by a large margin over a network trained for classification alone; the standard version always improved over the jointly trained network without test-time updates, and the online version never fell more than 0.2% below it (their Figure 1 and appendix). The standard version also improved accuracy on the uncorrupted test set slightly.

Tent (Wang et al., 2021), as its official repository describes it, adapts during testing by minimizing the entropy of the model’s predictions, one update per batch. Its official code updates only the scale and shift parameters of the batch normalization layers and forces them to use the current test batch’s statistics. The repository’s example on CIFAR-10-C at severity 5 with a WRN-28-10 lists mean error 43.5% without adaptation, 20.4% when only the normalization statistics are re-estimated on test batches, and 18.6% with Tent, and says it illustrates the method rather than reproducing the paper. The paper was not read.

Re-estimating normalization statistics on target data is the network version of the per-domain standardization in the example below.

Domain generalization

With no target images, a model can only be trained to depend less on what varies among its source domains. Li et al. (2017) introduced PACS for this: photo, art painting, cartoon and sketch images of seven classes, 9,991 images in all, with one domain held out for testing. Their low-rank parameterized network averaged 69.21% accuracy over the four held-out domains, and the baseline of fine-tuning an ImageNet-pretrained AlexNet on the pooled source domains (“Deep-All”) 67.05% (Table 3); on the older VLCS benchmark they found that this simple baseline outperformed all prior domain-generalization methods. In our reading, such comparisons also depend on how hyperparameters are chosen; the DomainBed suite distinguishes three model-selection criteria: validation data from the training domains, from a held-out domain that is neither training nor test, and from the test domain itself, which it calls oracle selection (DomainBed README). In WILDS (Koh et al., 2021, Table 2), Deep CORAL, IRM and Group DRO trained across source domains failed to improve on standard training on most datasets: for example, on Camelyon17-WILDS (tumour patches from held-out hospitals) average accuracy was 70.3% with standard training and 59.5% with CORAL, and on iWildCam2020-WILDS (held-out camera traps) macro F1 was 31.0% and 32.8%.

How to Apply It

Example: correlation alignment, and alignment under a prior shift

The script trains a logistic regression to tell a filled disc from a filled square of equal area in 24×24 grayscale images, at random sizes, positions and rotations. The features are fixed, as if taken from a frozen network: 32 random 5×5 filters with zero mean, a ReLU, and global average and maximum pooling, 64 numbers per image. The source style is a 0.8 object on a 0.2 background; the target style is brighter, lower in contrast and slightly blurred (0.9 on 0.4, Gaussian blur of σ = 0.5). The target appearance was fixed before any adaptation method was run, chosen so that a classifier trained on target labels learns the task well. Source data always have equal class frequencies.

Four conditions are compared: no shift (target images in the source style), appearance shift, prior shift only (source style, but 90% squares in the target), and appearance plus prior shift. Five methods see the 2,000 labelled source images and, except the first, the 2,000 unlabelled target images of each condition:

• source only: features standardized with the source statistics and used as they are on the target;
• standardize: each domain standardized with its own per-feature mean and standard deviation, the feature-level counterpart of re-estimating normalization statistics;
• CORAL with $\lambda = 1$, Sun et al.’s Algorithm 1, and with $\lambda = 0$, the exact solution; the paper leaves the choice open (it uses $\lambda = 1$ and reports little difference), so both are run;
• target oracle: trained on the labels of the same 2,000 target images, a reference no adaptation method has.

Each method is scored by its accuracy on 2,000 fresh target images. The script also prints the mean predicted frequency of squares on the target, and, for the four methods trained on source labels, the joint error $\varepsilon_S + \varepsilon_T$ (the source error measured on 2,000 source test images mapped the same way) and Zhao et al.’s lower bound computed with the predicted label distributions, $\tfrac{1}{2}\left(d_{JS}(Y_S, Y_T) - d_{JS}(\hat{Y}_S, \hat{Y}_T)\right)^2$. This form follows from their proof, which passes through the distributions of the predicted labels, and is at least as large as the bound with the features, because no classifier’s predictions can differ between domains more than its inputs do (their Lemma 4.6). The last line checks that the exact CORAL map makes the source covariance equal to the target’s. Data are drawn from seeds 0–4 and the table gives the mean and range. The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2; it took between about half a minute and two minutes on a laptop CPU in our runs (approximate, as the machine was shared), and two runs gave identical output.

import sys
import numpy as np
from numpy.lib.stride_tricks import sliding_window_view
from scipy.ndimage import gaussian_filter
from sklearn.linear_model import LogisticRegression

S = 24                                             # image size (pixels)
yy, xx = np.mgrid[0:S, 0:S] + 0.5
FILTERS = np.random.default_rng(1234).normal(size=(32, 5, 5))   # fixed random filters
FILTERS -= FILTERS.mean(axis=(1, 2), keepdims=True)
FILTERS /= np.linalg.norm(FILTERS, axis=(1, 2), keepdims=True)
STYLES = {  # background, object brightness, blur sigma, noise sigma
    "source": dict(bg=0.2, fg=0.8, blur=0.0, noise=0.05),
    "target": dict(bg=0.4, fg=0.9, blur=0.5, noise=0.05),   # brighter, lower contrast, blurred
}

def render(n, p1, style, rng):
    """n images: class 0 = filled disc, class 1 = filled square of equal area, P(y=1) = p1."""
    d = STYLES[style]
    y = (rng.random(n) < p1).astype(int)
    X = np.empty((n, S, S))
    for i in range(n):
        r = rng.uniform(4, 7)
        cx, cy = S / 2 + rng.uniform(-3, 3, 2)
        th = rng.uniform(0, np.pi / 2)
        u, v = xx - cx, yy - cy
        if y[i] == 0:
            m = u**2 + v**2 <= r**2
        else:
            a, b = np.cos(th) * u + np.sin(th) * v, -np.sin(th) * u + np.cos(th) * v
            m = (np.abs(a) <= 0.886 * r) & (np.abs(b) <= 0.886 * r)
        img = np.where(m, d["fg"], d["bg"])
        if d["blur"] > 0:
            img = gaussian_filter(img, d["blur"])
        X[i] = img + rng.normal(0, d["noise"], (S, S))
    return X, y

def features(X):
    """Fixed features: 32 random 5x5 filters, ReLU, global average and max pooling (64 numbers)."""
    P = sliding_window_view(X, (5, 5), axis=(1, 2))
    R = np.maximum(np.einsum("nijkl,fkl->nfij", P, FILTERS), 0)
    return np.concatenate([R.mean((2, 3)), R.max((2, 3))], axis=1)

def zscore(F, ref):
    return (F - ref.mean(0)) / ref.std(0)

def msqrt(C, power):
    """Symmetric matrix power C^power via eigendecomposition (pseudo-inverse for tiny eigenvalues)."""
    w, U = np.linalg.eigh(C)
    w = np.clip(w, 0, None)
    return (U * np.where(w > 1e-12, w ** power, 0.0)) @ U.T

def coral(Zs, Zt, lam):
    """Sun, Feng & Saenko (2016): the map that whitens the source and re-colours it with the
    target covariance; lam = 1 is their Algorithm 1, lam = 0 the exact solution of their Eq. (1)."""
    d = Zs.shape[1]
    Cs = np.cov(Zs, rowvar=False) + lam * np.eye(d)
    Ct = np.cov(Zt, rowvar=False) + lam * np.eye(d)
    return msqrt(Cs, -0.5) @ msqrt(Ct, 0.5)

def fit(Z, y):
    return LogisticRegression(C=1.0, max_iter=5000).fit(Z, y)

def djs(p, q):
    """Jensen-Shannon distance (base 2) between Bernoulli(p) and Bernoulli(q)."""
    P, Q = np.array([1 - p, p]), np.array([1 - q, q])
    M = (P + Q) / 2
    kl = lambda a, b: np.sum(np.where(a > 0, a * np.log2(np.where(a > 0, a, 1) / b), 0.0))
    return np.sqrt(max(0.5 * kl(P, M) + 0.5 * kl(Q, M), 0.0))

CONDITIONS = [  # name, target style, target P(y=1)
    ("no shift", "source", 0.5),
    ("appearance shift", "target", 0.5),
    ("prior shift", "source", 0.9),
    ("appearance + prior", "target", 0.9),
]
METHODS = ["source only", "standardize", "CORAL lam=1", "CORAL lam=0", "target oracle"]

def run(seed, n_src=2000, n_tgt=2000, n_test=2000):
    rng = np.random.default_rng(seed)
    Xs, ys = render(n_src, 0.5, "source", rng)            # labelled source
    Xs2, ys2 = render(n_test, 0.5, "source", rng)         # source test (for the joint error)
    Fs, Fs2 = features(Xs), features(Xs2)
    out, gap = {}, {}
    for cname, style, p1 in CONDITIONS:
        Xt, yt = render(n_tgt, p1, style, rng)            # target pool: labels used only by the oracle
        Xt2, yt2 = render(n_test, p1, style, rng)         # target test
        Ft, Ft2 = features(Xt), features(Xt2)
        for mname in METHODS:
            if mname == "target oracle":
                m = fit(zscore(Ft, Ft), yt)
                pred_s, pred_t = None, m.predict(zscore(Ft2, Ft))
            else:
                Zs, Zs2 = zscore(Fs, Fs), zscore(Fs2, Fs)
                ref_t = Fs if mname == "source only" else Ft   # whose statistics normalize the target
                Zt, Zt2 = zscore(Ft, ref_t), zscore(Ft2, ref_t)
                if mname.startswith("CORAL"):
                    lam = 1.0 if mname.endswith("1") else 0.0
                    A = coral(Zs, Zt, lam)
                    if lam == 0 and cname == "appearance shift":
                        gap[seed] = np.abs(np.cov(Zs @ A, rowvar=False) - np.cov(Zt, rowvar=False)).max()
                    Zs, Zs2 = Zs @ A, Zs2 @ A                 # same map for source test images
                m = fit(Zs, ys)
                pred_s, pred_t = m.predict(Zs2), m.predict(Zt2)
            acc_t = np.mean(pred_t == yt2)
            row = dict(acc=acc_t, phat=pred_t.mean())
            if pred_s is not None:
                eS, eT = np.mean(pred_s != ys2), 1 - acc_t
                dY = djs(ys2.mean(), yt2.mean())               # true label distributions
                dYhat = djs(pred_s.mean(), pred_t.mean())      # predicted label distributions
                bound = 0.5 * max(dY - dYhat, 0.0) ** 2
                row.update(joint=eS + eT, bound=bound, holds=eS + eT >= bound - 1e-12)
            out[(cname, mname)] = row
    out["gap"] = gap[seed]
    return out

seeds = [int(s) for s in sys.argv[1:]] if len(sys.argv) > 1 else [0, 1, 2, 3, 4]
res = [run(s) for s in seeds]
print(f"target accuracy, mean over seeds {seeds[0]}-{seeds[-1]} (min-max); "
      "P(yhat=1) on target; joint error eS+eT and Zhao et al.'s lower bound (means)")
for cname, _, p1 in CONDITIONS:
    print(f"\n{cname} (target P(y=1) = {p1})")
    print(f"{'method':>14} {'target acc':>20} {'P(yhat=1)':>10} {'eS+eT':>7} {'bound':>7} {'holds':>6}")
    for mname in METHODS:
        r = [o[(cname, mname)] for o in res]
        a = np.array([x["acc"] for x in r])
        line = f"{mname:>14} {a.mean():8.3f} ({a.min():.3f}-{a.max():.3f}) {np.mean([x['phat'] for x in r]):10.3f}"
        if "joint" in r[0]:
            line += (f" {np.mean([x['joint'] for x in r]):7.3f} {np.mean([x['bound'] for x in r]):7.3f}"
                     f" {'all' if all(x['holds'] for x in r) else 'NO':>6}")
        print(line)
print(f"\nCORAL lam=0, appearance shift: largest |cov(mapped source) - cov(target)| "
      f"over seeds = {max(o['gap'] for o in res):.1e}")

Output:

target accuracy, mean over seeds 0-4 (min-max); P(yhat=1) on target; joint error eS+eT and Zhao et al.'s lower bound (means)

no shift (target P(y=1) = 0.5)
        method           target acc  P(yhat=1)   eS+eT   bound  holds
   source only    0.937 (0.925-0.941)      0.488   0.123   0.000    all
   standardize    0.937 (0.925-0.942)      0.487   0.122   0.000    all
   CORAL lam=1    0.937 (0.925-0.941)      0.485   0.122   0.000    all
   CORAL lam=0    0.935 (0.920-0.943)      0.485   0.126   0.000    all
 target oracle    0.941 (0.930-0.950)      0.485

appearance shift (target P(y=1) = 0.5)
        method           target acc  P(yhat=1)   eS+eT   bound  holds
   source only    0.714 (0.596-0.784)      0.253   0.345   0.000    all
   standardize    0.839 (0.834-0.845)      0.500   0.220   0.000    all
   CORAL lam=1    0.841 (0.827-0.855)      0.493   0.218   0.000    all
   CORAL lam=0    0.824 (0.805-0.838)      0.496   0.235   0.000    all
 target oracle    0.889 (0.870-0.898)      0.488

prior shift (target P(y=1) = 0.9)
        method           target acc  P(yhat=1)   eS+eT   bound  holds
   source only    0.933 (0.927-0.944)      0.849   0.126   0.002    all
   standardize    0.679 (0.667-0.699)      0.582   0.380   0.049    all
   CORAL lam=1    0.674 (0.664-0.694)      0.576   0.386   0.050    all
   CORAL lam=0    0.672 (0.662-0.694)      0.574   0.388   0.051    all
 target oracle    0.961 (0.957-0.963)      0.915

appearance + prior (target P(y=1) = 0.9)
        method           target acc  P(yhat=1)   eS+eT   bound  holds
   source only    0.510 (0.260-0.673)      0.425   0.549   0.041    all
   standardize    0.642 (0.628-0.663)      0.560   0.417   0.056    all
   CORAL lam=1    0.645 (0.622-0.671)      0.557   0.415   0.056    all
   CORAL lam=0    0.639 (0.620-0.655)      0.551   0.420   0.058    all
 target oracle    0.946 (0.945-0.948)      0.921

CORAL lam=0, appearance shift: largest |cov(mapped source) - cov(target)| over seeds = 2.5e-14

No shift. All four methods trained on source labels agree to within a few thousandths, and close to the oracle: when the domains are the same, adaptation has nothing to do and does no harm. This is the base check for every compared method.

Appearance shift. The source-only classifier loses nearly a quarter of its accuracy, with a wide spread over seeds, and its predicted frequency of squares falls to about a quarter: the brighter, blurrier, lower-contrast images move every feature in the same direction, which this classifier reads as evidence for discs. Standardizing each domain with its own statistics removes that offset, brings the predicted frequency back to one half and recovers more than half of the loss. We expected CORAL to do at least as well as standardization. That expectation held only in part: CORAL with $\lambda = 1$ matches standardization in the mean but is lower in at least one seed (its range starts below standardization’s), and CORAL with $\lambda = 0$ is lower in the mean, although the last line shows that its source covariance matches the target’s to rounding error. In our reading, equal covariances do not mean equal class-conditional distributions, and the exact map also stretches directions in which the source varies little; the regularized map of Algorithm 1 changes the standardized features less. The oracle remains about five points higher: unlabelled target images closed most of the gap, not all of it.

Prior shift only. Here nothing about the images changed, and source only keeps its in-domain accuracy. Every alignment method loses about a quarter of it. The target contains 90% squares, so the target’s feature mean lies towards the squares; re-centring the target on its own mean pulls the squares towards the source boundary, and many are called discs. The predicted frequency of squares drops from about 0.85 to under 0.6, towards the source’s 0.5: the alignment has made the predictions nearly invariant to the domain, exactly the situation in which Zhao et al.’s bound forces a joint error. The bound holds in every row and seed, but it is loose here: about 0.05 against a joint error of about 0.38. The oracle, trained with the target’s frequencies, does best of all, by using the prior.

Appearance plus prior shift. Both problems act at once. Source only scores far below the 0.9 that always answering “square” would reach here, and varies widely by seed; the alignment methods improve on it but stay near 0.64, while the oracle reaches 0.95. Neither doing nothing nor aligning is adequate when the class frequencies move along with the appearance. The example does not test remedies; Zhao et al. recommend aligning label distributions as well, and estimating and correcting the target’s class prior, as in Lipton et al.’s method described in class imbalance, is another route in our reading.

What the example cannot show: the features are fixed and the classifier is linear, so it tests alignment as a preprocessing step, not methods that learn the features; and the shifts are a single synthetic change of rendering, whose size we chose. The prior-shift result follows from the construction and agrees with the theorem. How large the effect of an appearance shift is on real data, and how much of it alignment recovers, has to be measured on that data.

Practical procedure

1. Measure the gap first. Evaluate the source model on a labelled sample of the target domain, however small; without it, neither the size of the problem nor the effect of a method is known. Dataset Bias describes cross-dataset tests.
2. Start with cheap baselines. Re-normalize features or re-estimate normalization statistics on target data; apply CORAL to the features of a frozen network; fine-tune on whatever target labels exist. Sun et al. (2016) found CORAL competitive with deep adaptation methods on Office.
3. Check the class frequencies. If the target’s class frequencies may differ, estimate them (see class imbalance) before aligning marginal feature distributions.
4. Choose by the kind of shift. Low-level appearance (rendering, colour, texture) suits pixel-level translation; changes that leave the output structure intact suit output-space alignment; source data that cannot be shared call for source-free methods; continually changing conditions call for test-time adaptation.
5. Select hyperparameters honestly. State what data were used for model selection; selecting on target labels makes a method semi-supervised, and DomainBed calls this oracle selection.

Variants

• Instance re-weighting for covariate shift, which weights source examples by how target-like they are; Sun et al. (2016) list it among early unsupervised approaches.
• Feature-level alignment by statistics: MMD and DAN (Long et al., 2015), CORAL and Deep CORAL, moment matching across several sources (M³SDA, Peng et al., 2019).
• Feature-level adversarial alignment: DANN, ADDA, and later variants such as CDAN, which the SHOT and MME papers compare with.
• Output-space adversarial alignment for dense prediction (Tsai et al., 2018).
• Pixel-level translation: PixelDA, CyCADA, and the sim-to-real translators described in Sim-to-Real Transfer.
• Self-training and entropy-based methods: SHOT, minimax entropy.
• Normalization-statistics adaptation and test-time methods: re-estimating normalization statistics, test-time training, Tent.
• Domain generalization: training on several source domains for an unseen one, with or without invariance penalties, and data-side alternatives such as domain randomization and data augmentation.

Best Practices

• Report the source-only and target-oracle results with the same network and training procedure, and the share of the gap closed, as Ganin et al. (2016) do. SHOT’s two source-only rows (60.2% and 46.1% on Office-Home) show how much a baseline depends on its training.
• Use enough target images. Liang et al. (2020) suggest that SHOT’s weaker results on some Office tasks are due to the small DSLR and Webcam target domains.
• Do not trust divergence alone. A low proxy $\mathcal{A}$-distance or MMD shows that the domains are hard to tell apart, not that the classes are; Zhao et al.’s counterexample has zero divergence and joint error 1.
• Watch for label shift and missing classes. Check the predicted class frequencies on the target, as the example prints; a collapse towards the source’s frequencies after alignment is a warning.
• Prefer adapting a strong pretrained model. Many results above start from ImageNet-pretrained networks; the strongest gains in Sun et al. (2016) came from CORAL on features of a network fine-tuned on the source.

Pitfalls

• Selecting hyperparameters on target labels and reporting the result as unsupervised. PixelDA’s 1,000 labelled MNIST-M images and DomainBed’s oracle criterion are explicit about it.
• Comparing numbers across protocols. Ganin et al. report Office results under a “fully transductive” protocol that uses all source images, while Sun et al., citing the same protocol, report means over five random training/test splits; papers also reproduce one another’s numbers inconsistently: Sun et al.’s Table 2 lists ReverseGrad at 67.3% on Amazon→Webcam, while Ganin and Lempitsky’s own table gives 0.730.
• Aligning under prior shift. The example’s alignment methods lost about a quarter of the accuracy that doing nothing kept; Zhao et al. show the general limit, and Castro et al., as Medical Computer Vision reports, caution that adaptation may fail or harm when class prevalence changes.
• Expecting a method to transfer between tasks. CORAL fell below source only on MNIST→MNIST-M in PixelDA’s comparison; DANN failed on MNIST→SVHN (Ganin et al., 2016), not improving on the unadapted model’s accuracy of about 0.25, and on localization in Sim-to-Real Transfer’s example; in WILDS, CORAL lowered accuracy on Camelyon17-WILDS.
• Over-training adversarial alignment. Zhao et al. observed target accuracy peaking early and then declining under DANN.
• Batch statistics at test time. Methods that re-estimate normalization statistics need test batches representative of the target; with single images Sun et al. (2020) replaced batch normalization by group normalization.

Evidence and Impact

Domain adaptation is evaluated on benchmarks built from pairs or sets of domains with the same classes:

• Office (Saenko et al., 2010): Amazon, DSLR and Webcam images of 31 office objects; Office-Caltech10 adds Caltech-256 for 10 shared classes (Sun et al., 2016).
• Digits: MNIST, USPS, SVHN, MNIST-M (MNIST digits blended with photograph patches, Ganin & Lempitsky, 2015), and synthetic numbers and traffic signs (Ganin et al., 2016).
• Office-Home (Venkateswara et al., 2017): about 15,500 images of 65 categories in four domains, artistic images, clip art, product images and real-world photographs.
• VisDA-C: 12-class synthetic-to-real recognition with about 152,000 rendered and 55,000 real images, as Liang et al. (2020) describe it.
• DomainNet (Peng et al., 2019): six domains (clipart, infograph, painting, quickdraw, real and sketch), 345 categories and about 0.6 million images, which the authors call by far the largest UDA dataset when it was released.
• PACS (Li et al., 2017) for domain generalization.
• GTA5→Cityscapes and SYNTHIA→Cityscapes for segmentation, evaluated by mean intersection over union on the Cityscapes validation images, over 19 classes for GTA5 and, in Tsai et al. (2018), 13 for SYNTHIA.
• WILDS (Koh et al., 2021): ten datasets with shifts that arose in applications, such as hospitals for tumour patches, camera traps for wildlife and time and region for satellite images. Standard training scored lower out of distribution than in distribution on every dataset in its Table 1, for example 93.2% against 70.3% average accuracy on Camelyon17-WILDS and 47.0% against 31.0% macro F1 on iWildCam2020-WILDS, and the methods for shift that the authors tried did not close the gaps.
• EPIC-KITCHENS runs an unsupervised domain adaptation challenge for action recognition across recording years (EPIC-KITCHENS).

The evidence shows large gains on synthetic-to-real and digit shifts, and on segmentation: on GTA5→Cityscapes, CyCADA recovered approximately 40% of the performance lost to domain shift, in the authors’ words, and output-space adaptation added about six points over a ResNet-101 baseline, while the remaining gap to a target-trained model stayed above 20 points in both papers. On benchmarks of shifts that arose in practice, WILDS found alignment and other shift-specific penalties no better than standard training on most datasets. In our reading, Zhao et al.’s analysis gives one possible reason, which the example reproduces on a small scale: alignment is justified when only the appearance changes, and harmful when what is being aligned also carries the class frequencies.