Multi-Task Learning

Multi-task learning trains one model to solve several tasks at once. In vision the usual form is a single network that reads an image once, computes a shared representation with a common backbone, and passes it to a small head per task: one head segments the image, another predicts depth, a third detects objects. Training minimizes a combination of the tasks’ losses. The hope is twofold: one network is cheaper to run than several, and a representation that must serve several tasks may generalize better than one fitted to a single task. Both hopes can fail. Tasks can compete for the shared parameters, losses on different scales can let one task dominate, and a multi-task network often ends up worse than separate networks on some of its tasks. Much of the method’s literature is about when sharing helps, how to weight the losses, and which tasks to put together.

The idea predates deep learning; the Deep Learning book cites Caruana (1993) for it and presents it as a form of regularization (Goodfellow et al., 2016, §7.7).

Purpose

Multi-task learning addresses two problems.

Computation. A driving system needs detections, lane lines and drivable area from every frame; a robot needs depth, surface orientation and semantics. Running a separate network per task multiplies the cost. Sharing the backbone pays for the expensive part once. Eigen and Fergus (2015) combined their depth and surface-normal networks by sharing the first of their three scales and keeping the other two per task, which ran 1.6 times faster than two separate models. Drivable Area Segmentation describes driving networks such as YOLOP, with one encoder and heads for detection, drivable area and lane lines, and the half point of drivable-area mIoU that YOLOP’s README reports the sharing cost.

Generalization. The Deep Learning book’s argument is statistical: parameters shared by several tasks are fitted to the pooled examples of all of them, so they are more constrained toward good values, “assuming the sharing is justified” (§7.7). The prior behind it is that some of the factors explaining the data matter for more than one task. When they do not, the constraint becomes a bias. Kendall, Gal and Cipolla (2018) put the hope in vision terms: cues from one task, such as depth, can regularize and improve another, such as segmentation.

When to Use It

Multi-task learning fits when:

several outputs are needed from the same input at inference time, and running separate networks is too slow or too large;
the tasks plausibly depend on shared factors (geometry and semantics of the same scene; boxes and masks of the same objects);
one task has few labels and a related task has many, so that the data-rich task can shape the shared representation (Misra et al., 2016, report larger gains for “data-starved” segmentation categories than for the rest, and an improvement on attribute prediction, the task with fewer labels than detection).

It is a poor fit, in our reading, when accuracy on one task matters far more than cost and a dedicated network is affordable, or when the tasks share little. Standley et al. (2020) found that multi-task networks often lost to separate networks given the same compute each, and that the tasks that trained well together changed with network size and data size, so relatedness judged by intuition is not a reliable guide (Evidence and Impact below).

Multi-task learning is related to, but distinct from, transfer learning, where a model trained on a source task is adapted to a target task afterwards. Standley et al. draw the line by goal: transfer learning usually assumes a source model is given and seeks no benefit for the source task, while multi-task learning trains the tasks together and cares about all of them. Misra et al. describe fine-tuning as sequential multi-task learning, and Kendall et al. call it a basic example of multi-task learning with the first task as a pre-training step.

How It Works

The objective

With tasks $t = 1, \dots, T$ , shared parameters $\theta_{\text{sh}}$ and task-specific parameters $\theta_t$ , the common objective is a weighted sum

L(\theta_{\text{sh}}, \theta_1, \dots, \theta_T) = \sum_{t=1}^{T} w_t \, L_t(\theta_{\text{sh}}, \theta_t),

where $L_t$ is task $t$ ’s loss averaged over its training examples. The gradient with respect to the task parameters $\theta_t$ comes from task $t$ alone; the gradient with respect to the shared parameters is the weighted sum $\sum_t w_t \nabla_{\theta_{\text{sh}}} L_t$ . Everything that distinguishes multi-task methods happens in one of three places: what is shared (the architecture), how the $w_t$ are set (loss weighting), and how the per-task gradients on the shared parameters are combined (gradient manipulation).

Fast R-CNN’s detector loss, classification plus box regression with $\lambda = 1$ , is a two-task instance of this objective.

In hard sharing, all tasks use the same trunk and only the last layers are task specific. Standley et al., following Ruder’s survey, name this hard parameter sharing and cite UberNet (Kokkinos, 2017) as a known example in vision. Misra et al. (2016) call the family “split architectures”: the network is shared up to some layer and then splits into one branch per task. Splitting at the first layer gives two separate networks; splitting just before the output gives sibling prediction layers, as in Fast R-CNN. Their exhaustive experiment on AlexNet found that the best split depended on the pair of tasks: semantic segmentation and surface normals on NYU-v2 both did best when split at conv4, while detection and attribute prediction on PASCAL preferred different splits. For that second pair the paper disagrees with itself: Section 1.1 names conv2 for detection and fc6 for attributes, while Section 6.4 and Table 6 give Split fc7 as best for detection (44.8 mAP) and Split conv2 for attributes (61.0).

In soft sharing, each task keeps its own parameters, and the models are tied by a penalty or by exchanging information. The Deep Learning book gives the penalty form, $\Omega(w^{(A)}, w^{(B)}) = \lVert w^{(A)} - w^{(B)} \rVert_2^2$ , for two models whose parameters should be close (§7.9).

Cross-stitch units (Misra et al., 2016) exchange activations. Two networks, one per task, run side by side, and after selected layers each channel’s activations are mixed by a learned $2 \times 2$ matrix:

\begin{bmatrix} \tilde{x}_A \\ \tilde{x}_B \end{bmatrix} = \begin{bmatrix} \alpha_{AA} & \alpha_{AB} \\ \alpha_{BA} & \alpha_{BB} \end{bmatrix} \begin{bmatrix} x_A \\ x_B \end{bmatrix}.

Setting $\alpha_{AB} = \alpha_{BA} = 0$ makes a layer task specific; equal values make it shared. The units can therefore represent every split architecture and blends between them, and they are learned with the rest of the network. Misra et al. initialized the units as convex combinations, found results robust to the initial values, scaled their learning rate by $10^2$ to $10^3$ relative to the network’s, and found that initializing the two networks from single-task models worked better than from ImageNet. Learned values favoured task-specific representations at pool5, consistent with the split-at-conv4 result. The cost is that the network runs one branch per task, so inference costs as much as separate networks; Standley et al. note that soft sharing does not reduce inference time.

Multi-head vision networks

Several vision systems are multi-task by construction:

Depth, normals and semantics. Eigen and Fergus (2015) applied one multi-scale architecture to depth, surface normals and semantic labels, and shared the first scale between depth and normals. A compatibility constraint between predicted normals and normals computed from predicted depth improved the latter but neither task, so they did not use it.
UberNet. Kokkinos (2017) trained one VGG-16-based network for seven tasks (boundaries, normals, saliency, semantic segmentation, human parts, semantic boundaries, proposals and detection) from a union of datasets, each image labelled for only some tasks. A task’s loss is simply zero on images without its labels, and an asynchronous variant of SGD updates each task’s parameters only after enough examples of that task have been seen. A memory-saving backpropagation stores a subset of the trunk’s activations and frees each task branch after it has sent its gradient back, so memory no longer grows with the number of tasks; this allowed $921 \times 621$ inputs on a 12 GB GPU, against $321 \times 321$ without it, by the paper’s account.
Kendall et al. (2018) used a DeepLabV3-style encoder (ResNet-101 with atrous spatial pyramid pooling) and a small decoder per task for semantic segmentation, instance segmentation by regressing each pixel’s vector to its instance centroid, and inverse depth.
Detection and segmentation heads. Mask R-CNN adds a mask head beside Faster R-CNN’s box and class heads; the keypoint variant adds another.
Driving and localization networks. YOLOP and HybridNets (in Drivable Area Segmentation), and HF-Net in visual localization, which has one encoder with heads for keypoints, local descriptors and a global descriptor.

Encoder–decoder networks make dense multi-head designs natural: the encoder is shared and each task gets its own decoder.

Loss weighting

The weights $w_t$ matter because tasks’ losses have different units and scales. A depth loss in metres and the same loss in centimetres differ by a factor of $10^4$ in squared error, which changes the weighted sum’s optimum unless $w_t$ changes too. Kendall et al.’s Figure 2 shows how sensitive the result is: on Tiny Cityscapes (128 × 256, validation set), semantic-segmentation IoU ranged from 42.7% to 60.4% and inverse-depth error from 0.573 to 0.664 as the weight moved between the two tasks, and only some weightings beat both single-task models. Searching weights by grid costs a training run per point.

Uncertainty weighting (Kendall et al., 2018) treats each task’s output as a likelihood with a learned, input-independent noise scale $\sigma_t$ , the homoscedastic or task-dependent uncertainty. For two Gaussian regression outputs, the negative log-likelihood gives their Eq. 7,

L(W, \sigma_1, \sigma_2) = \frac{1}{2\sigma_1^2} L_1(W) + \frac{1}{2\sigma_2^2} L_2(W) + \log \sigma_1 \sigma_2,

with $L_t(W) = \lVert y_t - f^W(x) \rVert^2$ . A large $\sigma_t$ lowers task $t$ ’s weight; the $\log \sigma_t$ term stops $\sigma_t$ from growing without limit. For classification they scale the logits by $1/\sigma^2$ inside the softmax and, with an approximation that is exact as $\sigma \to 1$ , obtain a term $L_t / \sigma_t^2 + \log \sigma_t$ (their Eq. 10). In practice the network learns $s = \log \sigma^2$ for numerical stability, and the $\sigma_t$ are trained with the network by gradient descent.

At a fixed $W$ , setting the derivative with respect to $\sigma_t$ to zero gives $\sigma_t^2 = L_t$ (our derivation; the DUSt3R article derives the same minimizer for its per-pixel confidence). Each task’s weight $1/(2\sigma_t^2)$ is then $1/(2L_t)$ , as Chen et al. (2018) also state it: uncertainty weighting enforces $w_t \sim 1/L_t$ . Substituting back, the objective becomes $\tfrac{1}{2}\sum_t \log L_t$ plus a constant, so at the $\sigma$ -optimum uncertainty weighting minimizes the sum of log losses. Multiplying a task’s targets by a constant $c$ multiplies its $L_t$ by $c^2$ and adds a constant to the objective, so the fitted model does not depend on each task’s units, while with equal weights it does (our derivation). Navon et al. (2022) include a “scale-invariant” baseline that minimizes $\sum_k \log \ell_k$ directly. The example below checks both statements.

Kendall et al.’s Table 1 (Tiny Cityscapes, validation set) compares single-task models, an unweighted sum, approximately optimal weights from a search, and uncertainty weighting:

Loss	Segmentation IoU	Instance error (px)	Inverse-depth error (px)
Single-task models	59.4%	4.61	0.640
Unweighted sum (1/3 each)	50.1%	3.79	0.592
Approx. optimal weights (0.89, 0.01, 0.1)	62.8%	3.61	0.549
Uncertainty weighting, 3 tasks	63.4%	3.50	0.522

The unweighted sum lost 9 IoU points on segmentation against the single-task model, while the learned weights beat both the single-task models and the searched weights. At the end of training of their final model, the losses were weighted 43 : 1 : 0.16 for segmentation, depth and instance regression. The authors attribute the advantage over grid search to the grid’s resolution and to the weights changing during training.

GradNorm (Chen et al., 2018) sets the weights from gradients rather than likelihoods. At each step it measures, at the last shared layer, the gradient norm $G_t$ of each weighted task loss and each task’s training rate through the loss ratio $\tilde{L}_t = L_t(\text{step}) / L_t(0)$ . The target norm for task $t$ is the mean norm times $r_t^\alpha$ , where $r_t$ is $\tilde{L}_t$ divided by its mean over tasks, so a task that has trained more slowly gets a larger gradient. The weights are updated to reduce $\sum_t \lvert G_t - \bar{G}\, r_t^\alpha \rvert$ , with the targets held fixed, and renormalized after each step to sum to $T$ . The hyperparameter $\alpha$ sets how strongly training rates are pulled together; they used 0.12 on a symmetric toy problem and 1.5 on NYUv2. On their NYUv2 variant with 90,000 images and room-keypoint labels (Table 2), GradNorm lowered all three errors relative to equal weights for both backbones. In a search over 100 random fixed weightings (VGG backbone) trained for 15,000 steps, none reached the GradNorm network at the same step, and errors grew with the distance from GradNorm’s time-averaged weights. The same table lists uncertainty weighting below equal weights on the ResNet backbone (depth RMS error 0.702 m against 0.697) and on VGG with depth error between the two (0.649 m against 0.658 for equal weights and 0.629 for GradNorm). Discussing their toy problems, Chen et al. attribute uncertainty weighting’s weaker results to the unconstrained growth of its weights as losses fall, which raises the effective learning rate.

Gradient conflict

On the shared parameters, the update is a sum of per-task gradients. When two task gradients point in opposing directions, or one is much larger, the sum can serve one task at the other’s expense. Navon et al. (2022) name gradient conflict as a main reason multi-task networks lose to single-task ones, and describe a family of methods that compute every task’s gradient and combine them into one update by some rule:

MGDA (Sener & Koltun, 2018) finds the convex combination of task gradients with minimal norm, which Navon et al. describe as biased toward the task with the smallest gradient;
PCGrad (Yu et al., 2020) removes from each gradient its component that conflicts with the others, as Navon et al. describe it;
Nash-MTL (Navon et al., 2022) picks the update $\sum_t \alpha_t g_t$ that solves a Nash bargaining problem among tasks, with $\alpha$ the solution of $G^\top G \alpha = 1/\alpha$ (elementwise reciprocal), and is invariant to the scale of each loss.

Neither the MGDA nor the PCGrad paper was read for this article; both are described through Navon et al. These methods need one backward pass per task through the shared layers, which Navon et al. note is shared by every method that computes all task gradients. A geometric fact constrains what “conflict” can mean: at any minimum of a weighted sum, the weighted task gradients on the shared parameters add up to zero, so with two tasks they point in exactly opposite directions whenever they are not zero (our derivation; the example checks it). Conflict at the end of training is therefore built into any compromise, and what varies with the tasks is how large the opposing gradients are.

Which tasks to train together

Taskonomy (Zamir et al., 2018) measured relationships among 26 vision tasks for transfer learning, not joint training. They built a dataset of 4 million indoor images from about 600 buildings with labels for every task, trained a network per task, and then trained small readout functions from each task’s frozen encoder (and combinations of encoders) to every other task with little data. Normalized transfer affinities (with the Analytic Hierarchy Process) fed a binary integer program that chose which tasks to supervise fully and how to transfer to the rest under a supervision budget. They score the result by Gain (win rate against training from scratch with the same small data) and Quality (win rate against a fully supervised network trained on 120k images).

Standley et al. (2020) asked the multi-task question on the same dataset: which tasks should share a network, given an inference-time budget? They trained a network for every subset of five tasks (31 networks plus five half-size single-task networks) and chose the set of networks with the lowest total loss whose cost fit the budget, a problem they state is NP-hard in general and solve with a branch-and-bound search. To cut the training cost they propose predicting final performance from networks trained on 20% of the data (Pearson’s $r = 0.49$ with the final validation loss, about 20 times cheaper) or predicting a higher-order network’s per-task losses by averaging the pairwise networks’ results. Their main finding on task relations is that multi-task affinity did not correlate with Taskonomy’s transfer affinity in any of their settings (in the small-network setting, $r = -0.12$ , $p = 0.74$ ), and that it changed with network size and with training-set size.

How to Apply It

A workable sequence, drawn from the sources above:

Train single-task baselines with the same architecture and compute; every multi-task claim is a comparison with them. Standley et al. compare at equal total compute, which separate networks of reduced size make possible.
Start with hard sharing of the backbone and a light head per task, the design of the multi-head networks above.
Normalize targets and choose weights deliberately. Equal weights are a choice, and with losses on different scales a poor one. Use uncertainty weighting or GradNorm, or search a few weightings on validation data.
Report per-task metrics, not only an average. Navon et al.’s $\Delta_m$ averages each task metric’s relative change from the single-task baseline, signed so that a negative value is an improvement, and their mean rank summarizes the ordering of methods across tasks; a total loss across tasks with different scales hides a task that got worse.
If a task suffers, test the grouping before the optimizer. Removing a task or splitting the tasks across two networks can do more than reweighting (Standley et al.).

Example: shared representations, relatedness and loss weighting

The mechanics of sharing can be seen exactly in a linear model, where every fit has a closed form and no network has to be trained. The example asks three questions: when does fitting two tasks jointly through a shared low-dimensional representation beat fitting them separately, what do the task gradients on the shared parameters look like, and what does uncertainty weighting do with a change of units?

Data and tasks. Each $32 \times 32$ image holds one to four Gaussian blobs of random sign, size and position, plus pixel noise. Its features $\mathbf{h} \in \mathbb{R}^{64}$ are the means of its $4 \times 4$ -pixel blocks, standardized with statistics from 5,000 reference images. Each task maps the features to $q = 8$ outputs, $\mathbf{y}_t = W_t \mathbf{h} + \boldsymbol{\varepsilon}_t$ , with Gaussian noise of variance 1 per output. Both maps have rank 3: $W_t = C_t U_t^\top$ with $U_t \in \mathbb{R}^{64 \times 3}$ orthonormal and a random head $C_t$ , so each task reads a three-dimensional subspace of the features. The relatedness of the tasks is set by the principal angles between the two subspaces, all three equal to $\theta$ : at $\theta = 0$ the tasks read the same subspace through different heads, at $\theta = 90^\circ$ orthogonal ones. Each $W_t$ is scaled to unit signal variance per output, so the noise equals the signal. The convention throughout is row vectors in code: a fitted map is $B = \hat{W}^\top$ and predictions are $HB$ for a feature matrix $H$ with one image per row.

Methods. On each training set the code fits:

S-OLS: ordinary least squares for each task separately, which ignores the rank structure;
S-RRR: reduced-rank regression of rank 3 for each task separately, which knows each task’s rank but shares nothing;
J-RRR(3): one rank-3 reduced-rank regression on both tasks’ outputs stacked side by side, so the two tasks must share one three-dimensional representation, the linear analogue of hard sharing with a narrow trunk;
J-RRR(6): the same with rank 6, a shared representation wide enough for both tasks’ subspaces.

Reduced-rank regression minimizes the squared error over maps of rank at most $r$ in closed form: fit least squares, then project the fit onto the top $r$ right singular vectors of the fitted values $H B_{\text{OLS}}$ , as derived in the LoRA article. A joint fit factors as $B = A G^\top$ : the $64 \times r$ matrix $A$ is a shared linear encoder and the rows of $G$ belonging to each task form its head. Per-task loss weights $w_t$ enter by scaling each task’s columns by $\sqrt{w_t}$ before the fit and undoing the scaling afterwards.

Measure. The relative test error of task $t$ is $\sum \lVert (\hat{W}_t - W_t)\mathbf{h} \rVert^2 / \sum \lVert W_t \mathbf{h} \rVert^2$ over 2,000 test images, measured against the noise-free targets: 0 is perfect and 1 is the error of predicting zero. Training sets of $n = 96$ , 192, 384 and 1,536 images are nested, and each seed draws new images and noise, the same for all angles. The code reports the mean over seeds 0–99 and the share of seeds on which J-RRR(3) beats S-RRR.

Gradients. For the factored model with per-task loss $L_t = \frac{1}{2n}\lVert Y_t - H A G_t^\top \rVert^2$ , the gradient with respect to the shared encoder is $\nabla_A L_t = -\frac{1}{n} H^\top (Y_t - H A G_t^\top) G_t$ , checked against finite differences. On seed 0 with $n = 1{,}536$ , the code computes the cosine between the two tasks’ gradients at 200 random encoders, each with both heads fitted by least squares for that encoder (a stand-in for an early point in training), and the gradients at the joint rank-3 optimum.

Uncertainty weighting. At $\theta = 60^\circ$ and $n = 192$ , the code minimizes Kendall et al.’s Eq. 7, with $L_t$ the mean over training images of the squared error summed over the task’s outputs, by alternating two exact steps: the weighted rank-3 fit for fixed $\sigma$ , and $\sigma_t^2 = L_t$ for a fixed fit. It checks $\sigma_t^2$ against a separate numerical minimization over $\log \sigma_t^2$ , refits from a second starting point, and refits after task 2’s targets are multiplied by 10, as if re-expressed in units ten times smaller; equal weights get the same change of units. Errors are reported in the original units. The rank-3 fits cannot interpolate noisy targets at these sizes, so every $L_t$ stays positive; with noise-free targets $L_t \to 0$ would drive the objective to $-\infty$ .

The script sets single-threaded BLAS before importing NumPy; these lines were added after a pilot and before the first run, since the order of floating-point sums depends on the number of threads and can change the rounding-level digits of the exact checks. It ran with NumPy 2.3.5 and SciPy 1.16.3 in about 20–25 seconds on one CPU core, and two runs gave identical output. The script takes the number of seeds as an optional argument (100 by default).

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"      # single-threaded BLAS: identical output on every run
import sys
import numpy as np
from scipy.linalg import subspace_angles
from scipy.optimize import minimize_scalar

p, q, k = 64, 8, 3            # features, outputs per task, rank of each task's map
SIZES = (96, 192, 384, 1536)  # training-set sizes (nested)
ANGLES = (0, 30, 60, 90)      # principal angle between the two tasks' subspaces
SEEDS = range(int(sys.argv[1]) if len(sys.argv) > 1 else 100)


def make_images(rng, n):
    """32x32 images: 1-4 Gaussian blobs of random sign, size and position, plus pixel noise."""
    yy, xx = np.mgrid[0:32, 0:32].astype(float)
    imgs = np.zeros((n, 32, 32))
    for i in range(n):
        for _ in range(rng.integers(1, 5)):
            cy, cx = rng.uniform(4, 28, size=2)
            w, a = rng.uniform(1.5, 5.0), rng.uniform(-1, 1)
            imgs[i] += a * np.exp(-((yy - cy) ** 2 + (xx - cx) ** 2) / (2 * w * w))
    return imgs + rng.normal(0, 0.1, imgs.shape)


def block_means(imgs):        # 8x8 grid of 4x4-pixel block means: 64 features
    return imgs.reshape(len(imgs), 8, 4, 8, 4).mean(axis=(2, 4)).reshape(len(imgs), -1)


ref = block_means(make_images(np.random.default_rng(12345), 5000))
mu, sd = ref.mean(0), ref.std(0)
Sigma = ((ref - mu) / sd).T @ ((ref - mu) / sd) / len(ref)


def featurize(rng, n):
    return (block_means(make_images(rng, n)) - mu) / sd


def tasks(theta_deg, seed=7):
    """W_t = C_t U_t^T with U_2 at principal angle theta from U_1; unit signal variance per output."""
    rng = np.random.default_rng(seed)
    Q, _ = np.linalg.qr(rng.normal(size=(p, 2 * k)))
    th = np.deg2rad(theta_deg)
    Ws = []
    for U in (Q[:, :k], np.cos(th) * Q[:, :k] + np.sin(th) * Q[:, k:]):
        W = rng.normal(size=(q, k)) @ U.T
        Ws.append(W / np.sqrt(np.trace(W @ Sigma @ W.T) / q))
    return Ws


def ols(H, Y):
    return np.linalg.lstsq(H, Y, rcond=None)[0]


def rrr(H, Y, r, w=(1.0, 1.0)):
    """Reduced-rank regression: min sum_t w_t ||Y_t - H B_t||^2 over rank(B) <= r (closed form)."""
    s = np.repeat(np.sqrt(w[:Y.shape[1] // q]), q)
    B = ols(H, Y * s)
    Vr = np.linalg.svd(H @ B, full_matrices=False)[2][:r].T
    return B @ Vr @ Vr.T / s


def rel_err(W, B, H):         # relative test error against the noise-free targets
    return np.sum((H @ (B - W.T)) ** 2) / np.sum((H @ W.T) ** 2)


def fit_all(H, Y1, Y2):
    Y = np.hstack([Y1, Y2])
    J3, J6 = rrr(H, Y, k), rrr(H, Y, 2 * k)
    return {"S-OLS": (ols(H, Y1), ols(H, Y2)), "S-RRR": (rrr(H, Y1, k), rrr(H, Y2, k)),
            "J-RRR(3)": (J3[:, :q], J3[:, q:]), "J-RRR(6)": (J6[:, :q], J6[:, q:])}


Hte = featurize(np.random.default_rng(999), 2000)
METHODS = ("S-OLS", "S-RRR", "J-RRR(3)", "J-RRR(6)")

# 1. Data checks and the noise-free base check
H96 = featurize(np.random.default_rng(0), 96)
print("features: rank at n = 96: %d, condition number of covariance: %.0f" % (np.linalg.matrix_rank(H96), np.linalg.cond(Sigma)))
for th in ANGLES:
    W1, W2 = tasks(th)
    ang = np.rad2deg(subspace_angles(W1.T, W2.T))
    errs = fit_all(H96, H96 @ W1.T, H96 @ W2.T)
    print("theta %2d: angles %s, noise-free n = 96:" % (th, np.round(ang, 6)),
          " ".join("%s %.1e/%.1e" % (m, *(rel_err(W, b, Hte) for W, b in zip((W1, W2), errs[m]))) for m in METHODS))

# 2. Relative test error against n and relatedness, mean over seeds (noise variance 1 per output)
res = np.zeros((len(SEEDS), len(ANGLES), len(SIZES), len(METHODS), 2))
for si, seed in enumerate(SEEDS):
    rng = np.random.default_rng(seed)
    H, E = featurize(rng, SIZES[-1]), rng.normal(size=(SIZES[-1], 2 * q))
    for ai, th in enumerate(ANGLES):
        W1, W2 = tasks(th)
        Y1, Y2 = H @ W1.T + E[:, :q], H @ W2.T + E[:, q:]
        for ni, n in enumerate(SIZES):
            fits = fit_all(H[:n], Y1[:n], Y2[:n])
            for mi, m in enumerate(METHODS):
                res[si, ai, ni, mi] = [rel_err(W, b, Hte) for W, b in zip((W1, W2), fits[m])]
mean = res.mean(0)
win = (res[:, :, :, 2] < res[:, :, :, 1]).mean(0)   # share of seeds where J-RRR(3) beats S-RRR
print("\nrelative test error, task 1 / task 2, mean over %d seeds" % len(SEEDS))
print("theta     n " + "".join("%16s" % m for m in METHODS) + "   J3<S-RRR")
for ai, th in enumerate(ANGLES):
    for ni, n in enumerate(SIZES):
        print("%5d %5d " % (th, n) + "".join("   %6.3f/%6.3f" % tuple(mean[ai, ni, mi]) for mi in range(4))
              + "   %4.2f/%4.2f" % tuple(win[ai, ni]))

# 3. Task gradients with respect to the shared encoder A (B_t = A G_t^T), seed 0, n = 1536
rng = np.random.default_rng(0)
H, E = featurize(rng, 1536), rng.normal(size=(1536, 2 * q))


def grad_A(A, G, Y):          # gradient of (1/2n)||Y - H A G^T||^2 with respect to A
    return -H.T @ (Y - H @ A @ G.T) @ G / len(H)


def cos(a, b):
    return np.sum(a * b) / np.linalg.norm(a) / np.linalg.norm(b)


W1, W2 = tasks(60)
Y1, Y2 = H @ W1.T + E[:, :q], H @ W2.T + E[:, q:]
A, G = rng.normal(size=(p, k)), rng.normal(size=(q, k))
num = np.zeros_like(A)
for i in range(p):
    for j in range(k):
        D = np.zeros_like(A); D[i, j] = 1e-6
        f = lambda X: 0.5 * np.sum((Y1 - H @ X @ G.T) ** 2) / len(H)
        num[i, j] = (f(A + D) - f(A - D)) / 2e-6
print("\nfinite-difference check of grad_A: max relative error %.1e" % (np.abs(num - grad_A(A, G, Y1)).max() / np.abs(num).max()))
print("theta  cos at random encoders: mean  min   max  share<0 | at joint optimum: |g1|   |g2|   cos   |g1+g2|/|g1|")
for th in ANGLES:
    W1, W2 = tasks(th)
    Y1, Y2 = H @ W1.T + E[:, :q], H @ W2.T + E[:, q:]
    cs = []
    for _ in range(200):
        A = rng.normal(size=(p, k))
        G1, G2 = (ols(H @ A, Y).T for Y in (Y1, Y2))   # each head fitted for this encoder
        cs.append(cos(grad_A(A, G1, Y1), grad_A(A, G2, Y2)))
    B = ols(H, np.hstack([Y1, Y2]))
    V = np.linalg.svd(H @ B, full_matrices=False)[2][:k].T
    g1, g2 = grad_A(B @ V, V[:q], Y1), grad_A(B @ V, V[q:], Y2)
    print("%5d %31.3f %5.3f %5.3f %6.2f  | %22.4f %6.4f %6.3f %9.1e" % (th, np.mean(cs), np.min(cs), np.max(cs), np.mean(np.array(cs) < 0),
          np.linalg.norm(g1), np.linalg.norm(g2), cos(g1, g2), np.linalg.norm(g1 + g2) / np.linalg.norm(g1)))


# 4. Uncertainty weighting (Kendall et al.): sum_t L_t / (2 sigma_t^2) + log sigma_t, theta = 60, n = 192
def uw_fit(H, Y, s2=(1.0, 1.0)):
    """Alternate the closed-form weighted fit (weights 1/(2 sigma^2)) and sigma_t^2 = L_t."""
    s2 = np.array(s2)
    for it in range(1000):
        B = rrr(H, Y, k, w=1 / (2 * s2))
        L = np.array([np.mean(np.sum((Y[:, t*q:(t+1)*q] - H @ B[:, t*q:(t+1)*q]) ** 2, 1)) for t in (0, 1)])
        if np.max(np.abs(L - s2) / L) < 1e-13:
            break
        s2 = L
    return B, L, it


W1, W2 = tasks(60)
n, c = 192, 10.0              # c: task 2 re-expressed in units 10 times smaller
out, chk = np.zeros((len(SEEDS), 4, 2)), np.zeros((len(SEEDS), 5))
for si, seed in enumerate(SEEDS):
    rng = np.random.default_rng(seed)
    H, E = featurize(rng, SIZES[-1]), rng.normal(size=(SIZES[-1], 2 * q))
    H, Y = H[:n], np.hstack([H[:n] @ W1.T, H[:n] @ W2.T]) + E[:n]
    Yc = Y * np.repeat([1.0, c], q)
    Be, Bec = rrr(H, Y, k), rrr(H, Yc, k) / np.repeat([1.0, c], q)
    Bu, L, it = uw_fit(H, Y)
    Buc = uw_fit(H, Yc)[0] / np.repeat([1.0, c], q)
    for j, Bx in enumerate((Be, Bec, Bu, Buc)):
        out[si, j] = rel_err(W1, Bx[:, :q], Hte), rel_err(W2, Bx[:, q:], Hte)
    s2_num = [np.exp(minimize_scalar(lambda ls: L[t] / (2 * np.exp(ls)) + ls / 2, bounds=(-10, 10),
                                     method="bounded", options={"xatol": 1e-12}).x) for t in (0, 1)]
    chk[si] = (np.max(np.abs(np.array(s2_num) - L) / L), np.abs(Buc - Bu).max() / np.abs(Bu).max(),
               np.abs(uw_fit(H, Y, (1.0, 100.0))[0] - Bu).max() / np.abs(Bu).max(), it,
               np.abs(Bec - Be).max() / np.abs(Be).max())
print("\nuncertainty weighting, theta = 60, n = %d, relative test error task 1 / task 2 (original units), mean over seeds" % n)
for j, name in enumerate(("equal weights", "equal weights, task 2 x10", "uncertainty weighting", "uncertainty weighting, task 2 x10")):
    print("  %-34s %.3f / %.3f" % (name, *out[:, j].mean(0)))
print("max over seeds: |sigma^2 (numerical) - L| / L = %.1e, unit change %.1e, init (1, 100) %.1e, iterations %d"
      % tuple(chk.max(0)[:4]))
print("equal weights, unit change: max relative change of B, min over seeds %.2f" % chk[:, 4].min())

features: rank at n = 96: 64, condition number of covariance: 712
theta  0: angles [0. 0. 0.], noise-free n = 96: S-OLS 2.1e-30/2.5e-30 S-RRR 1.8e-30/1.9e-30 J-RRR(3) 2.5e-30/2.4e-30 J-RRR(6) 2.6e-30/2.9e-30
theta 30: angles [30. 30. 30.], noise-free n = 96: S-OLS 2.1e-30/2.5e-30 S-RRR 1.8e-30/2.8e-30 J-RRR(3) 3.4e-02/2.6e-02 J-RRR(6) 2.3e-30/2.5e-30
theta 60: angles [60. 60. 60.], noise-free n = 96: S-OLS 2.1e-30/3.5e-30 S-RRR 1.8e-30/3.1e-30 J-RRR(3) 1.7e-01/9.2e-02 J-RRR(6) 1.7e-30/3.3e-30
theta 90: angles [90. 90. 90.], noise-free n = 96: S-OLS 2.1e-30/2.8e-30 S-RRR 1.8e-30/2.5e-30 J-RRR(3) 1.8e-01/1.4e-01 J-RRR(6) 2.6e-30/4.5e-30

relative test error, task 1 / task 2, mean over 100 seeds
theta     n            S-OLS           S-RRR        J-RRR(3)        J-RRR(6)   J3<S-RRR
    0    96     2.635/ 2.580    1.072/ 1.052    0.591/ 0.485    1.390/ 1.300   1.00/1.00
    0   192     0.576/ 0.558    0.233/ 0.229    0.135/ 0.113    0.307/ 0.288   1.00/1.00
    0   384     0.217/ 0.213    0.088/ 0.087    0.053/ 0.044    0.117/ 0.110   1.00/1.00
    0  1536     0.046/ 0.045    0.019/ 0.018    0.011/ 0.009    0.025/ 0.023   1.00/1.00
   30    96     2.635/ 2.566    1.072/ 1.057    0.634/ 0.508    1.409/ 1.254   1.00/1.00
   30   192     0.576/ 0.555    0.233/ 0.229    0.172/ 0.139    0.311/ 0.281   1.00/1.00
   30   384     0.217/ 0.212    0.088/ 0.087    0.086/ 0.070    0.113/ 0.108   0.55/0.99
   30  1536     0.046/ 0.045    0.019/ 0.018    0.044/ 0.036    0.022/ 0.021   0.00/0.00
   60    96     2.635/ 2.532    1.072/ 1.051    0.787/ 0.599    1.345/ 1.182   0.98/1.00
   60   192     0.576/ 0.548    0.233/ 0.228    0.300/ 0.224    0.283/ 0.265   0.01/0.55
   60   384     0.217/ 0.209    0.088/ 0.086    0.206/ 0.153    0.100/ 0.096   0.00/0.00
   60  1536     0.046/ 0.044    0.019/ 0.018    0.162/ 0.115    0.020/ 0.019   0.00/0.00
   90    96     2.635/ 2.518    1.072/ 1.043    0.855/ 0.607    1.332/ 1.145   0.94/1.00
   90   192     0.576/ 0.545    0.233/ 0.227    0.334/ 0.239    0.272/ 0.256   0.00/0.31
   90   384     0.217/ 0.208    0.088/ 0.086    0.237/ 0.173    0.098/ 0.094   0.00/0.00
   90  1536     0.046/ 0.044    0.019/ 0.018    0.189/ 0.141    0.020/ 0.019   0.00/0.00

finite-difference check of grad_A: max relative error 1.6e-09
theta  cos at random encoders: mean  min   max  share<0 | at joint optimum: |g1|   |g2|   cos   |g1+g2|/|g1|
    0                           0.591 -0.440 0.877   0.01  |                 0.1300 0.1300 -1.000   9.5e-14
   30                           0.461 -0.041 0.822   0.01  |                 0.6319 0.6319 -1.000   2.5e-14
   60                           0.134 -0.523 0.515   0.23  |                 1.2784 1.2784 -1.000   1.1e-14
   90                           0.167 -0.510 0.600   0.20  |                 0.8082 0.8082 -1.000   1.0e-14

uncertainty weighting, theta = 60, n = 192, relative test error task 1 / task 2 (original units), mean over seeds
  equal weights                      0.300 / 0.224
  equal weights, task 2 x10          0.741 / 0.226
  uncertainty weighting              0.302 / 0.224
  uncertainty weighting, task 2 x10  0.302 / 0.224
max over seeds: |sigma^2 (numerical) - L| / L = 3.3e-08, unit change 1.0e-12, init (1, 100) 8.9e-13, iterations 63
equal weights, unit change: max relative change of B, min over seeds 0.40

The exact checks hold. With noise-free targets, every method that can represent both maps recovers them to about $10^{-30}$ , rounding error: separate fits at every angle, the rank-6 joint fit at every angle, and the rank-3 joint fit at $\theta = 0$ . The rank-3 joint fit at $\theta > 0$ cannot: six directions cannot fit into three, and its noise-free error (0.034/0.026 at 30°, up to 0.18/0.14 at 90°) is the floor that sharing imposes whatever the amount of data. The analytic gradient matches finite differences, and at the joint optimum the two task gradients cancel to $10^{-14}$ with cosine $-1$ , as the stationarity argument requires.

Related tasks gain at every size. At $\theta = 0$ the joint fit beats the separate rank-3 fits on every seed at every $n$ , with about half their error: 0.591/0.485 against 1.072/1.052 at $n = 96$ , 0.011/0.009 against 0.019/0.018 at $n = 1{,}536$ . Both tasks help estimate the shared subspace, which is the Deep Learning book’s pooling argument in its simplest form. The planned hypothesis that the advantage would shrink toward nothing as $n$ grew failed: the ratio stays near one half from the smallest to the largest set. In our reading, this is because the joint fit keeps fewer free parameters at every size, $3 \times (64 + 16 - 3) = 231$ against $2 \times 3 \times (64 + 8 - 3) = 414$ for the two separate fits, a ratio of 0.56 close to the printed error ratios, and in a correctly specified model that advantage does not wear off with more data.

Unrelated tasks interfere once the data allow it. At $\theta = 90^\circ$ the joint rank-3 fit is worse than separate fits from $n = 192$ on, and its error stops falling at about its noise-free floor (0.189/0.141 at $n = 1{,}536$ against 0.019/0.018). The planned hypothesis that it would be worse at every $n$ failed: at $n = 96$ it beats the separate fits on 94% and 100% of seeds. With 96 images, the separate fits’ estimation error (above 1, worse than predicting zero) is larger than the bias of forcing two unrelated tasks into one subspace, so the constrained model wins. Interference here is a bias that only shows once variance has fallen below it. At 30° the crossover happens inside the range: the joint fit wins at 96 and 192 images, at 384 is about level with them on task 1 (better on 55% of seeds) and still better on task 2 (99%), and loses at 1,536. The same model is helped or hurt by sharing depending on the amount of data, which is consistent with Standley et al.’s finding that task affinities change with the training-set size, though their direction differed (below).

Capacity is the other half. The rank-6 joint fit has room for both subspaces and approaches the separate fits at large $n$ at every angle, but it is worse than the rank-3 joint fit at $\theta = 0$ and worse than the separate fits at small $n$ : a shared representation wider than the tasks need gives each head more to estimate. Least squares without the rank structure (S-OLS) is far worse than every rank-3 fit at small $n$ . In this model, choosing the width of the shared representation matters as much as choosing what to share.

Gradient cosines are a weak signal. At random encoders the mean cosine between the tasks’ gradients is 0.59 for identical subspaces and 0.46 at 30°, and negative cosines become frequent (about a fifth of the encoders) at 60° and 90°. The planned hypothesis that the mean cosine would fall steadily with the angle failed: it is 0.13 at 60° and 0.17 at 90°. At the joint optimum the gradients always cancel exactly, and their size reflects how much each task gave up; the planned hypothesis that this size would grow with the angle failed too, since it peaks at 60° (1.28) and is lower at 90° (0.81). Both of these measurements use seed 0 only. A cosine at one point says which way two tasks pull, not how much the shared representation must give up, and in this example it does not rank the four levels of relatedness correctly.

Uncertainty weighting is unit-free; equal weights are not. With both tasks in their own units and the same noise, uncertainty weighting and equal weights give nearly the same errors (0.302/0.224 against 0.300/0.224). After task 2’s targets are multiplied by 10, the equal-weight fit changes completely: its matrix moves by at least 40% of its largest entry on every seed, task 2 now dominates the shared subspace, and task 1’s error rises from 0.300 to 0.741. Uncertainty weighting gives the same fit, after undoing the scale, to $10^{-12}$ , and the same errors. Its $\sigma_t^2$ equals the task’s mean loss to $3 \times 10^{-8}$ (the precision of the scalar minimizer), and starting from $\sigma^2 = (1, 100)$ reaches the same fit. Where uncertainty weighting helps, in this model, is in removing a choice that units would otherwise make; it does not find a better compromise between equally scaled tasks. The printed value for the equal-weight fit’s change was added before the first run, to match a check written in the plan that the script had left out.

Variants

Approach	What it changes	Examples
Hard sharing	One trunk, a head per task	Eigen & Fergus’s shared first scale; UberNet; Kendall et al.; Mask R-CNN; YOLOP
Split architectures	Where the trunk branches	Misra et al.’s split study
Soft sharing	Separate networks tied by a penalty or by exchanged activations	Parameter-norm tying (DL book §7.9); cross-stitch units
Loss weighting	The weights $w_t$	Uncertainty weighting; GradNorm; dynamic weight averaging and random weights (in Navon et al.’s comparison)
Gradient combination	How per-task gradients become one update	MGDA; PCGrad; Nash-MTL
Task grouping	Which tasks share a network	Standley et al.’s selection under a budget
Datasets with partial labels	Each image labelled for some tasks	UberNet’s zero loss for missing labels and asynchronous updates

Auxiliary tasks. A task can be included only to help another, and dropped at test time. In two-stream networks, the temporal stream was trained with one softmax head for UCF101 and one for HMDB51 and the two losses summed; on HMDB51 split 1 it reached 55.4%, against 46.6% trained on HMDB51 alone. Standley et al. found that adding surface normals improved the other task in 15 of 16 networks trained with it, while normals themselves tended to do worse when trained with another task. Deep supervision and auxiliary losses attached to intermediate layers of one task, as in DETR’s decoder, are a related but different device: the extra losses supervise the same task.

Multi-task distillation. HF-Net trains its three heads to reproduce existing networks’ outputs (NetVLAD for the global descriptor, SuperPoint for keypoints and local descriptors), so the tasks’ labels come from teachers (knowledge distillation).

Best Practices

Compare at equal cost. A multi-task network should be compared with single-task networks given the same total inference budget as well as with full-size ones; Standley et al.’s two comparisons give opposite answers for three to five tasks (below).
Normalize regression targets to comparable scales before choosing weights, as Fast R-CNN did for its box targets and Navon et al. did for QM9’s 11 properties; or use a weighting that is invariant to scale.
Tune, or learn, the loss weights, and report what was used; Kendall et al.’s and UberNet’s results move by several points with the weights.
Check every task. Report each task’s metric next to its single-task baseline; an aggregate can improve while one task gets worse.
Revisit task groupings when the network or the dataset changes size, since Standley et al. found the affinities to depend on both.
Give the shared representation enough capacity. UberNet’s author attributes the drop at seven tasks partly to the limited parameter budget of the shared trunk, and the example shows the same trade between width and sharing.

Pitfalls

Assuming related tasks help. Intuitively related tasks need not train well together: Standley et al. found that depth and normals, Taskonomy’s most transfer-related pair in their set, did not score highly on multi-task affinity.
Assuming multi-task learning helps most with little data. In Standley et al.’s setting with 5% of the data (about 200,000 images), most tasks suffered when trained with another, against the common assumption that pooling supervision helps most in low-data regimes. The linear example, where sharing helped most at small $n$ for related tasks, shows that the answer depends on the model and the tasks; neither direction is a law.
Letting one loss dominate. Kendall et al.’s unweighted sum and the example’s change of units both show it.
Judging by total loss. Summed losses on different scales weight tasks arbitrarily; Chen et al. evaluate their toy problem with loss ratios for this reason.
Reading a gradient cosine as relatedness. The example’s cosines did not order four levels of relatedness, and at any compromise the gradients oppose each other exactly.
Ignoring missing labels. When datasets are combined, each image’s loss should include only the tasks it is labelled for (UberNet), or the network is trained toward missing values.

Evidence and Impact

The evidence is mixed, and its direction depends on the tasks, the network size, the data and the baseline.

Gains.

Fast R-CNN’s box-regression loss raised VOC 2007 mAP by 0.8 to 1.1 points even with box regression unused at test time (Fast R-CNN article).
Mask R-CNN’s mask branch added 0.9 box AP, and joint training with person masks raised its keypoint variant from 62.7 to 63.1 keypoint AP on COCO test-dev (Mask R-CNN paper).
Cross-stitch networks improved both tasks over single-task networks on NYU-v2 (surface-normal mean angular error 34.1° against 34.8°, segmentation mIU 19.3 against 18.4) and on attribute PASCAL 2008 (detection 45.2 against 44.9 mAP, attributes 63.0 against 60.9), and beat the best split architecture on all four. An ensemble of two single-task detectors reached 46.1 mAP on detection, above the cross-stitch network.
Kendall et al.’s uncertainty-weighted three-task network beat all three single-task models on Tiny Cityscapes (table above).
In surgical video, EndoNet’s features trained for phase and tool presence beat features trained for phase alone on Cholec80.

Losses and interference.

UberNet’s detection reached 80.1 mAP on PASCAL VOC 2007 test when trained with segmentation, against 78.7 alone, but 77.8 with all seven tasks; the other tasks showed what the author calls a moderate but systematic drop at seven tasks. Raising the weight of normal estimation from 1 to 50 improved normals and lowered every other task, which the paper likens to communicating vessels; at the intermediate weight of 5 in the same table (Table 9), normals were slightly worse too.
In Mask R-CNN, adding the keypoint branch slightly lowered box and mask AP.
Standley et al. (Setting 1, small network) found that a two-task network had a 7.56% worse total loss on average than two single-task networks of full size each, and a five-task network 19.00% worse. Against single-task networks shrunk to the same total compute, three-, four- and five-task networks were better (4.23%, 4.86% and 0.34%), while two-task networks were still 1.15% worse. For budgets above 1.5 standard network times, their chosen groupings beat both one multi-task network and five separate ones.
Standley et al. also report that Gong et al. (2019) and Leang et al. (2020), who compared loss-weighting strategies, found no clear winner, with uniform weights performing similarly; neither paper was read here.

Weighting and gradient methods on shared benchmarks. Navon et al. compared twelve approaches over three seeds on NYUv2 (segmentation, depth, normals) and Cityscapes (segmentation, depth, at 128 × 256), following an earlier protocol with single-task SegNet baselines. On Cityscapes every multi-task method had higher depth error than the single-task model (absolute error 0.0125); equal weights reached 0.0155, uncertainty weighting 0.0140 and Nash-MTL 0.0129, while segmentation mIoU moved from 74.01 for the single-task model to 75.18 with equal weights and 75.41 with Nash-MTL. By their average relative drop $\Delta_m$ , Nash-MTL was best on NYUv2 (−4.04%, an improvement) and second best on Cityscapes (6.82%), where uncertainty weighting was best (5.89%). On NYUv2 with equal weights, multi-task training improved segmentation and depth but worsened surface normals (mean angle error 28.15° against 25.01° single-task). In Standley et al.’s comparison, MGDA (Sener & Koltun’s method) under-weighted segmentation and did worse than a plain sum of losses, and GradNorm was slightly worse than uniform weights, though they note this may be because they evaluated with uniform weights.

Task relations. Taskonomy turned transfer relationships among 26 tasks into supervision policies that reduce labelled-data needs. Standley et al. found no correlation between those transfer affinities and multi-task affinities in any of their settings, and multi-task affinities that changed between a small and a large network ( $r = 0.08$ between the two settings) and with data size.

In our reading, the evidence supports two firm conclusions: multi-task networks save inference cost at some loss on at least one task in most of the comparisons above, and gains on all tasks occur for particular pairs, weightings and sizes rather than generally. The linear example shows the mechanism behind both: shared parameters gain from pooled data when the tasks share structure, and pay a bias when they do not, with the amount of data deciding which effect is larger.