Multimodal Learning

Multimodal learning builds models that learn from, and relate, more than one kind of data: images and text, video and audio, color and depth. A captioning model turns a photograph into a sentence, a retrieval system finds photographs that match a sentence, and a vision–language assistant answers questions about an image. Its largest branch in computer vision is vision–language (or vision-and-language) research, for which this article is the hub.

Definition

A modality is a way in which information is experienced or recorded. Baltrušaitis, Ahuja and Morency (2019) call a problem multimodal when it includes several modalities, and define multimodal machine learning as building models that process and relate information from them. The modalities differ in kind: an image is a dense array of continuous values, a sentence a short sequence of discrete symbols. Liang, Zadeh and Morency (2024) put this heterogeneity first among their three principles, beside the connections between modalities and the interactions that arise when they are combined.

Intuition

A photograph of a dog on a beach and the sentence “a dog runs along the shore” carry overlapping information in incompatible forms. A multimodal model needs common ground: a vector space in which the photograph and the sentence land near each other, a network that reads both together, or a decoder that produces one from the other. Each modality can also fill the other’s gaps: text gives a date the camera cannot show, the image the color of the dog.

The Core Challenges

Going beyond the early-versus-late fusion split, Baltrušaitis et al. (2019) organized the field around five challenges:

• Representation: encoding multimodal data so that the modalities’ complementary and redundant information is exploited.
• Translation: mapping data from one modality to another, as in captioning, where many outputs can be correct; methods are example-based (retrieve a stored output) or generative.
• Alignment: finding direct relations between sub-elements of different modalities, such as the image region a phrase refers to or the video segment showing a recipe step; explicit alignment is the goal itself, implicit alignment a latent step inside another task.
• Fusion: joining information from several modalities to make a prediction, either model-agnostically (early, late or hybrid fusion) or inside the model (kernels, graphical models, neural networks).
• Co-learning: transferring knowledge from one modality to help a model of another, with parallel (paired), non-parallel or hybrid training data; zero-shot learning is one example.

Liang, Zadeh and Morency (2024) revised the taxonomy into six challenges: representation, alignment, reasoning, generation, transference and quantification. In their arXiv version (v2, 2023), fusion becomes one form of representation, beside coordination and fission (splitting data into more, factorized representations); translation becomes part of generation, with summarization and creation; co-learning sits under transference; and reasoning (composing knowledge, usually over several inferential steps) and quantification (studying heterogeneity, cross-modal interactions and the learning process) are new. Fusion has its own article, Multimodal Fusion.

Joint and Coordinated Representations

Baltrušaitis et al. (2019) distinguish two kinds of multimodal representation. A joint representation computes one vector from all modalities,

$$x_m = f(x_1, \dots, x_n),$$

with $f$ a neural network or another model; concatenating unimodal features is the simplest case (early fusion). A coordinated representation keeps a separate projection for each modality and constrains the outputs to be related,

$$f(x_1) \sim g(x_2),$$

where $\sim$ can mean small cosine distance, high correlation, or a partial order. Joint representations are mostly used when every modality is present at both training and test time; coordinated ones encode each modality alone, so images can be embedded once and searched with any sentence later, the basis of image–text retrieval and zero-shot classification.

A widely used coordinated method is image–text contrastive learning. For a batch of $N$ image–caption pairs, with image embeddings $u_i = f(x_i)$, caption embeddings $w_j = g(c_j)$, cosine similarity $s_{ij} = \mathrm{sim}(u_i, w_j)$ and temperature $\tau$, CLIP’s symmetric loss (Radford et al., 2021) averages a cross-entropy over captions for each image and over images for each caption:

$$\mathcal{L} = \frac{1}{2N} \sum_{i=1}^{N} \left[ -\log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N} \exp(s_{ij}/\tau)} \; - \log \frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N} \exp(s_{ji}/\tau)} \right].$$

Each term is an $N$-way softmax cross-entropy like the contrastive learning article’s NT-Xent, except that only the other modality supplies the positive and the negatives. Its gradient with respect to a similarity is

$$\frac{\partial \mathcal{L}}{\partial s_{ij}} = \frac{\big(P^{\text{i}\to\text{t}}_{ij} - \mathbb{1}[i = j]\big) + \big(P^{\text{t}\to\text{i}}_{ji} - \mathbb{1}[i = j]\big)}{2N\tau},$$

where $P^{\text{i}\to\text{t}}_{ij}$ is the softmax probability of caption $j$ for image $i$ and $P^{\text{t}\to\text{i}}_{ji}$ that of image $i$ for caption $j$. After training, an image is classified zero-shot by writing a caption for each class and choosing the closest, $\hat{y} = \arg\max_k \mathrm{sim}\big(f(x), g(\text{prompt}_k)\big)$; in the arXiv version, CLIP’s authors found the template “A photo of a {label}.” a good default.

Vision and Language: A Short History

Words as labels for regions (2002). Duygulu, Barnard, de Freitas and Forsyth (2002) cast object recognition as machine translation: images are segmented into regions, the regions are grouped into types by their features, and an EM method learns a lexicon between region types and the images’ keywords, as one learns a lexicon from aligned bilingual text. It is an early case of alignment learned from weakly paired data.

A shared embedding (2013). DeViSE (Frome et al., 2013) mapped a convolutional network’s image features, through a linear layer trained with a hinge rank loss, into the word-vector space of a skip-gram model learned from unannotated text. Because related labels have nearby word vectors, it could name images with labels the visual model never saw: 18.1% flat hit@5 on 1,589 unseen ImageNet labels close to its training classes. This is coordinated representation used for co-learning.

Generating and answering (2015). Show and Tell (Vinyals et al., 2015) connected a CNN image encoder to an LSTM decoder trained to maximize the caption’s likelihood, the encoder–decoder design of machine translation applied to images. VQA (Antol et al., 2015) posed free-form, open-ended visual question answering, with about 0.25 million images, 0.76 million questions and 10 million answers.

Pretrained transformers (2019). ViLBERT (Lu et al., 2019) processed image regions and text in two transformer streams joined by co-attentional layers, pretrained on Conceptual Captions and transferred to four tasks; LXMERT (Tan & Bansal, 2019) used object, language and cross-modality encoders with five pretraining tasks. Both build joint representations from Faster R-CNN region features. The family is the subject of Vision–Language Pretraining.

Contrastive pretraining at web scale (2021). CLIP (Radford et al., 2021) trained an image encoder, a ResNet or a Vision Transformer, and a text encoder with the loss above on 400 million image–text pairs from the internet. Zero-shot, it matched the ImageNet accuracy of the original ResNet-50 without using any of its 1.28 million training images.

Language models that see (2022–2023). Flamingo (Alayrac et al., 2022) bridged a frozen pretrained vision encoder and a frozen language model with a Perceiver Resampler and new cross-attention layers, and trained on web pages with interleaved images and text, so that one model could learn a new task from a few examples in its prompt. BLIP-2 (Li et al., 2023) bridged a frozen image encoder and a frozen large language model with a lightweight Querying Transformer pretrained in two stages. LLaVA (Liu et al., 2023) connected a CLIP ViT-L/14 encoder to the Vicuna language model with a linear projection and fine-tuned on 158K instruction-following samples generated by the text-only GPT-4. In these vision–language models (VLMs), image features condition a language model, as extra input tokens or through cross-attention, and it answers in text; Vision–Language Model covers the architecture.

In our reading, the trend is from hand-built correspondences to representations learned from ever larger, noisier paired data, and from task-specific models to one model reused across tasks, first by fine-tuning and then by prompting.

Tasks

Task	Input → output	Representative methods	Article
Image captioning	image → sentence	Show and Tell; encoder–decoders with attention; VLMs	Image Captioning
Visual question answering	image + question → answer	VQA baselines; ViLBERT, LXMERT; VLMs	Visual Question Answering
Image–text retrieval	sentence → ranked images, or image → ranked captions	DeViSE-style embeddings; CLIP	Image–Text Retrieval
Visual grounding	image + phrase → box or mask	referring-expression models; Grounding DINO	Visual Grounding
Zero-shot classification	image + class names → label	DeViSE; CLIP	CLIP
Text-to-image generation	sentence → image	text-conditioned generative models	(not yet covered)
Video–language tasks	video + text → text, score or segment	video captioning, retrieval, question answering	(not yet covered)

In the taxonomy above, captioning is translation, retrieval and zero-shot classification use coordinated representations, question answering needs fusion and often reasoning, and grounding is explicit alignment, close to object detection.

Datasets and Evaluation

Common benchmarks include COCO Captions (five human captions per image), Flickr30k, VQA v2, Visual Genome and RefCOCO; each task’s article describes its own. Captioning is scored with n-gram metrics such as BLEU, METEOR and CIDEr; VQA counts an answer as fully correct when at least three of the ten human annotators gave it, $\min(\#\text{humans}/3, 1)$ (Antol et al., 2015); retrieval reports recall at rank $K$.

Example: A Toy Coordinated Embedding and Zero-Shot Classification

The script trains a coordinated image–caption embedding from scratch. The images are synthetic $16 \times 16$ RGB pictures of a red, green or blue circle, square or triangle with small random shifts, sizes, brightness and noise; the captions are invented phrases such as “a red circle” over a seven-word vocabulary. The image encoder is a linear map from 768 pixel values to 16 dimensions, the text encoder a linear map from a bag-of-words vector. Two of the nine color–shape combinations, a red square and a blue triangle, never appear in training, although every color and every shape does.

Each step draws one fresh image per training caption, so no batch contains duplicate captions (which would act as false negatives), and takes an Adam step on the symmetric loss with $\tau = 0.3$; the hand-written gradient is checked against finite differences. New images of all nine combinations are then classified zero-shot. It runs in under a second with NumPy 2.3.

import sys
import numpy as np

S = 16
COLORS = {"red": (1.0, 0.25, 0.25), "green": (0.25, 1.0, 0.25), "blue": (0.25, 0.25, 1.0)}
SHAPES = ("circle", "square", "triangle")
VOCAB = ["a", "red", "green", "blue", "circle", "square", "triangle"]
HELD_OUT = {("blue", "triangle"), ("red", "square")}          # never shown in training
yy, xx = np.mgrid[0:S, 0:S] + 0.5

def draw(color, shape, rng):
    """One 16x16 RGB image: a colored shape on a dark background, jittered and noisy."""
    cy, cx = 8 + rng.uniform(-1, 1, size=2)
    r = rng.uniform(4.0, 5.5)
    y, x = yy - cy, xx - cx
    if shape == "circle":
        m = x**2 + y**2 < r**2
    elif shape == "square":
        m = (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r)
    else:
        m = (y < 0.8 * r) & (y > -r + 2 * abs(x))
    img = 0.1 + m[..., None] * np.array(COLORS[color]) * rng.uniform(0.7, 1.0)
    return img + 0.05 * rng.standard_normal((S, S, 3))

def caption_vector(caption):
    """Bag of words over the invented 7-word vocabulary."""
    v = np.zeros(len(VOCAB))
    for w in caption.split():
        v[VOCAB.index(w)] += 1
    return v

def symmetric_loss(Wi, Wt, X, T, tau, grad=True):
    """Symmetric contrastive loss for N image-caption pairs (row i matches row i)."""
    A, B = X @ Wi, T @ Wt                       # unnormalized image and text embeddings
    na, nb = np.linalg.norm(A, axis=1, keepdims=True), np.linalg.norm(B, axis=1, keepdims=True)
    U, V = A / na, B / nb
    L = U @ V.T / tau                           # logits: sim(u_i, v_j) / tau
    def softmax(Z):
        Z = np.exp(Z - Z.max(1, keepdims=True))
        return Z / Z.sum(1, keepdims=True)
    P_it, P_ti = softmax(L), softmax(L.T)       # image->text rows, text->image rows
    n = len(X)
    loss = -0.5 * (np.log(np.diag(P_it)).mean() + np.log(np.diag(P_ti)).mean())
    if not grad:
        return loss
    I = np.eye(n)
    G = 0.5 * ((P_it - I) + (P_ti - I).T) / (n * tau)   # d loss / d sim(u_i, v_j)
    dU, dV = G @ V, G.T @ U
    dA = (dU - (dU * U).sum(1, keepdims=True) * U) / na  # back through u = a / ||a||
    dB = (dV - (dV * V).sum(1, keepdims=True) * V) / nb
    return loss, X.T @ dA, T.T @ dB

def train(seed, steps=600, d=16, tau=0.3, lr=0.01, check=False):
    rng = np.random.default_rng(seed)
    pairs = [(c, s) for c in COLORS for s in SHAPES if (c, s) not in HELD_OUT]
    T = np.stack([caption_vector(f"a {c} {s}") for c, s in pairs])
    Wi = 0.01 * rng.standard_normal((S * S * 3, d))   # linear image encoder
    Wt = 0.1 * rng.standard_normal((len(VOCAB), d))   # bag-of-words text encoder
    m = [np.zeros_like(Wi), np.zeros_like(Wt)]
    v = [np.zeros_like(Wi), np.zeros_like(Wt)]
    for step in range(1, steps + 1):
        # one fresh image per training caption, so the batch has no duplicate captions
        X = np.stack([draw(c, s, rng).ravel() for c, s in pairs])
        loss, gWi, gWt = symmetric_loss(Wi, Wt, X, T, tau)
        if check and step == 1:
            num, crng = [], np.random.default_rng(100)   # separate stream: training is unchanged
            for W, g in ((Wi, gWi), (Wt, gWt)):
                for idx in [tuple(crng.integers(0, n) for n in W.shape) for _ in range(20)]:
                    old = W[idx]
                    W[idx] = old + 1e-6
                    lp = symmetric_loss(Wi, Wt, X, T, tau, False)
                    W[idx] = old - 1e-6
                    lm = symmetric_loss(Wi, Wt, X, T, tau, False)
                    W[idx] = old
                    num.append(abs((lp - lm) / 2e-6 - g[idx]))
            print(f"max |analytic - finite-difference| over 40 weights: {max(num):.1e}")
            print(f"loss at step 1: {loss:.3f}")
        for j, (W, g) in enumerate(((Wi, gWi), (Wt, gWt))):     # Adam
            m[j] = 0.9 * m[j] + 0.1 * g
            v[j] = 0.999 * v[j] + 0.001 * g**2
            W -= lr * (m[j] / (1 - 0.9**step)) / (np.sqrt(v[j] / (1 - 0.999**step)) + 1e-8)
    return Wi, Wt, loss, rng

def zero_shot(Wi, Wt, rng, n=50):
    """Classify fresh images of all 9 combinations against 9 caption prompts."""
    prompts = [(c, s) for c in COLORS for s in SHAPES]
    V = np.stack([caption_vector(f"a {c} {s}") for c, s in prompts]) @ Wt
    V /= np.linalg.norm(V, axis=1, keepdims=True)
    acc = {}
    for k, (c, s) in enumerate(prompts):
        X = np.stack([draw(c, s, rng).ravel() for _ in range(n)])
        U = X @ Wi
        U /= np.linalg.norm(U, axis=1, keepdims=True)
        acc[(c, s)] = ((U @ V.T).argmax(1) == k).mean()
    return acc

if __name__ == "__main__":
    seeds = [int(a) for a in sys.argv[1:]] or [4]
    for seed in seeds:
        Wi, Wt, loss, rng = train(seed, check=(len(seeds) == 1))
        acc = zero_shot(Wi, Wt, rng)
        seen = np.mean([a for p, a in acc.items() if p not in HELD_OUT])
        unseen = np.mean([acc[p] for p in sorted(HELD_OUT)])
        if len(seeds) == 1:
            print(f"loss after 600 steps: {loss:.3f} (ln 7 = {np.log(7):.3f} for a uniform guess)")
            print("zero-shot accuracy on 50 new images per caption (9-way):")
            for (c, s), a in acc.items():
                tag = "  <- never seen in training" if (c, s) in HELD_OUT else ""
                print(f"  a {c:5s} {s:8s} {a:.2f}{tag}")
        print(f"seed {seed}: seen combinations {seen:.3f}, unseen combinations {unseen:.3f}")

max |analytic - finite-difference| over 40 weights: 4.0e-10
loss at step 1: 1.893
loss after 600 steps: 0.177 (ln 7 = 1.946 for a uniform guess)
zero-shot accuracy on 50 new images per caption (9-way):
  a red   circle   1.00
  a red   square   1.00  <- never seen in training
  a red   triangle 1.00
  a green circle   1.00
  a green square   0.98
  a green triangle 1.00
  a blue  circle   1.00
  a blue  square   1.00
  a blue  triangle 0.76  <- never seen in training
seed 4: seen combinations 0.997, unseen combinations 0.880

The loss falls from about $\ln 7$ to 0.18 and the seen combinations are classified almost perfectly; the held-out combinations score 100% and 76%, against 11% for a random guess. Our explanation of why the transfer works: the text encoder is additive, so “a blue triangle” embeds as a sum of word vectors each trained in other combinations; the image encoder is linear too, and because every color here has some intensity in all three channels, a triangle contributes a shared signal whatever its color. Over seeds 0–9, seen-combination accuracy was 99.1–100% and unseen-combination accuracy 74–98% (mean 89%); 112 of the 114 errors on unseen images kept the color and confused the shape, most often naming the blue triangle a blue square. With shifts of up to ±2 pixels instead of ±1, unseen accuracy dropped to 45–75%; by our reasoning, a linear pixel encoder handles position changes poorly, one reason real systems use convolutional or transformer encoders.

By our reasoning the example also shows a limit: a bag-of-words encoder gives “a red circle and a blue square” and “a blue circle and a red square” the same embedding, so it cannot record which attribute belongs to which object.

Challenges

• Noisy alignment in web data. Uncurated web text may describe something absent from its image, or nothing visual at all.
• Modality imbalance. A model can lean on one modality and ignore the other. Liang et al. (2024, arXiv text) cite biases in the language side of VQA that let models ignore the image, which led to curated benchmarks such as VQA 2.0, and work showing that modalities overfit and generalize at different rates.
• Hallucination. Rohrbach et al. (2018) measured object hallucination, mentioning objects absent from the image, in COCO captioning models: the best models on standard sentence metrics did not always hallucinate least, and models that hallucinated more tended to make errors driven by language priors. Multimodal Hallucination covers the problem in VLMs.
• Bias. Social biases in web text and images pass into the models. CLIP’s authors say their unfiltered, uncurated pairs led CLIP to learn many social biases (Radford et al., 2021, arXiv version); Liang et al. (2024) survey compounding social biases in multimodal systems.
• Evaluation. Open-ended outputs have many correct forms, n-gram metrics may not capture hallucinated objects (Rohrbach et al., 2018), and a test set may overlap the web data a model was pretrained on; over 35 datasets CLIP’s authors found a median overlap of 2.2%, rarely shifting accuracy by more than 0.1% (Radford et al., 2021).

Common Misconceptions

• “Multimodal means concatenating features.” Concatenation is just early fusion; coordinated embeddings, cross-attention and translation are equally multimodal.
• “A VLM understands the image the way it understands text.” In the VLMs above, the image enters as features from a separately pretrained vision encoder; by our reasoning, detail that the encoder discards cannot be recovered by the language model, however fluent its answer.

Where It Is Used

Vision–language models answer questions about images, search image collections by text, label images for open-vocabulary image classification and detection, and describe images for blind and low-vision users, a motivation named by both Show and Tell and VQA. The same ideas combine video with audio, color with depth, and camera with LiDAR. Next: CLIP, Multimodal Fusion and Vision–Language Model; their recipes build on self-supervised learning and are used through transfer learning.