Multimodal Fusion

Multimodal fusion is the part of a multimodal model where information from different modalities is combined to make one prediction: an image and a question become an answer, a color image and a depth map become a segmentation. A fusion scheme is defined by where the streams meet, from raw input to final scores, and how they are combined, from concatenation to attention. A dual encoder such as CLIP is the limiting case with no fusion at all (see the misconceptions below).

Definition

Baltrušaitis, Ahuja and Morency (2019) define fusion as joining information from several modalities to predict a class or a continuous value. They divide model-agnostic approaches into early (feature-based) fusion, which integrates features right after extraction, often by concatenation; late (decision-based) fusion, which combines unimodal decisions by averaging, voting, weighting or a learned model; and hybrid fusion, which combines early fusion with unimodal predictors. Deep networks add intermediate (or mid) fusion: each modality is first processed by its own layers, and the streams merge inside the network.

Intuition

Suppose a model must answer “is it red?” or “is it a circle?” about a picture of one colored shape. Neither input alone determines the answer: an image-only model can recognize the color perfectly and still be at chance, because by construction a red picture is paired with a “no” question as often as with a “yes” one. The answer depends on an interaction between the image’s attributes and what the question asks. Late fusion cannot compute it, because it combines two opinions each formed without seeing the other input; every fusion method below is a different place and form for the interaction.

Formal Definition

Let $x_1, x_2$ be two inputs, $\phi_1, \phi_2$ unimodal encoders and $y$ the target.

Where. Early fusion applies one network to the joined inputs or low-level features, $\hat{y} = f([x_1; x_2])$. Late fusion combines unimodal predictions $p_m = p(y \mid x_m)$, for example by averaging, $\tfrac{1}{2}(p_1 + p_2)$, by summing logits, or by multiplying probabilities and renormalizing (a product of experts, $p \propto p_1 p_2$). Intermediate fusion combines features, $\hat{y} = f\big(g(\phi_1(x_1), \phi_2(x_2))\big)$, with a fusion operator $g$, possibly at several depths.

How. For an image vector $v \in \mathbb{R}^{d_v}$ and a question vector $q \in \mathbb{R}^{d_q}$:

• Concatenation: $g(v, q) = [v; q]$, usually followed by an MLP.
• Element-wise sum or product: $W_v v + W_q q$ or $W_v v \odot W_q q$, after projecting both to a common width.
• Bilinear pooling: every pair of components interacts, $z_k = v^\top W_k q + b_k$ for $k = 1, \dots, d_z$, with $d_v d_q d_z$ weights.
• Feature-wise modulation (FiLM): one modality rescales and shifts the feature maps of the other, $\mathrm{FiLM}(F_c) = \gamma_c(q)\, F_c + \beta_c(q)$ for each channel $c$ (Perez et al., 2018).
• Cross-attention: with $H_t$ the $n_t$ text tokens and $H_v$ the $n_v$ image tokens, queries come from one modality and keys and values from the other, $\operatorname{softmax}\big((H_t W_Q)(H_v W_K)^\top / \sqrt{d_k}\big)\, H_v W_V$. It is the self-attention formula with two different token sets, and the attention itself costs $O(n_t n_v d)$.

Properties

Bilinear pooling and its approximations. Concatenation followed by a linear layer is additive in $v$ and $q$; multiplicative interactions appear only through later nonlinearities. Fukui et al. (2016) argued that an outer product is more expressive but infeasible: with $d_v = d_q = 2048$ and 3,000 answer classes, $W$ would have 12.5 billion parameters. Their Multimodal Compact Bilinear pooling (MCB) approximates it by convolving random Count Sketches of the two vectors, computed as an element-wise product in FFT space. On VQA test-dev (trained on the train split), concatenation scored 57.49%, element-wise product 58.57% and MCB with a 16,000-dimensional output 59.83%; concatenation followed by two 4,096-unit layers, with about 46 million parameters to MCB’s 48 million, reached only 57.10%. Kim et al. (2017) replaced the sketch with a low-rank bilinear model (MLB), $z = P^\top\big(\sigma(U^\top v) \odot \sigma(V^\top q)\big) + b$, which constrains the bilinear weights to low rank; their attention model scored 65.08% on VQA test-dev with 51.9M parameters, against 64.2% for MCB with attention at 69.2M as listed in their table. MUTAN (Ben-younes et al., 2017) parametrizes the bilinear tensor by a Tucker decomposition and shows MCB and MLB as special cases.

FiLM costs two numbers per modulated feature map and does not scale with image resolution. In the FiLM model a GRU reads the question and predicts $(\gamma, \beta)$ for each of four residual blocks of a CNN over the image; it reached 97.7% on CLEVR, against 92.6% for humans, and roughly halved the error of the best previous method that used no extra supervision (program labels), from 4.5% to 2.3% (Perez et al., 2018).

Co-attention and cross-attention. Lu, Yang, Batra and Parikh (2016) argued that VQA models should decide which words to listen to as well as where to look. Their co-attention computes an affinity matrix $C = \tanh(Q^\top W_b V)$ between question and image features and attends over both, either in parallel or alternately, at word, phrase and question levels. In transformers the idea becomes cross-attention, which is directional: text queries over image keys bring image information to each word, and the reverse needs a separate layer. The transformer decoder’s encoder–decoder attention is cross-attention; DETR applies it from learned object queries to image features, and SAM’s mask decoder runs it both ways between prompt tokens and the image embedding.

Single-stream and dual-stream transformers. A single-stream model concatenates image tokens and word tokens and runs one transformer over the joint sequence, so every layer fuses through self-attention; UNITER feeds region features and WordPieces to one transformer (Chen et al., 2020). A dual-stream model keeps a transformer per modality and exchanges information through cross-attention. In ViLBERT’s co-attentional layers each stream computes queries, keys and values as usual, but the keys and values are passed to the other stream’s attention block (Lu et al., 2019). Its authors argued that separate streams allow different depths per modality and protect the pretrained language model, and ViLBERT beat their single-stream baseline, both pretrained on Conceptual Captions, 70.55% against 68.85% on VQA test-dev. UNITER’s authors later reported state-of-the-art results with a single stream and far fewer parameters (86M for UNITER-base against 221M for ViLBERT). The pretraining recipes of these models are covered in vision–language pretraining.

Bottleneck tokens. The cost of self-attention over both modalities grows as $(n_1 + n_2)^2$ per layer. Nagrani et al. (2021) route all cross-modal flow in an audio–visual transformer through $B$ shared bottleneck tokens ($B = 4$ by default) that each modality’s layers read and write. On AudioSet (trained on a 20,361-clip balanced subset), mid fusion beat both early fusion ($L_f = 0$) and late fusion ($L_f = 12$ of 12 layers), with bottlenecks best at $L_f = 8$. With fusion from the first layer, bottlenecks beat full cross-modal attention by over 2 mAP at less than half the compute.

Gated cross-attention and resampling. Flamingo inserts new cross-attention layers, with text queries and keys and values from visual features, between the frozen layers of a pretrained language model. Each new layer’s output is multiplied by $\tanh(\alpha)$ before the residual addition, with a learnable scalar $\alpha$ initialized to 0, so the model starts out as the unchanged language model; removing the gating cost 4.2% of the authors’ overall score and made training unstable. To keep cross-attention cheap, a Perceiver Resampler first reduces the visual features to 64 tokens by letting learned latent queries cross-attend to them (Alayrac et al., 2022); BLIP-2’s Q-Former uses 32 learned queries (Li et al., 2023).

Missing modalities and modality imbalance. Late fusion can still predict when a modality is missing, and can even be trained without paired data, but it ignores low-level interactions (Baltrušaitis et al., 2019). Early and intermediate fusion, as usually built, expect every input at test time. More inputs can even make a model worse: Wang, Tran and Feiszli (2020) found that the best unimodal network often beats the multimodal one. On Kinetics, an audio + RGB network fused late by concatenation reached 71.4% video top-1 validation accuracy against 72.6% for RGB alone. They attribute the drop to overfitting from extra capacity and to modalities that overfit and generalize at different rates, and propose Gradient-Blending, which weights each modality’s loss by its overfitting behavior.

Examples

RGB-D segmentation. Depth is a natural second modality for indoor semantic segmentation. In the journal version of FCN, on NYUDv2 test with FCN-32s, adding depth as a fourth input channel (early fusion) barely changed mean IU (31.7% against 31.6% for RGB), while late fusion of an RGB stream and a stream on the HHA depth encoding reached 33.3% (Shelhamer et al., 2017). FuseNet keeps separate RGB and depth encoders and adds the depth features into the RGB branch by element-wise summation at several depths. On SUN RGB-D, stacked four-channel input gave 31.95% IoU against 32.47% for RGB alone, while FuseNet-SF5 reached 37.29% (Hazirbas et al., 2017).

A question about a shape. The script below tests the intuition. Each $16 \times 16$ RGB image shows one red, green or blue circle, square or triangle at a random position and size, with noise; each question is a one-hot vector over six invented questions (“is it red?” … “is it a triangle?”). Half the answers are yes, and the image alone or the question alone predicts them at chance. Four models share the same two-layer convolutional encoder, which yields an $8 \times 8$ grid of 32-dimensional features:

• late fusion: the sum of an image-only logit and a question-only logit;
• concatenation: max-pooled image features concatenated with the question and passed through an MLP;
• FiLM: the question predicts $\gamma$ and $\beta$ for the encoder’s 32 channels;
• cross-attention: the question supplies one query per head (four heads) over the 64 image tokens, which carry learned position embeddings.

Each model trains for 1,000 Adam steps on fresh batches of 32 with binary cross-entropy, with gradients from PyTorch autograd, and is tested on 2,000 new pairs. The script also trains the seed-0 cross-attention model for 4,000 steps. It takes about 40 seconds on one CPU thread with PyTorch 2.14.1 and gave identical output on two runs.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
S = 16
COLORS = {"red": (1.0, 0.2, 0.2), "green": (0.2, 1.0, 0.2), "blue": (0.2, 0.2, 1.0)}
SHAPES = ("circle", "square", "triangle")
QUESTIONS = list(COLORS) + list(SHAPES)        # "is it red?" ... "is it a triangle?"
yy, xx = np.mgrid[0:S, 0:S] + 0.5

RGB = np.array(list(COLORS.values()))

def make_batch(n, rng):
    """n images of one colored shape at a random position, each with a question
    whose answer is yes or no with probability 1/2."""
    q = rng.integers(6, size=n)                     # 0-2 ask about color, 3-5 about shape
    yes = rng.random(n) < 0.5
    color, shape = rng.integers(3, size=n), rng.integers(3, size=n)
    other = (q % 3 + rng.integers(1, 3, size=n)) % 3        # an attribute value != q's
    asked = np.where(yes, q % 3, other)
    color = np.where(q < 3, asked, color)
    shape = np.where(q >= 3, asked, shape)
    cy, cx = rng.uniform(6, 10, size=(2, n, 1, 1))
    r = rng.uniform(4.5, 6.0, size=(n, 1, 1))
    y, x = yy - cy, xx - cx
    masks = np.stack([x**2 + y**2 < r**2,                                   # circle
                      (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r),             # square
                      (y < 0.8 * r) & (y > -r + 2 * abs(x))])                # triangle
    m = masks[shape, np.arange(n)]                                          # (n, 16, 16)
    tint = RGB[color] * rng.uniform(0.7, 1.0, size=(n, 1))
    img = 0.1 + m[:, None] * tint[:, :, None, None] + 0.05 * rng.standard_normal((n, 3, S, S))
    return (torch.tensor(img, dtype=torch.float32),
            F.one_hot(torch.tensor(q), 6).float(), torch.tensor(yes, dtype=torch.float32))

class Encoder(nn.Module):
    """Shared image encoder: 16x16x3 image -> 8x8 grid of 32-d features (64 tokens)."""
    def __init__(self):
        super().__init__()
        self.c1 = nn.Conv2d(3, 16, 3, padding=1)
        self.c2 = nn.Conv2d(16, 32, 3, stride=2, padding=1)
    def forward(self, x, film=None):
        h = self.c2(F.relu(self.c1(x)))
        if film is not None:                    # FiLM: gamma * h + beta, per channel
            gamma, beta = film
            h = gamma[:, :, None, None] * h + beta[:, :, None, None]
        return F.relu(h)

class Late(nn.Module):                          # score-level fusion: logit_I + logit_Q
    def __init__(self):
        super().__init__()
        self.enc, self.img_head, self.q_head = Encoder(), nn.Linear(32, 1), nn.Linear(6, 1)
    def forward(self, x, q):
        return (self.img_head(self.enc(x).amax((2, 3))) + self.q_head(q)).squeeze(1)

class Concat(nn.Module):                        # feature-level fusion: MLP([v; q])
    def __init__(self):
        super().__init__()
        self.enc = Encoder()
        self.mlp = nn.Sequential(nn.Linear(32 + 6, 64), nn.ReLU(), nn.Linear(64, 1))
    def forward(self, x, q):
        return self.mlp(torch.cat([self.enc(x).amax((2, 3)), q], 1)).squeeze(1)

class FiLM(nn.Module):                          # the question modulates image features
    def __init__(self):
        super().__init__()
        self.enc, self.gen, self.head = Encoder(), nn.Linear(6, 64), nn.Linear(32, 1)
    def forward(self, x, q):
        g = self.gen(q)
        film = (1 + g[:, :32], g[:, 32:])       # gamma starts near 1, beta near 0
        return self.head(self.enc(x, film).amax((2, 3))).squeeze(1)

class CrossAttn(nn.Module):                     # question queries, image keys and values
    def __init__(self, d=32, heads=4):
        super().__init__()
        self.enc, self.pos = Encoder(), nn.Parameter(0.02 * torch.randn(64, 32))
        self.wq, self.wk, self.wv = nn.Linear(6, d), nn.Linear(32, d), nn.Linear(32, d)
        self.mlp = nn.Sequential(nn.Linear(d + 6, 64), nn.ReLU(), nn.Linear(64, 1))
        self.h, self.dh = heads, d // heads
    def forward(self, x, q):
        B = len(x)
        tokens = self.enc(x).flatten(2).transpose(1, 2) + self.pos   # (B, 64, 32)
        Qh = self.wq(q).view(B, self.h, 1, self.dh)                  # one query per head
        K = self.wk(tokens).view(B, 64, self.h, self.dh).transpose(1, 2)
        V = self.wv(tokens).view(B, 64, self.h, self.dh).transpose(1, 2)
        A = torch.softmax(Qh @ K.transpose(2, 3) / self.dh**0.5, -1)  # (B, h, 1, 64)
        out = (A @ V).reshape(B, -1)                                  # concatenate heads
        return self.mlp(torch.cat([out, q], 1)).squeeze(1)

def run(model_cls, seed, steps=1000, batch=32):
    torch.manual_seed(seed)
    rng = np.random.default_rng(seed)
    model = model_cls()
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    for _ in range(steps):
        X, Q, Y = make_batch(batch, rng)
        loss = F.binary_cross_entropy_with_logits(model(X, Q), Y)
        opt.zero_grad(); loss.backward(); opt.step()
    X, Q, Y = make_batch(2000, np.random.default_rng(1000 + seed))   # fresh test images
    with torch.no_grad():
        correct = ((model(X, Q) > 0).float() == Y).numpy()
    is_color_q = (Q.argmax(1) < 3).numpy()
    return correct.mean(), correct[is_color_q].mean(), correct[~is_color_q].mean()

if __name__ == "__main__":
    seeds = [int(a) for a in sys.argv[1:]] or [0, 1, 2]
    models = {"late (sum of logits)": Late, "concatenation + MLP": Concat,
              "FiLM": FiLM, "cross-attention": CrossAttn}
    print(f"torch {torch.__version__}; accuracy on 2,000 new image-question pairs")
    print(f"{'':21s} {'seeds ' + ','.join(map(str, seeds)):20s}  color Qs  shape Qs")
    for name, cls in models.items():
        r = np.array([run(cls, s) for s in seeds])
        print(f"{name:21s} " + " ".join(f"{a:.3f}" for a in r[:, 0])
              + f"   {r[:, 1].mean():.3f}     {r[:, 2].mean():.3f}")
    print(f"cross-attention, seed 0, 4,000 steps: {run(CrossAttn, 0, steps=4000)[0]:.4f}")

torch 2.14.1; accuracy on 2,000 new image-question pairs
                      seeds 0,1,2           color Qs  shape Qs
late (sum of logits)  0.519 0.559 0.477   0.534     0.502
concatenation + MLP   0.997 0.935 0.995   1.000     0.950
FiLM                  0.930 0.994 0.986   1.000     0.939
cross-attention       0.915 0.917 0.890   1.000     0.811
cross-attention, seed 0, 4,000 steps: 0.9995

Late fusion stays near chance (48–56%), although its encoder is the same as the others’. In our explanation, a sum $s_I(x) + s_Q(q)$ can only shift one image score by a per-question offset, which cannot express “the asked attribute matches”; and because each input alone is uninformative by construction, training finds no unimodal signal to follow. (By our count, the best such sum could still reach about 71% on this data.) All three fusing models answer every color question correctly; their errors are on shape questions, which we take to demand more of the encoder. Concatenation reached 93.5–99.7% and FiLM 93.0–99.4% across seeds; cross-attention reached 89.0–91.7%, learning the shape questions more slowly: with 4,000 steps instead of 1,000, the seed-0 model reached 99.95%. On a problem this small, fusing at all matters far more than the choice of operator; the operators are told apart by large-scale comparisons such as those above.

Strategies at a Glance

Strategy	Where the streams meet	Operator	Example models
Early fusion	input or low-level features	channel stacking, concatenation	FCN RGB-D four-channel input
Late fusion	scores or decisions	averaging, voting, product of experts, logit sum, concatenation at the last layer	FCN RGB + HHA; late-fusion video networks (Wang et al.)
Intermediate fusion	inside the network, once or at several depths	sum, concatenation + MLP	FuseNet
Multiplicative pooling	pooled feature vectors	bilinear (MCB, MLB, MUTAN)	VQA models of 2016–2017
Feature modulation	feature maps of one stream	FiLM, $\gamma \odot F + \beta$	FiLM on CLEVR
Co-attention	between question and image features	affinity matrix, attention both ways	Hierarchical co-attention
Single-stream	every transformer layer	self-attention over joint tokens	UNITER
Dual-stream	selected layers	co-attentional / cross-attention	ViLBERT
Bottleneck	later layers, through $B$ tokens	shared latent tokens	attention bottlenecks (audio–visual)
Gated cross-attention	inserted into a frozen language model	tanh-gated cross-attention, resampler	Flamingo; BLIP-2 Q-Former
No fusion	similarity only	dot product of embeddings	CLIP, ALIGN

Common Misconceptions

• “Late fusion is always worse.” On NYUDv2 the FCN authors’ late-fused RGB + HHA model beat four-channel early fusion, and late fusion tolerates missing modalities. It fails when the answer depends on an interaction between inputs, as in the example.
• “Adding a modality cannot hurt.” Wang et al. (2020) measured multimodal networks below their best unimodal counterparts and traced the gap to overfitting and to modalities that learn at different rates.
• “A dual encoder fuses the modalities.” CLIP’s image and text never interact before the final dot product. That is what makes retrieval over large collections cheap (see image–text retrieval), and why such models need a fusion module for tasks like visual question answering.

Where It Is Used

Fusion is central to visual question answering, visual grounding, captioning decoders, and vision–language models that feed visual tokens to a language model or let it cross-attend to them. Outside language, it combines color with depth for indoor segmentation, camera images with LiDAR point clouds, and video frames with audio and optical flow for action recognition; promptable models such as SAM fuse points, boxes and masks with image features.