Prompt Engineering

Prompt engineering is the practice of adapting a pretrained model to a task by changing what goes into it rather than its weights. For a contrastive image–text model such as CLIP, the prompt is the text that turns a class name into a classifier, for example “a photo of a {label}.”; for a generative vision–language model, it is the instruction, the worked examples and the requested answer format placed before the question; for a text-to-image model, it is the description of the picture. CLIP’s authors used the term by analogy with the discussion of prompts for GPT-3 (Radford et al., 2021, ICML supplement, Appendix B). Two neighbouring ideas share the word. Prompt learning or prompt tuning replaces hand-written words, or adds input tokens, with vectors fitted by gradient descent while the model stays frozen; visual prompting edits the image itself, for instance by drawing a circle around the region a question is about. This article covers all three for vision; the prompts of promptable segmenters such as SAM, points, boxes and masks, are a different interface described with that model.

Purpose

A model trained on image–caption pairs rarely sees a class label on its own, a list of class names or a benchmark’s answer format. CLIP’s authors named the two gaps that prompts close in zero-shot classification: class names are often ambiguous (ImageNet contains both construction cranes and the birds; in Oxford-IIIT Pets, “boxer” is a dog breed that a text encoder could read as an athlete), and the text paired with images during training is rarely a single word (Radford et al., 2021, ICML supplement, Appendix B). A prompt supplies the missing context in the form the model learned from. For generative models the purpose widens: the prompt specifies the task itself, so that one frozen model can caption, answer questions or classify depending on what precedes the query (Flamingo was designed around this).

When to Use It

Prompting is the cheapest adaptation there is: it needs no gradients, no labelled data beyond a few examples, and one stored model serves every task. It is the first thing to try when the classes or the task can be stated in words the model has seen, when labels are scarce, and when the same backbone must serve many tasks. The documented gains are a few points on ImageNet for CLIP and larger on some specialized sets, such as textures in CuPL’s experiments (below).

It helps less when the gap is in the image rather than the text. In our reading, a prompt cannot make a model see what its encoder does not represent: CLIP’s zero-shot accuracy was weak on counting, satellite images and lymph-node tumours although each of these datasets had its own template in the released prompt file, and a linear probe on CLIP’s features matched zero-shot CLIP with a median of 5.4 labelled examples per class (both summarized in CLIP and the CLIP paper article). With labels in hand, a probe or fine-tuning is the stronger option. Learned prompts sit between the two: they need labelled examples and gradients through the frozen model but train only a few thousand parameters.

How It Works

Templates as classifier weights

A contrastive model has an image encoder $f$ and a text encoder $g$. Zero-shot classification over $K$ classes fills a template with each class name, $t_k$ = “a photo of a {$k$}.”, and scores an image $x$ with

$$p(y = k \mid x) = \frac{\exp\big(\cos(f(x), g(t_k)) / \tau\big)}{\sum_{j=1}^{K} \exp\big(\cos(f(x), g(t_j)) / \tau\big)},$$

where $\tau$ is the temperature learned in pretraining (CoCoOp’s review of CLIP writes it this way: Zhou et al., 2022, Eq. 1). The normalized text embeddings $g(t_k)$ are the weights of a linear classifier, so changing the template changes the classifier without touching the model. In CLIP’s experiments the default template “A photo of a {label}.” added 1.3 points on ImageNet over the bare label; task-specific context helped further on fine-grained sets (“A photo of a {label}, a type of pet.” for Oxford-IIIT Pets, “a type of food” for Food101, “a type of aircraft” for FGVC Aircraft), quotes around the text helped OCR datasets, and “a satellite photo of a {label}.” helped satellite sets (ICML supplement, Appendix B). The templates actually used are released for 26 of the 27 datasets (data/prompts.md): EuroSAT uses three variants of “a centered satellite photo of {}.”, MNIST ‘a photo of the number: “{}”.’, CLEVRCounts “a photo of {} objects.” and the PatchCamelyon histology set “this is a photo of {}”.

The class names are part of the prompt too. The official ImageNet notebook describes names edited by trial and error on the ImageNet training set, concentrating on the classes with the lowest top-1 and top-5 accuracy for three ResNet models: “nail” became “metal nail” because CLIP read it as a fingernail, and “kite” became “kite (bird of prey)”. The notebook reports that these edits added 1.5 points of top-1 for ViT-B/32 and that the list was never finished (notebook, commit d05afc4).

Ensembling

Averaging over several templates usually beats any one of them. CLIP builds the ensemble in embedding space,

$$\mathbf{w}_k = \frac{\bar{\mathbf{w}}_k}{\lVert \bar{\mathbf{w}}_k \rVert}, \qquad \bar{\mathbf{w}}_k = \frac{1}{M} \sum_{m=1}^{M} \frac{g(t_{m,k})}{\lVert g(t_{m,k}) \rVert},$$

rather than by averaging $M$ probability vectors, so that a single set of averaged class embeddings is cached and the ensemble costs no more at test time than one template (ICML supplement, Appendix B; the notebook’s zeroshot_classifier implements exactly this). On ImageNet, 80 templates added 3.5 points over the default prompt and prompt engineering with ensembling almost 5; averaged over 36 datasets the gain was also almost 5 points, which the authors liken to the gain from four times more compute (ICML supplement, Figure 13). The CLIP article’s example shows the averaging mechanism on synthetic embeddings.

The 80 ImageNet templates were not derived from a principle. The notebook describes them as gathered by intuition-guided trial and error on the ImageNet training set over about a year. Templates written to help ImageNet-R (renditions such as “a sketch of a {}.”, “the origami {}.”, “a {} in a video game.”) also raised standard ImageNet accuracy. After the 80 were fixed, sequential forward selection stopped at a subset of seven (“itap of a {}.”, “a bad photo of the {}.”, “a origami {}.”, “a photo of the large {}.”, “a {} in a video game.”, “art of the {}.”, “a photo of the small {}.”), which the notebook says performs slightly better than the full 80, especially for the smaller models.

Descriptions written by a language model

Hand-written templates must fit every class of a dataset, so they cannot say what distinguishes one class. CuPL (Customized Prompts via Language models; Pratt et al., 2023) asks a language model to describe each class and uses the descriptions as the prompts. For ImageNet, five hand-written “LLM-prompts” to GPT-3 (such as “What does a(n) {} look like?” and “Describe what a(n) {} looks like”) each generated 10 sentences per class at temperature 0.99, giving 50 image prompts per class, which are embedded and averaged as in CLIP’s ensemble (§2.1–2.2). With CLIP ViT-L/14 and GPT-3 davinci-002, the full setting raised top-1 accuracy over CLIP’s own prompts on all 15 datasets tested, from 75.54% to 76.69% on ImageNet and from 55.20% to 61.70% on the Describable Textures Dataset; a base setting using the same three LLM-prompt templates for all 15 datasets improved 13 of them and lowered Stanford Cars (−1.04) and RESISC45 (−2.14) (Table 1). The question still has to name the domain where words are ambiguous: the authors write “Describe a pet boxer:” for Oxford Pets, and “Tornado” in FGVC Aircraft is otherwise also weather (§2.2). Shapley-value maps suggested that the descriptions shift the model’s weight to the parts they mention, such as the tail of a Schipperke (§3.4).

Menon and Vondrick (2023) had a language model generate a structured list of attributes per class instead, reformatted into captions for CLIP, as CuPL’s authors describe it (§4.2). Their paper was not read for this article; in CuPL’s comparison with ViT-L/14 their descriptors scored 75.00% on ImageNet against 75.54% for CLIP’s prompts and 76.69% for CuPL (Pratt et al., 2023, Table 2).

Learned context: CoOp and CoCoOp

CoOp (Context Optimization; Zhou et al., IJCV 2022) replaces the words “a photo of a” with $M$ continuous vectors in the text encoder’s word-embedding space, shared by all classes:

$$t_k = [\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_M, \mathbf{c}_k],$$

where $\mathbf{c}_k$ is the embedding of the class name. The vectors are fitted with cross-entropy on a few labelled images per class, with gradients passing through the frozen text encoder; image and text encoders are not updated (CoCoOp paper, §3.1). The IJCV paper was not read here; this description follows CoCoOp’s review of it and the official code at commit ff61507. The code trains only the prompt_learner, initializes the context either from the embeddings of given words or randomly from a normal distribution with standard deviation 0.02, offers a class-specific context (CSC, one set of vectors per class) and places the class token at the end, middle or front of the prompt; the ResNet-50 configuration that COOP.md pairs with 16 context vectors for 8 and 16 shots (rn50.yaml) uses SGD with learning rate 0.002 and a cosine schedule for 200 epochs, while ImageNet uses 50-epoch configurations. CoCoOp’s abstract summarizes CoOp’s result as large improvements over intensively tuned manual prompts from only a few labelled images.

The base-to-new trade-off. Zhou et al. (CVPR 2022) found that CoOp’s context does not carry over to unseen classes of the same dataset. They split each of 11 datasets’ classes into equal base and new halves, trained on 16 shots per base class with CLIP ViT-B/16, a context of 4 vectors initialized from “a photo of a”, and scored base and new classes separately, with the harmonic mean $H$ to summarize both (§4). Averaged over the 11 datasets (Table 1a; results averaged over three runs):

Prompt	Base	New	H
CLIP, hand-written prompts	69.34	74.22	71.70
CoOp	82.69	63.22	71.66
CoCoOp	80.47	71.69	75.83

CoOp gained over 13 points on base classes and lost 11 on new ones, ending level with CLIP on $H$. The largest new-class losses were on UCF101, where new accuracy fell from 77.50% to 56.05%; on DTD, from 59.90% to 41.18%; and on Flowers102, from 77.80% to 59.67% (Table 1). The authors read this as the static context overfitting the base classes. The comparison carries a caveat the authors state themselves: CLIP’s manual prompts had been tuned using all classes in the test data (§4).

CoCoOp (Conditional Context Optimization) makes the context depend on the image. A small Meta-Net, Linear–ReLU–Linear with a 16-fold bottleneck, maps the image feature to one vector $\boldsymbol{\pi} = h_\theta(f(x))$ added to every context vector, $\mathbf{v}_m(x) = \mathbf{v}_m + \boldsymbol{\pi}$ (§3.2). It raised average new-class accuracy to 71.69% at a cost of 2.2 points on base classes, beat CoOp on new classes on all 11 datasets, and exceeded CLIP’s new-class accuracy on 4 of them (ImageNet, OxfordPets, Food101, SUN397) while still trailing it on 7 (§4.1, §5). A CoOp with 60 context vectors, matching CoCoOp’s parameter count, did worse than the 4-vector CoOp on ImageNet, so the gain does not come from size (Table 5). Initializing from “a photo of a” beat random initialization on both base and new classes, and longer contexts helped new classes when initialized randomly (Fig. 4). Two further settings favoured CoCoOp more mildly: context learned on ImageNet and transferred to 10 other datasets averaged 65.74% against CoOp’s 63.88% (Table 2), and on four ImageNet variants with domain shift both learned prompts beat CLIP, with CoCoOp ahead of CoOp on three (Table 3). The price is training cost: each image needs its own pass of the prompts through the text encoder, so CoCoOp was trained with batch size 1 for 10 epochs (§4, §5).

Visual prompts

Visual prompt tuning (VPT; Jia et al., 2022) is the image-side analogue: learnable tokens are prepended to the input of a frozen Vision Transformer, at the first layer only (VPT-shallow) or at every layer (VPT-deep), and trained with the task head. For ViT-B/16, 50 prompts add 0.04% (shallow) or 0.53% (deep) of the backbone’s parameters (§3.2). With an ImageNet-21k supervised ViT-B/16, VPT-deep beat full fine-tuning on 20 of 24 tasks, 5 fine-grained classification sets and the 19 VTAB-1k tasks (Table 1); transfer learning places it among the other parameter-efficient methods.

Marks drawn on the image need no training at all. Shtedritski et al. (2023) found that drawing a red circle around a region makes CLIP describe that region while keeping the rest of the image as context. They call this visual prompt engineering and distinguish it from visual prompt tuning (§1, footnote 1). Each candidate answer is a marked copy of the image and the text describes the question, so CLIP picks the marked location that best matches, for example, “an image of the right ear of a dog” (§3.1). With CLIP ViT-L/14 at 336 pixels, matching keypoint names to marked locations on CUB reached 46.5% where cropping around the keypoint reached 15.8% (58.2% and 25.5% with Sinkhorn normalization; Table 1). On zero-shot referring expression comprehension, scored as correct at box IoU above 0.5 on MAttNet proposals, red circles reached 49.8% on RefCOCO val, 55.3% on RefCOCO+ val and 59.4% on RefCOCOg val, ahead of ReCLIP with its relation rules on most splits but not on RefCOCO testB (39.9% against 47.1%) or RefCOCOg test (58.9% against 59.0%) (Table 5; the visual grounding article covers these benchmarks). The details of the mark matter. Averaged over the variants tested on CUB (colours and sizes for each shape, sizes for each colour), circles beat arrows, squares and crosses, 33.5% against 21.5% for crosses, and red beat green, purple, blue and yellow (Table 2). The ability grew with the size of the vision encoder and, among CLIP models, with the size of the pretraining set, though OpenCLIP models trained on LAION-2B did worse than OpenAI’s (Table 3). Among 10,000 YFCC15M images most likely to contain annotations, the authors found 70 with drawn marks, about 0.001% of the data by their estimate, which they take as the origin of the behaviour (§4.1).

Set-of-Mark prompting (Yang et al., 2023) does the same for generative models: segmentation models such as SAM, SEEM or Semantic-SAM partition the image, and each region receives a mask, box or numeric label that a model such as GPT-4V can name in its answer (official repository README). The released benchmark gives the language prompts used, for example “I have labeled a bright numeric ID at the center for each visual object in the image. Please enumerate their names,” with 100 images each from COCO and ADE20K for open-vocabulary segmentation, Flickr30K for phrase grounding and RefCOCOg for referring comprehension and segmentation (benchmark README). The paper is on arXiv only and was not read, so its results are not reported here.

Prompting generative vision–language models

In-context examples. Flamingo poses every task as completing an interleaved sequence of support examples, each an image followed by text in a fixed template, then the query image (Alayrac et al., 2022, §2.5, Appendix A.2); the Flamingo article gives its templates, its zero-shot format, its retrieval and ensembling of examples, and its single set of evaluation prompts for all benchmarks. The choice and order of examples matter. With 16 shots on ImageNet, Flamingo-80B scored 66.4% with random support examples, 76.0% when the 16 were retrieved from 5,000 by similarity to the query image (RICES) and 77.3% when log-likelihoods were also averaged over 6 orderings, still below the 77.9% of its own contrastive encoder zero-shot (supplement, Table 7). The authors list sensitivity to the order and format of examples among in-context learning’s limits (Appendix D.1).

Chain-of-thought. Wei et al. (2022) showed that adding worked reasoning steps to the few-shot examples of a language model raised its accuracy on arithmetic, commonsense and symbolic tasks, and that on arithmetic the gains appeared only with models of about 100 billion parameters; smaller models wrote fluent but illogical chains (§3.2). ScienceQA (Lu et al., 2022) brought the idea to multimodal questions. Its about 21,000 multiple-choice science questions carry a lecture and an explanation for most answers (83.9% have a lecture), split 12,726 / 4,241 / 4,241 (§3). GPT-3 (text-davinci-002) saw images only as captions from a captioning model and was prompted with two training examples whose answers were followed by “BECAUSE:” and the lecture and explanation, so the model generates the answer first and the explanation after it (§4.2, Fig. 5). On the test split (Table 3):

Model and prompt	Average accuracy (%)	With image context (%)
GPT-3, zero-shot	74.04	65.74
GPT-3, 2-shot, answer only	73.97	67.28
GPT-3, 2-shot, answer + lecture + explanation	75.17	67.43
UnifiedQA-base, fine-tuned, answer only	70.12	61.38
UnifiedQA-base, fine-tuned, answer + lecture + explanation	74.11	66.53
Humans	88.40	87.50

Chain-of-thought added 1.20 points to two-shot prompting and 3.99 points to fine-tuning, but the two-shot prompt without explanations was no better than zero-shot. Accuracy peaked at two examples and fell as more were added; prompts with explanations only, without lectures, varied most between runs; and choosing examples from the same topic, category or skill as the question made little difference (§5.4, Fig. 7, Table 5). Giving GPT-3 the gold lecture and explanation in the test prompt raised accuracy to 94.13%, 18.96 points above the chain-of-thought prompt, which the authors treat as an upper bound on what correct explanations could add (Table 6; the abstract calls it an improvement of GPT-3’s few-shot performance). Multimodal-CoT (Zhang et al., 2023) incorporates vision features in two training stages with the same architecture, rationale generation and then answer inference (official repository README); its paper was not read. LLaVA’s ScienceQA experiments with reasoning placed before or after the answer are summarized in the LLaVA paper article, and visual reasoning covers language models prompted to write programs instead of prose.

Format instructions. Instruction-tuned models are also steered by short format prompts appended to the question, such as a request for a single word; how such prompts are built into the training mixture, and what they cost on other benchmarks, is part of visual instruction tuning.

Prompts for text-to-image models

A text-to-image model’s prompt is its only description of the target, and whether the image depicts everything the prompt states is the model’s prompt adherence, whose mechanisms and evaluation are the subject of text conditioning. Prompt engineering for these models mostly rewrites what users type. DALL·E 3’s report rewrites short user prompts into the long, detailed descriptions its model was trained on, with GPT-4 (“upsampling”; summarized in text conditioning). Promptist (Hao et al., 2023) learns the rewriting for Stable Diffusion (v1.4 in the main experiments): GPT-2 is first fine-tuned on 360,000 pairs built from 90,000 human-engineered prompts collected from Lexica, then trained with PPO to maximize a reward that adds the gain in aesthetic score over images generated from the user’s input, from an aesthetic predictor on CLIP features, to a CLIP relevance score between the generated images and the original user input, with a KL penalty to the fine-tuned model (§2–3). The relevance term is $\min(20\,s_{\text{CLIP}} - 5.6,\ 0)$, so it only penalizes images whose CLIP similarity to the user’s text falls below 0.28 (Eq. 3). Typical rewrites append style modifiers: “A rabbit is wearing a space suit” became “A rabbit is wearing a space suit, digital Art, Greg rutkowski, Trending cinematographic artstation” (Table 3). On DiffusionDB prompts the aesthetic score rose from 5.47 for user input to 5.87 for human-engineered prompts and 6.26 for Promptist, while CLIP relevance to the user’s input fell from 0.28 to 0.26 for both (Table 4). Optimizing appearance and staying faithful to the input pull in different directions. Classifier-free guidance strength is the other inference-time knob that sets how strongly a sampler follows the prompt.

How to Apply It

Example: wording, ensembling and learned context on a tiny contrastive model

The script trains a CLIP-like dual encoder from scratch (PyTorch 2.14.1, NumPy 2.5.4, one CPU thread; about 70 seconds for three seeds). No pretrained weights are used. Each $32 \times 32$ image shows one circle, square, triangle, cross or ring in one of five colours, small or large, on a noisy dark background; a class is a colour and a shape (“red ring”), 25 in all. Pretraining captions come from four templates that always mention the size (“a photo of a small red ring”, “the ring is red and small”); the image encoder is three strided convolutions, the text encoder a one-layer transformer with mean pooling, trained with CLIP’s symmetric contrastive loss for 12 epochs on 8,000 images.

Two experiments follow. First, zero-shot classification of 1,000 clean test images with 16 single templates in four groups: the bare class name; training wordings with the size word dropped; training wordings with “small” or “large” filled in; and new orders of known words (“is there a red ring”; one of these four also contains a comma, which no training caption has); then the 16 templates ensembled in embedding space, as CLIP does, and in probability space. Second, a shifted downstream domain (gray gradient background, more noise, washed-out colours) on which a CoOp-style context of four vectors is learned from 4 labelled images of each of the 13 base classes, with frozen encoders, and scored on base and on the 12 new classes, each among its own classes as in CoCoOp’s protocol; both CoOp initializations are printed, random and from the words “a photo of a”. Every colour and every shape occurs in both groups. The script repeats everything for seeds 0, 1 and 2, regenerating the data and retraining, and prints seed 0 with the range over the three seeds.

Checks before the first run confirmed that the planned model learns the base task at this scale and that the shifted labels can be learned from shifted images, although the probe used for this never recognized at least one class. They also replaced the shifted domain first planned with the milder one in the script. The plan, after the second check and before the third, required “hand-template zero-shot accuracy between 0.30 and 0.80 on seed 0 (room to rise and to fall)”; the first domain fell just below that bound. The check scored the hand template among all 25 classes, not among the base or the new classes as the example does. This was settled before any learned-context code had run. The script shown is the one first run, without changes; two runs gave identical output.

"""Prompt templates, ensembling and learned context for a tiny contrastive
image-text model trained from scratch on synthetic shapes (PyTorch, CPU)."""
import math
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
SHAPES = ["circle", "square", "triangle", "cross", "ring"]
COLOURS = {"red": (1, .15, .15), "green": (.15, .9, .15), "blue": (.2, .3, 1),
           "yellow": (1, .9, .1), "magenta": (.95, .15, .9)}
CNAMES = list(COLOURS)
CLASSES = [(c, s) for c in range(5) for s in range(5)]          # 25 "colour shape" classes
BASE = [k for k, (c, s) in enumerate(CLASSES) if (c + s) % 2 == 0]   # 13 base classes
NEW = [k for k, (c, s) in enumerate(CLASSES) if (c + s) % 2 == 1]    # 12 new classes

# ---------------------------------------------------------------- images
def draw(rng, cls, shifted=False):
    c, s = CLASSES[cls]
    large = rng.random() < 0.5
    r = rng.uniform(9, 12) if large else rng.uniform(5, 7)
    cy, cx = rng.uniform(r + 1, 31 - r, size=2)
    yy, xx = np.mgrid[0:32, 0:32].astype(float)
    dy, dx = yy - cy, xx - cx
    d = np.hypot(dx, dy)
    shape = SHAPES[s]
    if shape == "circle":
        m = d <= r
    elif shape == "ring":
        m = (d <= r) & (d >= 0.6 * r)
    elif shape == "square":
        m = np.maximum(abs(dx), abs(dy)) <= 0.8 * r
    elif shape == "triangle":
        m = (dy >= -r) & (dy <= 0.8 * r) & (abs(dx) <= 0.55 * (dy + r))
    else:  # cross
        m = ((abs(dx) <= 0.3 * r) & (abs(dy) <= r)) | ((abs(dy) <= 0.3 * r) & (abs(dx) <= r))
    col = np.clip(np.array(COLOURS[CNAMES[c]]) + rng.uniform(-.1, .1, 3), 0, 1)
    if shifted:   # downstream domain: gray gradient background, more noise, washed-out colour
        a = rng.uniform(0, 2 * np.pi)
        g = (np.cos(a) * xx + np.sin(a) * yy)
        g = 0.3 * (g - g.min()) / (g.max() - g.min())
        img = np.repeat(g[..., None], 3, axis=2)
        col = 0.8 * col + 0.2 * col.mean()
        sd = 0.08
    else:
        img = np.full((32, 32, 3), 0.1)
        sd = 0.05
    img[m] = col
    img = img + rng.normal(0, sd, img.shape)
    return np.clip(img, 0, 1).transpose(2, 0, 1).astype(np.float32), ("large" if large else "small")

def make_set(rng, per_class, classes=range(25), shifted=False):
    X, y, sz = [], [], []
    for k in classes:
        for _ in range(per_class):
            im, size = draw(rng, k, shifted)
            X.append(im); y.append(k); sz.append(size)
    return torch.tensor(np.stack(X)), torch.tensor(y), sz

# ---------------------------------------------------------------- text
TRAIN_T = ["a photo of a {z} {c} {s}", "a {z} {c} {s}",
           "there is a {c} {s} and it is {z}", "the {s} is {c} and {z}"]
WORDS = ["<pad>", "a", "photo", "of", "the", "there", "is", "and", "it", ",",
         "small", "large"] + CNAMES + SHAPES
VOCAB = {w: i for i, w in enumerate(WORDS)}
MAXLEN = 12

def tokenize(texts):
    ids = torch.zeros(len(texts), MAXLEN, dtype=torch.long)
    for i, t in enumerate(texts):
        w = [VOCAB[x] for x in t.split()]
        ids[i, :len(w)] = torch.tensor(w)
    return ids

class ImageEncoder(nn.Module):
    def __init__(self, d=64):
        super().__init__()
        self.net = nn.Sequential(nn.Conv2d(3, 16, 3, 2, 1), nn.ReLU(),
                                 nn.Conv2d(16, 32, 3, 2, 1), nn.ReLU(),
                                 nn.Conv2d(32, 64, 3, 2, 1), nn.ReLU())
        self.proj = nn.Linear(64, d)
    def forward(self, x):
        return self.proj(self.net(x).mean((2, 3)))

class TextEncoder(nn.Module):
    def __init__(self, d=64):
        super().__init__()
        self.tok = nn.Embedding(len(WORDS), d)
        self.pos = nn.Parameter(0.02 * torch.randn(MAXLEN, d))
        self.layer = nn.TransformerEncoderLayer(d, 4, 128, dropout=0.0, batch_first=True)
        self.proj = nn.Linear(d, d)
    def from_embeddings(self, e, mask):          # e: [B, L, d], mask: [B, L] True = token
        h = self.layer(e + self.pos[:e.shape[1]], src_key_padding_mask=~mask)
        h = (h * mask[..., None]).sum(1) / mask.sum(1, keepdim=True)
        return self.proj(h)
    def forward(self, ids):
        return self.from_embeddings(self.tok(ids), ids != 0)

def captions(sizes, labels, rng):
    out = []
    for z, k in zip(sizes, labels.tolist()):
        c, s = CLASSES[k]
        out.append(TRAIN_T[rng.integers(4)].format(z=z, c=CNAMES[c], s=SHAPES[s]))
    return out

def pretrain(X, y, sz, seed, epochs=12, bs=128):
    torch.manual_seed(seed)
    rng = np.random.default_rng(seed + 100)
    img_enc, txt_enc = ImageEncoder(), TextEncoder()
    log_scale = nn.Parameter(torch.tensor(math.log(1 / 0.07)))
    opt = torch.optim.Adam(list(img_enc.parameters()) + list(txt_enc.parameters()) + [log_scale], lr=2e-3)
    n = len(y)
    for ep in range(epochs):
        perm = torch.randperm(n)
        for i in range(0, n - bs + 1, bs):
            idx = perm[i:i + bs]
            ids = tokenize(captions([sz[j] for j in idx.tolist()], y[idx], rng))
            zi = F.normalize(img_enc(X[idx]), dim=-1)
            zt = F.normalize(txt_enc(ids), dim=-1)
            logits = log_scale.exp() * zi @ zt.T
            target = torch.arange(len(idx))
            loss = (F.cross_entropy(logits, target) + F.cross_entropy(logits.T, target)) / 2
            opt.zero_grad(); loss.backward(); opt.step()
            with torch.no_grad():
                log_scale.clamp_(max=math.log(100))
    return img_enc.eval(), txt_enc.eval(), log_scale.exp().item(), loss.item()

# ---------------------------------------------------------------- zero-shot
TEMPLATES = {
    "bare class name": ["{c} {s}"],
    "seen wording, size dropped": ["a photo of a {c} {s}", "there is a {c} {s}", "the {s} is {c}"],
    "seen wording, size filled": [t.replace("{z}", z) for t in TRAIN_T for z in ("small", "large")],
    "unseen wording": ["{c} {s} , a photo", "is there a {c} {s}", "it is a {c} {s}",
                       "the photo of the {c} {s}"],
}
ALL_T = [t for ts in TEMPLATES.values() for t in ts]

@torch.no_grad()
def class_text(txt_enc, template, classes):
    ids = tokenize([template.format(c=CNAMES[CLASSES[k][0]], s=SHAPES[CLASSES[k][1]]) for k in classes])
    return F.normalize(txt_enc(ids), dim=-1)

@torch.no_grad()
def zero_shot(img_enc, txt_enc, scale, X, y, templates, classes=range(25), mode="embedding"):
    classes = list(classes)
    zi = F.normalize(img_enc(X), dim=-1)
    W = torch.stack([class_text(txt_enc, t, classes) for t in templates])      # [T, K, d]
    if mode == "embedding":
        p = zi @ F.normalize(W.mean(0), dim=-1).T
    else:                                                                    # probability space
        p = torch.softmax(scale * torch.einsum("nd,tkd->tnk", zi, W), -1).mean(0)
    pred = torch.tensor(classes)[p.argmax(1)]
    return (pred == y).float().mean().item()

# ---------------------------------------------------------------- learned context (CoOp-style)
def coop(img_enc, txt_enc, scale, X, y, classes, init, seed, M=4, steps=300):
    torch.manual_seed(seed)
    for p in list(img_enc.parameters()) + list(txt_enc.parameters()):
        p.requires_grad_(False)
    if init == "random":
        ctx = nn.Parameter(0.02 * torch.randn(M, 64))
    else:
        ctx = nn.Parameter(txt_enc.tok(tokenize(["a photo of a"]))[0, :M].clone())
    opt = torch.optim.Adam([ctx], lr=1e-2)
    zi = F.normalize(img_enc(X), dim=-1).detach()
    target = torch.tensor([classes.index(k) for k in y.tolist()])
    for _ in range(steps):
        logits = scale * zi @ ctx_text(txt_enc, ctx, classes).T
        loss = F.cross_entropy(logits, target)
        opt.zero_grad(); loss.backward(); opt.step()
    return ctx.detach()

def ctx_text(txt_enc, ctx, classes):
    names = tokenize([f"{CNAMES[CLASSES[k][0]]} {SHAPES[CLASSES[k][1]]}" for k in classes])[:, :2]
    e = torch.cat([ctx.expand(len(classes), -1, -1), txt_enc.tok(names)], 1)
    mask = torch.ones(e.shape[:2], dtype=torch.bool)
    return F.normalize(txt_enc.from_embeddings(e, mask), dim=-1)

@torch.no_grad()
def ctx_acc(img_enc, txt_enc, ctx, X, y, classes):
    zi = F.normalize(img_enc(X), dim=-1)
    pred = torch.tensor(classes)[(zi @ ctx_text(txt_enc, ctx, classes).T).argmax(1)]
    return (pred == y).float().mean().item()

def h(a, b):
    return 2 * a * b / (a + b)

# ---------------------------------------------------------------- one seed
def run(seed):
    rng = np.random.default_rng(seed)
    Xtr, ytr, sztr = make_set(rng, 320)
    Xte, yte, _ = make_set(rng, 40)
    Xsh_tr, ysh_tr, _ = make_set(rng, 4, BASE, shifted=True)
    Xb, yb, _ = make_set(rng, 40, BASE, shifted=True)
    Xn, yn, _ = make_set(rng, 40, NEW, shifted=True)
    img_enc, txt_enc, scale, loss = pretrain(Xtr, ytr, sztr, seed)
    out = {"final loss": loss, "logit scale": scale}
    for g, ts in TEMPLATES.items():
        accs = [zero_shot(img_enc, txt_enc, scale, Xte, yte, [t]) for t in ts]
        out[g] = (np.mean(accs), min(accs), max(accs))
    out["ensemble, embedding space"] = zero_shot(img_enc, txt_enc, scale, Xte, yte, ALL_T)
    out["ensemble, probability space"] = zero_shot(img_enc, txt_enc, scale, Xte, yte, ALL_T, mode="probability")
    hand = "a photo of a {c} {s}"
    rows = {}
    for name, ts in (("hand template", [hand]), ("16-template ensemble", ALL_T)):
        b = zero_shot(img_enc, txt_enc, scale, Xb, yb, ts, BASE)
        n = zero_shot(img_enc, txt_enc, scale, Xn, yn, ts, NEW)
        rows[name] = (b, n, h(b, n))
    for init in ("random", "a photo of a"):
        ctx = coop(img_enc, txt_enc, scale, Xsh_tr, ysh_tr, BASE, init, seed)
        b = ctx_acc(img_enc, txt_enc, ctx, Xb, yb, BASE)
        n = ctx_acc(img_enc, txt_enc, ctx, Xn, yn, NEW)
        rows[f"learned context, init {init}"] = (b, n, h(b, n))
    out["shift"] = rows
    return out

if __name__ == "__main__":
    results = {seed: run(seed) for seed in (0, 1, 2)}
    r = results[0]
    print(f"seed 0: final batch loss {r['final loss']:.3f}, logit scale {r['logit scale']:.1f}")
    print("clean test, 25 classes, zero-shot accuracy (seed 0 | range over seeds 0-2 of the group mean)")
    for g in TEMPLATES:
        m, lo, hi = r[g]
        ms = [results[s][g][0] for s in results]
        print(f"  {g:28s} mean {m:.3f} (min {lo:.3f}, max {hi:.3f}) | {min(ms):.3f}-{max(ms):.3f}")
    for k in ("ensemble, embedding space", "ensemble, probability space"):
        v = [results[s][k] for s in results]
        print(f"  {k:28s} {r[k]:.3f} | {min(v):.3f}-{max(v):.3f}")
    print("shifted domain: base (13 classes) / new (12 classes) / harmonic mean (seed 0 | range over seeds)")
    for name, (b, n, hm) in r["shift"].items():
        vb = [results[s]["shift"][name][0] for s in results]
        vn = [results[s]["shift"][name][1] for s in results]
        print(f"  {name:32s} {b:.3f} / {n:.3f} / {hm:.3f} | base {min(vb):.3f}-{max(vb):.3f}, new {min(vn):.3f}-{max(vn):.3f}")

seed 0: final batch loss 1.200, logit scale 27.6
clean test, 25 classes, zero-shot accuracy (seed 0 | range over seeds 0-2 of the group mean)
  bare class name              mean 0.977 (min 0.977, max 0.977) | 0.855-0.977
  seen wording, size dropped   mean 0.983 (min 0.979, max 0.987) | 0.819-0.983
  seen wording, size filled    mean 0.951 (min 0.943, max 0.964) | 0.847-0.951
  unseen wording               mean 0.977 (min 0.967, max 0.981) | 0.837-0.977
  ensemble, embedding space    0.985 | 0.881-0.985
  ensemble, probability space  0.988 | 0.860-0.988
shifted domain: base (13 classes) / new (12 classes) / harmonic mean (seed 0 | range over seeds)
  hand template                    0.588 / 0.623 / 0.605 | base 0.467-0.588, new 0.444-0.623
  16-template ensemble             0.585 / 0.590 / 0.587 | base 0.458-0.585, new 0.485-0.590
  learned context, init random     0.837 / 0.833 / 0.835 | base 0.727-0.837, new 0.769-0.833
  learned context, init a photo of a 0.856 / 0.854 / 0.855 | base 0.733-0.856, new 0.746-0.854

The plan expected that the “bare class name scores lowest; seen-wording templates score higher than unseen wordings”. On seed 0 the output supports neither: every group lies within about three points, the bare name (0.977) ties with the new word orders, of the two groups of training wordings one scores above the new orders and one below, and the only clear loss is from filling in a size word (0.951), which by construction describes only about half the images of each class. In our reading the gap CLIP’s authors describe, between one-word labels and sentence-length captions, is small here because every training caption contains the colour and shape words side by side and the encoder averages over tokens, so extra words shift all class embeddings alike. The variation between seeds (group means from 0.82 to 0.98) is larger than any wording effect, so a template ranking measured on one model need not hold for another. Ensembling helps a little: 0.985 and 0.988 on seed 0, and the embedding-space ensemble’s lowest seed (0.881) sits above the lowest seed of every single-template group (0.819–0.855). The two ways of averaging differ by less than the spread between seeds.

On the shifted domain the hand template falls to 0.59 on base and 0.62 on new classes, and ensembling does not recover it; in our reading the problem is in the image features, which a choice among wordings cannot move. Learned context raises both, to 0.83–0.86 on seed 0, and the ranges over seeds do not overlap with the hand template’s, so it helps on every seed. The plan expected that “new-class accuracy gains less or falls below the hand template (CoCoOp’s base-to-new gap)”. On seed 0 new classes do gain about four points less than base classes, but only because the hand template started higher on them; after learning, base and new accuracy are level, so no base-to-new gap of the kind CoCoOp reports appears. By the example’s construction, the shift is the same for every class and the new class names reuse the base classes’ colour and shape words, so a context that moves all class embeddings towards washed-out, noisy features helps any class; CoCoOp’s new classes are new concepts, where a context can absorb cues specific to the base classes. The example therefore shows the mechanics and the protocol of learned prompts but cannot reproduce their overfitting. The two initializations overlap across seeds and cannot be separated here.

Variants

• Hand-written templates, single or ensembled, per dataset or shared (CLIP).
• Generated descriptions, free sentences (CuPL) or attribute lists (Menon and Vondrick).
• Learned text context, shared (CoOp), class-specific (CoOp’s CSC option) or conditioned on the image (CoCoOp).
• Learned visual tokens in the input of a frozen encoder (VPT-shallow, VPT-deep).
• Pixel-space marks: circles, boxes, masks or numeric labels drawn on the image (red circles for CLIP, Set-of-Mark for generative models).
• In-context examples for generative models, chosen at random or retrieved (RICES), optionally ensembled over orderings.
• Chain-of-thought in the examples or the requested output, with the explanation before or after the answer.
• Prompt rewriting for generators, by a large language model (DALL·E 3) or a learned rewriter (Promptist).

Best Practices

• Match the training text. Prompts that look like the captions a model was trained on are the first improvement CLIP’s authors report (“A photo of a {label}.”); for generators, the same reasoning leads DALL·E 3 to expand short prompts to its long training captions.
• Disambiguate class names with a category word (“a type of pet”) or an edited name (“metal nail”), and check the lowest-scoring classes first, as the CLIP notebook did.
• Ensemble in embedding space when the classifier is reused for many images: it costs nothing at test time.
• Tune prompts on training or validation data, never on test images. The CLIP notebook tuned ImageNet templates and names on the training set; Flamingo fixed one set of evaluation prompts across benchmarks.
• Report base and new classes separately for learned prompts, with their harmonic mean, as CoCoOp does; a gain on the training classes says nothing about the others.
• Initialize learned context from words (“a photo of a”) rather than randomly; CoCoOp’s ablation found this better on both base and new classes.
• Choose in-context examples by similarity when a large support set exists, and average over several orderings (RICES and prompt ensembling in Flamingo).

Pitfalls

• Prompts tuned on the evaluation set. CoCoOp’s authors note that CLIP’s manual prompts were tuned using all classes in the test data, which flatters the hand-written baseline on new classes; CLIP’s authors acknowledge repeatedly querying validation sets while developing a model meant to be used zero-shot (summarized in CLIP). Any reported prompt gain should say what data the prompt was chosen on.
• Sensitivity hidden by a single number. Flamingo’s authors call in-context learning sensitive to the order and format of examples; ScienceQA’s prompts with explanations only varied most between runs; the marker’s shape changed CLIP’s average keypoint-naming accuracy on CUB from 33.5% for circles to 21.5% for crosses. The example above shows the reverse case too: with a different trained model the ranking of wordings changes. Report the spread over templates, orderings or seeds.
• Learned prompts that fit only their classes. CoOp’s average new-class accuracy fell 11 points below CLIP’s hand-written prompts. The learned vectors are also not words: they cannot be read or transferred to another model.
• More examples are not always better. ScienceQA’s chain-of-thought prompt peaked at two examples; Flamingo’s performance plateaued beyond 32 shots and the prompt length is capped by the sequence length used in training.
• Prompts that invoke learned associations. A red circle around a person raised the share of COCO person crops that ViT-L/14 at 336 pixels classified into “criminal” categories from 29.8% to 47.9%, and similar rises appeared for the other ViT encoders tested (Shtedritski et al., 2023, Table 6); the authors warn that the method must not be used on sensitive data. CLIP’s own bias probes showed that the label set offered to the model changes how often people are misclassified (CLIP).
• Rewriting that drifts from the user’s intent. Promptist’s optimized prompts raised aesthetic scores while lowering CLIP relevance to the user’s input, and a rewriter trained against one generator learns that generator’s preferences.

Evidence and Impact

Each result below holds only within its own set-up: the model, the datasets, the number of shots and the baseline prompt.

Technique	Set-up	Documented effect	Source
Default template	CLIP, ImageNet	+1.3 points over the bare label	Radford et al., 2021
80-template ensemble	CLIP, ImageNet	+3.5 over the default template; almost +5 with engineering, also on average over 36 datasets	Radford et al., 2021
LLM-written descriptions (CuPL, full)	CLIP ViT-L/14, 15 datasets	Better on all 15; ImageNet 75.54 → 76.69, DTD 55.20 → 61.70	Pratt et al., 2023
Learned context (CoOp)	ViT-B/16, 16 shots, 11 datasets	Base 69.34 → 82.69, new 74.22 → 63.22	Zhou et al., 2022 (CVPR)
Image-conditioned context (CoCoOp)	same	Base 80.47, new 71.69, H 75.83	Zhou et al., 2022 (CVPR)
VPT-deep	ImageNet-21k ViT-B/16, 24 tasks	Beat full fine-tuning on 20	Jia et al., 2022
Red circle	CLIP ViT-L/14@336, CUB keypoint naming	46.5% against 15.8% for crops	Shtedritski et al., 2023
Retrieved in-context examples	Flamingo-80B, 16 shots, ImageNet	66.4% random → 76.0% RICES → 77.3% with ensembling	Alayrac et al., 2022
Chain-of-thought	GPT-3 on captions, ScienceQA test	+1.20 points (2-shot), +3.99 with fine-tuned UnifiedQA	Lu et al., 2022
Learned prompt rewriting	Stable Diffusion v1.4, DiffusionDB	Aesthetic 5.47 → 6.26, relevance 0.28 → 0.26	Hao et al., 2023

The pattern across these results is that prompts recover knowledge a model already has, in the form it learned it: sentence-like captions for CLIP, red circles for a model trained on the web’s annotated photos, worked examples for a model trained on interleaved documents. Where the knowledge is missing, prompting gains little, and learned prompts trade the generality of words for accuracy on the classes they were fitted to. CoCoOp’s authors call CoOp the earliest work to bring continuous prompt learning to vision, and it showed that the adaptation could be automated with a few thousand parameters (2,048 for its 4-vector ImageNet context, CoCoOp Table 5); CoCoOp’s base-to-new protocol, with base and new classes reported separately, is the check of what such prompts generalize to that the example above adopts.