Learning and Transferring Mid-Level Image Representations Using Convolutional Neural Networks

“Learning and Transferring Mid-Level Image Representations Using Convolutional Neural Networks” is a paper by Maxime Oquab, Léon Bottou, Ivan Laptev and Josef Sivic, presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Columbus, Ohio, in June 2014; an INRIA technical report with the same title was posted to the HAL archive on 28 November 2013. It asked whether the layers of a convolutional network trained on ImageNet could serve a task with far less labelled data, and answered with a concrete recipe: copy the network up to its second fully connected layer, keep those weights fixed, add two new fully connected “adaptation” layers, and train them on overlapping image patches labelled from the target task’s bounding boxes. On Pascal VOC it reported the best published object classification results on VOC 2007 and, with a larger source set, on VOC 2012, and the best average on VOC 2012 action classification. Adapting a pretrained network in general, including which layers to freeze, is the subject of fine-tuning; this article covers the publication. It was read in the CVF open-access copy of the CVPR paper and compared with both HAL versions; details that differ between versions are marked.

Problem

In 2012 Krizhevsky et al.’s network won the ImageNet challenge (ILSVRC-2012) by a wide margin. Oquab et al. note that this network has more than 60 million parameters and ask whether every new recognition task would need millions of annotated images of its own. Learning that many parameters from the few thousand training images of a dataset like Pascal VOC was, in their words, problematic.

The obstacle was not only size. Citing Torralba and Efros’s study of dataset bias, the authors list three differences between the source and the target. ImageNet images typically show one centred object filling much of the frame, while VOC images show objects embedded in cluttered scenes, at several scales and in mutual occlusion (“dataset capture bias”). VOC scenes contain objects absent from the source data (“negative data bias”). And the labels differ: ImageNet separates breeds of dog that VOC calls simply dog, and no ImageNet class corresponds to an action such as reading or running (“label bias”).

At the time the best VOC classifiers were bag-of-features pipelines, typically densely sampled SIFT descriptors quantized into visual words and aggregated by histogram or Fisher-vector encodings. The paper argues that these pipelines resemble the first layers of early neural networks, and that they may have outperformed CNNs only because CNNs were hard to train on small datasets.

Contribution

• Transfer of a supervised representation. The convolutional layers and the first two fully connected layers of an ImageNet-trained network are treated as a generic extractor of what the authors call a mid-level image representation, reused unchanged on the target task. The related work section contrasts this with transfer methods that adapt classifiers or kernels on top of hand-designed features, and with earlier CNN transfer from unsupervised pseudo-tasks.
• An adaptation layer that remaps source labels to target labels, including a background class.
• Training and test procedures modelled on sliding-window detectors, which present the target data to the network as multi-scale patches resembling the centred source images.
• Experiments on VOC 2007 and VOC 2012 object classification and VOC 2012 action classification, an analysis of how the overlap between source and target classes affects the result, and qualitative object and action localization maps.

Method

Network and Adaptation Layers

The source network has the architecture of Krizhevsky et al.: five convolutional layers C1–C5 and three fully connected layers FC6–FC8 on a $224 \times 224$ input, with ReLU units and a softmax output. For the target task the authors remove FC8 and add two layers on top of FC7’s output $Y_7$:

$$Y_a = \sigma(W_a Y_7 + B_a), \qquad Y_b = \psi(W_b Y_a + B_b),$$

where $\sigma$ is the ReLU and $\psi$ the softmax. FCa has 2,048 units; FCb has one output per target category. The ablation in §4 gives FCb 21 outputs on VOC, the 20 object classes plus background. The parameters of C1–C5, FC6 and FC7 are trained on the source task, transferred and kept fixed; only FCa and FCb are trained on the target data (§3.1, Figure 2). In current terms, by our reading, this is a frozen backbone with a small nonlinear head, between a linear probe and fine-tuning.

Patches and Their Labels

Training is designed to bridge the gap between centred ImageNet objects and cluttered VOC scenes (§3.2). From each image the authors extract about 500 square patches on a regular grid, with at least 50% overlap between neighbours. A patch has side $s = \min(w, h)/\lambda$ for an image of width $w$ and height $h$, with

$$\lambda \in \{1, 1.3, 1.6, 2, 2.4, 2.8, 3.2, 3.6, 4\},$$

and is rescaled to $224 \times 224$. The text calls these “eight different scales”, but lists nine values of $\lambda$, in the CVPR paper and in both HAL versions.

A patch $P$ is labelled a positive example of class $o$ if some ground-truth box $B_o$ of that class satisfies $|P \cap B_o| \geq 0.2\,|P|$ (the object fills enough of the patch) and $|P \cap B_o| \geq 0.6\,|B_o|$ (the patch contains most of the object), and the patch overlaps no more than one object; $|A|$ is the area of box $A$. Figure 4 shows how the cases are handled. A patch that meets the criteria for two objects is deleted from the training set, rather than labelled. A patch whose objects are all too small or truncated, or which overlaps no object, is labelled background, so background patches can contain parts of objects. The failure-mode discussion confirms that patches with several objects are excluded from training.

Most patches are background. Instead of re-weighting the loss, the authors re-sample: a random 10% of the background patches is kept, and stochastic gradient descent draws its mini-batches from the resampled set (§3.2).

Scoring an Image

At test time the network scores each of the roughly 500 patches of the test image, and the image’s score for class $C_n$ is (§3.3, Eq. 1)

$$\operatorname{score}(C_n) = \frac{1}{M} \sum_{i=1}^{M} y(C_n \mid P_i)^k ,$$

where $y(C_n \mid P_i)$ is the network’s output for class $C_n$ on patch $P_i$ and $M$ is the number of patches. Larger $k$ concentrates the score on the highest-scoring patches; $k = 5$ was chosen on the validation set and fixed. The authors note that the same patch scores could be computed far more cheaply by running large convolutions over subsampled versions of the whole image, as Farabet et al. did for scene labelling, which would also allow denser coverage.

Training Details

All networks were trained with Krizhevsky et al.’s cuda-convnet code and its dropout and jittering, but without the RGB intensity alteration, on one GeForce GTX Titan GPU with 6 GB of memory (§4). The learning rate started at 0.01 and was divided by 10 whenever the training cross-entropy stabilized; training stopped after three such stages. The authors say they did not tune these parameters and observed no overfitting on the validation set.

Three source networks were trained:

• PRE-1000C, on the 1,000 classes and 1.2 million images of ILSVRC-2012, with 4,096-unit FC6 and FC7. Training took about one week and reached 18% top-5 error, against the 17% Krizhevsky et al. reported for one network; the authors suggest the missing intensity augmentation as the cause.
• PRE-1000R, on 1,000 classes drawn at random from the roughly 22,000 ImageNet classes, with about 25% fewer training images than ILSVRC-2012.
• PRE-1512, on the ILSVRC-2012 classes plus 512 classes chosen to overlap with VOC: every ImageNet class below the WordNet nodes hoofed mammal (276 classes), furniture (165), motor vehicle (48), public transport (18) and bicycle (5). FC6 and FC7 grew to 6,144 units, and training on the resulting 1.6 million images reached 21.8% top-5 error over the 1,512 classes.

NO PRETRAIN is the same architecture trained only on the target data.

Worked Example: Patch Sampling, Labelling and Scoring

The NumPy sketch below runs the patch machinery of §3.2–§3.3 on a synthetic $500 \times 375$ image with two objects, a tall rectangle (“person”) and a small disc (“sheep”). The paper does not give the grid stride, so the code tries strides of one half and one third of the patch side. In place of the network, a patch’s score for a class is the fraction of the patch covered by that object’s pixels; this stand-in exists only so that Eq. 1 and the localization map can be computed.

import numpy as np

# Synthetic 500 x 375 scene (w x h) with two objects drawn as filled shapes; boxes are (x1, y1, x2, y2).
W, H = 500, 375
img = np.zeros((H, W))
yy, xx = np.mgrid[0:H, 0:W]
boxes = {"person": (60, 40, 200, 360), "sheep": (300, 250, 380, 330)}
img[40:360, 60:200] = 1.0                                              # person: a rectangle
img[((xx - 340) / 40) ** 2 + ((yy - 290) / 40) ** 2 <= 1] = 2.0      # sheep: a disc

LAMBDAS = [1, 1.3, 1.6, 2, 2.4, 2.8, 3.2, 3.6, 4]   # the nine values printed in Sec. 3.2

def patches(w, h, f):
    """Square patches of side s = min(w,h)/lambda on a grid with stride about s/f (our choice of f)."""
    out = []
    for lam in LAMBDAS:
        s = min(w, h) / lam
        nx = int(np.floor((w - s) / (s / f))) + 1
        ny = int(np.floor((h - s) / (s / f))) + 1
        xs = np.linspace(0, w - s, nx)            # spread the grid evenly over the image
        ys = np.linspace(0, h - s, ny)
        out += [(x, y, x + s, y + s) for x in xs for y in ys]
    return out

def inter(a, b):
    return max(0, min(a[2], b[2]) - max(a[0], b[0])) * max(0, min(a[3], b[3]) - max(a[1], b[1]))
area = lambda a: (a[2] - a[0]) * (a[3] - a[1])

def label(P):
    """Sec. 3.2 rule: positive for o if |P∩B| >= 0.2|P| and |P∩B| >= 0.6|B|; two qualifying objects -> discarded."""
    hits = [o for o, B in boxes.items() if inter(P, B) >= 0.2 * area(P) and inter(P, B) >= 0.6 * area(B)]
    return "background" if not hits else (hits[0] if len(hits) == 1 else "discarded")

print("patches at stride s/2 (50% overlap):", len(patches(W, H, 2)), "| s/3 (67% overlap):", len(patches(W, H, 3)))
P = patches(W, H, 3)
labels = [label(p) for p in P]
print("patches:", len(P), {k: labels.count(k) for k in ["person", "sheep", "background", "discarded"]})
rng = np.random.default_rng(0)
bg = [i for i, l in enumerate(labels) if l == "background"]
keep = rng.choice(bg, size=round(0.1 * len(bg)), replace=False)   # "a random 10% of the background patches"
print("background patches kept for training:", len(keep))

# Stand-in for the network's patch output y(C|P): the fraction of the patch covered by each object's
# pixels (not a CNN; it only lets us run Eq. 1 and the output map).
def y(p, value):
    x1, y1, x2, y2 = (int(round(v)) for v in p)
    return float(np.mean(img[y1:y2, x1:x2] == value))

for name, value in (("person", 1.0), ("sheep", 2.0)):
    scores = np.array([y(p, value) for p in P])
    agg = {k: np.mean(scores ** k) for k in (1, 5)}
    print(f"{name}: Eq.1 score k=1 {agg[1]:.4f}  k=5 {agg[5]:.4f}  share of score from top 5% patches "
          f"k=1 {np.sort(scores)[-len(P)//20:].sum()/scores.sum():.2f}  "
          f"k=5 {np.sort(scores**5)[-len(P)//20:].sum()/(scores**5).sum():.2f}")
    # Output map: average the scores of all patches covering each pixel (Sec. 4, "Object localization").
    acc, cnt = np.zeros((H, W)), np.zeros((H, W))
    for p, sc in zip(P, scores):
        x1, y1, x2, y2 = (int(round(v)) for v in p)
        acc[y1:y2, x1:x2] += sc; cnt[y1:y2, x1:x2] += 1
    m = acc / cnt
    r, c = np.unravel_index(np.argmax(m), m.shape)
    B = boxes[name]
    print(f"   output-map peak at (x={c}, y={r}); inside its box: {B[0] <= c < B[2] and B[1] <= r < B[3]}")

patches at stride s/2 (50% overlap): 206 | s/3 (67% overlap): 432
patches: 432 {'person': 4, 'sheep': 22, 'background': 406, 'discarded': 0}
background patches kept for training: 41
person: Eq.1 score k=1 0.2947  k=5 0.1145  share of score from top 5% patches k=1 0.17  k=5 0.44
   output-map peak at (x=117, y=234); inside its box: True
sheep: Eq.1 score k=1 0.0343  k=5 0.0005  share of score from top 5% patches k=1 0.53  k=5 0.97
   output-map peak at (x=330, y=327); inside its box: True

With exactly 50% overlap the nine scales give only about 200 patches on an image of this size, so the paper’s “around 500” implies more overlap than the minimum it states; a stride of a third of the side comes closer (our arithmetic, since the paper gives no stride). The labelling rule is strict for elongated objects: the tall person box earns only 4 positive patches, because a patch must hold 60% of the box while the box fills 20% of the patch, whereas the compact sheep earns 22. Even before resampling, background patches outnumber positives about 16 to 1 here, which is why the authors subsample them. Raising $k$ from 1 to 5 moves most of each class’s score onto its few best patches, which keeps a small object from being drowned by the many patches that miss it. Averaging patch scores per pixel puts each class’s peak inside its object’s box, which is the mechanism behind the paper’s output maps; with a real network the scores come from appearance, not from known masks.

Results

All results are per-class average precision (AP, %) on the test sets, with the mean over classes (mAP), as printed in Tables 1–3 of the CVPR paper. Differences in the text are in percentage points.

VOC 2007 Object Classification (Table 1)

Method	mAP
INRIA (Marszałek et al., 2007 challenge winners)	59.4
NUS-PSL (Song et al., CVPR 2011)	70.5
PRE-1000C	77.7

The transferred representation beat INRIA by 18.3 points and NUS-PSL by 7.2, and was better than NUS-PSL on 17 of the 20 classes, losing on bottle, horse and sofa (our count from Table 1). The mean of NUS-PSL’s 20 printed per-class APs is 70.0, not the 70.5 printed as its mAP (our arithmetic); the other rows’ means match.

VOC 2012 Object Classification (Table 2)

Method	mAP
NUS-PSL (Yan et al., 2012 challenge winners)	82.2
NO PRETRAIN	70.9
PRE-1000C	78.7
PRE-1000R	76.3
PRE-1512	82.8

With the ILSVRC-2012 source, transfer stayed below the challenge winner; the text says “about 4%”, while the table gives 3.5 points. PRE-1000C was better on five of the twenty classes, which Table 2 marks in italics: bird, cat, dog, person and potted plant. Training the same network on VOC 2012 alone cost what the text calls “nearly 8%” (7.8 points in the table), the authors’ measure of what the transfer contributes.

The class-overlap experiments probe where the gain comes from. ILSVRC-2012 contains 59 bird species and 120 dog breeds related to VOC’s bird and dog. With 1,000 random ImageNet classes (PRE-1000R), mAP fell 2.4 points, which the authors read as a modest benefit from class overlap and a procedure robust to the choice of source classes; they note that PRE-1000R also had about 25% fewer images, another possible cause. Adding the 512 overlapping classes (PRE-1512) lifted mAP to 82.8, above the 2012 winner by 0.6 points, and better than it on 8 of the 20 classes (our count). The largest gains over PRE-1000C were on the classes the extra source classes target: cow (13.0 points), horse, sofa, sheep, table and chair (our arithmetic); on sofa, chair and table PRE-1512 still trails NUS-PSL. Every class satisfies PRE-1000R ≤ PRE-1000C ≤ PRE-1512, which the authors take to suggest that the number of source images and classes may be decisive for the target task.

In the PRE-1512 setup, using a single adaptation layer (FCb only, 21 outputs) or three (2,048, 2,048 and 21 units) each cost about 1 point of mAP compared with the two-layer FCa–FCb head. This paragraph is not in the technical report (HAL v1); it appears in the CVPR paper and HAL v2.

VOC 2012 Action Classification (Table 3)

The task has 4,588 training and 4,569 test images of people performing one of ten actions; since the person’s box is given at test time, training and testing use one square patch per person, centred on the box and padded by mirroring pixels where it leaves the image. Training samples with several simultaneous actions were excluded. The source network was PRE-1512.

Method	mAP
STANFORD (VOC 2012 challenge entry)	69.1
OXFORD (VOC 2012 challenge entry)	69.6
NO PRETRAIN	49.7
PRE-1512	68.4
PRE-1512U	70.2

The paper cites the two baselines only to the VOC 2012 web page. On the challenge’s results page (as of October 2026) the matching per-class APs belong to STANFORD_RF_MULTFEAT_SVM (Khosla, Zhang, Yao and Fei-Fei; random forests with SVMs on multiple features), entered in the competition restricted to VOC 2012 training data, and OXFORD_ALIGNED_BODYPARTS (Hoai, Ladicky and Zisserman; body-part detectors and aligned descriptors), entered in the competition that allows outside training data.

Transfer added 18.7 points over training from scratch (our arithmetic), the largest such gap in the paper. With the frozen layers, PRE-1512 was best on playing instrument and taking photo but below both STANFORD and OXFORD on average. For PRE-1512U, “inspired by” Collobert et al.’s transfer between language tasks, the authors also retrained upper pretrained layers on the action data. In the CVPR copy they retrain FC6, FC7, FCa and FCb with C1–C5 fixed. The change improved all ten classes, by 0.7 to 3.6 points (our count), and gave what the authors believe was the best published average on this task, 0.6 points above OXFORD; per class it beat both STANFORD and OXFORD only on playing instrument, reading and taking photo (our count).

Localization

The paper reports no localization metric. For a qualitative view, it builds an output map for each class by averaging the scores of all test patches covering each pixel (Figures 1 and 5), and the authors write that the maps show the system knows the sizes and locations of objects, concluding that “addressing the detection task seems within reach”. For actions, a network trained in the sliding-window manner with ground-truth person boxes at training time, but none at test time, produced maps that the authors say estimate where the action happens. This localization is not weakly supervised in the later sense: patch labels come from bounding boxes during training. Only the test-time localization goes without boxes.

Failure Modes

The authors list three: top-ranked false positives resemble the target classes and might need higher-level scene interpretation; co-occurring objects, such as a person on a chair, can be missed because multi-object patches are excluded from training, which a multi-label objective could fix; and very small or very large objects can fail under the sparse patch sampling, which a convolutional implementation of the sliding window could fix.

Versions

Version	Date	Notes
HAL v1, INRIA technical report	28 Nov 2013	11 pages with HAL cover; separate figures for object (Fig. 5) and action (Fig. 6) maps
CVPR 2014 (CVF open access)	June 2014	8 pages; IEEE Xplore pages 1717–1724
HAL v2	13 Sep 2014	Deposited as the conference version; text matches the CVF copy except the PRE-1512U layer list

Tables 1–3 are identical in the technical report and the CVPR paper. By our comparison of the texts, the CVPR version adds the paragraph on parallel work, the conclusion’s sentence placing the paper among that work (the report cites none of these four papers) and the adaptation-layer ablation, combines the object and action maps into one Figure 5, and replaces the report’s closing plan to release the mid-level representation (for combination with indexing and deformable models) with a statement that the code is available on the project page.

One method detail differs. In both HAL versions the PRE-1512U sentence copies C1–C5, FC6 and FC7 but lists only FC6, FCa and FCb as retrained, leaving FC7’s status unstated; the CVF copy lists FC6, FC7, FCa and FCb. All three versions introduce the experiment as training “without freezing the FC6 weights”, without naming FC7. The IEEE Xplore version is not open access, according to OpenAlex, and was not read; the CVF copy is the one followed here.

Context and Contemporaries

The paper was one of several in late 2013 and early 2014 that reused an ImageNet-trained network on other tasks. The CVPR version names them as efforts done in parallel, which the technical report did not yet mention: DeCAF (Donahue et al., arXiv October 2013), which trained classifiers on the network’s activations and tested them on Caltech-101, the Office domain-adaptation benchmark, Caltech-UCSD Birds and SUN-397; Zeiler and Fergus (arXiv November 2013), who retrained only the softmax classifier of their ImageNet network for Caltech-101, Caltech-256 and PASCAL VOC 2012; R-CNN (Girshick et al., arXiv November 2013), which fine-tuned the network for detection; and OverFeat (Sermanet et al., arXiv December 2013). The CVPR paper characterizes these as transfer to Caltech-256 classification, scene classification and object localization. By our reading, its own distinguishing elements are the trained adaptation head, the patch-based handling of scene clutter and the action task; the conclusion places the work within this “recent evidence” that CNNs learn rich mid-level features transferable to many recognition tasks.

Razavian et al. (arXiv March 2014, CVPR 2014 workshops) cite the technical report. In their VOC 2007 table, linear SVMs on off-the-shelf OverFeat features reached 73.9 mAP, and 77.2 with training-set augmentation (CNNaug-SVM); the caption notes that Oquab et al., who train two fully connected layers on VOC, achieve a better 77.7, indicating the potential of further adapting the representation to the target. The transfer learning article discusses DeCAF and Razavian et al. as linear probes.

Impact

The paper is widely cited as an early demonstration that ImageNet-trained CNN layers transfer to small datasets. Its recipe, a frozen pretrained trunk with a newly trained head, is the reference point the fine-tuning article uses for layer freezing; its PRE-1512U experiment is an early instance of retraining the upper pretrained layers together with the head. The class-overlap experiments were an early controlled look at how the choice of source classes affects transfer.

As of October 2026 the project page lists the CVPR paper and the technical report, offers a testing-only code release based on cuda-convnet (839 MB) that it says reproduces the CVPR results, and still announces Torch training and testing code as “coming soon”.

Limitations

• Small margins on VOC 2012. PRE-1512 beat the 2012 winner by 0.6 points and PRE-1512U the best listed action result by 0.6, each a single result with no variance reported. In our view the comparison also sets one network against challenge systems built in quite different ways.
• The best object result needed a source chosen for the target. The 512 added classes were selected by WordNet nodes matching VOC categories; with the standard ILSVRC-2012 source the method stayed 3.5 points below the winner on VOC 2012.
• Supervision is strong. Training patches are labelled from ground-truth boxes, so the method needs box annotations that image-level classification benchmarks do not otherwise require.
• Cost. Each image is processed as about 500 independent $224 \times 224$ patches, a cost the authors acknowledge and propose to remove with convolutional sliding windows.
• Unexplained figures. The conclusion’s claim that “only 12% of the ImageNet corpus” already gives the best VOC 2012 results does not state what the 12% is a fraction of. The “eight scales” with nine listed $\lambda$ values and the version difference over FC7 in PRE-1512U are noted above.
• Localization is qualitative. The maps are examples, with no localization measure.

What Came After

The same group took up the localization question in “Is Object Localization for Free?” (Oquab et al., CVPR 2015). It keeps the frozen pretrained layers of the 2014 setup but treats the adaptation layers as convolutions, so the patch scores of the 2014 sliding window come from one pass over the image, and adds a max-pooling layer over positions so that the network can be trained from image-level labels alone. In its VOC 2012 test table, the weakly supervised network reached 86.3 mAP, against 82.8 for the 2014 PRE-1512 setup, which it lists as a comparable method with object-level supervision; two other methods trained from image-level labels in that table, Chatfield et al. and NUS-HCP, were also above 82.8.

Yosinski et al. (2014) studied which layers to copy and whether to freeze them in controlled ImageNet experiments, and found that initializing with transferred layers and then fine-tuning them improved generalization; their results and the later practice of fine-tuning whole pretrained networks are described in the transfer learning and fine-tuning articles.