F1 Score

The F1 score summarizes a classifier’s positive decisions in one number: the harmonic mean of its precision and recall. It is high only when both are high, it never counts true negatives, and it is a common single score where positives are rare, from defect detection to the matching step of detection and tracking benchmarks. Its generalization $F_\beta$ weights recall $\beta$ times as much as precision. With several classes, the per-class scores are combined by macro, micro or weighted averaging, which can rank the same two models in opposite orders. Computed over the pixels of a binary mask, F1 is the Dice coefficient of segmentation. This article follows the definitions of scikit-learn 1.7.2 and notes where TorchMetrics differs.

Formula

For one class treated as positive, with TP, FP and FN the true positives, false positives and false negatives of the confusion matrix, precision $P$ and recall $R$,

$$F_1 = \frac{2PR}{P + R} = \frac{2\,\mathrm{TP}}{2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN}}, \qquad F_\beta = \frac{(1 + \beta^2)\,PR}{\beta^2 P + R} = \frac{(1 + \beta^2)\,\mathrm{TP}}{(1 + \beta^2)\,\mathrm{TP} + \beta^2\,\mathrm{FN} + \mathrm{FP}}.$$

Equivalently, $F_\beta$ is a weighted harmonic mean,

$$\frac{1}{F_\beta} = \frac{1}{1 + \beta^2}\,\frac{1}{P} + \frac{\beta^2}{1 + \beta^2}\,\frac{1}{R},$$

so $\beta > 1$ puts more weight on recall and $\beta < 1$ on precision. $F_\beta$ tends to $R$ as $\beta \to \infty$ and to $P$ as $\beta \to 0$; $F_2$ and $F_{0.5}$ are the common choices. scikit-learn computes the score from the counts, the right-hand forms, rather than from $P$ and $R$ (scikit-learn 1.7.2 User Guide, §3.4.4.9).

Origin. Van Rijsbergen (1979, ch. 7, pp. 133–134) derived an effectiveness measure for information retrieval from measurement theory,

$$E = 1 - \frac{1}{\alpha/P + (1 - \alpha)/R}, \qquad \alpha = \frac{1}{\beta^2 + 1},$$

where $\beta$ is defined through a user’s preferences: it is the ratio $P/R$ at which the user would trade a small gain in precision for an equal loss in recall, which the book reads as attaching $\beta$ times as much importance to recall as to precision. Substituting $\alpha$ shows that $1 - E = F_\beta$. The book notes that at $\beta = 1$, $E$ is the normalized symmetric difference $|A \,\triangle\, B| / (|A| + |B|)$ between the set $A$ of relevant documents and the set $B$ of retrieved ones; its complement $2|A \cap B|/(|A| + |B|)$ is the Dice coefficient. The name F-measure appears in the scoring of the Fourth Message Understanding Conference, where Chinchor (1992) describes combining recall and precision with “van Rijsbergen’s F-measure” and reports it with $\beta = 1$, 0.5 and 2.

Intuition

A harmonic mean is dominated by the smaller of its arguments. A detector with precision 1.0 and recall 0.1 has an arithmetic mean of 0.55 but $F_1 = 0.18$; neither calling everything positive (recall 1, precision equal to the share of positives) nor making a single safe call (precision 1, tiny recall) scores well.

The count form gives a second reading: $F_1$ is the number of correct positive calls divided by the average of the number of actual positives, $\mathrm{TP} + \mathrm{FN}$, and the number of predicted positives, $\mathrm{TP} + \mathrm{FP}$. True negatives do not appear, so adding easy negatives to a test set, such as background images in a detection set or background pixels around a mask, does not change it.

Hand and Christen (2018) showed a third reading: $F_1 = pR + (1 - p)P$, a weighted arithmetic mean with weight $p = (\mathrm{TP} + \mathrm{FN})/(2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN})$ on recall. The ratio of the weights, $p/(1-p)$, is the number of actual positives divided by the number of predicted positives, so it changes from one classifier to the next. In their view, comparing two methods by F1 therefore weighs precision against recall differently for each method, unless their thresholds are set to predict the same number of positives.

Interpretation

• Range. $F_\beta \in [0, 1]$; 1 means no false positives and no false negatives, and 0 means no true positives. $F_\beta$ lies between $P$ and $R$; $F_1$, as a harmonic mean, lies closer to the smaller.
• The baseline depends on prevalence. With a fraction $b$ of positives, a classifier that calls everything positive has $P = b$ and $R = 1$, so $F_1 = 2b/(1 + b)$: 0.67 when half the images are positive, 0.02 when one in a hundred is. A score of 0.6 is therefore poor on a balanced set and strong for a rare class. Lipton, Elkan and Naryanaswamy (2014, Theorem 2) proved that for a classifier whose scores carry no information about a label, the threshold that maximizes expected F1 predicts every example positive.
• It depends on which class is positive. In a use case of Chicco and Jurman (2020), a test set has 91 positives and 9 negatives and a classifier produces TP = 90, FN = 1, TN = 0 and FP = 9: F1 is 0.95, while the Matthews correlation coefficient is −0.03. It follows from the definition that taking the other class as positive gives TP = 0 and F1 = 0.
• Undefined cases. When TP + FP + FN = 0 (no positives in the data and none predicted, such as an empty mask predicted empty), F1 is $0/0$. scikit-learn’s zero_division parameter decides the value: the default "warn" returns 0 with an UndefinedMetricWarning, and 0.0, 1.0 or np.nan can be set; np.nan excludes the class from an average (option added in version 1.3). Because the score is computed from counts, precision can be undefined while F1 is not: a class that is never predicted but does occur has $F_1 = 0$ whatever zero_division says (scikit-learn 1.7.2 source code).

Averaging Over Classes

With $K$ classes, each class $k$ gets its own counts $\mathrm{TP}_k$, $\mathrm{FP}_k$ and $\mathrm{FN}_k$ against the rest and its own $F_{1,k}$. scikit-learn’s average parameter combines them (scikit-learn 1.7.2 User Guide, §3.4.4.1 and §3.4.4.9):

$$F_1^{\text{macro}} = \frac{1}{K}\sum_{k} F_{1,k}, \qquad F_1^{\text{weighted}} = \frac{\sum_k n_k F_{1,k}}{\sum_k n_k}, \qquad F_1^{\text{micro}} = \frac{2\sum_k \mathrm{TP}_k}{2\sum_k \mathrm{TP}_k + \sum_k \mathrm{FP}_k + \sum_k \mathrm{FN}_k},$$

with $n_k = \mathrm{TP}_k + \mathrm{FN}_k$ the support of class $k$.

• Macro gives each class equal weight, so rare classes count as much as frequent ones. The scikit-learn guide notes that this can over-emphasize the typically low performance on an infrequent class.
• Weighted weights each class by its support. scikit-learn’s documentation warns that it can give an F-score that is not between the averaged precision and recall.
• Micro pools the counts, so each decision counts once and frequent classes dominate. In single-label multi-class classification every error is a false positive for one class and a false negative for another, so with all labels included micro-averaged precision, recall and F1 all equal accuracy.
• Samples applies to multi-label data, where each image can carry several labels: F1 is computed per image between its sets of true and predicted labels and averaged over images.
• average=None returns the per-class scores, and average="binary", the default of f1_score and fbeta_score, reports only the class pos_label and raises an error for multi-class targets.

Two macro F1s. Some papers compute “macro F1” as the harmonic mean of macro precision and macro recall, $2\bar{P}\bar{R}/(\bar{P} + \bar{R})$, instead of the mean of per-class F1. Opitz and Burst (2019) proved that this “F1 of averages” is never below the mean of per-class F1, that the two are equal only when every class has equal precision and recall, and that the gap can approach 0.5: its supremum is 0.5 for an even number of classes and $0.5 - 1/(2K^2)$ for an odd number $K$. The larger form rewards classifiers that trade precision on some classes against recall on others, and the two can rank classifiers differently; they recommend the mean of per-class F1 and ask authors at least to state which they use. scikit-learn computes the mean of per-class scores, as its guide states.

Other libraries. TorchMetrics (v1.9.0 source code) computes $F_\beta$ from counts as scikit-learn does, with differences in defaults. Its task wrapper F1Score defaults to average="micro", while MulticlassF1Score and MultilabelF1Score default to "macro". In multi-class macro averaging, a class with TP + FP + FN = 0 is left out of the mean, whereas scikit-learn includes any class passed in labels at the zero_division value, unless that value is np.nan. zero_division defaults to 0 without a warning, and the binary and multi-label metrics accept scores and apply a threshold of 0.5 unless told otherwise. A macro F1 reported without its library, or its average setting, is therefore ambiguous.

Choosing the Threshold

F1 scores hard decisions. A classifier or segmenter that outputs scores yields a different confusion matrix, and a different F1, at every threshold, and nothing places the best F1 at the conventional threshold of 0.5. Lipton, Elkan and Naryanaswamy (2014, Theorem 1 and Corollary 1) showed that the F1-maximizing rule thresholds the likelihood ratio at a value that depends on the F1 the classifier reaches. In particular, if the scores are calibrated probabilities, the optimal threshold is half the maximum attainable F1: a model that can reach $F_1 = 0.8$ should call a pixel or image positive above probability 0.4. They also found that when the predictive features for rare labels are lost, thresholds tuned to maximize macro F1 over many labels predict those labels very often, because for a label the classifier cannot predict, calling everything positive maximizes its F1.

Tuning the threshold on the test set inflates the score, as in the precision and recall article’s example. Choose it on validation data (see train, validation and test splits) and report the test F1 at that threshold, or report a threshold-free summary such as average precision alongside. A higher $\beta$ moves the best threshold down, toward more positive calls, as the second example shows. The BSDS boundary benchmark reports its F-measure both at one threshold for the whole dataset (ODS) and at each image’s best threshold (OIS); its article explains why both are optimistic.

When to Use It

Use F1 when positives are the subject of interest, true negatives are uninformative or uncountable, and one number is needed at a fixed operating point: rare-class classification, defect and anomaly detection, and the scoring of matched objects. Use $F_\beta$ when the costs of misses and false alarms differ, with $\beta$ chosen from those costs before looking at the results. In vision it appears as:

• the Dice coefficient, the F1 of pixels in a binary mask (below);
• the boundary F-measure of edge detection benchmarks, over matched boundary pixels;
• the recognition quality of panoptic quality, an F1 over matched segments, and IDF1, an F1 over identity-matched detections in tracking;
• the F-score of 3D reconstruction benchmarks such as Tanks and Temples, the harmonic mean of the precision and recall of points within a distance threshold.

Failure Cases

• True negatives ignored. Chicco and Jurman (2020) argued that on imbalanced binary problems F1 and accuracy can be overoptimistic, as in their use case above, and recommended the Matthews correlation coefficient, which uses all four cells.
• Different class ratios. Because the baseline $2b/(1+b)$ moves with prevalence, F1 values from test sets with different class ratios are not comparable.
• Averages that disagree. Micro F1 can rank a classifier that abandons rare classes above one that recognizes them, macro F1 the reverse, and the two macro formulas differ again; the first example shows all three.
• Empty images and absent classes. When many images are empty, the zero_division convention, or a library’s choice to exclude a class, can move an average a lot.
• One threshold, hidden costs. One F1 value describes one operating point, and, read as Hand and Christen’s weighted mean, it weighs precision against recall differently for each classifier.

Variants

Dice coefficient. For a predicted binary mask $A$ and a ground-truth mask $B$, counting pixels as examples gives $\mathrm{TP} = |A \cap B|$, $\mathrm{FP} = |A \setminus B|$ and $\mathrm{FN} = |B \setminus A|$, so

$$F_1 = \frac{2\,|A \cap B|}{|A| + |B|} = D(A, B),$$

the Dice coefficient, and $F_\beta$ over pixels generalizes it to unequal costs. The Dice article covers its segmentation use, its relation to IoU, per-image versus pooled averaging, empty masks and the soft Dice loss.

Example

The first script classifies scikit-learn’s $8 \times 8$ digit images by the nearest class-mean image, trained on 900 images. The test set keeps up to 80 images of each digit 0–6 (80 in the split printed) and only 8 each of 7, 8 and 9. Classifier B uses the same distances but adds a penalty of 3.0 to the rare digits, so it predicts them only when they are clearly closer. The script computes per-class, macro, weighted and micro F1 from the counts, checks them against scikit-learn 1.7.2, computes the F1 of macro precision and recall under two zero_division settings, and repeats the comparison over ten random splits.

import warnings
import numpy as np
from sklearn.datasets import load_digits
from sklearn.metrics import f1_score, precision_recall_fscore_support

warnings.simplefilter("ignore")                  # undefined precisions are handled below
X, y = load_digits(return_X_y=True)              # 1,797 images of 8x8 pixels
X = X / 16.0
penalty = np.where(np.arange(10) < 7, 0.0, 3.0)  # classifier B: rare digits must be clearly closer

def run(seed):
    idx = np.random.default_rng(seed).permutation(len(y))
    tr, pool = idx[:900], idx[900:]
    # Imbalanced test set: up to 80 images each of digits 0-6, only 8 each of 7, 8 and 9.
    te = np.concatenate([pool[y[pool] == k][:80 if k < 7 else 8] for k in range(10)])
    means = np.stack([X[tr][y[tr] == k].mean(0) for k in range(10)])
    d2 = ((X[te][:, None, :] - means[None]) ** 2).sum(-1)   # squared distance to each class mean
    return y[te], {"A": d2.argmin(1), "B": (d2 + penalty).argmin(1)}

def counts(yt, yp, K=10):
    tp = np.array([np.sum((yp == k) & (yt == k)) for k in range(K)])
    fp = np.array([np.sum((yp == k) & (yt != k)) for k in range(K)])
    fn = np.array([np.sum((yp != k) & (yt == k)) for k in range(K)])
    return tp, fp, fn

def scores(yt, yp):
    tp, fp, fn = counts(yt, yp)
    f1 = 2 * tp / (2 * tp + fp + fn)             # per class, one class against the rest
    support = tp + fn
    return dict(per_class=f1, macro=f1.mean(), weighted=(f1 * support).sum() / support.sum(),
                micro=2 * tp.sum() / (2 * tp.sum() + fp.sum() + fn.sum()))

yt, preds = run(0)
print(f"{len(yt)} test images, support {np.bincount(yt).tolist()}")
for name, yp in preds.items():
    s = scores(yt, yp)
    sk = [f1_score(yt, yp, average=a) for a in ("macro", "weighted", "micro")]
    assert np.allclose(sk, [s["macro"], s["weighted"], s["micro"]])
    print(f"{name}: per-class F1 " + " ".join(f"{v:.2f}" for v in s["per_class"]))
    print(f"   macro {s['macro']:.3f}  weighted {s['weighted']:.3f}  micro {s['micro']:.3f}"
          f"  accuracy {np.mean(yp == yt):.3f}")
    for zd in (0.0, 1.0):                        # F1 of the macro-averaged precision and recall
        P, R, _, _ = precision_recall_fscore_support(yt, yp, average="macro", zero_division=zd)
        print(f"   zero_division={zd:.0f}: macro P {P:.3f}, macro R {R:.3f}, "
              f"F1 of these {2 * P * R / (P + R):.3f}")

wins = {"macro": 0, "micro": 0}
for seed in range(10):
    yt, preds = run(seed)
    a, b = scores(yt, preds["A"]), scores(yt, preds["B"])
    for k in wins:
        wins[k] += a[k] > b[k]
print(f"seeds 0-9: A beats B on macro F1 in {wins['macro']}/10, on micro F1 in {wins['micro']}/10")

Output:

584 test images, support [80, 80, 80, 80, 80, 80, 80, 8, 8, 8]
A: per-class F1 0.98 0.84 0.94 0.95 0.94 0.93 0.99 0.67 0.52 0.45
   macro 0.820  weighted 0.922  micro 0.911  accuracy 0.911
   zero_division=0: macro P 0.795, macro R 0.912, F1 of these 0.850
   zero_division=1: macro P 0.795, macro R 0.912, F1 of these 0.850
B: per-class F1 0.97 0.88 0.93 0.92 0.96 0.94 0.99 0.67 0.00 0.00
   macro 0.725  weighted 0.911  micro 0.925  accuracy 0.925
   zero_division=0: macro P 0.748, macro R 0.720, F1 of these 0.734
   zero_division=1: macro P 0.948, macro R 0.720, F1 of these 0.818
seeds 0-9: A beats B on macro F1 in 10/10, on micro F1 in 0/10

Classifier B never predicts an 8 or a 9, so it gives up all 16 of those test images, but it makes fewer errors on the frequent digits, and its micro F1, equal to its accuracy, is higher than A’s. Macro F1 gives each of the two abandoned digits a tenth of the score, so it drops by almost 0.1, and the ten splits show the same opposite rankings every time. Weighted F1, which counts each digit by its support, here favors A. The F1 of macro precision and recall is above the mean of per-class F1 for both classifiers, as Opitz and Burst proved. For B it depends on zero_division: precision is undefined for the two digits B never predicts, and setting it to 1 lifts that score from 0.734 to 0.818, while their per-class F1 stays 0 because they have false negatives.

The second script scores soft-edged synthetic disk masks on $48 \times 48$ images, each with a faint “shadow” region of probability 0.3. Every ground-truth pixel is drawn as foreground with exactly the predicted probability, so the predictions are calibrated by construction. It picks the threshold that maximizes pooled pixel $F_\beta$ on 20 validation images, applies it to 20 test images, checks per-image F1 against the Dice coefficient, and shows zero_division on an empty image.

import warnings
import numpy as np
from sklearn.metrics import f1_score

H = W = 48
yy, xx = np.mgrid[0:H, 0:W] + 0.5

def make_set(rng, n):
    """Foreground probability maps and masks sampled from them (calibrated by construction)."""
    probs, masks = [], []
    for _ in range(n):
        cx, cy, r = rng.uniform(14, 34), rng.uniform(14, 34), rng.uniform(6, 12)
        p = 1 / (1 + np.exp(-(r - np.hypot(xx - cx, yy - cy)) / 1.5))   # soft-edged disk
        sx, sy = rng.uniform(4, 44, size=2)                              # a faint "shadow"
        p = np.maximum(p, 0.3 * (np.hypot(xx - sx, yy - sy) < 5))
        probs.append(p)
        masks.append(rng.random((H, W)) < p)
    return np.array(probs), np.array(masks)

def f_beta(prob, mask, t, beta=1.0):
    pred = prob >= t
    tp = np.sum(pred & mask); fp = np.sum(pred & ~mask); fn = np.sum(~pred & mask)
    b2 = beta ** 2
    return (1 + b2) * tp / ((1 + b2) * tp + b2 * fn + fp)   # pooled over all pixels

rng = np.random.default_rng(0)
val_p, val_m = make_set(rng, 20)
test_p, test_m = make_set(rng, 20)
ts = np.round(np.arange(0.01, 1.00, 0.01), 2)          # candidate thresholds

# 1. Choose the threshold on validation images, then score the test images once.
chosen = {}
for beta in (0.5, 1.0, 2.0):
    fv = [f_beta(val_p, val_m, t, beta) for t in ts]
    ft = [f_beta(test_p, test_m, t, beta) for t in ts]
    chosen[beta] = t = ts[np.argmax(fv)]
    print(f"beta={beta}: validation picks t={t:.2f} (val F {max(fv):.3f}), test F "
          f"{f_beta(test_p, test_m, t, beta):.3f}; best on test {max(ft):.3f} at t={ts[np.argmax(ft)]:.2f}")
print("validation F1 at t = 0.10, 0.30, 0.44, 0.60, 0.80:",
      " ".join(f"{f_beta(val_p, val_m, t):.3f}" for t in (0.10, 0.30, 0.44, 0.60, 0.80)))

# 2. Per-image F1 over pixels is the Dice coefficient.
gap = 0.0
for p, m in zip(test_p, test_m):
    pred = p >= chosen[1.0]
    dice = 2 * np.sum(pred & m) / (pred.sum() + m.sum())
    gap = max(gap, abs(dice - f1_score(m.ravel(), pred.ravel())))
print(f"largest |Dice - sklearn f1_score| over 20 test masks: {gap:.1e}")

# 3. An image with no foreground: F1 is 0/0.
empty, nothing = np.zeros(H * W, bool), np.zeros(H * W, bool)
spurious = nothing.copy(); spurious[:3] = True
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    print(f"empty vs empty: default {f1_score(empty, nothing)} ({w[0].category.__name__}), "
          f"zero_division=1 -> {f1_score(empty, nothing, zero_division=1.0)}, "
          f"nan -> {f1_score(empty, nothing, zero_division=np.nan)}")
print(f"empty vs 3 false pixels: {f1_score(empty, spurious, zero_division=1.0)}")

Output:

beta=0.5: validation picks t=0.64 (val F 0.821), test F 0.808; best on test 0.808 at t=0.64
beta=1.0: validation picks t=0.44 (val F 0.780), test F 0.770; best on test 0.773 at t=0.36
beta=2.0: validation picks t=0.16 (val F 0.839), test F 0.836; best on test 0.836 at t=0.17
validation F1 at t = 0.10, 0.30, 0.44, 0.60, 0.80: 0.697 0.761 0.780 0.753 0.650
largest |Dice - sklearn f1_score| over 20 test masks: 0.0e+00
empty vs empty: default 0.0 (UndefinedMetricWarning), zero_division=1 -> 1.0, nan -> nan
empty vs 3 false pixels: 0.0

The best threshold moves from 0.64 for $F_{0.5}$ to 0.16 for $F_2$: weighting recall more pays for the shadow pixels and the soft rims. For F1, half the best validation score is 0.39, Lipton et al.’s optimum for calibrated scores; the grid searches land on 0.44 on validation and 0.36 on test because the curve is flat near its peak, so the random sampling of the masks moves its maximum. With the validation threshold the test F1 is 0.770, slightly below the 0.773 that picking on the test set itself would report. Per-image F1 equals Dice exactly. An empty prediction on an empty image scores 0, 1 or NaN depending only on the convention, while three false pixels score 0 under every convention.

Related Metrics

Precision and recall are F1’s components; average precision summarizes their curve over all thresholds, and the ROC AUC measures ranking quality without fixing a threshold. Accuracy counts true negatives and equals micro F1 in single-label multi-class classification. On pixels, F1 is the Dice coefficient, and intersection over union, $\mathrm{TP}/(\mathrm{TP} + \mathrm{FP} + \mathrm{FN})$, is the monotonic function $J = F_1/(2 - F_1)$ of it. The Matthews correlation coefficient,

$$\mathrm{MCC} = \frac{\mathrm{TP}\cdot\mathrm{TN} - \mathrm{FP}\cdot\mathrm{FN}}{\sqrt{(\mathrm{TP}+\mathrm{FP})(\mathrm{TP}+\mathrm{FN})(\mathrm{TN}+\mathrm{FP})(\mathrm{TN}+\mathrm{FN})}},$$

uses all four counts and does not change when the classes are swapped (Chicco & Jurman, 2020).