Facial Expression Recognition

Facial expression recognition (FER) assigns a description of facial movement to a face image or video: a category such as “happiness” or “surprise”, the set of facial action units that are active and how intense they are, or continuous values of valence and arousal. The datasets and papers cited here call the task facial expression recognition, facial expression analysis, emotion detection or affective computing from faces. Each benchmark defines its labels in its own way, by the expression a subject was asked to pose, by rules applied to codes from trained coders, or by what annotators perceived in the image, and none of them measures what the person felt. Barrett et al. (2019) reviewed the evidence on how far emotional states can be inferred from facial movements and concluded that such technology “detects facial movements, not emotional expressions”. This article describes the task as the computer vision literature poses it and the limits that literature and that review document. It does not judge what any expression reveals about the person.

A typical system follows the same pipeline as other face analysis: a face detector finds the face, facial landmark detection locates its features and supports alignment, and a classifier or regressor maps landmark geometry, image appearance or both to the output. The task differs from face recognition, which must ignore expression to match identity. For expression, identity is the nuisance factor, which is why subject-independent evaluation matters (see the example below).

Problem Definition

AffectNet’s authors list three ways of describing facial affect that datasets use (Mollahosseini, Hasani and Mahoor, 2019): a categorical model, which picks one label from a list such as the six basic emotions; a dimensional model, which places the face on continuous scales of valence (how positive or negative) and arousal (how calm or excited); and the Facial Action Coding System (FACS), which describes the facial movements themselves as action units and does not describe an affective state directly. A FER system predicts one of these from a still image or a video.

Basic emotion categories

Most categorical datasets use six categories that their papers attribute to Ekman and colleagues (anger, disgust, fear, happiness, sadness and surprise), often with neutral and sometimes with contempt. CK+ uses seven: anger, contempt, disgust, fear, happiness, sadness and surprise (Lucey et al., 2010). FER2013 and RAF-DB’s basic subset use the six plus neutral, and AffectNet uses eight (the six, neutral and contempt) inside a larger annotation scheme (Mollahosseini, Hasani and Mahoor, 2019; Li, Deng and Du, 2017). The DISFA authors describe such labels as “message-based” measurement, in Ekman’s terms: the implied meaning of the expression is labelled. They note that these expressions occur relatively rarely in their prototypic forms and can wrongly imply emotional states that are absent (Mavadati et al., 2013). Ekman and Friesen’s original papers were not read for this article, and the history here follows these dataset papers and Barrett et al. (2019).

Several datasets go beyond single categories. RAF-DB’s crowdsourced labels showed that many internet images received two dominant labels. Its authors formed a compound subset with twelve classes, such as “happily surprised” and “sadly fearful”, and dropped one rare class for an eleven-class benchmark (Li, Deng and Du, 2017). AffectNet adds None, Uncertain and Non-face categories to its eight and annotates valence and arousal for every image except those labelled Uncertain or Non-face (Mollahosseini, Hasani and Mahoor, 2019).

Facial action units

FACS, developed by Ekman and Friesen in the 1970s, describes nearly all visible facial movements as action units (AUs). Each AU rests on one or a few facial muscles: AU12, the lip corner puller, codes contraction of the zygomatic major muscle, and AU25 codes the lips parting (Mavadati et al., 2013). The 1978 edition had 44 AUs plus action descriptors. A 2002 revision reduced the number to slightly more than 30, revised the scoring criteria and widened intensity coding from three levels to five, written A (trace) to E (maximum) (Mavadati et al., 2013). Kanade, Cohn and Tian (2000) list the 44 AUs, 30 of them with their muscles, and already describe a five-point intensity scale, so the two sources differ on when the five levels arrived. FACS is purely descriptive. Emotion-specified expressions are not part of it and are coded in separate systems such as EMFACS, which maps combinations of AUs to emotion labels (Kanade, Cohn and Tian, 2000). AffectNet’s authors give the EMFACS example that AU6 with AU12 is taken as a sign of happiness. Barrett et al. (2019) make the same point: FACS is agnostic about whether movements express emotions or any other mental event.

Coding is slow and needs trained people. FACS coders must pass a standardized test (Kanade, Cohn and Tian, 2000). An hour or more of coding can be needed for each minute of video just to mark the onset and offset of each AU (Mavadati et al., 2013), and expert coders reach inter-rater reliabilities near 0.80 for individual AUs (Barrett et al., 2019, citing Jeni, Cohn and De la Torre). This is why AU datasets are small. They are also the most carefully labelled data in the field.

Input and Output

The input is a face image or a video, usually cropped and aligned from detector boxes and landmarks. The outputs by formulation are:

• Category: a probability vector over the $K$ classes listed above, scored as image classification.
• Compound or distribution labels: a label vector, as in RAF-DB’s seven-dimensional vote distribution (Li, Deng and Du, 2017).
• Action units: a multi-label output, one presence probability per AU, or an intensity on the 0–5 scale per AU and frame, as in DISFA (Mavadati et al., 2013).
• Valence and arousal: two real numbers inside the unit circle, as in AffectNet’s annotation tool, which kept values within the circumplex (Mollahosseini, Hasani and Mahoor, 2019).

For sequences, CK+ gives each video from a neutral onset frame to the peak expression and codes the peak (Lucey et al., 2010). DISFA codes AU intensity on every frame (Mavadati et al., 2013).

Variants

• Static against dynamic. FER2013, AffectNet and RAF-DB are still images collected from the web. CK+ and DISFA are video. Kanade, Cohn and Tian (2000) stressed that pattern recognition methods that depend on the timing of appearance change, such as hidden Markov models, need data whose timing is realistic.
• AU detection and AU intensity. Most AU work detects presence; DISFA codes graded intensity (Mavadati et al., 2013).
• Lab against in-the-wild. Lab datasets have frontal, well-lit faces of a few dozen to a few hundred people. Web datasets have thousands of identities and uncontrolled pose, lighting and occlusion, with labels from annotators rather than from the recording protocol.

Posed and Spontaneous Expressions

Early databases asked people to perform expressions. In the original Cohn-Kanade recordings, an experimenter instructed 210 adults to perform 23 facial displays, single AUs and combinations, each beginning and ending at a neutral face (Lucey et al., 2010). Kanade, Cohn and Tian (2000) warned that such directed facial actions may differ in appearance and timing from spontaneous behaviour. Citing earlier work, they wrote that deliberate and spontaneous facial actions are controlled by different motor pathways, so deliberate actions are often less finely controlled and less symmetric. Some spontaneous actions, such as lowering the lip corners (AU15) or raising and drawing together the inner brows (AU1+4), are hard for untrained people to produce on request, and their database lacked AU13, which most people find difficult to perform voluntarily. They concluded that a database without both deliberate and spontaneous actions would probably not suffice for methods that must handle the difference.

The DISFA authors repeat the point: posed expressions can differ markedly in configuration, intensity and timing from spontaneous ones, and spontaneous behaviour is less intense and harder to annotate reliably (Mavadati et al., 2013). DISFA was recorded to supply such data. Twenty-seven adults watched a four-minute video montage alone, in front of a stereo camera. CK+ added a small non-posed subset: 122 smiles from 66 subjects that occurred between tasks without a request, which groups of observers classified as amused, polite or embarrassed/nervous (Lucey et al., 2010).

Web images are often called spontaneous or “in the wild”, but they are not recordings of people in known situations. They are photographs that search engines returned for emotion keywords, labelled by what annotators perceived. RAF-DB’s cross-database study shows how far the two kinds of data differ. An SVM on HOG features trained on RAF-DB and tested on CK+ reached a mean diagonal of 62% on six classes, while training on CK+ and testing on RAF-DB gave 39%. The AUs that FACS coders found in RAF-DB images were more diverse than, and sometimes deviated from, those of the lab-controlled CK+ (Li, Deng and Du, 2017). Barrett et al. (2019) add that posed and spontaneous movements differ, sometimes substantially, so posed faces may record people’s beliefs about expressions rather than how they move their faces.

Challenges

• Identity confounds expression. Differences in face shape and appearance between people can be larger than the changes an expression makes; the example below builds a synthetic set in which they are. CK+’s baseline subtracts each sequence’s neutral first frame from its shape features (Lucey et al., 2010), which removes the subject’s resting shape.
• Subtle and ambiguous classes. In CK+, fear, sadness and contempt were the hardest classes. The authors call contempt subtle and easily confused with stronger expressions (Lucey et al., 2010). In AffectNet, two annotators agreed on the category of 60.7% of 36,000 doubly annotated images. Agreement was highest for happiness and non-faces and lowest for the None category, and the valence and arousal labels of the two annotators differed by a root mean square of 0.34 and 0.36 (Table 5; a later passage of the text gives 0.367 and 0.481; Mollahosseini, Hasani and Mahoor, 2019).
• Imbalance. Search engines return mostly happy and neutral faces. In AffectNet, queries for disgust, fear and contempt yielded those labels for only 2.7%, 4% and 2.4% of the images (Table 4 of the authors’ copy). Its manual labels range from 146,198 happy faces to 5,135 contempt faces (Mollahosseini, Hasani and Mahoor, 2019).
• Label noise. Single annotators make mistakes and disagree on hard cases; SCN, below, is built for this (Wang et al., 2020).
• Pose, occlusion and resolution. FER2013’s $48 \times 48$ greyscale faces are not registered, and the AffectNet authors report that most landmark detectors fail at that resolution and quality.
• Few subjects in AU data. DISFA has 27 people. The CK+ authors estimated that a robust system would need 5,000 to 10,000 reliably coded examples per action (Lucey et al., 2010).

Approaches

Two families of hand-designed features came first: geometric features computed from landmarks and appearance features computed from the pixels of the aligned face. Learned convolutional networks replaced both once large web datasets existed. The RAF-DB authors divide the earlier literature into appearance-based and AU-based methods, the latter recognizing an expression by first detecting AUs (Li, Deng and Du, 2017).

Landmark geometry

A geometric method describes the face by the positions of its landmarks after removing translation, rotation and scale. The CK+ baseline tracked 68 vertices with an active appearance model, a joint linear model of shape and appearance (see facial landmark detection). Its similarity-normalized shape feature (SPTS) is the 136 coordinates of those vertices with the rigid similarity removed and the neutral first frame subtracted (Lucey et al., 2010). DISFA ships 66 AAM landmarks per frame for the same purpose (Mavadati et al., 2013), and RAF-DB provides five manually placed and 37 automatically detected landmarks (Li, Deng and Du, 2017). In our summary, geometry is compact and largely unaffected by lighting and skin tone. It misses movements that change texture more than contour, and in low-resolution web images the landmarks themselves are unreliable.

Appearance features

Appearance methods warp the face to a canonical frame and describe its texture. CK+’s canonical appearance feature (CAPP) warps each triangle of the tracked mesh onto the base shape, giving an $87 \times 93$ greyscale image with the non-rigid shape variation removed (Lucey et al., 2010). Texture descriptors computed on aligned faces were the standard baselines into the late 2010s:

• RAF-DB’s baselines use local binary pattern histograms on $10 \times 10$ cells, a 4,000-dimensional HOG vector and a bank of 40 Gabor filters, each with a multiclass SVM (Li, Deng and Du, 2017).
• DISFA’s intensity benchmark computes the same three descriptors in patches around the 66 landmarks (Mavadati et al., 2013).
• AffectNet’s conventional baseline is HOG with PCA and a linear SVM (Mollahosseini, Hasani and Mahoor, 2019).

The CK+ results show the two feature types to be complementary. With a linear SVM per AU under leave-one-subject-out evaluation, shape features did better for AUs 1, 2, 4, 23, 25 and 27 and appearance for AUs 5, 6, 7, 9, 11, 12, 15, 17, 20, 24 and 26, some by small margins. The authors explain that the brow movements of AUs 1, 2 and 4 lie on the mesh, while AUs 6, 9 and 11 change texture through wrinkles more than contour. The weighted average area under the ROC curve was 90.0 for shape, 91.4 for appearance and 94.5 for the calibrated fusion of both (Table 4). For the seven emotion classes, contempt was recognized in about 22–25% of cases with either feature alone and in 84.4% with fusion (Tables 5–7; Lucey et al., 2010).

Deep networks

A convolutional network learns its features from the aligned crop. In the ICML 2013 representation-learning contest that produced FER2013, the winner combined a deep convolutional network with an SVM (Li, Deng and Du, 2017, citing the contest’s results). The AffectNet baselines are AlexNet classifiers trained on $224 \times 224$ crops of unregistered faces. The authors compared four answers to class imbalance: training on the imbalanced data, down-sampling, up-sampling, and a cross-entropy loss with class weights (Mollahosseini, Hasani and Mahoor, 2019). Up-sampling, which replicated contempt and disgust samples almost twenty times, overfitted those classes. The weighted loss gave the best skew-normalized scores.

Two lines of work address the label problems specific to FER:

• Feature geometry for ambiguous classes. RAF-DB’s DLP-CNN adds a locality-preserving loss to the softmax loss. The loss pulls each training feature toward the mean of its $k$ nearest neighbours of the same class, $L_{lp} = \tfrac{1}{2}\sum_i \lVert x_i - \tfrac{1}{k}\sum_{x \in N_k(x_i)} x \rVert_2^2$. Unlike center loss, which the authors show is a special case with $k$ equal to the class size minus one, it allows a class to have several clusters, as compound expressions do. They trained the network from scratch on RAF-DB, without face-recognition pretraining, and classified its 2,000-dimensional features with an SVM (Li, Deng and Du, 2017).
• Uncertain labels. The Self-Cure Network (SCN) of Wang et al. (2020) learns an importance weight for each image with a self-attention module, regularizes the weights so that a high-importance group of each batch stays above a low-importance group by a margin, and relabels low-importance samples when the network is confident in another class. Its backbone is a ResNet-18 pretrained on the MS-Celeb-1M face recognition dataset, so it uses transfer learning from identity features.

Action unit detection

AU detection is a multi-label problem over local regions whose AUs are correlated. DRML (Zhao, Chu and Zhang, 2016) adds a region layer, a design between locally connected and ordinary convolutional layers in which each facial region learns its own filters, and trains all AUs jointly with a multi-label loss. JAA-Net (Shao et al., 2018) predicts landmarks and AUs in one network and uses the predicted landmarks to place attention maps on each AU’s region of interest. The DISFA baseline itself is non-deep: patch descriptors, Laplacian eigenmap dimensionality reduction and an RBF SVM for each AU’s six intensity levels (Mavadati et al., 2013).

Reported results

The numbers below are not comparable across rows. They differ in classes, splits, metric and training data, as the protocol columns show.

Dataset	Protocol	Metric	Method	Result	Source
CK+ (posed, 7 classes, 327 sequences)	Leave-one-subject-out, neutral frame subtracted	Per-class hit rate	AAM shape + appearance, linear SVM (fused)	65.2% (fear) to 100% (happiness)	Lucey et al. (2010), Table 7
RAF-DB basic (7 classes)	Official train/test split (5:1)	Mean diagonal of confusion matrix	Gabor + SVM	65.12% (77.28% accuracy)	Li, Deng and Du (2017)
RAF-DB basic	Same	Mean diagonal	DLP-CNN features + SVM	74.20%	Li, Deng and Du (2017), Table 4
RAF-DB basic	12,271 train / 3,068 test	Overall accuracy	SCN, ResNet-18 (+ AffectNet training data)	87.03% (88.14%)	Wang et al. (2020), Table 7
AffectNet (8 classes)	Authors’ test set (doubly annotated images)	Accuracy, original / skew-normalized	AlexNet, weighted loss	0.64 / 0.63	Mollahosseini, Hasani and Mahoor (2019), Table 9
AffectNet (8 classes)	Validation set (500 per class)	Mean class accuracy	SCN, ResNet-18, oversampling	60.23%	Wang et al. (2020), Table 7
DISFA (12 AUs, intensity 0–5)	Leave-one-subject-out	ICC / accuracy	Gabor + SVM	0.77 / 85.71%	Mavadati et al. (2013), Table 4
DISFA (8 AUs, presence)	Trained on BP4D, tested on DISFA	Average F1-frame / AUC	DRML	26.7 / 52.3	Zhao, Chu and Zhang (2016), Table 3
DISFA (8 AUs, presence)	Subject-exclusive 3-fold, initialized from BP4D	Average F1-frame	JAA-Net	56.0	Shao et al. (2018), Table 2

Two of these rows show how protocols get mixed when papers compare methods.

• DLP-CNN on RAF-DB. SCN’s comparison table lists DLP-CNN at 84.22% overall accuracy. The RAF-DB paper itself reports DLP-CNN only as a 74.20% mean diagonal, the metric it chose because the classes are imbalanced (Li, Deng and Du, 2017; Wang et al., 2020).
• DRML on DISFA. JAA-Net’s DISFA table puts DRML’s results, which came from a model trained on BP4D, beside its own three-fold results on DISFA. The column it heads “accuracy” for the earlier methods repeats the values that the DRML paper reports as AUC (Shao et al., 2018; Zhao, Chu and Zhang, 2016).

SCN’s table also lists AffectNet’s weighted-loss baseline at 58.00, the validation-set accuracy in an appendix table of the AffectNet authors’ copy (Table 15), not the 0.64 test-set accuracy of its Table 9.

Datasets and Benchmarks

Dataset	Content	Labels	Subjects	Protocol
CK+ (2010)	593 posed sequences from neutral to peak; 122 non-posed smiles	FACS on peak frames; 7 validated emotion labels on 327 sequences	123	Leave-one-subject-out; ROC area for AUs, confusion matrix for emotions
FER2013 (2013)	35,887 web images, $48 \times 48$ greyscale	6 basic + neutral	Not reported in the sources read	28,709 / 3,589 / 3,589 (train / validation / test), as SCN reports it for FERPlus
AffectNet (2017–19)	Over 1 million web faces, about 450,000 annotated by hand	8 expressions + None, Uncertain, Non-face; valence and arousal	Not given (web images)	Imbalanced training set, balanced validation set of 500 per class, authors’ test set
RAF-DB (2017)	29,672 web images	~40 crowd votes per image; 7 basic (15,339 images), 11 compound (3,954)	Not given (web images)	Train/test about 5:1 with matched class distributions; mean diagonal
DISFA (2013)	27 spontaneous videos, 4,845 frames each (about 130,000 frames)	Intensity 0–5 of 12 AUs per frame; 66 landmarks	27	Leave-one-subject-out (original); 8 AUs with subject-exclusive 3 folds (later work)

CK+. The Extended Cohn-Kanade dataset grew from the Cohn-Kanade database of Kanade, Cohn and Tian (2000). That database was distributed as 486 FACS-coded sequences from 97 subjects. CK+ has 593 sequences from 123 subjects, each 10 to 60 frames from the neutral onset to the peak. The peak frames are fully FACS coded; about 15% were coded by a second certified coder, with mean kappas of 0.82 at the apex and 0.75 frame by frame (Lucey et al., 2010). CK+’s main correction concerns the emotion labels. The original labels described the expression that subjects had been asked to perform, and these performances often differed from the stereotypical FACS definitions. CK+ therefore relabelled every sequence in three steps: a strict match of the FACS codes against the FACS manual’s Emotion Prediction Table, a looser check of extra or missing AUs against criteria such as “AU12 must be present” for happiness (Table 2), and a perceptual check of the whole clip. Only 327 of the 593 sequences met the criteria for one of seven classes, ranging from 18 for contempt to 83 for surprise (Table 3).

The paper proposes leave-one-subject-out cross-validation (123 folds for AUs, 118 for emotions), area under the ROC curve for AUs, and confusion matrices for emotions. It describes the recording pool of 210 adults as aged 18 to 50, 69% female, 81% Euro-American, 13% Afro-American and 6% other groups. It gives no separate breakdown for the 123 CK+ subjects. The paper is not internally consistent here: the row totals implied by the percentages in its emotion confusion matrices (for example multiples of 1/32 for contempt) do not match the class counts of its Table 3, and the fear row is identical in the shape and appearance confusion matrices (Tables 5 and 6). As of November 2026, access is requested through a University of Pittsburgh portal. The portal records the request and the acceptance of an end-user licence agreement, sends it to an authorizer for review, and emails approved requesters a single-use, password-protected link valid for seven days (CK+ request portal).

FER2013. FER2013 was introduced for a facial expression contest in the Challenges in Representation Learning workshop at ICML 2013. The contest’s report, by Goodfellow and co-authors, could not be read for this article: the publisher, arXiv and the journal version are on sites this project does not use, and the Kaggle contest page returned only a script shell. The description here therefore comes from later papers. Images were found with the Google image search API using 184 emotion-related keywords, resized to $48 \times 48$ pixels and converted to greyscale. Human labellers rejected wrongly labelled images, corrected crops and removed some duplicates, leaving 35,887 images, among them 547 disgust images (Mollahosseini, Hasani and Mahoor, 2019; Li, Deng and Du, 2017). SCN uses the split of 28,709 training, 3,589 validation and 3,589 test images for FERPlus, which it describes as extended from FER2013 with eight classes including contempt (Wang et al., 2020).

AffectNet. Mollahosseini, Hasani and Mahoor queried Google, Bing and Yahoo with 1,250 emotion-related terms in English, Spanish, Portuguese, German, Arabic and Farsi and kept over one million images with detected faces and landmarks. Twelve annotators, hired and trained at the University of Denver rather than recruited through crowdsourcing, labelled about 450,000 images, each by one annotator. The category counts in the paper’s Table 3 sum to 456,349, and the dataset page says about 440,000. Of the queried terms, only happiness returned its own label for a large share of images (48.9%). About 15% of query results were neutral and about 15% non-faces. The test set is the 36,000 doubly annotated images. Disagreements were resolved in favour of the queried emotion where one annotator chose it, which happened for 29.5% of the disagreements, and otherwise at random. It is heavily imbalanced, with more than 11,000 happy and only about 1,000 contempt images. The validation set has 500 random images per class, and the training set is the rest.

The paper estimates the composition of the faces with a commercial face API on 50,000 random images: 49% men, mean estimated age 33.0 years (standard deviation 17.0). These are model estimates, not annotations. As of November 2026, the dataset page states that AffectNet is released for research purposes only, as part of AffectNet+, a later extension with soft labels, and that users must cite the paper (AffectNet page).

RAF-DB. Li, Deng and Du collected 29,672 images from Flickr with emotion keywords. 315 annotators, students and staff given a one-hour tutorial, labelled them, and each image received about 40 independent labels. An expectation–maximization algorithm estimated each annotator’s reliability. The labels of the 285 annotators who were kept had a Cronbach’s alpha of 0.966. Each image’s label is a reliability-weighted seven-dimensional distribution, and an image belongs to the single-label (basic) subset when only one class exceeds the mean of its distribution. The authors manually annotated sex, five age ranges and race for every face: ages 0 to 70; 52% female, 43% male and 5% unsure; 77% Caucasian, 8% African-American and 15% Asian (Li, Deng and Du, 2017).

The training set is five times the size of the test set, with nearly identical class distributions. The paper’s metric is the mean diagonal of the confusion matrix, not overall accuracy, because the classes are imbalanced: happiness makes up 38.84% of the basic subset and fear 2.31%. Later work reports overall accuracy on the 12,271/3,068 split (Wang et al., 2020). As of November 2026, the database page restricts RAF-DB to non-commercial research. It forbids redistribution or copying beyond internal use at a single site, says that the images come from the Internet and are not the laboratory’s property, and gives the password on request from a university email address (RAF-DB page).

DISFA. Twenty-seven adults (12 women and 15 men, aged 18 to 50; three Asian, 21 Euro-American, two Hispanic and one African-American) watched a 242-second montage of nine video segments, mostly from YouTube. A $1024 \times 768$ stereo camera recorded them at 20 frames per second (Mavadati et al., 2013). One certified coder scored the intensity of 12 AUs (1, 2, 4, 5, 6, 9, 12, 15, 17, 20, 25 and 26) on a 0–5 scale for every frame. A second coder annotated ten participants, and the intra-class correlation between coders ranged from 0.80 to 0.94. All participants consented to distribution of their video for non-commercial research, and 25 of the 27 to publication of their images. The original benchmark is leave-one-subject-out intensity estimation, scored by accuracy and by ICC(3,1), the consistency form of the intra-class correlation:

$$\mathrm{ICC}(3,1) = \frac{\mathrm{BMS} - \mathrm{EMS}}{\mathrm{BMS} + (k-1)\,\mathrm{EMS}},$$

with between-target mean square $\mathrm{BMS}$, residual mean square $\mathrm{EMS}$ and $k$ judges. Unlike accuracy, ICC penalizes predicting 1 for a true 4 more than predicting 5.

Later AU detection work changed the task to presence detection of 8 of the 12 AUs. Frames with intensity of at least 2 count as positive in JAA-Net, and frames of intensity C or higher, which is 3 on DISFA’s numeric scale, in DRML, so even the positive class differs between papers. Training uses subject-exclusive three-fold cross-validation, often starting from a model trained on BP4D (Shao et al., 2018). DRML’s text describes DISFA as providing 8 AU annotations, although the dataset codes 12 (Zhao, Chu and Zhang, 2016; Mavadati et al., 2013). As of November 2026, the DISFA page offers the data for research purposes through an agreement form (DISFA page).

Evaluation Metrics

• Accuracy and mean class accuracy. Overall accuracy is dominated by happy and neutral faces on web datasets. RAF-DB’s mean diagonal and SCN’s mean class accuracy on AffectNet weight each class equally; the two figures for RAF-DB’s Gabor baseline in the table above show how much the choice matters.
• Skew normalization. AffectNet’s authors report every metric both on the imbalanced test set and averaged over 200 random class-balanced subsamples of it. Following Jeni, Cohn and De la Torre, they note that every metric they studied except the ROC area depends on the skew of the test set (Mollahosseini, Hasani and Mahoor, 2019).
• F1 and ROC area for AUs. AU detection reports per-AU F1 (F1-frame) and ROC AUC, averaged over AUs. The two can rank methods differently: in DRML’s DISFA table, LCN has the highest F1 for AU25 (46.2) and APL the highest AUC (50.4), while DRML has the highest average of both (Zhao, Chu and Zhang, 2016).
• Area under the ROC curve with an uncertainty bound. CK+ asks for the ROC area for each AU, averaged across the leave-one-subject-out folds, with an upper bound on its uncertainty, $\sqrt{A'(1-A')/\min(n_p, n_n)}$ for $n_p$ positive and $n_n$ negative examples (Lucey et al., 2010).
• ICC for AU intensity, as above, and Cohen’s kappa or Krippendorff’s alpha for agreement with human labels (Mollahosseini, Hasani and Mahoor, 2019).
• Regression errors for valence and arousal: RMSE, correlation, sign agreement and the concordance correlation coefficient (Mollahosseini, Hasani and Mahoor, 2019).

Barrett et al. (2019) recommend translating “accuracy” into “agreement”, “consensus” or “reliability” (their Table 6). In FER the reference is a requested pose, a FACS-based rule or an annotator’s judgement.

Example: Subject-Dependent and Subject-Independent Splits

CK+ proposed leave-one-subject-out evaluation because earlier papers had used random train/test splits of the same database, which made results hard to compare (Lucey et al., 2010). The example measures what a random split does when frames of one person, and of one sequence, can land on both sides. It builds synthetic faces as 20 landmarks: brows, eye outlines, nose tip, four mouth points and chin. Each of 30 subjects has a structured identity shape and a skin tone, hair band, marks and background for rendering. Each subject has one sequence for each of six classes, with four frames near the peak and one neutral frame. A class is a displacement of the landmarks that loosely follows the AU combinations of CK+’s Table 2 (lip corners up and out for the happiness label, brows up and jaw dropped for surprise, and so on). Each subject scales the displacement by a personal gain and adds a personal style, so the same subject’s frames of one class share a pattern that other subjects do not. These displacements are invented for the example and make no claim about real faces. Frames receive landmark jitter, as from a detector, and a random rotation, scale and shift, and are rendered into $64 \times 64$ images.

Three representations are compared:

• G-raw: landmarks aligned to the mean shape by a similarity transform, as in CK+’s SPTS without the neutral frame subtracted.
• G-neutral: the same minus the subject’s aligned neutral frame, CK+’s neutral-frame normalization.
• A-HOG: HOG on the image warped to a canonical frame from the noisy eye centres and nose tip.

Two classifiers are used: multinomial logistic regression and a 1-nearest-neighbour rule. Three five-fold protocols are compared. The frame-level protocol splits frames at random with stratification. The sequence-held-out protocol keeps each sequence in one fold, though its subject appears in training. The subject-independent protocol keeps each subject in one fold, which is grouped cross-validation by subject. Neutral subtraction and the choice of representation are open modelling choices, so the code prints every combination. It loops over five seeds and prints the mean and range.

import numpy as np
from skimage.draw import ellipse, line_aa, polygon, polygon_perimeter, disk
from skimage.transform import SimilarityTransform, warp
from skimage.feature import hog
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import StratifiedKFold, GroupKFold

CLASSES = ["anger", "disgust", "fear", "happiness", "sadness", "surprise"]
# 20 landmarks in a canonical frame (x right, y down, eye centres at (-0.5, 0) and (0.5, 0))
MEAN = np.array([
    [-0.75, -0.35], [-0.50, -0.42], [-0.22, -0.36],          # 0-2 brow (outer, mid, inner)
    [0.22, -0.36], [0.50, -0.42], [0.75, -0.35],             # 3-5 brow (inner, mid, outer)
    [-0.72, 0.0], [-0.50, -0.08], [-0.28, 0.0], [-0.50, 0.07],  # 6-9 eye: outer, top, inner, bottom
    [0.28, 0.0], [0.50, -0.08], [0.72, 0.0], [0.50, 0.07],   # 10-13 eye: inner, top, outer, bottom
    [0.0, 0.55],                                             # 14 nose tip
    [-0.38, 0.95], [0.0, 0.88], [0.38, 0.95], [0.0, 1.02],   # 15-18 mouth: corner, top, corner, bottom
    [0.0, 1.45]])                                            # 19 chin
L = len(MEAN)

def proto(moves):
    d = np.zeros((L, 2))
    for idx, v in moves.items():
        d[idx] = v
    return d

# Prototype displacements at full intensity, loosely after the AU combinations of CK+ Table 2
PROTO = np.stack([
    proto({0: (0, .03), 5: (0, .03), 1: (0, .06), 4: (0, .06), 2: (.04, .08), 3: (-.04, .08),       # anger: AU4,
           9: (0, -.03), 13: (0, -.03), 16: (0, .03), 18: (0, -.03), 15: (.03, 0), 17: (-.03, 0)}),  # AU7, AU23/24
    proto({16: (0, -.08), 14: (0, -.03), 15: (.02, -.02), 17: (-.02, -.02), 2: (.02, .04),          # disgust: AU9/10
           3: (-.02, .04), 9: (0, -.03), 13: (0, -.03), 18: (0, .01)}),
    proto({0: (0, -.08), 1: (0, -.08), 4: (0, -.08), 5: (0, -.08), 2: (.03, -.07), 3: (-.03, -.07),  # fear: AU1+2+4,
           7: (0, -.04), 11: (0, -.04), 15: (-.07, .02), 17: (.07, .02), 18: (0, .06), 19: (0, .04)}),  # AU5, AU20
    proto({15: (-.07, -.09), 17: (.07, -.09), 16: (0, -.02), 18: (0, .03), 9: (0, -.03), 13: (0, -.03)}),  # happy: AU6+12
    proto({2: (.02, -.07), 3: (-.02, -.07), 1: (0, -.02), 4: (0, -.02), 15: (.01, .07),             # sadness: AU1+4+15
           17: (-.01, .07), 18: (0, .02), 7: (0, .02), 11: (0, .02)}),
    proto({0: (0, -.12), 1: (0, -.12), 2: (0, -.12), 3: (0, -.12), 4: (0, -.12), 5: (0, -.12),      # surprise: AU1+2,
           7: (0, -.04), 11: (0, -.04), 18: (0, .18), 19: (0, .15), 15: (.03, .05), 17: (-.03, .05),  # AU5, AU26
           16: (0, .02)}),
])
N_SUBJ, FRAMES = 30, (0.55, 0.70, 0.85, 1.00)
OUT = 48
CANON_PTS = np.array([[-0.5, 0.0], [0.5, 0.0], [0.0, 0.55]]) * 16 + [24, 18]   # eye centres, nose tip in 48x48


def identity_shape(rng):
    s = MEAN.copy()
    s[:, 0] *= rng.normal(1, 0.06)                          # face width
    s[s[:, 1] > 0.3, 1] *= rng.normal(1, 0.06)              # lower-face height
    s[0:6, 1] += rng.normal(0, 0.04)                        # brow height
    e = rng.normal(0, 0.04)
    s[6:10, 0] -= e; s[10:14, 0] += e                       # eye spacing
    s[14, 1] += rng.normal(0, 0.05)                         # nose length
    m = rng.normal(0, 0.05)
    s[15, 0] -= m; s[17, 0] += m                            # mouth width
    s[15:19, 1] += rng.normal(0, 0.04)                      # mouth position
    return s + rng.normal(0, 0.03, s.shape)


def render(pts_img, subj, rng):
    img = np.full((64, 64), subj["bg"])
    c = pts_img
    centre = (c[6] + c[12]) / 2 + (c[19] - (c[6] + c[12]) / 2) * 0.35
    iod = np.linalg.norm(((c[10] + c[12]) / 2) - ((c[6] + c[8]) / 2))
    ang = np.arctan2(c[12, 1] - c[6, 1], c[12, 0] - c[6, 0])
    rr, cc = ellipse(centre[1], centre[0], 1.15 * iod * subj["fh"], 0.95 * iod * subj["fw"],
                     shape=img.shape, rotation=-ang)
    img[rr, cc] = subj["skin"]
    top = (rr < centre[1] - 0.75 * iod)
    img[rr[top], cc[top]] = subj["hair"]                      # hair band
    for mx, my in subj["marks"]:                              # subject-specific marks
        p = centre + iod * np.array([mx * np.cos(ang) - my * np.sin(ang), mx * np.sin(ang) + my * np.cos(ang)])
        r_, c_ = disk((p[1], p[0]), 1.0, shape=img.shape)
        img[r_, c_] = subj["skin"] - 0.25
    for a, b in [(0, 1), (1, 2), (3, 4), (4, 5)]:             # brows
        r_, c_, v = line_aa(*np.round(c[a, ::-1]).astype(int), *np.round(c[b, ::-1]).astype(int))
        ok = (r_ >= 0) & (r_ < 64) & (c_ >= 0) & (c_ < 64)
        img[r_[ok], c_[ok]] = np.minimum(img[r_[ok], c_[ok]], subj["skin"] - 0.4 * v[ok])
    for eye in (c[6:10], c[10:14]):
        r_, c_ = polygon_perimeter(eye[:, 1], eye[:, 0], shape=img.shape)
        img[r_, c_] = subj["skin"] - 0.35
        r_, c_ = disk((eye[:, 1].mean(), eye[:, 0].mean()), 1.2, shape=img.shape)
        img[r_, c_] = 0.05
    r_, c_ = disk((c[14, 1], c[14, 0]), 1.0, shape=img.shape)
    img[r_, c_] = subj["skin"] - 0.2
    mouth = c[[15, 16, 17, 18]]
    r_, c_ = polygon(mouth[:, 1], mouth[:, 0], shape=img.shape)
    img[r_, c_] = subj["skin"] - 0.3
    r_, c_ = polygon_perimeter(mouth[:, 1], mouth[:, 0], shape=img.shape)
    img[r_, c_] = subj["skin"] - 0.45
    return np.clip(img + rng.normal(0, 0.03, img.shape), 0, 1)


def to_image(shape, rng):
    th = np.deg2rad(rng.uniform(-10, 10)); s = rng.uniform(18, 22)
    R = np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
    return s * shape @ R.T + np.array([32, 24]) + rng.uniform(-3, 3, 2)


def procrustes(pts):
    t = SimilarityTransform(); t.estimate(pts, MEAN)
    return t(pts)


def appearance(img, pts):
    src = np.array([pts[[6, 8]].mean(0), pts[[10, 12]].mean(0), pts[14]])
    t = SimilarityTransform(); t.estimate(CANON_PTS, src)      # output (canonical) -> input image
    can = warp(img, t, output_shape=(OUT, OUT), mode="edge")
    return hog(can, orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2))


def make_data(seed):
    rng = np.random.default_rng(seed)
    Xg, Xn, Xa, y, subj_id, seq_id = [], [], [], [], [], []
    ident_norm, expr_norm = [], []
    for s in range(N_SUBJ):
        ident = identity_shape(rng)
        subj = dict(bg=rng.uniform(0.1, 0.35), skin=rng.uniform(0.5, 0.85), hair=rng.uniform(0.05, 0.4),
                    fw=rng.normal(1, 0.06), fh=rng.normal(1, 0.06),
                    marks=rng.uniform([-0.8, 0.2], [0.8, 1.0], size=(3, 2)))
        gain = rng.uniform(0.6, 1.4)
        ident_norm.append(np.linalg.norm(ident - MEAN))
        neutral = procrustes(to_image(ident + rng.normal(0, 0.01, MEAN.shape), rng))
        for k in range(len(CLASSES)):
            disp = gain * PROTO[k] + rng.normal(0, 0.03, MEAN.shape)   # subject's own style for this class
            expr_norm.append(np.linalg.norm(PROTO[k]))
            for inten in FRAMES:
                shape = ident + inten * disp + rng.normal(0, 0.01, MEAN.shape)
                pts = to_image(shape, rng)
                img = render(pts, subj, rng)
                g = procrustes(pts)
                Xg.append(g.ravel()); Xn.append((g - neutral).ravel()); Xa.append(appearance(img, pts))
                y.append(k); subj_id.append(s); seq_id.append(s * len(CLASSES) + k)
    return (dict(zip(["G-raw", "G-neutral", "A-HOG"], map(np.array, (Xg, Xn, Xa)))),
            np.array(y), np.array(subj_id), np.array(seq_id), np.mean(ident_norm), np.mean(expr_norm))


def splits(protocol, y, subj, seq, seed):
    if protocol == "frame-level":
        return StratifiedKFold(5, shuffle=True, random_state=seed).split(y, y)
    if protocol == "sequence-held-out":
        return GroupKFold(5).split(y, y, seq)
    return GroupKFold(5).split(y, y, subj)


CLFS = {"linear": lambda: make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=2000)),
        "1-NN": lambda: make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1))}
PROTOCOLS = ["frame-level", "sequence-held-out", "subject-independent"]
SEEDS = range(5)
res = {}
for seed in SEEDS:
    X, y, subj, seq, inorm, enorm = make_data(seed)
    if seed == 0:
        # label-free check: nearest noise-free prototype (full intensity, mean gain 1.0) on neutral-subtracted shape
        protos = PROTO.reshape(len(CLASSES), -1)
        pred = np.argmin(((X["G-neutral"][:, None, :] - protos[None]) ** 2).sum(-1), axis=1)
        print(f"label-free check (seed 0): mean |identity - mean shape| = {inorm:.3f}, "
              f"mean |full expression prototype| = {enorm:.3f} (inter-ocular units)")
        print(f"  oracle prototype matcher on neutral-subtracted geometry: {np.mean(pred == y):.3f} "
              f"(chance {1 / len(CLASSES):.3f}); {len(y)} frames, {N_SUBJ} subjects")
    for fname, F in X.items():
        for cname, mk in CLFS.items():
            for prot in PROTOCOLS:
                correct = 0
                for tr, te in splits(prot, y, subj, seq, seed):
                    clf = mk().fit(F[tr], y[tr])
                    correct += np.sum(clf.predict(F[te]) == y[te])
                res.setdefault((fname, cname, prot), []).append(correct / len(y))

print(f"\naccuracy, mean over seeds {list(SEEDS)} [min-max]")
print(f"{'features':10s} {'classifier':10s} " + " ".join(f"{p:>22s}" for p in PROTOCOLS))
for fname in ["G-raw", "G-neutral", "A-HOG"]:
    for cname in CLFS:
        cells = []
        for prot in PROTOCOLS:
            a = np.array(res[(fname, cname, prot)])
            cells.append(f"{a.mean():.3f} [{a.min():.3f}-{a.max():.3f}]")
        print(f"{fname:10s} {cname:10s} " + " ".join(f"{c:>22s}" for c in cells))

Output (NumPy 2.3.5, scikit-image 0.25.2, scikit-learn 1.7.2):

label-free check (seed 0): mean |identity - mean shape| = 0.314, mean |full expression prototype| = 0.207 (inter-ocular units)
  oracle prototype matcher on neutral-subtracted geometry: 0.653 (chance 0.167); 720 frames, 30 subjects

accuracy, mean over seeds [0, 1, 2, 3, 4] [min-max]
features   classifier            frame-level      sequence-held-out    subject-independent
G-raw      linear        0.867 [0.825-0.906]    0.609 [0.576-0.649]    0.570 [0.525-0.622]
G-raw      1-NN          0.990 [0.988-0.993]    0.024 [0.015-0.037]    0.438 [0.412-0.465]
G-neutral  linear        0.984 [0.976-0.990]    0.899 [0.876-0.943]    0.906 [0.892-0.943]
G-neutral  1-NN          0.996 [0.990-0.999]    0.747 [0.721-0.792]    0.768 [0.721-0.803]
A-HOG      linear        0.398 [0.367-0.438]    0.256 [0.228-0.287]    0.362 [0.322-0.410]
A-HOG      1-NN          0.378 [0.336-0.410]    0.065 [0.050-0.083]    0.239 [0.225-0.265]

The script ran for 136 s and 173 s in its first two runs on a ten-core laptop shared with other jobs, longer than the 60–90 s estimated in the plan, and was kept unchanged; two later runs on the same machine took 90 s and 12 s, so the runtime depends mostly on the load. Its output was identical in all four runs, and no setting was changed after the first run. Before any training, the label-free check shows that the synthetic task is learnable but not trivial. The identity deviation of the landmarks is larger than a full expression displacement (0.314 against 0.207 inter-ocular units), and a template matcher that compares neutral-subtracted geometry with the noise-free class prototypes is right 65% of the time against 17% chance.

The frame-level protocol is optimistic for every representation and classifier, and the size of the optimism depends on how much identity the features carry. With raw geometry, logistic regression drops from 0.87 to 0.57 when subjects are kept apart, and the nearest-neighbour rule drops from 0.99 to 0.44. Its near-perfect frame-level score comes from the sibling frames of the same sequence that sit in the training folds. Subtracting the neutral frame removes most of the identity: subject-independent accuracy rises to 0.91, and the gap narrows to about 8 points for logistic regression. It does not close, because each subject’s own style for a class is still shared between training and test frames in the random split. CK+ uses both remedies, neutral-frame subtraction and subject-by-subject evaluation, although its paper motivates leave-one-subject-out by comparability and the amount of training data rather than by identity (Lucey et al., 2010). The size of the gap belongs to this synthetic set, not to any benchmark.

Parts of the plan’s hypothesis failed. The plan expected “frame-level > sequence-held-out > subject-independent accuracy” for every feature and classifier. Instead sequence-held-out folds score below the subject-independent protocol in five of the six rows (raw geometry with logistic regression is the exception, 0.61 against 0.57), and raw geometry with a nearest-neighbour rule falls to 0.02, far below chance. In this design each subject has exactly one sequence per class, so once a sequence is held out, all of that subject’s remaining training frames carry other labels. Identity-dominated features make those frames the nearest neighbours, and so they are confidently wrong. In our reading, any dataset in which each subject contributes few sequences per class has this structure, so a split that holds out sequences but not subjects is a trap rather than a compromise. The plan also expected the gap to be “largest for 1-NN and for features that carry identity (G-raw, A-HOG)”, with “1-NN frame-level near 100%”, and “smallest for G-neutral with the linear classifier”. The nearest-neighbour gap is larger than the logistic-regression gap for every representation, and raw geometry has the largest gaps, but HOG features are weak under every protocol (mean accuracy at most 0.40): their nearest-neighbour frame-level score stays far below 100%, and HOG with logistic regression, not neutral-subtracted geometry, has the smallest gap. The plan’s last two expectations held: under subject-independent folds, neutral-subtracted geometry beats raw geometry and HOG trails it. The rendered drawings’ HOG cells are dominated by subject-specific skin, hair and marks and by pose jitter, so this result reflects the crude rendering and says nothing about appearance features in general; in CK+ the appearance features did slightly better than shape on average for AUs (Lucey et al., 2010). The protocol, and the distinction between identity and expression in the features, should be reported with every FER result, as the train, validation and test split article argues for grouped data in general.

Limits of Inferring Emotion from Faces

Barrett et al. (2019), in a review for Psychological Science in the Public Interest, examined the evidence for what they call the common view: that each of anger, disgust, fear, happiness, sadness and surprise is signalled by a distinctive configuration of facial movements, reliably and specifically enough to diagnose it. They name automated emotion detection by technology companies among the uses of that view. They state three shortcomings of the research: limited reliability (instances of one category are neither reliably expressed through, nor perceived from, a common set of movements), lack of specificity (no unique mapping from a configuration to a category), and limited generalizability (the effects of context and culture have not been sufficiently documented and accounted for). Their main findings, as they state them:

• Production. In a meta-analysis of mostly laboratory studies, people made the hypothesized configurations more often than chance in every category tested except fear, but with weak reliability. In an expanded meta-analysis by Duran and Fernández-Dols (131 effect sizes from 76 studies, 4,487 participants), the average correlation between a configuration’s intensity and a measure of the corresponding emotion was 0.31, ranging from 0.06 to 0.45 by category. The configuration appeared on average in 22% of emotional events, with a range of 11% to 35% by category. Neither meta-analysis reported an overall assessment of specificity, because most studies do not report how often a configuration occurs without the emotion. A Duchenne smile can be produced deliberately and often signals affiliation or submission rather than happiness.
• Perception studies. Of the 87 emotion-perception studies in the most recent meta-analysis they cite (Elfenbein and Ambady, 2002), 95% used posed faces. The strongest support for the common view comes from urban, industrialized samples asked to choose from a short list of emotion words, and it does not generalize robustly. Free labelling and reverse correlation find much weaker evidence, and the authors write that studies of small-scale remote cultures further undermine the common view.
• Automated AU coding. Accuracy against expert coders exceeds 90% on frontal, well-lit, posed laboratory images and drops for unconstrained images. In the 2017 EmotioNet challenge, accuracy was below 83% and F1 below 0.65, and the authors judged that algorithms could not yet replace expert coders for everyday facial movements.
• Conclusion. It is not possible to confidently infer happiness from a smile, anger from a scowl or sadness from a frown. Facial movements that express emotion are more variable and context-dependent than the common view allows, with many-to-many mappings between configurations and categories. They describe the six standard configurations as closer to Western stereotypes than to prototypes. Efforts to read internal states from facial movements alone, without context, are at best incomplete and at worst lack validity, however sophisticated the algorithm. The authors add that facial movements are not meaningless: they carry social information that depends on context.

The computer vision sources make related points about their own labels, as the sections above show, and the RAF-DB authors add that cultural differences in perceiving facial emotion make it hard to define prototypical AUs for each expression (Li, Deng and Du, 2017). In the terms of these sources, a FER model trained on these datasets predicts which label a FACS rule, a recording protocol or a group of annotators would assign to a face. Whether that label matches what the person felt is a separate question, and the datasets do not measure it.

The demographic composition of the benchmarks is reported unevenly. CK+ describes its recording pool, DISFA its 27 participants, and RAF-DB its annotated sex, age range and race. AffectNet reports only automatic estimates, and no composition was found for FER2013. None of the papers cited here reports accuracy separately by group.

Applications

The dataset papers name the uses that motivated them. CK+ lists security, human–computer interaction, driver safety and health care (Lucey et al., 2010). DISFA lists human–computer interaction, social robots, consumer photography, automated pain detection, marketing, biometrics and behavioural and neuroscience research, and gives as research examples distinguishing posed from spontaneous smiles and categorizing pain-related expressions (Mavadati et al., 2013). AffectNet’s authors aim at affect-aware human–machine interaction and compare their baselines with a commercial emotion API, which did well on neutral and happy faces (accuracy 0.94 and 0.85) and poorly on fear, disgust and contempt (0.25, 0.27 and 0.04) on their test set (Mollahosseini, Hasani and Mahoor, 2019). Barrett et al. (2019) see automated AU coding as a useful research tool, used with human coders to study facial movements in large collections of images. They regard it as premature to use the technology to conclude what people feel. For video-level analysis of behaviour beyond the face, see video understanding.

Open Problems

• Labels that mean what they claim. Posed labels record requests, FACS-based labels record movement rules, and web labels record perception. Barrett et al. (2019) call for studies that record facial movements together with context in everyday life, sampled across cultures, before inferences about emotion are drawn.
• Spontaneous, in-context data with AU coding. AU datasets have tens of subjects; web datasets have many people but no context and no FACS coding.
• Comparable protocols. Papers evaluate the same dataset on different sets, with different metrics and with cross-dataset and within-dataset results side by side, as the reported results above show.
• Subject-independent generalization. Identity and individual style leak into frame-level splits, and splits that hold out sequences but not subjects can mislead in the other direction, as the example shows. Cross-dataset tests show larger drops than within-dataset ones (Li, Deng and Du, 2017).
• Reporting by group. None of the papers cited here reports accuracy by group (see dataset bias).
• Intensity and dynamics. Graded AU intensity and the timing of actions, which the DISFA authors consider important to their meaning, remain less studied than presence detection and still-image classification (Mavadati et al., 2013).