Textualized and Feature-based Models for Compound Multimodal Emotion
Recognition in the Wild

Abstract

Systems for multimodal emotion recognition (ER) are commonly trained to extract features from different modalities (e.g., visual, audio, and textual) that are combined to predict individual basic emotions. However, compound emotions often occur in real-world scenarios, and the uncertainty of recognizing such complex emotions over diverse modalities is challenging for feature-based models. As an alternative, emerging large language models (LLMs) like BERT and LLaMA can rely on explicit non-verbal cues that may be translated from different non-textual modalities (e.g., audio and visual) into text. Textualization of modalities augments data with emotional cues to help the LLM encode the interconnections between all modalities in a shared text space. In such text-based models, prior knowledge of ER tasks is leveraged to textualize relevant non-verbal cues such as audio tone from vocal expressions, and action unit intensity from facial expressions. Since the pre-trained weights are publicly available for many LLMs, training on large-scale datasets is unnecessary, allowing to fine-tune for downstream tasks such as compound ER (CER). This paper compares the potential of text- and feature-based approaches for compound multimodal ER in videos. Experiments were conducted on the challenging C-EXPR-DB dataset in the wild for CER, and contrasted with results on the MELD dataset for basic ER. Our results indicate that multimodal textualization provides lower accuracy than feature-based models on C-EXPR-DB, where text transcripts are captured in the wild. However, higher accuracy can be achieved when the video data has rich transcripts. Our code is available at: github.com/nicolas-richet/feature-vs-text-compound-emotion.

Keywords: Emotion Recognition, Compound Expressions, Multimodal Learning, Multimodal Textualization, Large Language Models

Figure 1: Models for compound multimodal ER in videos. (a) In the feature-based approach, a dedicated feature extractor produces embeddings for each input modality. A feature-level fusion module then combines embeddings from all different modalities to produce joint feature representations for classification. (b) In the text-based approach, textual descriptions are extracted for nonverbal modalities, such as audio and visual. These texts are combined with verbal cues (i.e., text transcripts) and fed to an LLM as a joint textual representation for classification.

1 Introduction

Emotion recognition (ER) plays a critical role in human behavior analysis, human-computer interaction, and affective computing [13, 60]. Research in ER mainly focuses on recognizing the seven basic emotions – anger, surprise, disgust, enjoyment, fear, sadness, and contempt [4, 46]. Recently, there has been growing interest in recognizing complex emotions commonly occurring in real-world scenarios, such as compound emotions, where a mixture of emotions is exhibited [26]. ER is more challenging for complex emotions because they are often ambiguous and subtle, and can be easily confused with basic emotions. Despite the availability of multiple modalities that can potentially help to recognize these complex emotions [70], they can introduce additional uncertainty and conflict [23, 63].

Multimodal information, e.g., faces, voice, and text extracted from videos, has been used extensively to develop robust ER models [1, 60, 62]. Convolutional and transformer-based backbones are commonly trained to extract discriminant features from each modality. These include vision backbones such as ResNet [18] and audio backbones such as VGGish [19]. A fusion model is required to combine features from verbal (spoken text) and nonverbal cues (visual and audio), thereby producing contextualized features to predict accurate emotion classes [39]. Multimodal learning allows for building complex joint feature representations that can achieve high accuracy for ER. This feature-based approach has driven much progress in ER [60]. Indeed, it requires simple and minimal emotion-class annotations – typically enough to allow these models to learn to automatically extract and combine relevant features from different modalities for prediction. However, using only a single emotion class for CER in real-world scenarios and without any other guidance, such as output supervision or input cues, is challenging, especially using videos captured in the wild [30].

Recently, another multimodal learning approach called TextMI has been proposed for sentiment analysis [17] in videos. In contrast with the feature-based approach, input modalities like audio and visual are textualized. Based on prior knowledge of the task, the authors propose extracting nonverbal cues deemed relevant to the task in the form of a descriptive text. This can include a textual description of action unit (AU) intensity from visual [12, 14] and the tone of the audio. This conversion of modalities to text could be seen as an expert-based data augmentation with emotion-related cues used as inputs for the model during training and evaluation. This augmentation provides the models with direct guidance and emotional context to learn the task at hand. Since all modalities are formatted as text, a language model is required to better understand the interconnections between words and modalities and their relations to emotions. However, state-of-the-art language models are typically large, and training them requires a considerable amount of text data, which is not always available. The recent surge of large language models (LLMs) [72] made their use possible [17]. Powerful LLMs such as BERT [10] and LLaMA [56] have been pre-trained, and their weights have been made public, allowing us to fine-tune these models for downstream tasks [22]. Their application in multimodal ER provides a simple method for multimodal fusion, and results shown in [17] are promising. Multimodal textualization remains largely unexplored in the literature. In a very recent work [5], modality descriptions are employed to model output supervision for emotional reasoning and recognition. This differs from TextMI [17], which uses these cues as inputs. In this paper, we analyze text-based approaches like TextMI for multimodal CER.

Figure 2: A common feature-based approach used for multimodal CER.

This paper focuses on the following question: how does textualized modeling perform against feature-based modeling for CER in videos? As shown in Fig. 1(a), feature-based methods employ different backbone models to automatically extract features from each modality. Feature-level fusion models also allow us to automatically learn joint feature representations for ER, although combining diverse modalities over videos remains a challenge [39]. The textualized approach [17] simplifies fusion since all nonverbal modalities (visual and audio) are converted into a single textual modality (see Fig. 1(b)). However, a bottleneck of the approach is that textualizing nonverbal modalities requires the choice of textual descriptions. For instance, the same audio segment can be textualized into the high/low valence/arousal spectrum, or the vocal intonation can be textualized. Similarly, for the facial modality, the choice of AUs and the granularity and context window are some of the design choices that can heavily affect the overall performance. The manual selection and design of textual descriptions require the intervention of domain experts to construct relevant nonverbal cues, making them application-dependent. This is similar to the contrast between using learned versus handcrafted features and the challenges of the latter.

In this paper, we compare the performance of state-of-the-art deep learning models that follow standard feature-based vs. text-based modeling approaches. An extensive set of experiments was conducted on the challenging C-EXPR-DB video dataset for CER in the context of the 7th Workshop and Competition on Affective Behavior Analysis in-the-Wild (ABAW) [28, 33, 30, 29, 25, 24, 32, 31, 27]. To further assess the benefits of using a textualized approach in basic ER, experiments were also conducted on the MELD video dataset.

2 Feature-based Modeling

This approach extracts features from audio (vocal) and video (facial) modalities and text transcripts for multimodal CER in videos. Feature embedding is combined for feature-level classification (see Fig. 2). The rest of this section provides more details on this approach.

Feature Representation. In ER applications, the de facto strategy to leverage different modalities extracted from videos is to extract their features [1, 34, 54, 62, 75]. These modalities typically include vision and audio. Textual modality, such as audio transcripts, is also included when available. Other ER applications such as pain estimation leverage bio-signals such as physiological modalities [62, 64]: electrodermal activity (EDA), electromyograph (EMG), and electrocardiogram (ECG). The general motivation behind combining these modalities is to leverage their complementary information over a video sequence.

Each modality typically employs a dedicated pre-trained feature extractor, which can be pre-trained on different large-scale datasets. In addition, public weights pre-trained on related datasets can be employed. For visual modality, ResNet [18] backbone is commonly used, which is followed by a module that leverages temporal information such as temporal convolutional network (TCN) [2]. 3D models such as R3D-CNN [57] can better leverage spatio-temporal dependency between frames directly at feature extraction. For audio modality, a variety of public pre-trained feature extractors are available, such as VGGish [19], Wav2Vec 2.0 [61], and HuBERT [21]. In addition, traditional audio features can be easily computed, such as spectrograms and MFCCs [65]. Multiple text feature extractors are available for the text modality, such as BERT [10] and RoBERTa [43]. Feature extractors are typically kept frozen while the subsequent modules are fine-tuned to avoid expensive computational costs.

Figure 3: A common text-based approach used for multimodal CER, where non-verbal modalities are textualized.

Feature-Level Fusion. A bottleneck in feature-based models is fusion (Fig. 1). Different methods rely on temporal models to combine features from over a video, like LSTMs [7, 54, 48], or rely on simple concatenation [34, 73]. Recent works focus more on self- and cross-modal attention and transformers [59] to perform attention-based fusion [35, 44, 51, 58, 62, 69, 74]. This has shown promising results as they can capture inter- and intra-modality relationships. Note that aligning modalities in a video setup is challenging as well. For example, what part of audio or text could be assigned to a single frame remains unclear. Usually, an empirical sliding window is employed to align other modalities with a frame.

This paper follows the recent work [69] to experiment with a feature-based approach for the ER task (Fig. 2). Their method achieved good results for continuous ER (valence/arousal) in videos in a recent ABAW challenge. In our case, it was adapted to perform emotion classification in videos as well. In particular, we use ResNet50 [18], pre-trained on MS-CELEB1M [16] and FER⁺ [3] datasets for visual modality. For audio modality, we employ VGGish [19], while BERT [10] is used over text modality. Temporal information is further exploited by using temporal convolutional network (TCN) [2] after each feature extractor. We employ the co-attention block [69] to attend to features from different modalities. This builds a single embedding per frame while leveraging a contextual window. The per-frame feature is then fed to a classifier head to predict emotions.

3 Text-based Modeling

This approach extracts textual descriptions from audio (vocal) and visual (facial) modalities for multimodal CER in videos, and combines them with text transcripts. This joint textual description is processed by an LLM for classification (see Fig. 3). The rest of this section provides more details on this approach.

Audio and Visual Text Description. The API of Hume Inc. is employed over a sliding window to analyze the tone of the audio. Their model is trained on millions of human interactions. The API scores each tone characteristic, allowing us to sort and select the top 10. Examples of tone characteristics include confusion, anxiety, disappointment, distress, and even basic emotions. The name of each tone characteristic and a "Low" or "High" prefix, determined using a threshold on the score, are used to describe the tone textually. We also use a fine-tuned Wav2Vec 2.0 model [61] to predict scores for arousal, valence and dominance for each audio clip. Those scores are then categorized as "Low" or "High" using a threshold. The textual description produced from audio concatenates all tone and arousal-valence-dominance textual descriptions.

Face cropping and alignment are first performed at each frame using RetinaFace [8]. Py-feat library [6] is then used to extract action units (AUs) intensity [12, 14] along with basic emotion probabilities. AUs codebook [4] maps each facial expression to a set of action units. For instance, the expression "Happy" is associated with "AU6" (Cheek raise), "AU12" (Lip corner puller), and "AU25" (lips part). Typically, a set of these facial AUs activate at once. Over a sequence of frames, we select the maximum intensity of each action unit over time. Then, we use a threshold to determine which AUs are activated. In this case, the text description will be the concatenation of the name of each selected action unit as depicted in Fig. 3. The same procedure is repeated with basic emotions, where the top 3 emotions are selected, and their textual name is used as a description.

Combination of Transcripts with Audio and Visual Texts. Some multimodal datasets like MELD [49] and CMU-MOSEI [68] provide text transcriptions. We use Whisper-large-v2 [52] to generate transcripts when they are unavailable. Once all textual descriptions are acquired, we combine them into a single prompt and feed it to an LLM such as LLaMA-3 [47], in our case. Our used prompts follow this template:
"
Speech transcription of the video : [transcription];
Facial Action Units activated during the video : [visual_AU_text];
Emotions predicted from visual modality: [visual_emotions_text];
Characteristics of the prosody : [tone_description];
Audio emotional state : [arousal-valence-dominance_text]
".
A fully connected layer (FC) follows LLaMA-3 to perform classification. Both the LLM and the FC layer are fine-tuned using ground-truth class labels.

4 Results and Discussion

This paper compares feature- and text-based approaches for multimodal CER in videos in the wild. For a fair comparison, our experimental setup is constrained to be as similar as possible for the two approaches. This section provides the experimental methodology, results, and discussion. We also include results on basic emotions in videos.

4.1 Experimental Methodology

(1) Datasets. Two video-based datasets for emotion recognition are used: C-EXPR-DB (compound emotion), and MELD (basic emotion).

a) C-EXPR-DB [26]: It is composed of 56 videos taken from 7th ABAW CER Challenge test set. The full C-EXPR-DB [26] contains 400 videos with 200,000 frames in total. However, we do not have access to the 400, but only to the 56 test videos of the challenge. Each frame has an annotation with twelve compound emotions. For the 7th ABAW Challenge, only seven compound emotions are considered from the original twelve, which are: Fearfully Surprised, Happily Surprised, Sadly Surprised, Disgustedly Surprised, Angrily Surprised, Sadly Fearful, and Sadly Angry. In this work, these 56 videos used for the test are referred to as C-EXPR-DB. The challenge organizers provide it without annotation. To perform experiments earlier than the challenge deadline, we annotated these 56 videos by our internal expert team. Each video may have different parts where there are compound emotions. The annotation is done at frame level with the same seven compound emotions of the challenge in addition to the class "Other" that represents any other emotion, compound or basic, that is different from the considered seven ones. Once annotated, we cut each original video into segments using the annotation timeline. Each segment contains only one compound emotion. We obtained 125 segments. We refer to this dataset as C-EXPR-DB^∗. Its class distribution is presented in Table.1. We split C-EXPR-DB^∗ into 5-cross validation. Performance is reported on the validation set of each fold. The evaluation is done only on compound emotions while discarding the class "Other". However, the training may or may not include this extra class. More details about our annotation are provided later.

Compound class		Number of segments		Total duration (secs)
Angrily Surprised		8		53.92
Sadly Angry		16		100.60
Fearfully Surprised		24		98.85
Happily Surprised		15		150.57
Sadly Fearful		19		107.56
Disgustedly Surprised		10		82.45
Sadly Surprised		13		182.17
Other∗		20		71.00
Total		125		847.15

Table 1: Class distribution and duration of our internally labeled C-EXPR-DB^∗ dataset from the 56 videos of the 7th ABAW CER Challenge taken from C-EXPR-DB dataset.
^∗: In some of our experiments, the class "Other" has been used only for training. Only the seven compound emotions are considered for evaluation.

Code		Definition
Fearfully Surprised		Experiencing a sudden shock or surprise accompanied by fear. This might happen if something unexpected occurs that also appears threatening or dangerous. Must include at least one cue related to fear and a cue related to surprise, either occurring simultaneously or closely following each other.
Happily Surprised		Experiencing a sudden and unexpected event that brings joy or happiness. This type of surprise is pleasant and delightful. Must include at least one cue related to happiness and at least one cue related to surprise. They should occur simultaneously or closely following each other.
Sadly Surprised		Experiencing a sudden and unexpected event that brings sadness or disappointment. This type of surprise is unpleasant and upsetting. Must include at least one cue related to sadness and at least one cue related to surprise. They should occur simultaneously or closely following each other.
Disgustedly Surprised		Experiencing a sudden shock or surprise that also causes feelings of disgust or revulsion. This might happen if something unexpected occurs that is also repulsive. Must include at least one cue related to disgust and at least one cue related to surprise. They should occur simultaneously or closely following each other.
Angrily Surprised		Experiencing a sudden shock or surprise that provokes anger. This might happen if something unexpected occurs that is also infuriating. Must include at least one cue related to anger and at least one cue related to surprise. They should occur simultaneously or closely following each other.
Sadly Fearful		Feeling both sadness and fear simultaneously. This can occur when facing a situation that is both threatening and sorrowful. Must include at least one cue related to sadness and at least one cue related to fear. They should occur simultaneously or closely following each other.
Sadly Angry		Feeling both sadness and anger simultaneously. This can happen when dealing with a situation that evokes both sorrow and frustration or rage. Must include at least one cue related to sadness and at least one cue related to anger. They should occur simultaneously or closely following each other.

Table 2: Codebook used by our experts to annotate compound C-EXPR-DB^∗ emotions.

b) MELD (basic emotions) [49]: This multi-party dataset was created from video clipping of the TV show "Friends" utterances. The train, validation, and test sets consist of 9988, 1108, and 2610 utterances, respectively. Each utterance has one global label from seven basic emotions: anger, sadness, joy, neutral, fear, surprise, or disgust. In addition, a transcript of each utterance is provided. This dataset is unbalanced, where neutral is the most dominant label with 4710 utterances, while disgust is the least frequent label with 271 utterances.

(2) Annotation of Compound Emotions in C-EXPR-DB^∗. The 56 test videos of C-EXPR-DB for the 7th ABAW CER Challenge were annotated by two expert annotators, both with a psychology background and one with extensive experience with emotion recognition. Each video was annotated by both annotators and subsequently triangulated to create one unified annotation file. The triangulation included a discussion to create agreement on the compound emotions found in each video, the segment of the video where it could be found (time stamps), and the reasoning behind the choice of compound emotion identified.

Annotators followed a codebook (Table.2), created specifically for the challenge, where the seven compound emotions (Fearfully Surprised, Happily Surprised, Sadly Surprised, Disgustedly Surprised, Angrily Surprised, Sadly Fearful, Sadly Angry) were properly described. In addition, the individual emotions (Happiness, Sadness, Anger, Fear, Disgust, and Surprise) were described and broken down in specific cues, including behavioral responses and facial, language, audio, and body language markers for each of the basic emotions. This allowed for the use of multi-modalities to properly identify the presence of the emotions. Compound emotions were annotated when both emotions occurred simultaneously or closely followed each other. Additionally, other emotions not related to the compound emotions were tagged as "Other". Videos were annotated using the software ELAN 6.8 since both annotators had previous experience with this annotation tool.

(3) Baseline Models. For a fair comparison, recent models are considered. For the feature-based approach, we follow the work in [69] where we used ResNet50 [18] for feature extraction over visual modality. It is pre-trained on MS-CELEB1M [16] dataset as a facial recognition task. Then, it is fine-tuned on the FER⁺ [3] dataset. VGGish [19] is used for audio modality. Over text modality, the BERT Base Uncased model is used [10]. To leverage temporal dependency from videos, we used temporal convolutional network (TCN) [2], which has shown to yield good results in previous ABAW challenges [50, 69]. For the fusion module, we used the co-attention method (LFAN) proposed in [69], followed by a classification head. All three feature extractors are frozen, and every subsequent module is fine-tuned. For the text-based approach, once the text of all modalities is acquired, it is fed to a language model, which is followed by a dense layer to perform classification. In our experiments, we used LLaMA-3 8B [47], a recent open LLM that can fit in an average GPU. Both modules are fine-tuned using emotion labels as supervision.

(4) Training Protocol. The following learning strategies are used:
a) 7th ABAW CER Challenge: We train our model on MELD over the seven basic emotions. We then evaluate the final model over the 56 unlabeled videos of the test set C-EXPR-DB, following the challenge protocol. Since the model is trained over basic emotions, for each pair of the seven compound emotions, we sum their corresponding probabilities and pick the pair with the highest score to predict the compound emotion. We submitted different cases of feature and text approaches based on the training size of MELD: with 1% and 100% of total training samples. We alsp explored zero-shot predictions of a multimodal large language model (MLLM) using the LLaVA-NeXT-Video [71], which uses visual modality.
b) Feature vs. text comparison: Over C-EXPR-DB^∗ and MELD, we perform supervised learning on each dataset separately. For C-EXPR-DB^∗, we report the performance on the validation set of each of the 5-cross validation, while we report test performance on MELD.

The next presents our experimental details of each approach:
1) Feature-based approach: The pre-processing of videos is carried out as follows. For the visual modality, RetinaFace [8] is used to crop and align faces from each frame, which are then resized to ${48\times 48}$. A sliding window over frames is used for training. The size is estimated using validation among ${16*n}$ where ${n}$ spans from 2 to 18. Windows overlaps with the size of 16. A window of frames is fed to a pre-trained ResNet50 [18]. The audio of a video is initially converted to WAV format with a sampling rate of 16k. To synchronize with frames, we set the hop length to be $1/\text{frame rate}$. For audio feature extraction, we used VGGish [19]. For text modality, we used the transcripts provided by MELD. However, for C-EXPR-DB and C-EXPR-DB^∗, we used Whisper-large-v2 [52] to generate each video transcript. BERT [10] is then used to extract features aligned with frames. The three feature extractors are frozen. Only the subsequent modules are fine-tuned. The model [69] is fine-tuned on frame-level. When only a video-level emotion label is available, this same label is transferred to each frame in the video. Stochastic gradient descent (SGD) is used for optimization with a batch size estimated by validation from ${2,4,8,16,18}$ and weight decay of ${10^{-4}}$. For MELD dataset with 1%, we trained for 1000 epochs, while 300 epochs are used for the case of full data (100%) due to long computation time (37 hours on an NVIDIA A100 GPU). For C-EXPR-DB^∗, we trained for 100 epochs. Standard cross-entropy loss is used for training.
2) Text-based approach: The pre-processing of each modality is performed as follows. For the visual modality, the Py-Feat library is used to extract the intensity of $20$ Face Action Units for each frame of a video. A maximum is then applied over a sliding window of frames to summarize the action unit scores for each frame. This sliding window is used as the context for each frame. The audio parts of the video are used as input to the prosody model using the API of hume.ai, and the top-10 tone characteristics are then used. We also used a fine-tuned Wav2Vec 2.0 model [61] to predict scores for arousal, valence, and dominance for each audio clip. We use QLoRA [9] in order to efficiently fine-tune LLaMA-3 8B [47]. The training is done at the frame level, and in the case where only video-level emotion is available, this same label is transferred to each frame in the video. To reduce the computation time on the MELD dataset and avoid too many prompt duplicates, we only use a few frames with their context from each video during the training while the test is conducted over all frames. SGD is also used for optimization with a batch size estimated by validation from ${\{8,10,14\}}$ and weight decay of $10^{-3}$. For MELD dataset with $1\%$, we use a learning rate estimated by validation from ${\{2\times 10^{-3},5\times 10^{-3},7\times 10^{-3}\}}$ while learning rates from ${\{7\times 10^{-3},10^{-2},3\times 10^{-2}\}}$ are used for the case of full data. For the C-EXPR-DB^∗ dataset, we set a window hop size to decrease the number of redundant prompts used in training. We estimate the batch size from ${\{6,8\}}$ and the learning rate from ${\{2\times 10^{-3},5\times 10^{-3},7\times 10^{-3}\}}$. The window size is also selected from ${\{10,15,20,30\}}$, and we use a hop size of 10. Standard cross-entropy loss is used for training.
3) Zero-shot MLLM: The previous two approaches are compared with a zero-shot MLLM based on LLaVA-Next-Video [71], an open-source MLLM trained on text-image data and fine-tuned on video data. This model excels at strong zero-shot modality transfer, outperforming existing open-source MLLMs specifically trained for videos and achieving comparable performance to proprietary MLLMs. For basic ER and CER, we used raw videos and specific prompts to constrain the MLLM’s output to single responses related to discrete emotions: "Are the persons in the video either surprised, angry, joyful, sad, fearful, disgusted, or neutral? Choose a single answer.", and "Identify the primary compound emotion displayed by the individuals in the video from the following options: disgust-surprise, anger-surprise, fear-sadness, anger-sadness,
fear-surprise, happy-surprise, or sad-surprise.". This limitation affects the MLLM’s performance, as it was designed to provide a rich description for understanding video content rather than making decisions on a single category. As a result, for some short videos, the MLLM returned non-valid emotions, such as descriptions of scenes or other text. We post-processed the outputs in these cases, replacing them with the "neutral" category.

(5) Performance Measures. To assess model performance, we use the average $F1$ score required in the 7th ABAW Challenge. It is defined as follows,

$$\left\{\begin{aligned} &F1_{c}=\frac{2\times{\text{Precision}}_{c}\times{\text{Recall}}_{c}}{{\text{Precision}}_{c}+{\text{Recall}}_{c}};\\ &{\text{Precision}}_{c}=\frac{{TP}_{c}}{{TP}_{c}+{FP}_{c}};\quad{\text{Recall}}_{c}=\frac{{TP}_{c}}{{TP}_{c}+{FN}_{c}};\\ &\quad F1=\sum_{c=1}^{7}w_{c}\times F1_{c};\end{aligned}\right.$$ (1)

where $c$ represents the class ID, $TP$ represents True Positives, $FP$ represents False Positives, and $FN$ represents False Negatives. Average ${F1}$ is computed with ${w_{c}=1/7}$ while weighted ${F1}$ is computed with ${w_{c}}$ set to be the proportion of each class.

Following the literature, on the MELD dataset, we use the weighted $F1$ score, which accounts for unbalanced classes, similarly to C-EXPR-DB^∗. We note that evaluation over C-EXPR-DB^∗ is done over frame level similarly to C-EXPR-DB.

With the MELD dataset, only global video-level labels are available where a class is assigned to the entire video. Evaluation must also be performed at the video level, not the frame level. For video-level prediction, we post-process the frame-level predictions using three different strategies:

1) Majority voting: Majority voting is performed over the predicted classes of all the frames. The winning class becomes the prediction for the video.

2) Average logits: For each class, the average its logits are assesses across frames. This yields an average logit vector. The video class is the class with the maximum logit.

3) Average probabilities: This is similar to average logits, but it is perfromed over probabilities instead.

(6) Frame-based Ensembling. For the ABAW CER challenge, we perform a submission with ensembling over prediction labels of different models at the frame level. Given a set of models, we first perform single model prediction at frame level. Then, we perform label prediction aggregation at each frame to have a final frame label. To do so, at a time $t$, we consider a window of length 10 that covers the frame $t$ and its previous frames. Then, we perform majority voting over the prediction label across all frames and models within that window. The label result of the vote is assigned as the final label prediction for the frame $t$.

		Feature-based Approach		Text-based Approach
Modalities		MELD	C-EXPR-DB∗	MELD	C-EXPR-DB∗
Visual only		37.88	$\bm{53.46}$	25.18	19.82
Audio only		39.51	25.45	43.97	$\bm{50.96}$
Text (transcript) only		$\bm{59.78}$	33.09	$\bm{62.55}$	46.11
All 3 modalities		60.87	42.74	64.22	48.09

Table 3: Ablation study: weighted $F1$ scores for unimodal (visual, audio, and text) and multimodal cases. Results are reported for video-level (MELD) and frame-level (C-EXPR-DB^∗). For C-EXPR-DB^∗ dataset, we consider fold 0 and the case w/o "Other".

4.2 Results for Basic (MELD) and Compound (C-EXPR-DB^∗) Emotions

Ablation studies were conducted on text- and feature-based approaches over single and multimodal cases (see Table 3). Both datasets were considered – MELD with relatively controlled setup, and C-EXPR-DB^∗, an extreme case of in-the-wild. The first observation is the contribution of a single modality, which depends on the dataset. On MELD, where there are rich transcripts, both approaches achieved their highest $F1$ score on the text modality with 59.78% for feature-based and 62.55% for text-based. However, other modalities contributed less. We note that visual textualization achieved a poor $F1$ score of 25.18% compared to audio with 43.97%. This suggests that either there is less information in the visual modality or the textual cues used are less efficient in capturing the emotion. This pattern is consistent with the feature-based approach but with less decline in performance between the two modalities.

On C-EXPR-DB^∗, the text transcript is very limited. In most videos, people shout, scream, talk very briefly, or do not talk and only express compound emotions visually. Such characteristics seem to be tied to compound emotions, making their prediction difficult. Interestingly, each approach leverages a different modality to deal with compound emotion prediction. In the feature-based method, visual modality seems to be the strongest, yielding an $F1$ score of 53.46%. However, in the text-based method, audio seems to be the strongest modality with an $F1$ score of 50.96%, while text modality ranks second with 46.11%, and visual modality yields a poor score of 19.82%. This very low score of visual modality may indicate that its textualization yields inadequate information since this result contradicts the feature-based approach which scores the highest with 53.46% suggesting that visual modality holds rich information. The choice of converting a modality into a set of textual descriptions can lead to drastic and irreversible loss of information. Handcrafting these textual cues requires expertise, which may lead to poor performance. Modality textualization is very difficult to apply in real-world applications. An additional observation that can be drawn from these results is that finetuned LLMs, as in our case where we used a fully connected layer for classification on top of it, are powerful over single text modality when the text is rich. This can be observed over MELD dataset where the text-based method yields a score of 62.55% over transcript only. However, the performance drops when using a poorer transcript over the compound emotion dataset (C-EXPR-DB^∗) with a score of 46.11%.

Another observation from Table 3 is the impact of multiple modalities on performance. On MELD, using multimodal seems to improve performance compared to a single modality but leads to a considerable performance drop on compound emotion for both approaches. This may be explained by the conflicting emotions seen simultaneously through each modality. For example, the transcript "OK" can carry out the emotion ’Neutral’. However, the way it is said, and the facial appearance of saying it can change the emotion to suggest the dual emotions of ’Surprise’ and ’Happy’. In other cases, multi-conflicting instances of the same modality can be an issue, like in audio, where it is difficult to separate the sound source. For example, the case where a person is commenting on an event where we are interested in the emotion of the commentator. The overlay of the event and a person’s audio signals could easily reflect conflicting emotions. Videos in the wild are extremely challenging, and there are many different cases to consider. Focusing on the right instance in a modality remains a challenge [40], but it may be easier with the visual modality than with audio and text.

				Feature-based		Text-based		MLLM zero-shot
Prediction Method				${1\%}$	${100\%}$	${1\%}$	${100\%}$	LLaVa-NeXT-Video [71] (Visual)
Majority voting				43.64	59.92	$\bm{33.31}$	$\bm{65.50}$
Average logits				$\bm{45.58}$	$\bm{60.87}$	33.12	65.35	36.25
Average probabilities				43.34	59.72	33.09	65.29

Table 4: Video-level weighted $F1$ score on MELD test set for feature- and text-based approaches. Three methods are used to extract video-level prediction from frame-level prediction. Training is performed on 1% and 100% of the original MELD training set.

A comparison between text- and feature-based approaches for basic (MELD) and compound (C-EXPR-DB^∗) ER is presented in Tables 4 and 5, respectively. On MELD, the text-based approach yields the best performance with ${\approx 4\%}$ above the feature-based approach. As discussed earlier, this is mainly due to the high quality of transcripts and the LLM. However, on C-EXPR-DB^∗, the feature-based approach is ahead of the text-based approach. Given the poor transcript quality, textualization does not bridge the gap with the feature-based approach. Based on our experiments, one should attempt to use textualization over feature-based methods only when rich transcripts are available.

Methods		Fold 0	Fold 1	Fold 2	Fold 3	Fold 4	Mean $\pm$ std
Training w/ "Other"
$\cdot$ Text-based		34.82	54.47	36.47	51.06	48.08	44.98 $\pm$ 07.90
$\cdot$ Feature-based		$\bm{39.07}$	$\bm{64.73}$	$\bm{41.61}$	$\bm{55.65}$	$\bm{70.08}$	$\bm{54.22\pm 12.26}$
Training w/o "Other"
$\cdot$ Text-based		$\bm{48.09}$	48.09	34.59	$\bm{59.24}$	55.83	49.17 $\pm$ 08.49
$\cdot$ Feature-based		42.74	$\bm{65.56}$	$\bm{41.66}$	54.74	$\bm{70.65}$	$\bm{55.07\pm 11.70}$

Table 5: Frame-level weighted $F1$ score on C-EXPR-DB^∗ validation sets.

Experiments on C-EXPR-DB^∗ with ensembling at the frame level (see Table 6) indicate that combining the prediction of relatively good models yields better performance. Regarding the performance of Zero-shot MLLM LLaVa-NeXT-Video [71], we obtained low performance over both datasets. Acknowledging that we have constrained its output to be a single label is important. This may have limited its performance since a single label may not be enough to express an emotion. Despite this, the model produced decent performance over MELD with an $F1$ score of 36.25%. However, limited performance is reported over compound emotion dataset C-EXPR-DB^∗. We observed that the model predicts the class ’Fearfully Surprised’ on this specific dataset almost at every frame. Predicting two emotions simultaneously seems more challenging than a single emotion for this model.

Methods		Fold 0	Fold 1	Fold 2	Fold 3	Fold 4		Mean $\pm$ Std
Text-based		48.09	48.09	34.59	59.24	55.83		${49.16\pm\text{\ }9.49}$
Feature-based		42.74	65.56	41.66	54.74	70.65		${55.07\pm 13.08}$
Zero-shot MLLM LLaVa-NeXT-Video [71]		9.31	4.83	8.27	12.18	8.87		${\text{\ }8.69\pm\text{\ }2.63}$
Frame-based ensembling over all 3 previous methods		$\bm{54.21}$	65.86	35.63	63.39	69.04		${58.48\pm 15.40}$
Weighted Frame-based ensembling: Feature- and text-based only		50.75	$\bm{69.08}$	$\bm{49.57}$	$\bm{63.62}$	$\bm{75.15}$		${\bm{61.63\pm 11.24}}$

Table 6: Frame-level weighted $F1$ score on C-EXPR-DB^∗ validation sets for our three compared methods in addition to their predictions ensembling. We consider the case w/o "Other" in Table 5. The weighted frame-based ensembling gives twice the weight to the feature-based predictions compared to text-based predictions.

We note that our obtained results over MELD are competitive with state-of-the-art performance as presented in Table 7. The text-based method over the case without contextual information achieved a new state-of-the-art $F1$ score of 65.50%.

Model (year)	Modalities	Context Info	Weighted F1
HCAM (’24)[11]	T+A	✓	65.80
SDT (’23)[45]	T+A+V	✓	66.60
DF-ERC (’23)[37]	T+A+V	✓	67.03
EACL (’24)[66]	T	✓	67.12
TelME (’24)[67]	T+A+V	✓	67.37
InstructERC (’23)[36]	T	✓	69.15
CKERC (’24)[15]	T	✓	69.27
SMCN (’22)[20]	T+A	✗	62.3
HCAM Stage I (’24)[11]	T	✗	63.3
SSE-FT (’20)[55]	T+A+V	✗	63.9
Text-based	T+A+V	✗	65.50
Feature-based	T+A+V	✗	60.87
Zero-shot MLLM LLaVa-NeXT-Video [71] (’24)	V	✗	36.25

Table 7: Comparison of recent results on the MELD dataset.

We conducted additional experiments over C-EXPR-DB^∗ where we investigated the impact of weights initialization by comparing random vs MELD pretrained weights over both text- and feature-based method. Results are reported in Table 8. These results convey a mixed message where in some cases pretrained weights can help and sometimes, it is better to start from random initialization depending on the fold. However, we note that feature-based method is more robust to initialization as we observe only slight performance shift (around 2%) between random and pretrained case. However, for the case of text-based method, in some folds the performance shift is large an can go up to 8% and 14%.

Table 10 present video-level weighted $F1$ score on C-EXPR-DB^∗ validation sets. At video-level, feature-based method has a clear winning marge compared to text-based method overall.

Methods		Fold 0	Fold 1	Fold 2	Fold 3	Fold 4
Feature-based
$\cdot$ Random init.		42.74	65.56	$\bm{41.66}$	$\bm{54.74}$	$\bm{70.65}$
$\cdot$ Init. over MELD		$\bm{43.96}$	$\bm{68.14}$	38.87	54.07	68.45
Text-based
$\cdot$ Random init.		$\bm{48.09}$	48.09	34.59	$\bm{59.24}$	55.83
$\cdot$ Init. over MELD		34.99	$\bm{56.30}$	$\bm{37.94}$	58.53	$\bm{57.13}$

Table 8: Impact of pre-training: Frame-level weighted $F1$ score on C-EXPR-DB^∗ validation sets with different weights initialization: random vs pre-trained over MELD. We consider the case w/o "Other". In the pre-trained case, the fully connected last layer is randomly initialized.

Case/Approach		Text-based	Feature-based	Zero-shot MLLM LLaVa-NeXT-Video [71]
Train time 1 epoch
MELD		9.3min	6.5min	–
C-EXPR-DB∗		2.1min	23sec	–
Inference time per-frame		35ms	0.12ms	715ms
Total n. params.		7.51B	223.91M	7.06B
N. learnable params.		3.44M	5.00M	–
N. FLOPs (TFLOPs)		597.52	1.87	–
N. MACs		298.75TMACs	938.76 GMACs	–

Table 9: Comparison of computation time, number of parameters, number of FLOPs/MACs of text- and feature-based approach used in our experiments, in addition to a zero-shot MLLM.

Folds / Case		Training w/ "Other"		Training w/o "Other"		Zero-shot LMM LLaVa-NeXT-Video [71] (Visual)
		Text-based	Feature-based	Text-based	Feature-based
Majority voting
Fold 0		17.96	32.83	32.91	42.74
Fold 1		44.46	46.59	30.08	49.47
Fold 2		18.61	45.45	31.87	45.67
Fold 3		35.10	30.15	34.06	40.73
Fold 4		28.15	54.32	26.05	53.96
Average logits
Fold 0		20.76	36.56	32.91	46.52	12.02
Fold 1		42.12	49.01	26.92	43.98	34.20
Fold 2		18.61	45.45	31.87	44.66	25.06
Fold 3		35.10	29.90	39.21	37.14	29.23
Fold 4		28.15	54.32	16.50	53.96	26.45
Average probabilities
Fold 0		17.96	36.56	32.91	42.74
Fold 1		44.46	40.98	26.92	40.38
Fold 2		18.61	36.55	31.87	34.80
Fold 3		35.10	29.90	34.06	40.73
Fold 4		28.15	54.32	22.35	53.96

Table 10: Video-level weighted $F1$ score on C-EXPR-DB^∗ validation sets.

4.3 7th ABAW CER Challenge Results

Table 11 presents our results in the 7th ABAW CER Challenge Results. In our first four submissions, we presented our text-based (100% of MELD), feature-based (1% and 100% of MELD), and zero-shot MLLM models. Except for the zero-shot case, the training is done over basic emotion dataset MELD, as described in Section 4.1. Then, we combined the predicted labels to construct a compound emotion prediction on C-EXPR-DB. Over these four submissions, the feature-based method leads with an $F1$ score of 22.64%, followed by the text-based method with a score of 19.86%. Our final submission, which performs frame-level prediction fusion across the four submissions, achieved the highest score of 25.90%, leading to rank three in this challenge.

Methods		Average $F1$ Score
Netease Fuxi AI Lab, Liu et al [41]		60.63
HSEmotion, Savchenko [53]		32.43
HFUT-MAC2, Liu et al [42]		22.81
AIPL-BME-SEU, Li et al [38]		16.44
ETS-LIVIA (Ours methods)
$\cdot$ Text-based (100% of MELD)		19.86
$\cdot$ Feature-based (100% of MELD)		22.64
$\cdot$ Feature-based (1% of MELD)		16.20
$\cdot$ Zero-shot MLLM LLaVa-NeXT-Video [71]		17.67
$\cdot$ Frame-based ensembling of our 4 submissions		25.90

Table 11: ABAW CER Challenge: frame-level average $F1$ score on C-EXPR-DB test set.

4.4 Computation Time

We present in Table 9 the computation time and the number of parameters of different methods. Computations are done on an NVIDIA A100 GPU. The reported time here does not account for data pre-processing such as face detection/cropping/alignment, audio transcript extraction, cues extraction (action units, basic emotions, audio tone), and feature extractions in frozen encoders (feature approach). The train time is measured with a batch size of 8. A window of 224 with a hop length of 16 is used for the feature-based. A window of 15 with a hop length of 10 is used for text-based. The number of parameters in the feature-based method is decomposed as follows: 218.91M for the three feature extractors (37.28M for ResNet50 [18], 72.14M for VGGish [19], 109.48M for BERT [10]), and 5M for the fusion module. VGGish and BERT models are used only to pre-compute and store the features on disk to speed up training and evaluation. They are not used in any computations afterward. The number of parameters in the text-based method is controlled by QLoRA [9].

The total number of FLOPs and MACs is computed on the same device, an NVIDIA A100 GPU using the library pypi.org/project/calflops (V0.3.2) over a sequence of 224 frames with roughly 9 seconds length. For the feature-based model, the entire sequence is processed at once. The text input has roughly 12 words. The total studied model includes the 3 backbones, with 1.87 TFLOPs and 938.76 GMACs in total. This amounts to 8.39 GFLOPs and 4.19 GMACs per frame. Our experiments use offline audio and text backbones to extract the corresponding features. This leads to 1.36 TFLOPs and 680.31 GMACs for the process of the full sequence. Most of the total computations are consumed by the visual backbone (ResNet50 [18]) alone, with 1.35 TFLOPs and 678.10 GMACs. For the text-based model, we use one prompt per frame with a context window size of 20. This amounts to a total of 597.52 TFLOPs and 298.75 TMACs, which is equivalent to 2.67 TFLOPs and 1.33 TMACs per frame. These results indicate that text-based model consumes more than 300 times the computation needed by feature-based. This makes text-based approach quite computationally expensive. This approach also requires a lot of preprocessing by large models to extract additional textual information such as action units, emotions, and audio cues from visual and audio modalities. Note that all measurements reported here depend on the sequence length and account only for the forward computations while excluding all preprocessing steps such as face detection.

4.5 Challenges of Textualizing Modalities

Using deep feature-based models for multimodal ER is the most common approach. It is easy to use and it requires less effort and expertise from the user to implement. Adequate features can be automatically extracted and combined from different modalities by a feature extractor without the need for manual intervention. Since we provide full data to the model, we can assume that there is no loss of information at the model input. In addition, it is easily transferable to other tasks without much change. The publicly available pre-trained feature extractors make it more attractive. However, a well-known bottleneck of this approach is the multimodal and spatio-temporal fusion of diverse modalities [39].

Modality textualization is a very recent topic [17]. While it can leverage the very recent progress of LLMs, it still faces several limitations to be a practical and competitive approach to the feature-based method. We can mention two main limitations. The first one is the need for domain experts to select the cues to be extracted from each modality and how they should be textualized. For example, cues other than AUs could be extracted in the visual modality, and there could also be different ways to convert them into text. Handcrafting of cues is challenging as they are not guaranteed to provide optimum textualization. Moreover, each modality requires its specific textualization, which depends on the task. Changing the application or task requires new domain experts and new textual cues that are most suitable. Models are less transferable to other applications, which limits their usage.

Another issue is the loss of information. The process of textualization performs a discretization of a modality from row data to textual descriptions. This mapping will most likely lead to information loss and poor performance. This has been observed on C-EXPR-DB^∗ where the feature-based approach outperforms on visual, whereas its counterpart yields the lowest performance, indicating a potential loss of information during our choice for textualization.

A main benefit of using a text-based approach is to leverage the potential of LLMs when dealing with a dataset with rich transcripts. Combining rich transcripts with textualized modalities may lead to higher ER accuracy than a feature-based method, as observed over MELD. However, conflicting modalities could lead to poor performance over compound emotions. Feature-based methods leverage fusion, especially late fusion, which may balance this conflict. However, text-based approaches lack such explicit modules as all modalities are treated indistinguishably as they are all processed by the same module. Although the authors of [17] motivated textualization as an easy way to perform multimodal fusion, such said fusion could be its limitation. Performing very early fusion by manually converting all modalities into text to be processed by the same module, such as an LLM, could hinder the benefit of a modality and make it less efficient, as most of the information could already be lost. Most of the work needs to be done by the LLM to recover the missing information. However, in the feature-based approach, the specialized feature extractor per modality does a lot of work leading to reliable features. This eases the late fusion. We note that text-based approach could be computationally expensive with 597.52 TFLOPs for inference over a sequence of 224 frames while feature-based method yields 1.87 TFLOPs.

The choice of the best approach for CER or basic ER remains an open question. Similarly, the choice and design of textual cues of different modalities are still in their early stages. The feature-based models are easier to apply, yet text-based models can yield better results when dealing with rich text.

5 Conclusion

Multimodal ER is a challenging task, especially when recognizing the complex compound emotions that are captured in real-world unconstrained videos. The central question in this study is: how textualized modeling performs compared to feature-based modeling for CER in videos?. We performed several experiments to compare the performance of deep feature-based and text-based models on CER and basic ER datasets. Our results have uncovered several challenges related to modality textualization on C-EXPR-DB, where text transcripts are captured in the wild. Feature-based methods may still provide better accuracy in this case. However, a text-based approach may yield better results when videos have high-quality transcripts.