Incremental Extension of GraphSmile? EDL Reliability Estimation and Edge-Weight Scaling

Multimodal Emotion Recognition in Conversation (MERC) is an advanced research area that integrates cross-modal understanding and contextual reasoning through text-speech-visual fusion, with applications spanning diverse scenarios including student emotion monitoring in high school classroom interactions. Although existing research has made progress in multimodal alignment and dialogue relationship modeling through architectures such as graph neural networks and pre-trained language models, challenges persist in dataset overfitting and underexplored generative approaches. In this study, a generative MERC framework based on Multimodal Large Language Models (MLLMs) is proposed, employing Video-LLaMA, an open-source and advanced tri-modal foundation model, for end-to-end multimodal emotion reasoning. Carefully crafted structured prompts are used to align emotion semantics with dataset annotations, combined with Low-Rank Adaptation (LoRA) for parameter-efficient optimization. The method achieves a state-of-the-art weighted F1-score of 68.57% on the MELD benchmark. Further, exploratory experiments on dynamic modality combinations and fine-tuning strategies offer actionable insights for MLLM-based MERC research. This work not only advances emotion understanding in dialogues but also highlights MLLMs’ potential in complex multimodal reasoning tasks.

No abstract available

No abstract available

Emotion Recognition in Conversation (ERC) has gained considerable attention due to its importance in human-computer interaction. In ERC task, the combination of multimodal information and contextual information is necessary since it can help the model understand emotional changes in the context from multiple perspectives. As Graph Neural Networks (GNNs) have shown the superiority in relation modeling, many graph-based methods have been proposed to improve the performance of emotion recognition by utilizing the edges to mine the contextual relationship and multimodal relationship in a conversation. However, the existence of numerous redundant edges and excessively complex modality interaction in the graph hinders the model from capturing the truly effective dependency information for emotion recognition. In this paper, we propose a Hypergraph based Contextual Relationship Modeling Method (HyperCRM) to carry out the ERC task. HyperCRM models a conversation as a hypergraph instead of a graph, which defines two types of hyperedges, namely speaker-level hyperedge and sequence-level hyperedge, to represent the contextual relationship within the same speaker and the local sequence of the conversation, respectively. Multimodal information is leveraged here as the node feature representation by the feature concatenation. In addition, an improved hypergraph convolution method is designed to capture the long-range contextual information by three-stage information propagation in the hypergraph, including node-hyperedge, hyperedge-hyperedge and node-hyperedge. The extensive experiments on two public datasets shows the new State-Of-The-Art (SOTA) results, to further demonstrate that the proposed method can simply make use of the multimodal information and effectively model the complex contextual relationships in the conversation.

Multimodal Emotion Recognition in Conversation (MERC) focuses on detecting the emotions expressed by speakers in each utterance. Recent research has increasingly leveraged graph-based models to capture interactive relationships in conversations, enhancing the ability to extract emotional cues. However, existing methods primarily focus on explicit utterance-level relationships, neglecting both the implicit connections within individual modality and the differences in implicit relationships across modalities. Moreover, these methods often overlook the role of sentimental features in conversation history in cross-modal semantic alignment. To address these issues, we propose a novel model that employs modality-adaptive hybrid relational graphs to enrich the dialogue graph by inferring implicit relationships between nodes within each modality. Furthermore, we introduce historical sentiment through a progressive strategy that utilizes contrastive learning to refine cross-modal semantic alignment. Experimental results demonstrate the superior performance of our approach over state-of-the-art methods on the IEMOCAP and MELD datasets. Our code is available at https://github.com/cgao-comp/HRG-SSA.

No abstract available

No abstract available

,

Emotion recognition in conversation (ERC) is a vital task that requires deciphering human emotions through analysis of contextual and multimodal information. However, extant research on ERC concentrates predominantly on investigating multimodal fusion while overlooking the model's constraints in dealing with unimodal representation discrepancy and speaker dependencies. To address the aforementioned problems, this paper proposes a Hierarchical decision fusion-based Local-Global Graph Neural Network for multimodal ERC (HiMul-LGG). HiMul-LGG employs a hierarchical decision fusion strategy to ensure feature alignment across modalities. Moreover, HiMul-LGG also adopts a local-global graph neural network architecture to reinforce inter-modality and intra-modality speaker dependency. Additionally, HiMul-LGG utilizes a cross-modal multi-head attention mechanism to promote interplay between modalities. We evaluate HiMul-LGG on two emotion recognition datasets, IEMOCAP and MELD, where HiMul-LGG outperforms existing methods. The results of the ablation study also imply the effectiveness of the proposed hierarchical decision fusion strategy and local-global structure of Graph construction.

Robust emotion recognition is a prerequisite for natural, fluid human–computer interaction, yet conversational settings remain challenging because emotions are shaped simultaneously by global topic flow and local speaker-to-speaker dependencies. Here, we introduce GASMER—Graph-Adaptive Structure for Multimodal Emotion Recognition—a unified architecture that tackles both issues. It uses the correlation structure based on graph neural networks (GNNs) to model the complex dependencies in the conversation, while adaptively learning the graph structure for GNNs. The experiments indicate that our model has strong performance that outperforms all existing graph-based approaches, and remains competitive when compared to recent multimodal fusion models, underscoring the importance of combining fine-grained multimodal fusion with adaptive graph learning for conversational emotion recognition. On the IEMOCAP dataset, GASMER improves accuracy by 2.7% and the weighted F1-score by 3.6% compared to the best baseline. On the MOSEI dataset, it achieves a 1.2% gain in binary classification accuracy (ACC-2).

Emotion recognition in conversations (ERC) has garnered significant attention for its critical role in human-computer interaction systems. ERC benefits from multimodal data, which offers diverse perspectives on emotional states, and commonsense knowledge (CSK), which enriches the context by incorporating real-world understanding of human behavior. However, existing ERC studies have not fully exploited the potential of multimodal-CSK interactions for complementary information learning from these sources. To address this, we innovatively propose a Knowledge-Aware Multimodal Interaction Network (KA-MIN). KA-MIN is designed to capture complementary emotional information from CSK-multimodal interactions, thereby facilitating the ERC task. To achieve this, KA-MIN begins by combining six relation types of CSK, leveraging their differences between multimodal emotional information. The fused CSK features are then refined to incorporate context and emotional information using multimodal contextual guidance. Subsequently, we construct a novel knowledge-aware multimodal graph structure that allows the CSK information to interact with multimodal information, leading to more comprehensive multimodal and context modeling. During the graph learning process, the CSK-multimodal interactions capture the complementary emotional information between CSK and multimodal features. Finally, we dynamically fuse the multimodal emotional information with the informative CSK and textual guidance to obtain the final utterance representations, which encompass effective emotional information from both multimodal and CSK features. Extensive experiments on two popular multimodal ERC datasets demonstrate the superiority and effectiveness of the proposed KA-MIN framework.

Multimodal Emotion Recognition in Conversation (MERC) is a critical task for comprehending complex emotions from multiple sources in human-computer interaction. Despite existing methods having made progress, they still struggle to capture hierarchy characteristics and high-order interactions between modalities. Recent research adopts Euclidean space geometry for multimodal representation learning and fusion, resulting in representation and distance distortions. To tackle these challenges, we propose a multimodal hyperbolic embedding and hyperbolic hypergraph fusion framework. A hyperbolic embedding learning module projects modality representations into the Poincaré ball model to perceive hierarchies. To enhance representation discriminative and semantic consistency, it incorporates radial and angular contrastive learning objectives. A hyperbolic hypergraph fusion module further exploits high-order latent relationships and improves fusion results. Experimental results demonstrate that the proposed framework outperforms state-of-the-art methods and augments stability on IEMOCAP and MELD datasets.

No abstract available

Multimodal Emotion Recognition in Conversation (ERC) is a task of predicting the emotion of each utterance in a conversation by utilizing both verbal and non-verbal modalities. However, existing approaches often struggle to bridge cross-modal gaps, resulting in misaligned features and frequent misclassification of minority emotions into semantically similar majority emotions. To address these challenges, we propose MERNet, a framework that employs cross-modal knowledge distillation and contrastive learning to align multimodal features and effectively distinguish subtle emotions in conversations. Our framework consists of two stages: 1) guiding non-verbal modalities with the text modality to transfer knowledge and align their features, and 2) applying contrastive learning with emotion labels as anchors to distinguish subtle differences between similar emotions and address the class imbalance problem. Experiments conducted on two benchmark datasets, IEMOCAP and MELD, demonstrate that our MERNet outperforms existing state-of-the-art models.

Multimodal emotion recognition in conversation (MERC) involves predicting the emotion category of a conversation on the basis of textual, acoustic, and visual modalities. Information from these diverse modalities can reinforce each other to enhance the accuracy of emotion prediction. However, some information modalities may be absent in real-world applications and information from various modalities may be difficult to integrate. Therefore, a suitable strategy is required to compensate for missing modalities by using information from the available modalities and prioritizing important information. Consequently, this study developed a graph convolutional network (GCN) model with a feature compensation module and dual-channel second-order pooling module for MERC. This model initially uses a GCN to compensate for missing features by aggregating features corresponding to the same utterance node. Subsequently, it applies dual-channel second-order pooling to sift through and integrate all features. Empirical evaluations of the proposed model against other baseline models on two benchmark datasets, namely the IEMOCAP and MELD datasets, indicated that the proposed model outperformed the other models; this finding underscores the effectiveness of the proposed model in recognizing the emotions expressed in multimodal dialogue data.

Multimodal Emotion Recognition in Conversation (MERC) is an important element in human-machine interaction. It allows machines to automatically identify and track the emotional status of speakers during a conversation in a multimodal setting. However, the conversations involving various audio and visual cues aligned with textual cues are very complex. Recent works have tried integrating the audio and visual modalities with textual to improve the performance of emotion recognition in conversation. Although many MERC models leverage textual, audio, and visual modalities, those models assume that the speaker’s textual utterance, audio speech, and facial sequences are present. However, a conversation may contain multiple parties, among which only one is the speaker. Previous MERC assumed the availability of all modalities, but in many instances, one or more modalities may be unavailable during multiparty conversations. To tackle these issues, we propose the Possible Speaker Informed Multimodal Emotion Recognition in Conversation framework (PSI). PSI is specifically tasked to extract audio (speech) and visual (face) sequences of a possible speaker in the presence of multiple parties. Further, PSI seamlessly extracts the rich unimodal features and fuses them while addressing the unavailability of specific modalities. PSI demonstrates competitive performance with existing state-of-the-art models through experiments with a benchmark dataset.

Emotion Recognition in Conversation (ERC) plays a significant role in enhancing the responsiveness of dialogue systems by inferring users' emotions through various modalities, including audio, visual, and text. However, previous ERC studies have primarily focused on learning contextual and speakerdependent features from the text modality, while non-verbal modalities have not been fully utilized. Additionally, many multimodal ERC approaches assign equal importance to all modalities and fuse them accordingly, which limits the effective use of the text modality that carries strong emotional cues. To address these limitations, we propose a method that applies Knowledge Distillation (KD), where the text modality is used as the teacher model and the non-verbal modalities serve as student models. Additionally, we incorporate Emotion Label-Guided Contrastive Learning (ELCL) to enhance representation learning of emotions. In addition, instead of using the commonly used Supervised Contrastive Loss (SCL) in contrastive learning, we design an Focal Contrastive Loss (FCL) that accounts for minority classes and hard positive samples with low similarity, thereby improving both training stability and representational balance. Experimental results demonstrate that our proposed model, MERC-KDC, achieves higher overall emotion recognition accuracy on the IEMOCAP dataset compared to existing methods. In particular, the model shows significant improvement in recognizing minority class emotions.

Multimodal emotion recognition in conversation (MERC) requires modeling complex, context-dependent affective cues from heterogeneous modalities. However, most existing methods treat all modalities with equal importance throughout the dialogue, lacking a mechanism to adaptively filter redundant cues or emphasize salient signals as emotional dynamics shift. To address this, we propose the Adaptive Compression with Semantic Disentanglement Network (ACoSDN). ACoSDN introduces a context-aware controller that adjusts compression strength based on four dialogue structure signals—emotion shift, speaker gap, prediction entropy, and utterance position—enabling adaptive information retention. Furthermore, we disentangle compressed cross-modal representations into shared, unique, and synergistic semantic factors via a variational module, enhancing emotional interpretability and robustness. Experiments on two public datasets demonstrate that ACoSDN effectively handles emotion recognition tasks, showing robust performance particularly on ambiguous or minority emotion classes.

No abstract available

Multimodal emotion recognition in conversations is inherently challenging due to ambiguous cues, modality conflicts, and temporal dynamics, all of which contribute to complex and diverse uncertainty sources. While some recent methods incorporate uncertainty modeling, they often focus on overall prediction confidence, without explicitly distinguishing the different sources of uncertainty introduced by underlying factors. To address these challenges, we propose a novel Emotion-Specific Evidence Decomposition framework (ESED) that leverages evidential deep learning to explicitly model and disentangle multimodal emotional uncertainty. Rather than directly fusing features, ESED decomposes each modality's evidence into three interpretable components: (1) emotion-consistent evidence, capturing shared emotional cues across modalities; (2) emotion-specific evidence, highlighting the unique emotional role of each modality; and (3) dynamic evidence, modeling utterance-level temporal variations. These components are adaptively weighted based on emotional intensity, ambiguity, and dynamicity, quantified via prediction entropy, inter-modal divergence, and temporal variance. The final prediction is obtained through an adaptive fusion of these weighted components. Extensive experiments demonstrate that ESED outperforms the state-of-the-art methods on the MELD and IEMOCAP datasets, demonstrating the effectiveness of our proposed method.

The dynamics and complexity of emotional expressions in multi-party dialogues are frequently overlooked by traditional approaches. These expressions are characterised by participants' predisposition towards indirect emotional language, contextual dependency of emotions, and changes in emotional state with topic transitions. A semantic-enhanced heterogeneous dialogue graph model is presented to effectively capture emotional expressions in multi-party talks in order to overcome these issues. In order to extract implicit information from utterances, the model incorporates external emotional knowledge and uses graph neural networks to reconstruct adjacency matrices. Additionally, the impact of topic information on emotions is taken into consideration by using an improved variational autoencoder. A topic segmentation technique is used to create dialogue subgraphs for emotion categorization, together with semantic and topic information. Experimental results on the IEMOCAP, MELD, and EmoryNLP datasets show that the proposed model outperforms the state-of-the-art baseline models in terms of classification accuracy, greatly improving the performance of multi-party discourse emotion analysis.

Conversational Aspect-based Sentiment Quadruple Extraction refers to identify fine-grained sentiment quadruples from dialogue data, constituting an important AI application. While existing studies have proposed various methods to enhance dialogue representations, they often fall short in effectively modeling annotation labels and composite relational structures in dialogues. To address these limitations, we introduces a Label- and Neighborhood-aware Heterogeneous Graph Network (LnHGN) to better capture discourse structural information. Our framework incorporates three key innovations: (1) a node type-aware heterogeneous graph reconstruction method that fully utilizes sentiment element annotations to enhance representation learning by reducing intra-class node distances; (2) a neighborhood-aware stacked Simplified Graph Convolution (SGC) model that adaptively weights multi-order meta-paths to effectively aggregate multi-hop relational information while avoiding noise and limited expressiveness from manual meta-path design; and (3) a Focal Loss optimization strategy to mitigate severe class imbalance in entity and relation classification caused by sparse sentiment annotations. The experimental results on two benchmark datasets substantiate the effectiveness of the proposed LnHGN, achieving average improvements of 1.43% and 1.56% respectively over state-of-the-art baselines.

Multimodal Conversational Aspect-based Sentiment Analysis (MCA BSA) is a challenging task for multimodal dialogue understanding. Existing works often treat the entire dialogue as a flat sequence and feed it into Large Language Models (LLMs) for pipeline-style generation. However, these methods sometimes accumulate errors and overlook critical discourse structure and fine-grained inter-word relations that are essential for accurate sentiment reasoning. To address these limitations, we propose SDG-MLLM, a unified generative framework that integrates Structured Dialogue Graphs into Multimodal LLM (MLLM) for an end-to-end MCABSA. Specifically, we construct heterogeneous dialogue graphs that capture diverse structural relations, including syntactic dependencies, coreference links, speaker turns, reply flow, semantic role labeling, and sentiment propagation paths. These graphs are encoded using a heterogeneous dialogue graph encoder, and the resulting structure-aware graph features are injected into the embedding layer of LLM. Furthermore, SDG-MLLM incorporates aligned multimodal features such as image, audio, and video cues at the utterance level to enable unified and context-aware multimodal reasoning. Experiments on the MCABSA dataset show that SDG-MLLM significantly outperforms strong baselines across multiple tasks. In addition, our method also achieved top performance in the ACM MM 2025 Grand Challenge of MCABSA. Our code is available at https://github.com/Liuxj-Anya/SDG-MLLM.

Multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) are two crucial tasks for understanding human affective behavior, aiming to detect sentiment polarity and emotion category from multimodal cues in conversational videos. Despite recent advances leveraging dialogue-level graph structures and Transformer-based frameworks to capture cross-modal contextual dependencies, existing approaches still struggle with two persistent challenges: the fragmentation in modeling narrative-affective evolution and the semantic inconsistency caused by heterogeneous modality drift. To address these issues, we propose EvoGraph-Former, a unified framework for multimodal dialogue affective understanding across both MSA and ERC tasks. Specifically, EvoGraph-Former constructs a bi-channel evolutionary graph that explicitly integrates narrative and affective trajectories to capture complex intra-dialogue affective dynamics. Furthermore, an affect-anchor alignment mechanism is introduced to perform dual-stage semantic calibration on multimodal features, both before and after graph construction, thereby mitigating modality discrepancy at its source. Extensive experiments on four widely used benchmarks demonstrate that EvoGraph-Former outperforms prior state-of-the-art models across MSA and ERC, highlighting its effectiveness and generalizability.

No abstract available

No abstract available

The goal of multimodal aspect-based sentiment analysis (MABSA) is to identify the sentiment polarity associated with each aspect target by analyzing pairs of text and images. Most existing methods overlook the heterogeneity between different modalities and fail to fully integrate the data features of different modalities. In addition, connections between modalities are made in a fully-connected approach, leading to increased noise and reduced model performance. This research introduces a Multimodal Heterogeneous Graph Neural Network (MHGNN) based on local connectivity. MHGNN aims to efficiently integrate features from various modalities while minimizing noise by constructing a multimodal heterogeneous interaction graph and establishing precise local connections between different modal nodes. Specifically, to efficiently integrate the features of different modalities, we construct a heterogeneous graph to connect different modalities. Heterogeneous graph can more effectively represent relationships between various modalities and facilitate the integration of the features of different modalities. Additionally, to reduce the effect of noise, we constructed syntactic dependency trees for textual modals. The syntactic dependency tree represents the local connection relationship between text nodes and aspect nodes. And we established local connections between aspect nodes and image nodes by Token-patch alignment. The local connection reduces the introduction of noise and enhances the model's overall performance. Extensive experiments on three multimodal datasets indicate that MHGNN outperforms existing methods.

Sentiment analysis remains challenging due to the complex, intertwined relationships among sentiment expressions, contextual cues, and emotional features distributed across heterogeneous data sources. Conventional deep learning and transformer-based models often treat sentiments as isolated units, failing to capture these rich, multi-perspective interactions. To address these limitations, this study introduces a Two-Fold Multi-Perspective Heterogeneous Graph Neural Network (TFMPHGNN) that jointly models sentiment, emotion, and contextual dependencies within a dual-stage heterogeneous graph framework. The first stage employs a meta-path-based encoder integrated with a capsule network to capture hierarchical semantic relationships among sentiment-emotion-context nodes, while the second stage utilizes a multi-channel graph convolutional network (MC-GCN) to learn complementary topological, semantic, and collaborative representations of sentiment-emotion pairs. A variational autoencoder (VAE) further denoises and refines latent embeddings. Experiments on the newly developed VaKSent-2025 corpus show that TFMPHGNN outperforms eight state-of-the-art graph-based baselines by 4.67% in accuracy, 2.7% in F1-micro, and 4.2% in F1-weighted, with statistical significance testing confirming the reliability of these gains. An extended ablation analysis further demonstrates that the collaborative fusion channel achieves 0.9387 accuracy, representing improvements of 7.5% and 7.9% over the topological-only and semantic-only channels, respectively, underscoring the synergistic value of integrating multiple graph perspectives. Collectively, these results indicate that TFMPHGNN effectively captures complex sentiment-emotion interdependencies and offers a robust, interpretable framework for fine-grained sentiment understanding.

This paper presents an integrated framework for cross-language hotel review sentiment analysis that combines multi-agent federated learning with heterogeneous graph attention networks to address privacy preservation and multilingual data processing challenges in hospitality reputation management. Our system enables collaborative model training across distributed review platforms while maintaining data locality requirements and achieving improved cross-language sentiment classification performance. Beyond sentiment analysis, we developed dynamic reputation management and fake review detection capabilities that enable proactive intervention strategies for hospitality businesses. The heterogeneous graph architecture captures complex relationships between multilingual textual content, user behaviors, temporal patterns, and service attributes through specialized attention mechanisms. Experimental evaluation on a comprehensive multilingual dataset of 154,680 reviews across four languages demonstrates 89.7 ± 0.007 accuracy in sentiment classification with 0.925 privacy preservation score (Table 6), representing 2.6% point improvement over the strongest baseline XLM-RoBERTa large (87.1 ± 0.008 accuracy, paired t-test p = 0.002). The dynamic reputation management component provides real-time monitoring capabilities with early warning detection, achieving 93.4 ± 0.012 fake review identification accuracy and 66.2% reduction in response time compared to traditional centralized approaches (Table 9). The system offers practical applications for hospitality businesses seeking proactive reputation management while ensuring compliance with international data privacy regulations including GDPR and CCPA. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-026-41500-8.

Aspect-based sentiment analysis aims to predict sentiment polarities of given aspects in text. Most current approaches employ attention-based neural methods to capture semantic relationships between aspects and words in one sentence. However, these methods ignore the fact that sentences with the same aspect and sentiment polarity often share the structure and semantic information in a domain, which leads to lower model performance. To mitigate this problem, we propose a heterogeneous aspect graph neural network (HAGNN) to learn the structure and semantic knowledge from intersentence relationships. Our model is a heterogeneous graph neural network since it contains three different kinds of nodes: word nodes, aspect nodes, and sentence nodes. These nodes can pass structure and semantic information between each other and update their embeddings to improve the performance of our model. To the best of our knowledge, we are the first to use a heterogeneous graph to capture relationships between sentences and aspects. The experimental results on five public datasets show the effectiveness of our model outperforming some state-of-the-art models.

No abstract available

No abstract available

No abstract available

No abstract available

In multimodal sentiment analysis, existing methods often struggle with dynamic context modeling in multi-speaker dialogues, speaker interaction, and effective multimodal fusion. To address this, we propose a model based on a heterogeneous directed acyclic graph neural network. It uses BiLSTM and Gated TCN for multi-granularity context modeling and introduces speaker embeddings to capture individual expression differences. The DAG structure helps avoid cyclic dependencies and oversmoothing in traditional GNNs. Experiments show weighted F1 scores of $\mathbf{7 0. 6 8 \%}$ on IEMOCAP and $\mathbf{6 0. 8 3 \%}$ on MELD.

No abstract available

No abstract available

Evidential Deep Learning (EDL) has emerged as a widely adopted framework for quantifying uncertainty in classification tasks by modeling a higher-order prior on classification probabilities using a Dirichlet distribution. Recent advances in EDL have notably progressed, primarily by calibrating the predicted Dirichlet distribution to enhance uncertainty estimation. However, these methods often overlook the role of feature representations, the fundamental components of deep learning models that directly impact both classification performance and uncertainty estimation. This paper adopts a feature-centric perspective to explore the relationship between predicted evidence and feature representations in EDL. We demonstrate that existing EDL approaches struggle to learn sufficiently discriminative features as the number of classes increases, which significantly undermines their capacity to produce reliable uncertainty estimates. To tackle this issue, we propose Tangential Gradient Descent (TGD), a novel feature optimization method that explicitly decomposes the loss gradient with respect to feature representations into tangential and radial components. By accelerating updates along the tangential component, TGD promotes feature alignment to a simplex Equiangular Tight Frame (ETF), thereby maximizing inter-class angular separability for enhanced classification. At the same time, updates along the radial component foster discriminative feature projections that effectively differentiate between certain and uncertain samples, leading to improved uncertainty estimation. Extensive experiments on benchmarks such as CIFAR-10, CIFAR-100, and ImageNet-200, covering both OOD detection and generalization across various class scales, reveal that our approach consistently surpasses conventional EDL methods in both uncertainty estimation and scalability.

Cervical optical coherence tomography (OCT) is crucial in diagnosing cervical diseases due to its high-resolution imaging and non-invasive characteristics. However, recent improvements in classifying cervical OCT images have mainly focused on accuracy, often neglecting the importance of uncertainty and robustness in disease predictions. Uncertainty estimation faces challenges in collecting evidence for the minority classes, leading to fake confidence in evidential deep learning. There are similar issues with cervical OCT image classification. This study proposes a novel evidential learning process for cervical disease screening using OCT. It provides a measure of decision margin for each class with two novel mechanisms to address the hinge posted by class imbalance. Specifically, we introduce a reweighted margin adjustment strategy to learn less biased evidence among classes and make reliable uncertainty estimations. Extensive experiments on internal and external cervical OCT datasets demonstrate the proposed method outperforms competitive baseline methods, providing more reliable and robust predictions, particularly in out-of-distributions scenarios. By improving uncertainty estimation for class imbalance, our method offers valuable insights for practical clinical applications, highlighting the importance of measuring uncertainty in medical AI-aided diagnosis systems.

In computer-aided diagnosis, the extreme imbalance in disease incidence rates often results in the omission of rare conditions, leading to out-of-distribution (OOD) samples during testing. To prevent unreliable diagnostic outputs, detecting these OOD samples becomes essential for clinical jnsafety. While Evidential Deep Learning (EDL) and its variants have shown great promise in detecting outliers, their clinical application remains challenging due to the variability in medical images. We find that when encountering samples with high data uncertainty, the Kullback-Leibler divergence (KL) in EDL tends to suppress inherent ambiguity, resulting in an over-penalty effect in evidence estimation that impairs discrimination between ambiguous in-distribution cases and true outliers. Motivated by the confirmatory and differential diagnostic process in clinical practice, we propose Differential Evidential Deep Learning (D-EDL), a simple but effective method for robust OOD detection. Specifically, we treat KL as a confirmatory restriction and innovatively replace it with a Ruling Out Module (ROM) for differential restriction, which reduces over-penalty on ambiguous ID samples while maintaining OOD sensitivity. Considering extreme testing scenarios, we introduce test-time Raw evidence Inference (RI) to bypass instability in uncertainty estimation with refined evidence and further improve robustness and precision. Finally, we propose the Balanced Detection Score (BDS) to quantify the potential on clinical performance when optimally balancing misdiagnoses and missed diagnoses across varying sensitivities. Experimental results on ISIC2019, Bone Marrow Cytomorphology datasets and EDDFS dataset demonstrate that our D-EDL outperforms state-of-the-art OOD detection methods, achieving significant improvements in robustness and clinical applicability. Code for D-EDL is available at https://github.com/KellaDoe/Differential_EDL.

Evidential Deep Learning (EDL), grounded in Evidence Theory and Subjective Logic (SL), provides a robust framework to estimate uncertainty for out-of-distribution (OOD) detection alongside traditional classification probabilities. However, the EDL framework is constrained by its focus on evidence that supports only single categories, neglecting the other collective evidences that could corroborate multiple in-distribution categories. This limitation leads to a diminished estimation of uncertainty and a subsequent decline in OOD detection performance. Additionally, EDL encounters the vanishing gradient problem within its fully-connected layers, further degrading classification accuracy. To address these issues, we introduce hyper-domain and propose Hyper-opinion Evidential Deep Learning (HEDL). HEDL extends the evidence modeling paradigm by explicitly integrating sharp evidence, which supports a singular category, with vague evidence that accommodates multiple potential categories. Additionally, we propose a novel opinion projection mechanism that translates hyper-opinion into multinomial-opinion, which is then optimized within the EDL framework to ensure precise classification and refined uncertainty estimation. HEDL integrates evidences across various categories to yield a holistic evidentiary foundation for achieving superior OOD detection. Furthermore, our proposed opinion projection method effectively mitigates the vanishing gradient issue, ensuring classification accuracy without additional model complexity. Extensive experiments over many datasets demonstrate our proposed method outperforms existing OOD detection methods.

This article proposes Uncertainty-Guided Evidential Deep Learning (UGEDLS) for robust electromagnetic inverse scattering reconstruction and uncertainty estimation. We address the limitation of the standard evidential deep learning scheme (EDLS), where models tend to excessively inflate uncertainty for samples with high nonlinearity to minimize loss rather than improving prediction accuracy. UGEDLS mitigates this by penalizing unreasonable uncertainty inflation, forcing the model to focus on accurate mean reconstruction. Extensive experiments on numerical and experimental data demonstrate that UGEDLS outperforms EDLS for highly nonlinear samples in reconstruction accuracy and generalization ability, offering interpretable uncertainty estimates that effectively highlight prediction errors, particularly at scatterer boundaries.

In order to navigate safely and reliably in novel environments, robots must estimate perceptual uncertainty when confronted with out-of-distribution (OOD) obstacles not seen in training data. We present a method to accurately estimate pixel-wise uncertainty in semantic segmentation without requiring real or synthetic OOD examples at training time. From a shared per-pixel latent feature representation, a classification network predicts a categorical distribution over semantic labels, while a normalizing flow estimates the probability density of features under the training distribution. The label distribution and density estimates are combined in a Dirichlet-based evidential uncertainty framework that efficiently computes epistemic and aleatoric uncertainty in a single neural network forward pass. Our method is enabled by three key contributions. First, we simplify the problem of learning a transformation to the training data density by starting from a fitted Gaussian mixture model instead of the conventional standard normal distribution. Second, we learn a richer and more expressive latent pixel representation to aid OOD detection by training a decoder to reconstruct input image patches. Third, we perform theoretical analysis of the loss function used in the evidential uncertainty framework and propose a principled objective that more accurately balances training the classification and density estimation networks. We demonstrate the accuracy of our uncertainty estimation approach under long-tail OOD obstacle classes for semantic segmentation in both off-road and urban driving environments.

No abstract available

No abstract available

No abstract available

,

Multimodal emotion analysis can capture and understand human emotions more comprehensively by integrating information from different modalities, thereby improving the accuracy and robustness of emotion recognition. However, existing approaches often suffer from feature redundancy and low modal interaction efficiency in multimodal sentiment analysis. To address these challenges, this paper proposes a deep learning framework that integrates multi-scale features and cross-modal attention mechanisms. We conduct a systematic evaluation of five fusion architectures (CMAC, OTE, HSTEC, NaiveCat, NaiveCombine) and multiple feature extractors (InceptionV3, WideResNet50, MobileNetV2, ResNet50/101, BERT/RoBERTa). Extensive experiments demonstrate that: 1) The combination of BERT and WideResNet50 achieves the optimal balance between accuracy and computational efficiency; 2) The CMAC framework significantly outperforms baseline models through hierarchical feature fusion, with its adaptive weight allocation mechanism effectively mitigating overfitting; 3) Channel attention mechanism exhibits the best performance in text feature extraction. Moreover, ablation experiments further confirm that text features are the dominant source of information.

Multimodal emotion recognition involves leveraging complementary relationships across modalities to enhance the assessment of human emotions. Networks that integrate diverse information sources outperform single-modal approaches while offering greater robustness against noisy or missing data. Current emotion recognition approaches often rely on cross-modal attention mechanisms, particularly audio and visual modalities; however, these methods do not assume the complementary nature of the data. Despite making this assumption, it is not uncommon to see non-complementary relationships arise in real-world data, reducing the effectiveness of feature integration that assumes consistent complementarity. While audio–visual co-learning provides a broader understanding of contextual information for practical implementation, discrepancies between audio and visual data, such as semantic inconsistencies, pose challenges and lay the groundwork for inaccurate predictions. In this way, they have limitations in modeling the intramodal and cross-modal interactions. In order to address these problems, we propose a multimodal learning framework for emotion recognition, called the Hybrid Multi-ATtention Network (HMATN). Specifically, we introduce a collaborative cross-attentional paradigm for audio–visual amalgamation, intending to effectively capture salient features over modalities while preserving both intermodal and intramodal relationships. The model calculates cross-attention weights by analyzing the relationship between combined feature illustrations and distinct modes. Meanwhile, the network employs the Hybrid Attention of Single and Parallel Cross-Modal (HASPCM) mechanism, comprising a single-modal attention component and a parallel cross-modal attention component, to harness complementary multimodal data and hidden features to improve representation. Additionally, these modules exploit complementary and concealed multimodal information to enhance the richness of feature representation. Finally, the efficiency of the proposed method is demonstrated through experiments on complex videos from the AffWild2 and AFEW-VA datasets. The findings of these tests show that the developed attentional audio–visual fusion model offers a cost-efficient solution that surpasses state-of-the-art techniques, even when the input data are noisy or missing modalities.

To address the problem of insufficient intra-modal feature extraction and inter-modal interaction information fusion in current multimodal sentiment analysis, a multi-feature fusion multimodal sentiment analysis model based on cross-attention mechanism is proposed. The model first uses subnetworks and self-attention mechanism to obtain important features of multiple modalities such as text, audio, and video. Then, it calculates the correlation between modalities through cross-modal cross-attention mechanism to achieve the interaction and fusion of multimodal information. After that, soft attention mechanism is used to assign attention weights to each modality feature. Finally, the modalities features are spliced to output the final sentiment classification result. The experimental results show that compared with the benchmark model on the public datasets CH-SIMS and CMU-MOSEI, this model has certain improvements in two-class accuracy, three-class accuracy, and F1 value.

This paper presents an engineering-oriented study of edge-aware graph neural network baselines for protein function prediction on the OGBN-Proteins benchmark. The benchmark represents a large protein-protein interaction graph whose edges carry eight-dimensional association evidence and whose proteins have 112 functional labels. The study focuses on practical design choices that strongly affect reproducibility and deployment cost: construction of node features from edge evidence, use of scalar edge weights inside message passing, normalization under species-level distribution shift, and post-hoc decision calibration. We compare multilayer perceptrons, GraphSAGE, and GIN baselines in PyTorch Geometric, using mean, sum, and max edge-to-node aggregation, Batch Normalization, Layer Normalization, and a species-conditioned Layer Normalization variant. Results are reported over three seeds with ROC-AUC, micro-F1, calibrated micro-F1, expected calibration error, training time, memory use, and parameter count. Sum aggregation is consistently the strongest edge-to-node construction. GraphSAGE with sum-based features forms the best accuracy-cost trade-off, with Batch Normalization reaching the highest ROC-AUC and conditional Layer Normalization retaining stronger fixed-threshold behavior. Per-label temperature scaling and per-label thresholds substantially improve multi-label decision quality with negligible change in ROC AUC, while light label-correlation smoothing yields small additional gains. The resulting protocol provides a compact, reproducible baseline for large edge-attributed biological graph settings. Together, these findings give practitioners clear default choices for feature construction, normalization, and decision calibration on edge-attributed protein graphs.

Server performance anomaly detection is a critical task in data science and system management, aiming to identify unusual patterns in server performance metrics. However, server data often exhibits high dimensionality and complexity, which makes it challenging to detect subtle and rare anomalies. Accurately modeling the relationships between various performance indicators, such as CPU load, memory usage, and network bandwidth, is especially difficult. Moreover, the scarcity of labeled anomaly data for training supervised models can limit the effectiveness of traditional anomaly detection methods. To address these challenges, we propose a novel method called POINGRAPH-AE(Poincare Embedding + Graph Attention Networks + Auto Encoder). It enhances anomaly detection performance by embedding server performance data into a graph structure, allowing us to capture both spatial and dependencies more effectively. Additionally, we incorporate a gain factor to improve edge weight modeling, ensuring that spatial relationships are preserved. Experimental results demonstrate that optimizing edge weights significantly improves detection performance, outperforming traditional methods. We propose an edge weight-based analysis method using graph neural networks, which provides interpretable detection results and offers insights into the anomaly’s context. The proposed method is shown to be robust and scalable, making it well-suited for detecting anomalies in complex, multimodal server performance data.

Multimodal sentiment analysis recognizes emotions through text, audio, and visual modalities, but data incompleteness is a major challenge. Existing methods often focus on specific types of deficiencies and perform poorly when multiple types of noise are present simultaneously. To address this issue, we propose a noise-prompted adversarial training framework with a multimodal interaction model to enhance the model’s robustness to missing modalities. The model first extracts common and unique features from each modality using a BERT text encoder and a shared-private encoder. Correlation measurements are then used to calculate the similarity between modalities, and a weighting mechanism is applied to the shared features. These features are deeply fused using a Transformer, and adversarial training combined with semantic reconstruction supervision helps the model learn a unified representation of noisy and clean data. Experimental results show that this method significantly improves the performance of multimodal sentiment analysis.

In multimodal sentiment analysis tasks, we propose a novel method of multi-feature skip-connection fusion model based on representation learning (MSFR). The MSFR maps all feature representations extracted by raw videos into modality-invariant and modality-disparate subspaces using shared and private encoders. By this work, we learn modality-invariant and modality-disparate representations, which can narrow the heterogeneity gap among the three modalities. Additionally, emotional redundancy and conflicting information between modalities significantly impact the fusion effectiveness. To alleviate this problem, we design a text-centered multi-feature skip-connection fusion module. The final fused representations of modality-invariant and modality-disparate subspaces are used for the final sentiment analysis tasks. The proposed MSFR is extensively tested on two public benchmark datasets, CMU-MOSI and CMU-MOSEI. Compared to previous advanced methods, experimental results demonstrate the effectiveness of MSFR.

Multimodal sentiment analysis aims to integrate visual, audio, and text modal information to more accurately understand human sentiment. However, existing methods often encounter challenges such as semantic confusion, unreliable modal weights, and low contributions during the fusion stage, which have adverse effects on the model.To address this issue, we propose a Credibility-Driven Fusion Network (CDFN) for MSA, which incorporates structured feature reconstruction, contribution-consistent supervision and weak-modality enhancement.Specifically, the Modal Decoupling and Reconstruction (MDR) module achieves semantic consistency by separating modality-shared and modality-private representations and reconstructing modality features.The Dynamic Modality-aware Calibration (DMC) module supervise the consistency between fusion weights and contributions of modalities to enhance reliability.Finally, the Gated Multimodal Fusion (GMF) module activates latent information from low-contribution modalities to improve the completeness of fusion.Experiments conducted on the MOSI and MOSEI datasets demonstrate the effectiveness of CDFN in enhancing multimodal fusion, yielding competitive performance compared with state-of-the-art methods. The code is publicly available at https://github.com/LIU70KG/CDFN/.

Multimodal emotion recognition (MER) is essential for understanding human emotions from diverse sources such as speech, text, and video. However, modality heterogeneity and inconsistent expression pose challenges for effective feature fusion. To address this, we propose a novel MER framework combining a Dynamic Weighted Graph Convolutional Network (DW-GCN) for feature disentanglement and a Cross-Attention Consistency-Gated Fusion (CACG-Fusion) module for robust integration. DW-GCN models complex inter-modal relationships, enabling the extraction of both common and private features. The CACG-Fusion module subsequently enhances classification performance through dynamic alignment of cross-modal cues, employing attention-based coordination and consistency-preserving gating mechanisms to optimize feature integration. Experiments on the CMU-MOSI and CMU-MOSEI datasets demonstrate that our method achieves state-of-the-art performance, significantly improving the ACC7, ACC2, and F1 scores.

Multimodal Affective Analysis (MAA) has advanced significantly through sophisticated representation-learning and fusion methods, yet two challenges remain. First, current methods often lack an explicit mechanism to determine when modality-specific (private) cues should complement modality-invariant (shared) representations and when they should be suppressed. This matters because private cues are not uniformly useful: for some instances they provide complementary affective evidence, while for others they may be noisy, redundant, or even conflicting. Second, even when strong multimodal embeddings are learned, conventional neural prediction heads can struggle in overlapping and ambiguous embedding spaces, where clear decision boundaries are difficult to form. This thesis presents a unified multimodal framework that combines shared-only fusion, private feature regulation, and a tolerance-near-set-based predictor. The framework first learns robust shared representations, then selectively incorporates private features only when they add reliable complementary evidence. For decision making, it applies a tolerance-based ambiguity-aware prediction framework over learned embeddings. Additionally, this thesis introduces a novel distance measure that jointly models geometric proximity and label consistency, producing semantically coherent tolerance neighborhoods. In addition, the tolerance-based prediction framework is extended to regression settings alongside classification. The framework is evaluated on eight benchmark datasets covering key areas of affective computing in both tri-modal and bi-modal settings, including cross-lingual data. Results show consistent gains over strong baselines and state-of-the-art methods across regression and classification tasks, demonstrating the effectiveness of our approach.

Multimodal Sentiment Analysis (MSA) aims to identify sentiment polarity and intensity in media. Current methods typically employ a two-stage pipeline: extracting features from each modality, then predicting sentiment based on fused representations. However, most fusion strategies align features from different modalities in a single step, leading to conflicts during cross-modal interactions and hindering the modeling of hierarchical sentiment dependencies. Additionally, existing methods often overlook the dominant role of textual modality in high level latent fusion space, causing explicit linguistic sentiment cues to be obscured by redundant information. To address these issues, DDSE (Decoupled Dual-Stream Enhanced framework) is proposed in this work, which decouples features into public and private representations for improved feature enhancement and cross-modal interaction. The proposed TC-Mamba module enables progressive cross-modal interactions within shared state transition matrices under a text-guided fusion paradigm, effectively preserving sentiment cues and minimizing redundancy. Additionally, DDSE adopts a multi-task learning strategy to further enhance overall performance. Extensive experiments on the MOSI and MOSEI datasets demonstrate that DDSE achieves state-of-the-art results, with Acc-5 improvements of 3.06% and 0.1%, respectively, underscoring its effectiveness in MSA. Ablation studies confirm the critical contributions of each component within the framework. Code is available at https://anonymous.4open.science/r/DDSE-76D6.

Multimodal emotion recognition aims to identify human emotions from text, audio, and visual modalities. Previous methods either explore correlations between different modalities or design sophisticated fusion strategies. However, the serious problem is that the distribution gap and information redundancy often exist across heterogeneous modalities, resulting in learned multimodal representations that may be unrefined. Motivated by these observations, we propose a Feature-Disentangled Multimodal Emotion Recognition (FDMER) method, which learns the common and private feature representations for each modality. Specifically, we design the common and private encoders to project each modality into modality-invariant and modality-specific subspaces, respectively. The modality-invariant subspace aims to explore the commonality among different modalities and reduce the distribution gap sufficiently. The modality-specific subspaces attempt to enhance the diversity and capture the unique characteristics of each modality. After that, a modality discriminator is introduced to guide the parameter learning of the common and private encoders in an adversarial manner. We achieve the modality consistency and disparity constraints by designing tailored losses for the above subspaces. Furthermore, we present a cross-modal attention fusion module to learn adaptive weights for obtaining effective multimodal representations. The final representation is used for different downstream tasks. Experimental results show that the FDMER outperforms the state-of-the-art methods on two multimodal emotion recognition benchmarks. Moreover, we further verify the effectiveness of our model via experiments on the multimodal humor detection task.

Multimodal emotion analysis aims to understand human emotional states by integrating information from multiple modalities, such as language, audio, and visual. Current multimodal emotion analysis methods face many challenges. On the one hand, the heterogeneity between different modalities complicates the feature extraction and fusion process. On the other hand, interference and noise between modalities can negatively impact the results of emotion analysis. To address these issues, this paper proposes a decoupled cross-modal attention mechanism. Features are decoupled into modality - irrelevant and modality - specific parts via shared and private encoders and are optimized with homogeneous and heterogeneous loss functions. The language-guided attention is then employed to enhance the interaction and fusion of cross-modal features. Experimental results show that this model outperforms existing methods, with significant improvements in seven-class accuracy, binary accuracy, and F1 score, thereby validating the effectiveness of the proposed method.

To address the semantic gap and complex feature entanglement inherent in multimodal emotion recognition, we propose the Dynamic Heterogeneous Graph Temporal Network (DHGTN), an end-to-end framework designed to model dynamic cross-modal interactions effectively. Utilizing a robust backbone of Wav2vec 2.0, VideoMAE, and BERT, we introduce a “Shared Private” subspace projection mechanism that explicitly disentangles emotion common features from modality-specific noise through contrastive learning to ensure strict semantic alignment. Furthermore, our collaborative Dynamic Heterogeneous Graph and Transformer module overcomes static fusion limitations by constructing time-varying graphs for instantaneous associations and employing global attention to capture long-range temporal dependencies. Extensive experiments on the IEMOCAP and MELD benchmarks demonstrate that DHGTN significantly outperforms state-of-the-art baselines, achieving weighted F1-scores of 73.86% and 66.87%, respectively, which confirms the method’s effectiveness and robustness.

This paper aims to address modality noise and continuous changing of reliabilities of emotional cues in multimodal sentiment analysis. Such dynamic perceptual characteristics are challenging for continual sentiment perception, especially when there are modality conflicts. The authors propose a Continuously self-Adaptive Multimodal Sentiment Analysis method (CAMSA). CAMSA decouples modality features into shared and private features and introduces a Variational Information Bottleneck method to perform probabilistic modeling and information compression on shared features, and then continually suppressing redundant and noisy information during the progressively emerging of emotional cues. In addition, a multidimensional adaptive feature fusion mechanism is designed to evaluate modality reliability from the perspective of semantic certainty, modal consistency and sentiment extremity. Experimental results on CMU-MOSI, CMU-MOSEI and CH-SIMS demonstrate that CAMSA outperforms existing models on most evaluation metrics, validating its effectiveness in handling modality noise and modal conflicts. From the requirements of continual sentiment perception, this paper proposes a continuously self-adaptive CAMSA framework, which combines shared feature refinement with a multidimensional adaptive fusion mechanism, enabling robust cross-modal sentiment modeling in dynamic and evolving perceptual scenarios. The CAMSA approach can effectively address the problem of fixed fusion strategies in complex dynamic sentiment perception tasks.

Multimodal affective analysis benefits from combining textual, acoustic, and visual cues, yet many methods implicitly mix modality-invariant information with modality-specific factors, which can reduce robustness when modalities vary in reliability. We propose DiMoE, a representation-first framework that integrates feature disentanglement with sparse Mixture of Experts (MoE) fusion. DiMoE first decomposes each modality into shared (modality-invariant) and private (modality-specific) representations. It then fuses these factors using top-k sparse MoE routing, enabling input-adaptive expert selection. We study two routing strategies with a shared expert pool, a joint router that operates on combined shared and private factors, and separate routers that process shared and private factors independently. We evaluate DiMoE on five widely used benchmarks spanning sentiment intensity prediction and emotion classification, covering both trimodal and bimodal settings. Across benchmarks, DiMoE consistently achieves strong performance against recent competitive baselines, indicating that explicit modality disentanglement combined with Mixture of Experts fusion yields effective multimodal affective representations.

This study focuses on English teachers’ classroom language expression, emotional changes, and teacher-student interaction behaviors, and proposes an intelligent evaluation model based on multimodal representation learning—Bimodal Modality Representation and Fusion Network (BMRN). The model integrates text, speech, and visual multimodal information, introduces a shared-private feature decoupling structure, and combines four types of losses (similarity, difference, cycle consistency, and reconstruction) to achieve cross-modal deep semantic fusion and accurate modeling. Experiments are conducted based on public datasets CMU- Multimodal Opinion-Level Sentiment Intensity (MOSI), Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI), and self-built teacher teaching data. Results show that BMRN outperforms existing advanced models in both classification and regression metrics. Specifically, on the CMU-MOSEI dataset, the mean absolute error is reduced to 0.547, and the Pearson correlation coefficient reaches 0.781; on the self-built dataset, the four-classification accuracy reaches 66.1%. Ablation experiments indicate that the text modality and cycle consistency loss contribute the most to model performance. Robustness tests verify the effectiveness of the cross-modal reconstruction mechanism in cases of modality missing. This model improves the ability of semantic understanding and emotion recognition for multimodal teaching behaviors, with strong generalization and practical application potential. This study provides strong technical support and data basis for teacher behavior evaluation and professional development in smart classrooms.

Understanding a speaker's emotional states in conversation is essential for dialogue systems. Emotion prediction in conversation (EPC) aims to predict a speaker.s future emotional states based on context. Most existing studies focus on modeling dynamic interactions in the historical context but often overlook the effects of temporal dynamic interactions. To address this limitation, we propose the emotion interaction prediction (EIP) model, which captures temporal dynamic interactions between the speaker and interlocutors. Specifically, EIP characterizes emotional dynamics by integrating emotion similarity for long-term changes and emotion shifts for short-term changes. In this study, we investigate the efficacy of temporal dynamic interaction by comparing the EIP model with mainstream EPC models and emotion recognition in conversation (ERC)-based models. Then, we analyze the impact of long-term and short-term temporal dynamic interactions. Next, we assess the effectiveness of key architectural components in the EIP model. Subsequently, we evaluate the contributions of different modality-specific representations to EIP performance. After that, we explore the correlation between context length and EPC performance. Finally, we analyze the robustness of EPC models across different conversation scenarios. Experimental results demonstrate that 1) the EIP model outperforms established EPC and ERC-based methods, and 2) the inclusion of emotion similarity and emotion shifts enhances temporal interaction modeling. Furthermore, 3) key components such as the individual and interaction layers contribute to performance. Moreover, 4) the integration of speech and text representations improves EIP performance. Additionally, 5) context length affects EPC performance. Notably, 6) the EIP model exhibits improved robustness across different conversation scenarios.

The lack of non-verbal signals in text-based communication forces dialogue systems to rely entirely on dynamic textual sentiment changes to infer user intent and emotional shifts. However, substantial class imbalance and the static, post-hoc nature of existing text analysis models prevent dialogue systems from recognizing these emotional dynamics and predicting conversational outcomes in real time. This research presents a hybrid machine learning framework to address these challenges through integrated sentiment-aware analysis and real-time conversation outcome prediction. We first utilize a pre-trained RoBERTa model to assign sentiment scores to individual messages, which is required for two subsequent tasks: (i) conversational outcome classification to classify conversations into Interested in Islam, Wants to Convert, and Accepted Islam, and (ii) message impact prediction to assess the emotional influence of agents’ responses before deployment. In the initial task, our Enhanced Bi-LSTM model (enhanced with multi-head attention mechanisms and a boosted input capacity of 2500 tokens) achieves a macro-F1 score of 0.76, a balanced accuracy of 74.3%, and an overall accuracy of 90.0% on the full, unbalanced dataset when addressing the 3-class outcome classification task. By conducting an ablation study, we investigate the main source of error, the extremely ambiguous “Wants to Convert” class, and show the resilience of the model on well-defined classes, revealing that accuracy may exceed 99% by simplifying the task to a binary classification. In the second task, a fine-tuned BERT model achieves 80% accuracy in message-level sentiment prediction. The framework enables real-time feedback to refine adaptive responses and guide conversations toward desired outcomes. We assess our proposed framework using a case study of domain-specific conversations (such as religious outreach), supported by a dataset from a partner organization, to demonstrate applicability. The framework's flexibility makes it applicable to customer service, education, and healthcare, where effective communication depends on real-time sentiment analysis and outcome prediction. ABSTRAK: Kekurangan isyarat bukan lisan dalam komunikasi berasaskan teks memaksa sistem dialog bergantung sepenuhnya pada perubahan sentimen tekstual dinamik membuat kesimpulan tentang niat pengguna dan perubahan emosi. Walau bagaimanapun, ketidakseimbangan kelas yang ketara dan sifat statik dan pasca-hoc model analisis teks sedia ada menghalang sistem dialog daripada mengenali dinamik emosi ini dan meramalkan hasil perbualan pada masa nyata. Kajian ini membentang rangka kerja pembelajaran mesin hibrid bagi menangani cabaran melalui analisis kesedaran sentimen bersepadu dan ramalan hasil perbualan masa nyata. Pada permulaan, model RoBERTa dilatih terlebih dahulu bagi memberi skor sentimen kepada mesej individu. Ini diperlukan untuk dua tugasan seterusnya: (i) klasifikasi hasil perbualan kepada Berminat tentang Islam, Ingin Beranut Islam dan Menerima Islam, dan (ii) ramalan impak mesej bagi menilai pengaruh emosi respons ejen sebelum penggunaan. Melalui tugasan awal, dapatan model Peningkatan Bi-LSTM (dipertingkatkan dengan mekanisme perhatian berbilang kepala dan peningkatan kapasiti input sebanyak 2500 token) mencapai skor makro-F1 sebanyak 0.76, ketepatan seimbang sebanyak 74.3%, dan ketepatan keseluruhan sebanyak 90.0% pada set data penuh tidak seimbang apabila menangani tugasan pengelasan hasil 3 kelas. Dengan menjalankan kajian ablasi, kami menyiasat punca ralat utama, kelas "Ingin Menukar" yang sangat samar-samar, dan menunjukkan daya tahan model pada kelas yang ditakrif baik, mendedahkan bahawa ketepatan mungkin melebihi 99% dengan memudahkan tugasan kepada pengelasan binari. Menerusi tugasan kedua, model BERT yang ditala halus mencapai ketepatan 80% dalam ramalan sentimen peringkat mesej. Rangka kerja ini memudahkan maklum balas masa nyata bagi memperhalusi respons adaptif dan membimbing perbualan ke arah hasil yang diingini. Kami menilai rangka kerja yang dicadangkan menggunakan kajian kes yang mengandungi perbualan khusus domain (seperti jangkauan keagamaan). Ini disokong oleh set data daripada organisasi rakan kongsi, bagi menunjukkan kebolehgunaan. Fleksibiliti rangka kerja ini boleh digunakan untuk khidmat pelanggan, pendidikan dan penjagaan kesihatan, yang mana komunikasi berkesan bergantung pada analisis sentimen masa nyata dan ramalan hasil.

The task of dialogue sentiment analysis aims to identify the sentiment polarity of utterances in the context of a dialogue. Pre-trained models often struggle to capture the logical structure of a dialogue, making this task challenging. To address this issue, we propose a dialogue sentiment analysis framework that leverages pre-training on dialogue structure. Our proposed framework includes three sub-tasks for pre-training: utterance order sorting, sentence backbone regularization, and sentiment shift detection. These tasks are designed to improve the model’s ability to mine dialogue logical relationships and sentiment interactions. By focusing on learning the logical structure of dialogues and the perception of sentiment interactions, our framework is able to improve the performance of pre-trained models on recognizing the sentiment polarity of dialogues. This is demonstrated by the convincing results obtained on the public MEISD dataset.

No abstract available

No abstract available

No abstract available

Multimodal Emotion Recognition in Conversation (MERC) has emerged as a prominent research focus, aiming to accurately interpret emotional states by analyzing textual, audio, and visual modalities within dynamic conversational interactions. Despite significant progress in this field, existing methods still struggle with effectively modeling intra- and inter-modal dependencies, difficulty aligning heterogeneous feature representations, and capturing dynamic emotional transitions, which often leads to inconsistent predictions across dialogue turns. To overcome these limitations, we present a novel Graph-Guided Cross-Modal Attention Network (GGCMANet) equipped with Emotion Shift Modeling. First, we introduce a Multi-Graph Attention Mechanism (MGAM) that models each modality as a graph, where utterances are represented as nodes and both intra- and inter-modal dependencies are encoded as edges, thereby capturing intra-modal context and inter-modal features. Second, we propose a Cross-Modal Attention Network (CMAN) to align and integrate heterogeneous modality-specific features. By enabling dynamic cross-modal attention, the model selectively attends to the most informative inter-modal features, enhancing representation coherence and contextual alignment. Third, we design the Emotion Shift Model (ESM), which captures temporal variations in emotional states across dialogue turns and mitigates prediction inconsistencies over time. Extensive experiments on three benchmark datasets demonstrate that our approach consistently outperforms state-of-the-art methods in multimodal emotion recognition.