多源数据融合的解耦与跨视图互动及融合
多源异构数据融合的研究综述与总体方法论框架
合并三组中的综述/总论类文献:多源数据融合的技术类型、算法协同与评价范式,跨域推荐算法族系比较,以及多视图/多模态表征学习的任务谱系与评价指标。共同作用是为全文界定"多源异构—解耦—互动—融合—决策"的问题边界与术语体系,是后续各章方法铺陈的上位视角。
- A Review of Multi-Source Data Fusion and Analysis Algorithms in Smart City Construction: Facilitating Real Estate Management and Urban Optimization(Bing-Lin Liu, Qian Li, Zhihua Zheng, Yan-Jiao Huang, Shuguang Deng, Qiong Huang, Wei-Jian Liu, 2025, Algorithms)
- Advances in multi-source data fusion for precipitation estimation: remote sensing and machine learning perspectives(Vahid Nourani, M. Tosan, Jinhui Jeanne Huang, M. Gebremichael, S. Kantoush, Mehdi Dastourani, 2025, Earth-Science Reviews)
- An advanced multi-source data fusion method utilizing deep learning techniques for fire detection(Shi-Kuan Wang, Meng-Quan Wu, Xing Wei, Xiaodong Song, Qingtong Wang, Yanchun Jiang, Jinkun Gao, Lingyi Meng, Zhipeng Chen, Qiyue Zhang, Yi-Ke Zhang, Shuang Liang, Shengtao Wei, Long-Xing Liu, Chang-Bo Yi, Jin-Yi Lv, 2025, Engineering applications of artificial intelligence)
- Recent Advances in Multi-source Data Fusion for Traffic Flow Prediction: A Review(Xianhui Zong, He Yan, Yong Qi, 2025, Archives of Computational Methods in Engineering)
- A survey of representation learning for multi-modal emotion recognition(En-De Wang, Suyao Sun, Kejuan Xue, Yi-Fan Ji, Taiyang Ma, Kuiming Jiang, Jia-Geng Liu, 2026, Multimedia tools and applications)
- Modern Algorithmic Strategies in Cross-Domain Recommender Systems: A Comparative Review of Knowledge Transfer, Representation Learning, and Evaluation Trends (2018–2025)(Amandeep Kaur Gahier, Supreet Kaur Gujral, 2026, Journal of Intelligent Decision Making and Information Science)
基于互信息、信息瓶颈与结构约束的解耦表示学习理论
合并原三组中所有以信息论/层级结构为解耦目标函数的工作:用互信息估计、条件信息瓶颈、软分割、比例化约束与自监督分布约束显式地把共享(一致)信息与独有(特有、冗余、噪声)信息分离,并给出信息量配比的可解释分析。构成论文"解耦"部分的理论内核。
- Conditional disentangled information bottleneck for multi-view molecular graph representation learning(Jiaxin Dai, Dong-yang Fu, Zhongwei Qiu, Ling-Wei Ma, Da-Wei Zhang, 2026, Expert systems with applications)
- MVAIBNet: Multiview Disentangled Representation Learning With Information Bottleneck(Ming Yin, Xin Liu, Junli Gao, Haoliang Yuan, Taisong Jin, Shengwei Zhang, Lingling Li, 2024, IEEE Transactions on Industrial Informatics)
- Mutual Information Estimation-Based Disentangled Representation Network for Medical Image Fusion(Wan-Wan Huang, Han Zhang, Zeyuan Li, Yan-Bin Yin, 2022, IEEE International Conference on Bioinformatics and Biomedicine)
- Flexible disentangled representation learning with soft-splitting for multi-view data(Xun Yao, Ming Yin, Yonghua Wang, Yi Guo, 2025, Image and Vision Computing)
- Disentangled and Proportional Representation Learning for Multi-View Brain Connectomes(Yanfu Zhang, L. Zhan, Shan-Dong Wu, Paul M. Thompson, Heng Huang, 2021, International Conference on Medical Image Computing and Computer-Assisted Intervention)
- On Hierarchical Disentanglement of Interactive Behaviors for Multimodal Spatiotemporal Data with Incompleteness(Jiayi Chen, Aidong Zhang, 2023, Knowledge Discovery and Data Mining)
- Self-supervised disentangled representation learning with distribution alignment for multi-view clustering(Zhen-Qiu Shu, Teng Sun, Zheng-Tao Yu, 2025, Digit. Signal Process.)
- An Information-Theoretic Latent Disentanglement Framework for Multimodal Hyperspectral Image Compression(Jia-Hui Liu, Li-Li Zhang, Xin-Bo Li, 2026, IEEE Transactions on Geoscience and Remote Sensing)
多视图聚类中的共享-特理解耦与语义/一致性引导
合并原三组中面向多视图聚类与一致性表征的共享-特理解耦工作:通过子空间自表达、变分推断、语义标签引导、内容-风格分离、一致锚图与自适应多图融合,保证共享成分与视图特有成分的独立性并抑制表征塌缩。是论文"解耦"主线最直接的方法与对比基线来源。
- Disentangled representation learning for multi-view clustering via von Mises-Fisher hyperspherical embedding(Zhi-Xiang Li, Zhi Luo, N. Bouguila, Wei-Feng Su, Wen-Tao Fan, 2025, Neural Networks)
- Disentangled Contrastive Multi-View Clustering via Semantic Relevance Invariance(Peng-Yuan Li, Dong-Xia Chang, Yi-Ming Wang, Zi-Sen Kong, Lin-Hua Kong, Yao Zhao, 2026, IEEE Transactions on Knowledge and Data Engineering)
- Shared-Specific-Disentangled Variational Autoencoder for Multi-View Clustering(Xiao-Yun Yu, Hong Jia, 2025, International Conference on Computational Intelligence and Security)
- Disentangling Consistent and Specific Information for Double Incomplete Multi-View Multi-Label Classification(Jie Wen, Lian Zhao, Xiao-Huan Lu, Chengliang Liu, Li Shen, Chao Huang, Yong Xu, 2026, IEEE Transactions on Pattern Analysis and Machine Intelligence)
- Content-style Disentanglement Guided Representation Learning for Deep Incomplete Multi-view Clustering(En-Ze Ji, Meng Liu, Yu-Zhe Li, Zhi-Kui Chen, 2026, No journal)
- Multi-view subspace clustering with consistent and view-specific latent factors and coefficient matrices*(Run-kun Lu, Jian-wei Liu, Xin Zuo, Wei-min Li, 2021, IEEE International Joint Conference on Neural Network)
- Robust multi-view clustering via adaptive fusion and disentangled representation learning(Cheng-Quan Huang, Li Jiang, Shi Zhou, Yu-Qin Liu, Xian Wen, Lihua Zhou, 2026, International Journal of Computer Applications)
- Robust and Consistent Anchor Graph Learning for Multi-View Clustering(Su-Yuan Liu, Qing Liao, Si-Wei Wang, Xin-Wang Liu, En Zhu, 2024, IEEE Transactions on Knowledge and Data Engineering)
- Multi-View Urban Region Embedding via Commonality-Specificity Disentanglement(Zechen Li, H. Jia, Kai Zhao, Weiming Huang, Meng Chen, 2026, Knowledge Discovery and Data Mining)
缺失视图、不完整标签与增量视图下的解耦与补全建模
三组中均散落着处理"数据不完整"的解耦工作:部分视图/缺失视图的多标签与聚类学习、跨时间点视图补全、增量视图到达、双缺失(视图与标签)情形下的理论驱动表示。共同点是在不完整输入下仍维持共享-特有分解的可辨识性,对应论文面向真实多源数据(模态缺失)时的鲁棒性设计。
- Incomplete Multi-View Multi-Label Learning via Disentangled Representation and Label Semantic Embedding(Xudong Yan, Jun Yin, Jie Wen, 2025, Computer Vision and Pattern Recognition)
- Structure-guided decoupled contrastive framework for partial multi-view incomplete multi-label classification(Linqian Yang, Tao Zhuang, Xing Jin, Tian-Yang Cao, Xi-Yin Wu, 2025, Journal of King Saud University: Computer and Information Sciences)
- LEOPARD: missing view completion for multi-timepoint omics data via representation disentanglement and temporal knowledge transfer(Si-Yu Han, Shi-Xiang Yu, Mengya Shi, M. Harada, Jian-Hong Ge, Jie-Sheng Lin, C. Prehn, A. Petrera, Ying Li, F. Sam, G. Matullo, J. Adamski, K. Suhre, C. Gieger, S. Hauck, C. Herder, M. Roden, F. Casale, N. Cai, Annette Peters, R. Wang-Sattler, 2024, bioRxiv)
- Anchor-based disentanglement framework for incremental multi-view clustering(Bo Zhong, Peng-Yuan Li, Zi-Sen Kong, Dong-Xia Chang, Yi-Ming Wang, Yao Zhao, 2026, Neural Networks)
- Incomplete Disentangled Multi-View Representation Learning with Graph Propagation(An-Ge Xu, Zuxin Wang, Zhongyi Hu, Hui Huang, Nan Zhang, 2026, Information Fusion)
- Theory-Driven Label-Specific Representation for Incomplete Multi-View Multi-Label Learning(Quan-Jiang Li, Tian-Xiang Xu, Ting-Jin Luo, Yan Zhong, Yang Li, Yi-Yun Zhou, Chen-Ping Hou, 2025, Neural Information Processing Systems)
- Incomplete multi-view clustering based on kernel graph fusion tensor under self-expression framework(Mu-Ke Chen, Yong-Ming Cai, Zhanpeng Huang, 2025, Neurocomputing)
多视图多标签学习、无监督特征选择与张量子空间表征
合并原分组中处理"结构化低维表征"的一类:一致-特有标签特定表示、多视图无监督特征选择、双重自表达张量特征选择、自适应加权图张量与二分图秩约束、协同张量嵌入聚类与锚图张量聚类。共同点是用张量/低秩/稀疏结构约束把多视图信息压缩为统一低维子空间,是论文融合环节的降维与选择技术支撑。
- Consistent and specific multi-view multi-label learning with correlation information(Yiting Li, Jia Zhang, Han-Rui Wu, Guo-Dong Du, Jin-Yi Long, 2025, Information Sciences)
- Multi-view Unsupervised Feature Selection Based on Hessian Regularization and Consensus Tensor Fusion(Hai-Xin Zeng, Yi-Fan Shi, Kai-Xiang Yang, Jian-Qing Zhu, Zhi-Wen Yu, Huan-Qiang Zeng, 2026, IEEE Signal Processing Letters)
- Feature selection and double self-expressive tensor fusion for multi-view subspace clustering(Tian-Qi Zheng, Xiang-Cheng Li, You-Ming Sun, Ji-Peng Guo, Tian-Chuan Yang, Hai-Qiang Chen, 2026, Information Sciences)
- Auto-weighted graph tensor and rank-constrained bipartite graph fusion for multi-view clustering(Jie Zhang, Xiao-Qian Zhang, Jinghao Li, Yong-Ying Yang, Zhen-Wen Ren, Rong Tang, Dong Wang, 2025, Neurocomputing)
- Embedded Multi-View Clustering via Collaborative Tensor Subspace Representation and Multi-Graph Fusion(Jing-Yu Wang, Ting-Quan Deng, Ming Yang, Jia-Yi Wang, 2025, IEEE Signal Processing Letters)
- Anchor-based tensor clustering of multi-view data through high order relational fusion(Yan-cheng Gong, Tao Yang, Yan-Ying Mei, Yan-Hua Shao, 2026, Applied Intelligence)
图结构增强、跨视图图滤波与多视图对比融合
合并自"图结构与时空建模"与"图与推荐系统中的多视图交互"两处内容:可信跨视图图增强后再融合、双向跨视图图滤波、自适应多图对比融合、多视图对比学习的自适应聚合以及时空交互与跨视图融合。共同点是"先增强/滤波再融合",用跨视图图结构交互抑制视图噪声,与论文的跨视图互动机制形成方法论呼应。
- Enhance Before Fusion: Multi-View Graph Clustering With Graph Trend Filter(Peng-Lei Wang, Ji-Tao Lu, Dan-Yang Wu, Rong Wang, Feiping Nie, 2026, IEEE Transactions on Pattern Analysis and Machine Intelligence)
- Adaptive Fusion Multi-View Contrastive Learning with Interest Aggregation for Collaborative Filtering(Run-Ze Feng, 2025, Poster Volume Ⅱ The 2025 Twenty-First International Conference on Intelligent Computing July 26-29, 2025 Ningbo, China)
- AdaMCL: Adaptive Fusion Multi-View Contrastive Learning for Collaborative Filtering(Guanghui Zhu, Wang Lu, C. Yuan, Y. Huang, 2023, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval)
- Bidirectional Fusion With Cross-View Graph Filter for Multi-View Clustering(Xiaojun Yang, Tuoji Zhu, Dan-Yang Wu, Peng-Lei Wang, Yu-Jia Liu, Fei-Ping Nie, 2024, IEEE Transactions on Knowledge and Data Engineering)
- STIEFormer: Spatio-Temporal Interaction and Cross-View Fusion Enhanced Transformer for Traffic Forecasting(Shi-Yong Li, Hui Jia, Yan-Ping Chen, 2026, ICON)
跨视图注意力与几何先验引导的视图间交互机制
合并三组中"跨视图互动"最纯粹的视觉类工作:视差引导跨视图 Transformer、多平面跨视图交互、多步跨视图变化检测交互、免标定 3D 的特征级跨视图融合、外视角-第一人称融合、跨视角手语识别与稳定人体位姿估计。共同点是借助注意力与几何先验显式建模视图间对应关系,是论文"跨视图互动"关键词最直接的参照。
- Disparity-Guided Cross-View Transformer For Stereo Image Super-Resolution(Bing-Ting Li, Wenjing Shang, Yong-Shun Gong, Qiang-Chang Wang, Xinxin Zhang, Yi-Long Yin, 2025, IEEE International Conference on Acoustics, Speech, and Signal Processing)
- Multiplane-Based Cross-View Interaction Mechanism for Robust Light Field Angular Super-Resolution(Rong-Shan Chen, Hao Sheng, Da Yang, Rui-Xuan Cong, Zheng-Long Cui, Si-Zhe Wang, Wei Ke, 2025, IEEE Transactions on Visualization and Computer Graphics)
- Cross-View and Multi-Step Interaction for Change Captioning(Tian-Tao Xian, Zhi-Heng Zhou, Wen-Lve Zhou, De-Lu Zeng, Bo Li, 2026, IEEE transactions on multimedia)
- Feature-Level Cross-View Fusion for Calibration-Free 3D Human Pose Estimation(Krasimir Tonchev, Nikolay N. Neshov, Radostina Petkova, N. Christoff, A. Manolova, 2025, International Symposium on Wireless Personal Multimedia Communications)
- Cross-view Exocentric and Egocentric Fusion for Robust Microsurgical Anastomosis Understanding(Yuxuan Liu, Yu-Yang Zhuge, Xin-Yao Zhou, Ya-Ting Luo, Yun-Fei Luan, Yao Guo, Guang-Zhong Yang, 2026, IEEE International Conference on Robotics and Automation)
- Cross-View Isolated Sign Language Recognition via View Synthesis and Feature Disentanglement(Xin Shen, Xin-Yu Wang, Lei Shen, Kai-Hao Zhang, Yu Xin, 2025, IEEE International Conference on Computer Vision)
- Towards Stable Human Pose Estimation via Cross-View Fusion and Foot Stabilization(Li'an Zhuo, Jian Cao, Qi Wang, Bang Zhang, Lie-Feng Bo, 2023, Computer Vision and Pattern Recognition)
跨视图引导解耦与多尺度一致性/语义交互融合
合并原"跨视图特征交互与几何一致性建模"与"视图级与特征级融合架构"中的相关工作:跨视图引导的解耦器、跨视图注意力交互融合算法、多尺度全局感知特征学习、多尺度一致性增强、隐秩最小化的跨视图交互、2D-3D 交互融合与联合信息交互语义融合。共同主张不直接混合特征,而是先借视图间关系解耦、再做多尺度语义交互与融合。
- CVGD: Cross-View Guided Disentangler for Multi-View Radar Semantic Segmentation(Yaoyu He, Yuan Feng, Sha Tao, Nan Wang, 2026, IEEE Signal Processing Letters)
- Cross-View Attention Interaction Fusion Algorithm for Stereo Super-Resolution(Yaru Zhang, Jiantao Liu, Tong Zhang, Zhi-Biao Zhao, 2023, Applied Sciences)
- MGCFI-Net: Multi-scale globally aware feature learning with cross-view feature interaction for multi-view stereo(M. Han, Hui Yin, Aixin Chong, Hua Huang, 2026, Neural Networks)
- Integrating Multiscale Consistency and Enhanced Feature Interaction for Cross-View Geo-Localization(Mo Yang, Luo Chen, N. Jing, 2026, IEEE Transactions on Geoscience and Remote Sensing)
- Implicit Rank-Minimizing Representation-Based Cross-View Interaction and Fusion for Multiview Clustering(Shang-Hui Deng, Xiao Zheng, Wei-Qing Yan, Kun Sun, Chang Tang, Xin-Wang Liu, 2026, IEEE Transactions on Systems, Man & Cybernetics. Systems)
- MFINet: Multi-view Fusion and 2D-3D Interaction Enhancement for Real-Time LiDAR Semantic Segmentation(Nan Ma, Zhijie Liu, Yiheng Han, 2026, AAAI Conference on Artificial Intelligence)
- Joint Information Interaction and Semantic Fusion for Multi-View Unsupervised Feature Selection(Yong Mi, Hong-Mei Chen, Zhong Yuan, Chuan Luo, S. Horng, Tian-Rui Li, 2026, IEEE transactions on multimedia)
视图一致对齐与视图特定建模:重识别、跨视角位姿与单目视觉任务
保留原"跨视图交互与视图一致对齐"组的独特设定:以重识别、单目 HOI、XR 位姿等跨相机/跨视角任务为载体,级联跨视图特征交互、视图特定记忆与处理层、视图一致对齐与文本引导跨注意力,在解耦视角/尺度/运动因素的同时保留视图不变判别信息。与前一组的区别在于其目标是把视角差异显式建模为可分离因素。
- HALO: Hierarchical Adaptive Feature Learning and Cross-View Interaction for Object-Level Geo-Localization(Ze-Hao Zhang, Yu-Qi Sun, Lei Ding, Huayu Zhang, Yu-Feng Wang, Wen-Rui Ding, 2025, IEEE Transactions on Geoscience and Remote Sensing)
- VM-TAPS: View-specific Memory with Temporal and Scale Awareness Framework for Video-based Cross-View Person Re-Identification(Md. Rashidunnabi, Kailash A. Hambarde, João C. R. Neves, Vasco Lopes, Hugo Proença, 2025, 2025 IEEE International Joint Conference on Biometrics (IJCB))
- CrossHOI: Learning Cross-View Representations for Monocular 3D Human-Object Interaction Reconstruction(Pei Geng, Shan-Shan Zhang, Jian Yang, 2026, Computer Vision and Pattern Recognition)
- CVAF: A CLIP-Based View-Consistent Alignment Framework for Aerial-Ground Person Re-Identification(Dong-Xu Mao, Shangzhi Teng, Xueqiang Lyu, 2025, ACM Transactions on Multimedia Computing, Communications and Applications)
- Enhanced Vehicle Re-Identification Multi-Branch Network with Cross-View Feature Fusion(Ping Zhang, Wei Liang, Zongwen Liu, 2025, 2025 International Conference on Computer, Internet of Things and Smart City (CIoTSC))
- Odometry-Free Cross-View Pose Fusion for XR Interaction via Visibility-Guided Alignment(Sung-Jin Hong, Cho-Rong Yu, Youn-Hee Gil, 2026, Proceedings of the 32nd ACM Symposium on Virtual Reality Software and Technology)
跨模态注意力融合网络(CMAF)的架构设计与多任务部署
合并原三组中"以跨模态注意力为融合算子"的通用架构类文献:双向/多查询/层级/级联/轻量化跨模态注意力融合网络,覆盖虚假内容检测、编程意图识别、认知障碍评估、博物馆文物分类、RGB-D 与显著目标检测、图像-掩膜与文本-图像对齐等任务,关注融合结构设计与可部署性,为论文融合模块提供结构范式。
- CIVA-Net: Cross Modal Attention Fusion for AI Generated Content Detection(Abdul Momen, Suvo Saha, Shakil Khan, Tithi Biswas, Abdus Salam, 2026, 2026 5th International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE))
- NeuroMouse-CodeNet: Cross-Modal Attention Fusion for Programming Intent Recognition(Yanmei Ma, Yun-Gui Luo, K. Zhang, 2026, 2026 7th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT))
- Multi-Query Cross-Modal Attention Fusion for Cognitive Impairment Recognition(Ming-Hui Zhao, Hong-Xiang Gao, Xin-Ru Qi, Hai-Yan Yin, Yu-Lei Song, Ya-Mei Bai, Jian-Qing Li, Lu-Lu Zhao, Cheng-Yu Liu, 2025, IEEE transactions on neural systems and rehabilitation engineering)
- Multimodal Lightweight Cross-Modal Attention Fusion for Efficient Hate Meme Detection(Feng Cheng, Xinying Cui, 2025, Proceedings of the 2025 5th International Conference on Computational Modeling, Simulation and Data Analysis)
- HCAFNet: Hierarchical Cross-Modal Attention Fusion Network for HSI and LiDAR Joint Classification(Jiajia Bai, Na Chen, Jiang-Tao Peng, Lan-Xin Wu, Wei-Wei Sun, Zhi-Jing Ye, 2025, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing)
- CACFNet: Cross-Modal Attention Cascaded Fusion Network for RGB-T Urban Scene Parsing(Wu-Jie Zhou, Shaohua Dong, Mei-Xin Fang, Lu Yu, 2024, IEEE Transactions on Intelligent Vehicles)
- A Museum artifact classification model based on cross-modal attention fusion and generative data augmentation(Ying Lu, Jiaxi Li, Lin Li, Chongxin Yuan, 2025, Scientific Reports)
- CMA-SOD: cross-modal attention fusion network for RGB-D salient object detection(Kexuan Wang, Chenhua Liu, Rongfu Zhang, 2024, The Visual Computer)
- Cross-modal attention fusion network for RGB-D semantic segmentation(Qian-Kun Zhao, Ying-Cai Wan, Ji-Qian Xu, Li-Jin Fang, 2023, Neurocomputing)
- A Multi-Modal Attention Fusion Framework for Road Connectivity Enhancement in Remote Sensing Imagery(Yong-Qi Yuan, Yong Cheng, Bo Pan, Ge Jin, De Yu, Meng-Jie Ye, Qian Zhang, 2025, Mathematics)
跨模态注意力融合在故障诊断、异常检测与缺失模态场景中的鲁棒应用
保留原"CMAF 多任务应用"组的工程化分支:电网故障诊断、ECG-PCG 与日志-性能指标、多模态异常检测、缺失模态 MRI、语义统一等场景下的跨模态注意力融合与生成式增强。与通用 CMAF 组的区别在于其重点是模态可靠性不均与模态缺失条件下的融合鲁棒性。
- A grid fault diagnosis framework based on adaptive integrated decomposition and cross-modal attention fusion(Jiang-Xun Liu, Zhu Duan, Hui Liu, 2024, Neural Networks)
- HiCAN: Hierarchical Cross-Modal Attention Fusion Network for Rumor Detection(Zongliang Han, Shun Mao, Jie Sui, 2025, IEEE International Joint Conference on Neural Network)
- CMAF-Net: a cross-modal attention fusion-based deep neural network for incomplete multi-modal brain tumor segmentation(Kangkang Sun, Jiangyi Ding, Qixuan Li, Wei Chen, Heng Zhang, Jiawei Sun, Zhu-Qing Jiao, Xinye Ni, 2024, Quantitative Imaging in Medicine and Surgery)
- Cross-Modal Attention Networks for Multi-Modal Anomaly Detection in System Software(Su-Chuan Xing, Yihan Wang, 2025, IEEE Open Journal of the Computer Society)
- Cross-Modal Attention Fusion: A Deep Learning and Affective Computing Model for Emotion Recognition(Himanshu Kumar, Martin Aruldoss, M. Wynn, 2025, Multimodal Technologies and Interaction)
- TF-crossnet: a cross-modal attention fusion network for cardiovascular disease classification using pcg and ecg signals(Xing-Guang Li, Yutong Hou, Kai-Yao Shi, Yu-Jian Cai, 2025, Biomedical engineering and physics express)
- CMAFGAN: A Cross-Modal Attention Fusion based Generative Adversarial Network for attribute word-to-face synthesis(Xiaodong Luo, Xiang Chen, Xiao-Hai He, Lin-Bo Qing, Xin-Yue Tan, 2022, Knowledge-Based Systems)
- Cross-modal attention fusion guided semantic unification for image-text matching(Xiao-Qing Liu, Jin-Hui Hou, Zhi-Wen Yu, Huan-Qiang Zeng, Jun Jiang, C. L. Philip Chen, 2026, Expert systems with applications)
不确定性感知与混合专家路由驱动的自适应动态融合
独立保留该分组(三组中仅此一处集中讨论融合的"可靠性"与"动态路由"):证据不确定性编码、von Mises-Fisher 集中度加权、稀疏与上下文感知混合专家路由、解耦表征下的混合专家、混合融合网络与校准。共同解决模态冲突/冗余下"如何安全地加权融合",是解耦之后融合阶段的关键策略来源。
- Trust, but Verify: Uncertainty-Driven Evidential Multimodal Representation Learning(Yu-Peng Han, Kai Zhang, Xian-Quan Wang, Zhi-Hong Pan, Zeng-Hao Liu, Ji-Yuan He, 2026, No journal)
- DiMoE: Disentangled Representation Learning with Mixture of Experts Fusion for Sentiment Intensity Prediction and Emotion Classification(M. S. Sran, S. Ramanna, K. Kotecha, 2026, Algorithms)
- DMCAR: Disentangled Mixture-of-Experts with Context-Aware Routing for Multi-View Clustering(Baili Xiao, Ke Liang, Jiaqi Jin, Jun Wang, Yin-Bo Xu, Si-Wei Wang, En Zhu, 2026, AAAI Conference on Artificial Intelligence)
- DvMF: Uncertainty-Aware Disentangled Representation Learning for Multi-View Clustering(Jing-Yi Chen, Zhi-Bin Dong, Tie-Jun Li, Jian-Min Zhang, Yi-Bo Han, Tai-Chun Zhou, Lin-Xu Long, Chun-Xi Lu, Hong-Jing Song, 2026, Proceedings of the 34th ACM International Conference on Multimedia)
- DRLCCI: A hybrid fusion network leveraging disentangled representation learning and cross-modal collaborative interaction for multi-modal sentiment analysis(Xue Lei, Miao Ma, Yu-Tong Li, 2025, Neurocomputing)
多模态情感分析中的解耦-对齐-融合
合并原"多模态情感分析"类工作:解耦多任务双曲空间对齐、自监督引导的模态解耦、Transformer 解耦表征、多样性与平衡约束的情感融合、基于互信息的模态配比。共同点是针对文本-声学-视觉的异质性与分布差异,先解耦共享/特有语义再对齐融合,作为论文范式在高语义任务上的验证。
- Disentanglement Multi-Task Learning Based on Hyperbolic Space for Multimodal Sentiment Analysis(Si-Yuan Ni, Xian-Yong Li, Dong Huang, Ya-Jun Du, Xiao-Liang Chen, 2026, 2026 11th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA))
- Self-Supervised Guided Modality Disentangled Representation Learning for Multimodal Sentiment Analysis and Schizophrenia Assessment(Hsin-Yang Chang, An-Sheng Liu, Yi-Ting Lin, Chen-Chung Liu, Lue-En Lee, Feng Chen, Shu-Hui Hung, Li-Chen Fu, 2025, IEEE journal of biomedical and health informatics)
- Disentangled Representation Learning via Transformer with Graph Attention Fusion for Depression Detection(Luntian Mou, Siqi Zhen, Sha-Sha Mao, Nan Ma, 2025, Proceedings of the 1st International Workshop on Cognition-oriented Multimodal Affective and Empathetic Computing)
- Diversity and Balance: Multimodal Sentiment Analysis Using Multimodal-Prefixed and Cross-Modal Attention(Meng Li, Zhen-Fang Zhu, Ke-Feng Li, Hong-Li Pei, 2025, IEEE Transactions on Affective Computing)
- Multimodal Sentiment Analysis With Mutual Information-Based Disentangled Representation Learning(Hao Sun, Zi-Wei Niu, Hong-Yi Wang, Xin-Yao Yu, Jia-Qing Liu, Yen-Wei Chen, Lan-Fen Lin, 2025, IEEE Transactions on Affective Computing)
多模态生理与行为信号理解中的任务共享—模态特有解耦
合并原"多模态情感与生理信号"与"步态类"文献:心理压力/EEG-ECG-眼动、细粒度解耦与模态无关融合、跨模态表示学习框架、多层级解耦、三阶段渐进式学习、不确定性感知步态表征与步态性别分类。共同点是生理/行为信号噪声大、模态互补性强,需先分离任务共享与模态独有语义再做融合。
- Mental Stress Recognition Based on Multimodal Representation Learning of Heart–Brain Interactions(Qun-Li Yao, Heng Gu, Shao-Di Wang, Xiao-Li Li, 2026, IEEE Transactions on Emerging Topics in Computational Intelligence)
- Joint Fine-grained Disentanglement and Modal-agnostic Fusion Multi-task Framework for Multimodal Sentiment Analysis(Yao Zheng, Chi Zhang, Yu-Jun Wen, Xin Qi, Peng-Zhou Zhang, 2025, ACM Multimedia Asia)
- A cross-modal representation learning framework for immersive digital human interaction modeling(Xin-Ge Wang, Li-Li Xu, Yong-Shuai Guan, Bao-Chang Chu, Cheng-Zhi Xiong, 2026, No journal)
- Multilevel Representation Disentanglement Framework for Multimodal Sentiment Analysis(N. Jia, Z. Bai, Tiancheng Xiong, Ming-Yang Guo, 2025, IEEE Signal Processing Letters)
- Three-Stage Progressive Learning for Simplifying Multimodal Emotion Recognition with Singlemodality Inference(Jian-Lin Fu, Zhencheng Li, Huqin Weng, Chen Li, Yizhi Luo, Xinhua Li, Chuang-Quan Chen, 2025, 2025 International Conference on Electronic Information, Computer and Aerospace Remote Sensing (EICARS))
- Uncertainty-Aware Multimodal Gait Representation Learning for Scoliosis Screening.(Hai-Qing Li, Yin-Hao Wu, Jing-Quan Yan, Yuzhi Guo, Hehuan Ma, Wen-Liang Zhong, Jean Gao, Jun-Zhou Huang, 2026, IEEE Transactions on Medical Imaging)
- Research on Gait-Based Gender Classification via Fusion of Multiple Views(De Zhang, 2015, No journal)
医学影像配准、融合与跨模态合成中的模态解耦
独立保留医学影像分支:多模态形变配准的自适应递归推断与表示解耦、脑部多模态融合的互补/冗余分支解耦、CBCT-to-CT 合成中的内容-风格-伪影分离。三篇均以临床多模态数据验证解耦表征的有效性与泛化性,与视觉生成类解耦工作任务属性不同,故单列。
- ARDMR: Adaptive recursive inference and representation disentanglement for multimodal large deformation registration(Yi-Bo Hu, Qi Zhang, Ziqi Zhao, X. Ding, W. Li, Ji-Jin Yang, Jian-Qi Sun, L. Xu, 2025, Medical image analysis)
- A Disentangled Representation Based Brain Image Fusion via Group Lasso Penalty(An-Qi Wang, Xiao-Qing Luo, Zhan-Cheng Zhang, Xiao-Jun Wu, 2022, Frontiers in Neuroscience)
- Breath-Hold CBCT-Guided CBCT-to-CT Synthesis via Multimodal Unsupervised Representation Disentanglement Learning(Yi-Wen Zhang, Chuanpu Li, Z. Dai, L. Zhong, Xue-Tao Wang, Wei Yang, 2023, IEEE Transactions on Medical Imaging)
任务驱动的解耦-再融合:视觉与信号中的模态特有/共享分量重组
合并原"任务驱动的解耦融合"与多模态融合框架类文献:红外-可见光融合、医学图像融合、多模态高光谱压缩、超声 CAD、临床表征融合、细粒度解耦与结构化解耦融合框架。共同点是解耦只是中间步骤,关键在如何通过 AdaIN 注入、注意力增强、互信息正则把特有与共享分量可控地重组回融合表征。
- DCDR-GAN: A Densely Connected Disentangled Representation Generative Adversarial Network for Infrared and Visible Image Fusion(Yuan Gao, Shiwei Ma, Jing-Jing Liu, 2023, IEEE transactions on circuits and systems for video technology (Print))
- DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion(Wan-Wan Huang, Han Zhang, Yu Cheng, Xiong-Wen Quan, 2023, Frontiers in Physiology)
- Channel-wise joint disentanglement representation learning for B-mode and super-resolution ultrasound based CAD of breast cancer(Yu-Hang Zheng, Jiale Xu, Qing Hua, Xiao-Hong Jia, Xue-Qin Hou, Yan-Feng Yao, Zhengqi Wei, Yulu Zhang, FangGang Wu, Wei Guo, Yuan Tian, Jun Wang, Shu-Jun Xia, Yi-Jie Dong, Jun Shi, Jian-Qiao Zhou, 2026, Medical image analysis)
- A Representation Fusion Framework for Decoupling Diagnostic Information in Multimodal Learning(Sana Tonekaboni, S. Friedman, Xin-Yi Zhang, M. Maddah, Caroline Uhler, 2025, npj Digital Medicine)
- IG-GRD: A Model Based on Disentangled Graph Representation Learning for Imaging Genetic Data Fusion(Shuang Feng, Le-Tian Wang, Chang Li, Xiao-Hua Wan, Fa Zhang, Bin Hu, 2024, International Conference on Intelligent Computing)
- A Disentangled Representation-Based Multimodal Fusion Framework Integrating Pathomics and Radiomics for KRAS Mutation Detection in Colorectal Cancer(Zhilong Lv, Rui Yan, Yuexiao Lin, Ling Gao, Fa Zhang, Ying Wang, 2024, Big Data Mining and Analytics)
多模态与跨域推荐中的模态解耦及兴趣表示学习
合并三组中推荐/用户建模的解耦类文献:序列推荐的多兴趣解耦、基于余弦相似度的多模态跨域推荐、多模态融合的解耦表示、跨模态对比解耦推荐、解耦多兴趣重学、多视图用户解耦、跨视图知识融合与去噪、知识感知推荐解耦对比。共同点是同时处理多模态异质噪声与模态共享/独有知识分离,验证解耦在弱结构强噪声多源数据上的价值。
- Disentangled Multi-interest Representation Learning for Sequential Recommendation(Ying-Peng Du, Ziteng Wang, Zhu Sun, Yining Ma, Hong-Zhi Liu, Jie Zhang, 2024, Knowledge Discovery and Data Mining)
- Multimodal Cross-Domain Recommendation with Cosine Similarity-Based Representation Disentanglement(Hiroto Okada, Soh Yoshida, M. Muneyasu, 2025, Global Conference on Consumer Electronics)
- Learning Disentangled Representation for Multi-Modal Fusion in Resume-Job Matching(Da-Yong Shen, Xiang Jiang, Chenfeng Hu, Shiyi Xu, Zhong-Shan Zhang, Feng Yao, Tao Wang, 2025, 2025 11th International Conference on Big Data and Information Analytics (BigDIA))
- Contrastive Modality-Disentangled Learning for Multimodal Recommendation(Xi-Xun Lin, Rui Liu, Yanan Cao, Li-Xin Zou, Qian Li, Yongxuan Wu, Yang Liu, Da-Wei Yin, Guandong Xu, 2025, ACM Transactions on Information Systems)
- DMIMRec: Disentangled Multi-Interest Representation Learning for Multimodal Recommendation Systems(Chengyi Zhou, Min-He Nuo, Yuanyuan Ren, Rui Li, Hui Liu, 2025, Data Intelligence)
- Learning disentangled user representation with multi-view information fusion on social networks(Wenyi Tang, Bei Hui, Ling Tian, Guang-Chun Luo, Zao-Bo He, Zhi-Peng Cai, 2021, Information Fusion)
- Knowledge Fusion and Denoising with Cross-View Contrastive Learning: A Framework for Efficient Sparse Data Processing(Tao Qin, Zhiqiang Huang, Xin Wang, Chen-Xu Wang, Ping-Hui Wang, 2026, ACM Transactions on the Web)
- Disentangled Contrastive Learning for Knowledge-Aware Recommender System(Shuhua Huang, Chenhao Hu, Wei-Yang Kong, Yubao Liu, 2023, International Workshop on the Semantic Web)
多视图自监督解耦与超图协同的多侧面用户建模
保留原推荐组中偏"多视图/超图结构"的一支:多视图自监督解耦学习、模态内/模态间统一的表示、非线性增强解耦对比超图、多视图多侧面下一篮推荐。与上一组的区别在于其视图来自图结构(超图、多侧面交互)而非纯模态,属于结构化多源数据的跨视图互动扩展。
- MVSDL: a multi-view self-supervised disentangled learning approach for web API recommendation(F. Song, Bin Wang, Wen-Long Zhang, Xin-Qiang Xie, Zhi-Gao Zhang, Yi-Nan Liu, Ya-Tong Sun, R. Pu, Yan-Feng Zhang, 2026, Journal of King Saud University: Computer and Information Sciences)
- Unifying Within and Across: Intra-Modality Multi-View Fusion and Inter-Modality Alignment for Knowledge Graph Completion(Zhen Li, Ji-Bin Wang, Zhuo Chen, Kun Wu, Meng Ai, Lei-Ke An, Liqiang Wang, Haoxuan Li, 2025, IEEE International Conference on Acoustics, Speech, and Signal Processing)
- NE-DCHL: Nonlinear Enhanced Disentangled Contrastive Hypergraph Learning for Next Point-of-Interest Recommendation(Hongwei Zhang, Guo-Long Wang, Xiao Yan, 2025, Inf.)
- Multi-view Multi-aspect Neural Networks for Next-basket Recommendation(Zhiying Deng, Jianjun Li, Zhiqiang Guo, Wei Liu, Li Zou, Guo-Hui Li, 2023, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval)
生物医药多视图相似性网络对齐与跨视图对比学习
独立保留生物医药分支:药物-靶标/药物-药物相互作用预测中,把相似性网络、分子图、基序图、SMILES 序列对齐到共享锚空间,再用关系感知图 Transformer、节点自适应融合、药效团感知跨视图对比与交叉注意力融合。体现"多视图一致性对齐+动态融合"的完整链条,可作论文融合策略的对照案例。
- Drug–target interaction predictions with multi-view similarity network fusion strategy and deep interactive attention mechanism(Wei Song, Lewen Xu, Chen-Guang Han, Zhen Tian, Quan Zou, 2024, Bioinformatics)
- Relation-Aware Graph Transformer with Multi-View Consistency Alignment and Dynamic Fusion for Drug-Drug Interaction Prediction(Jue-Ming Li, Sheng-Yuan Yang, Xian-Fang Tang, Qiang Zhu, 2025, IEEE International Conference on Bioinformatics and Biomedicine)
- A Node-Adaptive Feature Fusion Network for Drug–Target Interaction Prediction Based on Multi-View Graphs(Lin Xie, Hong-Mei Xu, Pinglu Zhang, Jianshe Xiong, Jing Li, 2026, Biomolecules)
- CMCL-DDI: Pharmacophore-aware cross-view contrastive learning for drug-drug interaction prediction(Ye-Hong Han, Lin Du, 2026, PLoS ONE)
生成与3D视觉任务中的解耦表示:视图-动作分解、图像融合与隐式重建
合并原"视觉任务中的解耦表示"与多视图3D识别文献:把观测分解为视图与动作成分并互为推荐、把图像分解为场景属性与传感器属性分别融合、隐空间中细粒度3D语义解耦、以及多视图3D表征解耦。展示解耦不仅是表征约束,也可直接驱动融合质量与重建可解释性。
- Dual-Recommendation Disentanglement Network for View Fuzz in Action Recognition(Wenxuan Liu, X. Zhong, Zhuo Zhou, Kui Jiang, Zheng Wang, Chia-Wen Lin, 2023, IEEE Transactions on Image Processing)
- DRF: Disentangled Representation for Visible and Infrared Image Fusion(Han Xu, Xinya Wang, Jia-Yi Ma, 2021, IEEE Transactions on Instrumentation and Measurement)
- NaviNeRF++: Towards Interpretable 3D Reconstruction via Unsupervised Disentangled Representation Learning.(Ba-Ao Xie, Zequn Zhang, Huanting Guo, Qiu-Yu Chen, Hu Zhu, Bo-Han Li, Wen-Jun Zeng, Xin Jin, 2025, IEEE Transactions on Pattern Analysis and Machine Intelligence)
- Learning Disentangled Representation for Multi-View 3D Object Recognition(Jing-Jia Huang, Wei Yan, Ge Li, Thomas H. Li, Shan Liu, 2022, IEEE transactions on circuits and systems for video technology (Print))
自适应加权与张量低维嵌入的统一融合策略
保留关注"融合策略本身"的独立分组:层次化自适应多模态融合、基于张量学习与梯度下降求统一低维嵌入的多视图融合、以及多源监测下的多任务联合预测。三篇均不强调显式解耦,而是通过可学习权重或低维子空间适配各数据源贡献度,为论文融合环节提供加权聚合与统一表征的技术支撑。
- AHSD: adaptive multimodal fusion with hierarchical semantic disentanglement for sentiment analysis(Zheng-Yang Shi, Wei Huang, 2026, Multimedia Systems)
- A Multi-View Fusion Method via Tensor Learning and Gradient Descent for Image Features(Laihang Yu, Dongyan Zhang, Ningzhong Liu, Wen-Gang Zhou, 2021, IEEE Access)
- Advanced prediction of pipeline vertical deformation and axial strain via multi-source data fusion and multi-task deep learning(Zhen Sun, Xin Wang, Tian-Ran Han, Jia-Lin Liu, Longxiang Wang, Xing Wei, Huang Huang, Xi Huang, Zhi-Shen Wu, 2025, Structural Health Monitoring)
工业、能源与农业场景的多源数据融合工程实践
合并原工程实践类文献:多源振动/声发射/声纹的故障诊断、风电智能故障预警、电网负荷预测、大豆产地溯源、可靠多源融合(可信度未知)与 ResNet-Transformer 应用。共同点是在真实数据质量不均、可信度未知条件下做数据级/特征级/决策级融合的鲁棒性设计,为论文提供落地验证思路。
- MF-MSRNet: a fault diagnosis method for train bogie bearing based on multi-source data fusion and multi-scale residual network(Hai-Meng Sun, De-Qiang He, Zhenpeng Lao, Zhen-Zhen Jin, Ze-Xian Wei, Jin-Xin Wu, 2025, Nondestructive Testing and Evaluation)
- Intelligent Fault Warning Method for Wind Turbine Gear Transmission System Driven by Digital Twin and Multi-Source Data Fusion(Tiantian Xu, Xue-Dong Zhang, Wenlei Sun, 2025, Applied Sciences)
- Multi-Source Data Fusion-Based Grid-Level Load Forecasting(Hai Ye, Xiao-Bi Teng, Bing-Bing Song, Kai-Ming Zou, Moyan Zhu, Guang-Yu He, 2025, Applied Sciences)
- Multi-source data fusion for soybean origin traceability: Stable isotopes, elemental composition, & volatile organic compounds.(Khushboo Soni, Russell Frew, Biniam Kebede, 2025, Food Chemistry)
- RMDF-CV: A Reliable Multi-Source Data Fusion Scheme With Cross Validation for Quality Service Construction in Mobile Crowd Sensing(Ke-Jia Fan, Jia-Lin Guo, Runsheng Li, Yuanye Li, An-Feng Liu, Jian-Heng Tang, Tian Wang, Mian-Xiong Dong, H. Song, 2025, IEEE Transactions on Services Computing)
- Application of the ResNet-Transformer Model for Runoff Prediction Based on Multi-source Data Fusion(Shibang Zhu, Zhao-Cai Wang, Wen-Ting Zhang, Jing-Qi Yang, 2025, Water resources management)
遥感与海事多源异构传感器融合
保留遥感/海事分支:遥感多视图低秩融合优化、雷达极化-多普勒多特征融合海事检测、AIS/雷达/可见光异构传感器协同的水面目标检测与跟踪。其特点是跨传感器(而非跨语义模态)的一致性/互补性整合、低信杂比与盲区鲁棒性,并提供多源轨迹关联与融合数据集,为论文提供工程化评价参照。
- Multi-view fusion optimization method via low-rank tensor decomposition for remote sensing image classification(Laihang Yu, Ningzhong Liu, Dongyan Zhang, Shi Dong, 2024, Journal of Applied Remote Sensing)
- Multifeature Fusion Radar Detection for Maritime Small Targets via Disentangled Representation Learning(Ze-Zhou Wu, C. Pang, Guo-Qing Wu, Yong-Zhen Li, Shun-Ping Xiao, 2026, IEEE Sensors Journal)
- Surface Vessels Detection and Tracking Method and Datasets with Multi-Source Data Fusion in Real-World Complex Scenarios(Wen-Bin Huang, Hui Feng, Hai-Xiang Xu, Xu Liu, Jianhua He, Lang-Xiong Gan, Xiao-Qian Wang, Shanshan Wang, 2025, Italian National Conference on Sensors)
将三份初始分组去重合并后,137 篇文献被组织为 23 个并列分组,形成\"综述界定—解耦—跨视图互动—自适应融合—领域落地\"的完整叙事链:①第 1 组以综述与表征学习框架界定多源异构融合的问题空间与算法谱系;②解耦主线最为厚重,按目标函数(互信息/信息瓶颈/结构约束,第 2 组)、任务范式(多视图共享-特理解耦,第 3 组)、数据不完整性(缺失/增量视图,第 4 组)、结构化低维表征(多标签/特征选择/张量子空间,第 5 组)四个互补角度切分,避免笼统化;③跨视图互动主线按\"图结构增强与图滤波(第 6 组)—注意力与几何先验交互(第 7 组)—跨视图引导解耦与多尺度语义交互(第 8 组)—视图一致对齐与视图特定建模(第 9 组)\"层层递进,覆盖从方法机制到具体视觉任务的差别;④融合主线拆为跨模态注意力融合架构(第 10 组)、其工业鲁棒应用(第 11 组)、不确定性/混合专家自适应融合(第 12 组)与张量低维统一嵌入(第 21 组),兼具体系区分度与工程可行性;⑤应用域层面单列多模态情感(13)、生理行为信号(14)、医学影像(15)、任务驱动解耦-再融合(16)、多模态/跨域推荐(17)、多视图超图用户建模(18)、生物医药(19)、生成与3D视觉(20)、工业农业(22)、遥感海事(23),每个领域保留其独特任务约束与评价方式。整体统一结论为:多源融合的趋势已从\"早期拼接式融合\"转向\"先解耦出共享/特有成分—再以跨视图交互建立视图对应与互补—最后按可靠性与贡献度自适应聚合\"的三段式范式,这一脉络可直接支撑论文以解耦与跨视图互动为核心构建统一融合框架的立论与对比实验设计。
合并三组中的综述/总论类文献:多源数据融合的技术类型、算法协同与评价范式,跨域推荐算法族系比较,以及多视图/多模态表征学习的任务谱系与评价指标。共同作用是为全文界定"多源异构—解耦—互动—融合—决策"的问题边界与术语体系,是后续各章方法铺陈的上位视角。相关文献:Bing-Lin Liu et. al, 2025 等 6 篇文献
合并原三组中所有以信息论/层级结构为解耦目标函数的工作:用互信息估计、条件信息瓶颈、软分割、比例化约束与自监督分布约束显式地把共享(一致)信息与独有(特有、冗余、噪声)信息分离,并给出信息量配比的可解释分析。构成论文"解耦"部分的理论内核。相关文献:Jiaxin Dai et. al, 2026 等 8 篇文献
合并原三组中面向多视图聚类与一致性表征的共享-特理解耦工作:通过子空间自表达、变分推断、语义标签引导、内容-风格分离、一致锚图与自适应多图融合,保证共享成分与视图特有成分的独立性并抑制表征塌缩。是论文"解耦"主线最直接的方法与对比基线来源。相关文献:Zhi-Xiang Li et. al, 2025 等 9 篇文献
三组中均散落着处理"数据不完整"的解耦工作:部分视图/缺失视图的多标签与聚类学习、跨时间点视图补全、增量视图到达、双缺失(视图与标签)情形下的理论驱动表示。共同点是在不完整输入下仍维持共享-特有分解的可辨识性,对应论文面向真实多源数据(模态缺失)时的鲁棒性设计。相关文献:Xudong Yan et. al, 2025 等 7 篇文献
合并原分组中处理"结构化低维表征"的一类:一致-特有标签特定表示、多视图无监督特征选择、双重自表达张量特征选择、自适应加权图张量与二分图秩约束、协同张量嵌入聚类与锚图张量聚类。共同点是用张量/低秩/稀疏结构约束把多视图信息压缩为统一低维子空间,是论文融合环节的降维与选择技术支撑。相关文献:Yiting Li et. al, 2025 等 6 篇文献
合并自"图结构与时空建模"与"图与推荐系统中的多视图交互"两处内容:可信跨视图图增强后再融合、双向跨视图图滤波、自适应多图对比融合、多视图对比学习的自适应聚合以及时空交互与跨视图融合。共同点是"先增强/滤波再融合",用跨视图图结构交互抑制视图噪声,与论文的跨视图互动机制形成方法论呼应。相关文献:Peng-Lei Wang et. al, 2026 等 5 篇文献
合并三组中"跨视图互动"最纯粹的视觉类工作:视差引导跨视图 Transformer、多平面跨视图交互、多步跨视图变化检测交互、免标定 3D 的特征级跨视图融合、外视角-第一人称融合、跨视角手语识别与稳定人体位姿估计。共同点是借助注意力与几何先验显式建模视图间对应关系,是论文"跨视图互动"关键词最直接的参照。相关文献:Bing-Ting Li et. al, 2025 等 7 篇文献
合并原"跨视图特征交互与几何一致性建模"与"视图级与特征级融合架构"中的相关工作:跨视图引导的解耦器、跨视图注意力交互融合算法、多尺度全局感知特征学习、多尺度一致性增强、隐秩最小化的跨视图交互、2D-3D 交互融合与联合信息交互语义融合。共同主张不直接混合特征,而是先借视图间关系解耦、再做多尺度语义交互与融合。相关文献:Yaoyu He et. al, 2026 等 7 篇文献
保留原"跨视图交互与视图一致对齐"组的独特设定:以重识别、单目 HOI、XR 位姿等跨相机/跨视角任务为载体,级联跨视图特征交互、视图特定记忆与处理层、视图一致对齐与文本引导跨注意力,在解耦视角/尺度/运动因素的同时保留视图不变判别信息。与前一组的区别在于其目标是把视角差异显式建模为可分离因素。相关文献:Ze-Hao Zhang et. al, 2025 等 6 篇文献
合并原三组中"以跨模态注意力为融合算子"的通用架构类文献:双向/多查询/层级/级联/轻量化跨模态注意力融合网络,覆盖虚假内容检测、编程意图识别、认知障碍评估、博物馆文物分类、RGB-D 与显著目标检测、图像-掩膜与文本-图像对齐等任务,关注融合结构设计与可部署性,为论文融合模块提供结构范式。相关文献:Abdul Momen et. al, 2026 等 10 篇文献
保留原"CMAF 多任务应用"组的工程化分支:电网故障诊断、ECG-PCG 与日志-性能指标、多模态异常检测、缺失模态 MRI、语义统一等场景下的跨模态注意力融合与生成式增强。与通用 CMAF 组的区别在于其重点是模态可靠性不均与模态缺失条件下的融合鲁棒性。相关文献:Jiang-Xun Liu et. al, 2024 等 8 篇文献
独立保留该分组(三组中仅此一处集中讨论融合的"可靠性"与"动态路由"):证据不确定性编码、von Mises-Fisher 集中度加权、稀疏与上下文感知混合专家路由、解耦表征下的混合专家、混合融合网络与校准。共同解决模态冲突/冗余下"如何安全地加权融合",是解耦之后融合阶段的关键策略来源。相关文献:Yu-Peng Han et. al, 2026 等 5 篇文献
合并原"多模态情感分析"类工作:解耦多任务双曲空间对齐、自监督引导的模态解耦、Transformer 解耦表征、多样性与平衡约束的情感融合、基于互信息的模态配比。共同点是针对文本-声学-视觉的异质性与分布差异,先解耦共享/特有语义再对齐融合,作为论文范式在高语义任务上的验证。相关文献:Si-Yuan Ni et. al, 2026 等 5 篇文献
合并原"多模态情感与生理信号"与"步态类"文献:心理压力/EEG-ECG-眼动、细粒度解耦与模态无关融合、跨模态表示学习框架、多层级解耦、三阶段渐进式学习、不确定性感知步态表征与步态性别分类。共同点是生理/行为信号噪声大、模态互补性强,需先分离任务共享与模态独有语义再做融合。相关文献:Qun-Li Yao et. al, 2026 等 7 篇文献
独立保留医学影像分支:多模态形变配准的自适应递归推断与表示解耦、脑部多模态融合的互补/冗余分支解耦、CBCT-to-CT 合成中的内容-风格-伪影分离。三篇均以临床多模态数据验证解耦表征的有效性与泛化性,与视觉生成类解耦工作任务属性不同,故单列。相关文献:Yi-Bo Hu et. al, 2025 等 3 篇文献
合并原"任务驱动的解耦融合"与多模态融合框架类文献:红外-可见光融合、医学图像融合、多模态高光谱压缩、超声 CAD、临床表征融合、细粒度解耦与结构化解耦融合框架。共同点是解耦只是中间步骤,关键在如何通过 AdaIN 注入、注意力增强、互信息正则把特有与共享分量可控地重组回融合表征。相关文献:Yuan Gao et. al, 2023 等 6 篇文献
合并三组中推荐/用户建模的解耦类文献:序列推荐的多兴趣解耦、基于余弦相似度的多模态跨域推荐、多模态融合的解耦表示、跨模态对比解耦推荐、解耦多兴趣重学、多视图用户解耦、跨视图知识融合与去噪、知识感知推荐解耦对比。共同点是同时处理多模态异质噪声与模态共享/独有知识分离,验证解耦在弱结构强噪声多源数据上的价值。相关文献:Ying-Peng Du et. al, 2024 等 8 篇文献
保留原推荐组中偏"多视图/超图结构"的一支:多视图自监督解耦学习、模态内/模态间统一的表示、非线性增强解耦对比超图、多视图多侧面下一篮推荐。与上一组的区别在于其视图来自图结构(超图、多侧面交互)而非纯模态,属于结构化多源数据的跨视图互动扩展。相关文献:F. Song et. al, 2026 等 4 篇文献
独立保留生物医药分支:药物-靶标/药物-药物相互作用预测中,把相似性网络、分子图、基序图、SMILES 序列对齐到共享锚空间,再用关系感知图 Transformer、节点自适应融合、药效团感知跨视图对比与交叉注意力融合。体现"多视图一致性对齐+动态融合"的完整链条,可作论文融合策略的对照案例。相关文献:Wei Song et. al, 2024 等 4 篇文献
合并原"视觉任务中的解耦表示"与多视图3D识别文献:把观测分解为视图与动作成分并互为推荐、把图像分解为场景属性与传感器属性分别融合、隐空间中细粒度3D语义解耦、以及多视图3D表征解耦。展示解耦不仅是表征约束,也可直接驱动融合质量与重建可解释性。相关文献:Wenxuan Liu et. al, 2023 等 4 篇文献
保留关注"融合策略本身"的独立分组:层次化自适应多模态融合、基于张量学习与梯度下降求统一低维嵌入的多视图融合、以及多源监测下的多任务联合预测。三篇均不强调显式解耦,而是通过可学习权重或低维子空间适配各数据源贡献度,为论文融合环节提供加权聚合与统一表征的技术支撑。相关文献:Zheng-Yang Shi et. al, 2026 等 3 篇文献
合并原工程实践类文献:多源振动/声发射/声纹的故障诊断、风电智能故障预警、电网负荷预测、大豆产地溯源、可靠多源融合(可信度未知)与 ResNet-Transformer 应用。共同点是在真实数据质量不均、可信度未知条件下做数据级/特征级/决策级融合的鲁棒性设计,为论文提供落地验证思路。相关文献:Hai-Meng Sun et. al, 2025 等 6 篇文献
保留遥感/海事分支:遥感多视图低秩融合优化、雷达极化-多普勒多特征融合海事检测、AIS/雷达/可见光异构传感器协同的水面目标检测与跟踪。其特点是跨传感器(而非跨语义模态)的一致性/互补性整合、低信杂比与盲区鲁棒性,并提供多源轨迹关联与融合数据集,为论文提供工程化评价参照。相关文献:Laihang Yu et. al, 2024 等 3 篇文献
总计 137 篇相关文献
Multi-view clustering has gained significant attention due to its ability to integrate data from diverse perspectives, frequently outperforming single-view approaches. However, existing methods often assume a Gaussian distribution within the latent embedding space, which can degrade performance when handling high-dimensional data or data with complex, non-Gaussian distributions. These limitations complicate effective data alignment, hinder meaningful information fusion across views, and impair accurate similarity measurement. To overcome these challenges, we propose a novel contrastive multi-view clustering framework that leverages hyperspherical embeddings by explicitly modeling the latent space using the von Mises-Fisher (vMF) distribution. Additionally, the framework incorporates a contrastive learning paradigm guided by alignment and uniformity losses, facilitating more discriminative and disentangled representations within the hyperspherical latent space. Specifically, the alignment loss enhances consistency across embeddings of different views from the same instance, while the uniformity loss ensures distinctiveness among embeddings from different samples within each cluster. By jointly optimizing these objectives, our method substantially improves intra-cluster cohesion and inter-cluster separability across multiple views. Extensive experiments conducted on several benchmark datasets confirm that the proposed approach significantly outperforms state-of-the-art methods, particularly in scenarios involving high-dimensional and complex datasets. The source code of our model is publicly accessible at https://github.com/jcdh/DRMVC.
In incomplete multi-view multi-label learning scenarios, it is crucial to use the incomplete multi-view data to extract consistent and specific representations from different data sources and to fully exploit the missing label information. However, most previous approaches ignore the separation problem between view-shared and specific information. To address this problem, in this paper, we propose a method that can separate view-consistent features from view-specific features under the Variational Autoen-coder (VAE) framework. Specifically, we first introduce cross-view reconstruction to capture view-consistent features and extract shared information from different views through unsupervised pre-training. Subsequently, we develop a disentangling module to learn specific features by minimizing the variational upper bound of mutual information between consistent and specific features. Finally, we utilize prior label relevance information derived from training data to guide the learning of the distribution of label semantic embeddings, aggregating relevant semantic embeddings and maintaining the label relevance topology in the semantic space. In extensive experiments, our model outperforms existing state-of-the-art algorithms on several real-world datasets, which fully validates its strong adaptability to missing views and labels.
No abstract available
No abstract available
Multi-view clustering aims to fully explore the consistency and complementarity of multi-view data and has attracted increasing attention in recent years. However, most existing methods generally treat all views equally during the view fusion process, which cannot adaptively adjust the weights in terms of the quality and contribution of each view. Meanwhile, the disentanglement performance of those methods between view-common attribute and view-specific attributes is inadequate, thereby restricting the expressiveness of the learned features. To tackle these challenges, a novel framework of robust multi-view clustering via adaptive fusion and disentangled representation learning (AFDR-MVC) is proposed. Firstly, a multi-head attention mechanism is introduced to model inter-view correlations, which facilitates interaction and alignment among multiple views. Subsequently, a learnable gating network is employed to adaptively assign weights to each view based on its importance, allowing the model to learn a discriminative common representation under significant variations in view quality and noise. Finally, a mutual information minimization constraint is introduced to explicitly disentangle view-common variable and view-specific variables within a variational inference framework, yielding more discriminative common representation and specific representation which can improve clustering accuracy. Experimental results on multi-view datasets demonstrate that the proposed AFDR-MVC achieves superior clustering performance.
Multi-view clustering aims to exploit complementary information from multiple modalities to learn discriminative representations without supervision. However, existing methods often suffer from feature entanglement, where view-shared semantics are mixed with view-specific variations. This issue leads to less discriminative representations and degrades clustering performance. To address this issue, we propose DvMF (Disentangled Multi-View Fusion with von Mises-Fisher Distribution), a unified framework that combines representation disentanglement with uncertainty-aware feature fusion. Specifically, we introduce a tripartite mutual information objective to control dependencies among latent variables. It enhances cross-view consistency while reducing redundancy between shared and private representations, thus promoting the separation of view-shared semantics and view-specific factors. To robustly integrate disentangled representations, we project them onto a hyperspherical latent space modeled by the von Mises-Fisher (vMF) distribution. Fusion is then performed via a Product-of-Experts mechanism. The concentration parameter in the vMF distribution encodes representation reliability, enabling uncertainty-aware aggregation that down-weights less reliable features and mitigates the influence of noisy view-specific components in the fused embedding. Extensive experiments on multiple benchmark datasets show that DvMF consistently outperforms recent state-of-the-art multi-view clustering methods.
No abstract available
No abstract available
3D object recognition is a hot research topic. Particularly, view-based methods, which represent a 3D object with a collection of its rendered views on the 2D domain, play an important role in this field. Currently, view-based researches tend to aggregate information from multiple views via pooling based strategies to endow the models with the characteristic of view permutation invariance, at the cost of inevitable loss of useful features. In this paper, we introduce a new method that learns a more comprehensive descriptor for a 3D object from its views while successfully keeping its robustness to the variation of view permutation. Our method disentangles the information in the set of multi-view images into a global category-related feature and a set of view-permutation related features. To unbind these two parts, an encode-decoder based disentangling architecture is proposed, which barely bring extra computations compared to the baseline model. Systematic experiments are conducted for this new method to demonstrates the effectiveness and the competitive performance based on ModelNet40, ModelNet10, and ShapeNetCore55 datasets. Codes for our paper will be released soon on “https://github.com/hjjpku/multi_view_sort”.
Diffusion MRI-derived brain structural connectomes or brain networks are widely used in the brain research. However, constructing brain networks is highly dependent on various tractography algorithms, which leads to difficulties in deciding the optimal view concerning the downstream analysis. In this paper, we propose to learn a unified representation from multi-view brain networks. Particularly, we expect the learned representations to convey the information from different views fairly and in a disentangled sense. We achieve the disentanglement via an approach using unsupervised variational graph auto-encoders. We achieve the view-wise fairness, i.e. proportionality, via an alternative training routine. More specifically, we construct an analogy between training the deep network and the network flow problem. Based on the analogy, the fair representations learning is attained via a network scheduling algorithm aware of proportionality. The experimental results demonstrate that the learned representations fit various downstream tasks well. They also show that the proposed approach effectively preserves the proportionality.
No abstract available
Recently, much effort has been devoted to modeling users' multi-interests (aka multi-faceted preferences) based on their behaviors, aiming to accurately capture users' complex preferences. Existing methods attempt to model each interest of users through a distinct representation, but these multi-interest representations easily collapse into similar ones due to a lack of effective guidance. In this paper, we propose a generic multi-interest method for sequential recommendation, achieving disentangled representation learning of diverse interests technically and theoretically. To alleviate the collapse issue of multi-interests, we propose to conduct item partition guided by their likelihood of being co-purchased in a global view. It can encourage items in each group to focus on a discriminated interest, thus achieving effective disentangled learning of multi-interests. Specifically, we first prove the theoretical connection between item partition and spectral clustering, demonstrating its effectiveness in alleviating item-level and facet-level collapse issues that hinder existing disentangled methods. To efficiently optimize this problem, we then propose a Markov Random Field (MRF)-based method that samples small-scale sub-graphs from two separate MRFs, thus it can be approximated with a cross-entropy loss and optimized through contrastive learning. Finally, we perform multi-task learning to seamlessly align item partition learning with multi-interest modeling for more accurate recommendation. Experiments on three real-world datasets show that our method significantly outperforms state-of-the-art methods and can flexibly integrate with existing multi-interest models as a plugin to enhance their performances.
The rapid growth of Web APIs has made accurate API recommendation a core requirement for service composition and software development. API correlations are driven by both ID co-occurrence patterns and textual semantic features, which suffer from inherent modality conflicts: semantically irrelevant APIs may often co-occur frequently due to contextual factors, while semantically related APIs are not necessarily compatible. Coupled with data sparsity and cold-start problems, existing methods fail to achieve a good balance among recommendation accuracy, robustness, and interpretability. To this end, this paper proposes MVSDL, a multi-view self-supervised disentangled learning framework. Firstly, the global API co-occurrence structure and fine-grained textual semantics are encoded into a unified representation space via graph attention networks and pre-trained language models. Secondly, a self-supervised disentanglement mechanism is designed to separate ID co-occurrence representations from semantic representations and quantify their respective contributions. Finally, counterfactual-inspired supervision is introduced to dynamically identify the dominant modality and guide adaptive gating to achieve context-aware modality fusion. Extensive experiments on two real-world datasets demonstrate that MVSDL significantly outperforms state-of-the-art methods, and exhibits stronger robustness and interpretability in data sparsity and cold-start scenarios.
Multi-View Clustering (MVC) aims to enhance clustering performance by integrating multi-source complementary information. However, existing deep MVC methods face inherent challenges in balancing the learning of shared consensus representations with the preservation of view-specific information: independent encoders hinder effective cross-view collaboration, while a single shared encoder tends to sacrifice representation diversity. Although the recently introduced Mixture-of-Experts (MoE) model offers a novel approach to facilitating view collaboration, its flattened expert pool design often leads to entanglement between shared and specific information, and its routing mechanism limits collaboration potential by neglecting cross-view context. To address these challenges, this paper proposes a novel deep multi-view clustering framework—Decoupled Mixture-of-Experts with Context-Aware Routing for Multi-View Clustering (DMCAR-MVC). At its core is an innovative Decoupled MoE (D-MoE) architecture. We establish a public expert pool to learn cross-view shared representations while equipping each view with an independent private expert pool to capture its unique information, thereby structurally enforcing the decoupling of shared and specific representations. Building on this, we further design a Context-Aware Hierarchical Routing (CAHR) mechanism. When routing for the public expert pool, this mechanism introduces a global context vector to guide expert selection, enabling more efficient and globally informed cross-view collaboration. Finally, to optimize the model, we adopt a multi-level contrastive learning paradigm: on one hand, a cross-view alignment loss ensures semantic consistency in shared representations; on the other, an orthogonality constraint is imposed to further enhance separability between shared and specific representations. Extensive experiments on multiple benchmark datasets demonstrate that DMCAR-MVC significantly outperforms state-of-the-art methods across key clustering metrics. Additionally, comprehensive ablation studies thoroughly validate the effectiveness and necessity of each proposed component.
Next Point-of-Interest (POI) recommendation is a crucial task in personalized location-based services, aiming to predict the next POI that a user might visit based on their historical trajectories. Although sequence models and Graph Neural Networks (GNNs) have achieved significant success, they often overlook the diversity and dynamics of user preferences. To address these issues, researchers have begun to employ Hypergraph Convolutional Networks (HGCNs) for disentangled representation learning. However, two critical problems have received less attention: (1) the limited expressive capacity of conventional hypergraph convolution layers, which restricts the modeling of complex nonlinear user–POI preference interactions and consequently weakens generalization performance, and (2) the inadequate utilization of contrastive learning mechanisms, which prevents fully capturing cross-view collaborative signals and limits the exploitation of complementary multi-view information. To tackle these challenges, we propose a Nonlinear Enhanced Disentangled Contrastive Hypergraph Learning (NE-DCHL) for next POI recommendation. The proposed model enhances nonlinear modeling capability and generalization by integrating ReLU activation, residual connections, and dropout regularization within the hypergraph convolution layer. A K-Nearest Neighbor (KNN)-based weighted adjacency matrix is employed to construct the geographical-view hypergraph, reducing computational complexity while maintaining essential spatial correlations. Moreover, a mini-batch InfoNCE loss and the GRACE (deep GRAph Contrastive rEpresentation learning) framework are utilized to improve efficiency and cross-view collaboration. Extensive experiments on two real-world datasets demonstrate that NE-DCHL consistently outperforms the original DCHL and other state-of-the-art approaches.
Next-basket recommendation (NBR) is a type of recommendation that aims to recommend a set of items to users according to their historical basket sequences. Existing NBR methods suffer from two limitations: (1) overlooking low-level item correlations, which results in coarse-grained item representation; and (2) failing to consider spurious interests in repeated behaviors, leading to suboptimal user interest learning. To address these limitations, we propose a novel solution named Multi-view Multi-aspect Neural Recommendation (MMNR) for NBR, which first normalizes the interactions from both the user-side and item-side, respectively, aiming to remove the spurious interests, and utilizes them as weights for items from different views to construct differentiated representations for each interaction item, enabling comprehensive user interest learning. Then, to capture low-level item correlations, MMNR models different aspects of items to obtain disentangled representations of items, thereby fully capturing multiple user interests. Extensive experiments on real-world datasets demonstrate the effectiveness of MMNR, showing that it consistently outperforms several state-of-the-art NBR methods.
Effective multimodal learning in real-world scenarios depends on a nuanced treatment of uncertainty, which arises at three levels: (1) Intrinsic Uncertainty from modality-specific noise or ambiguity; (2) Relational Uncertainty due to cross-modal conflicts or redundancy; and (3) Aggregated Uncertainty when fusing potentially inconsistent signals. Most existing methods overlook this hierarchy, applying a single uncertainty model. We propose Adaptive Evidential Multimodal Representation Learning (AEMRL), a framework aligned with this multi-level view. To address intrinsic uncertainty, Disentangled Evidential Uncertainty Encoding (D-EUE) provides interpretable, class-aware reliability scores per modality. For relational uncertainty, Uncertainty-Conditioned Dynamic Factorization (UDF) uses a hypernetwork to dynamically extract complementary cues and suppress conflict. To resolve aggregated uncertainty, Adaptive Fusion & Conflict-Aware Calibration (AFCAC) adaptively weights evidence streams and calibrates final predictions based on detected conflicts. Extensive experiments on five diverse benchmark datasets show that AEMRL consistently enhances task accuracy, reduces calibration error, and improves robustness to noise, semantic conflict, and missing modalities.
No abstract available
In the context of the booming construction of smart cities, multi-source data fusion and analysis algorithms play a key role in optimizing real estate management and improving urban efficiency. In this review, we comprehensively and systematically review the relevant algorithms, covering the types, characteristics, fusion techniques, analysis algorithms, and their synergies of multi-source data. We found that multi-source data, including sensors, social media, citizen feedback, and GIS data, face challenges such as data quality and privacy security when being fused. Data fusion algorithms are diverse and have their own advantages and disadvantages. Data analysis algorithms help urban management in areas such as spatial analysis and deep learning. Algorithm collaboration can improve decision-making accuracy and efficiency and promote the rational allocation of urban resources. In the future, algorithm development will focus on data quality, real-time, deep mining, interdisciplinary research, privacy protection, and collaborative application expansion, providing strong support for the sustainable development of smart cities.
Mobile Crowd Sensing is a prevalent and efficient paradigm for multi-source data collection, where Multi-source Data Fusion (MDF) plays a crucial role in constructing quality data collection services. Current MDF methods often require the majority of participating sensing sources to be credible, or assume that the workers’ credibility is either prior known or easily calculable. However, due to the presence of uncredible environments and the problem of Information Elicitation Without Verification (IEWV), these methods are impractical. It may lead to a vicious cycle where the recruitment of uncredible workers affects the quality of the estimated truth, which can further lead to misjudgments of worker credibility, thereby exacerbating the quality of subsequent recruitment. In this article, a Reliable Multi-source Data Fusion scheme with Cross Validation (RMDF-CV) is proposed to obtain reliable truth for service construction. Specifically, we first introduce the Combinatorial Multi-Armed Bandit (CMAB) model to recruit high-credibility workers by balancing exploration and exploitation. Then, we establish three-stage truth data through three different data sources: Unmanned Aerial Vehicles, credible workers, and Deep Matrix Factorization. Theoretical analyses and extensive simulations confirm the excellent performance of our RMDF-CV scheme.
No abstract available
ABSTRACT As the core component of the railway train, the healthy state of the bogie bearing is essential for the safe operation of the train. The traditional bearing fault diagnosis methods typically rely on a single source signal, which cannot fully capture fault feature information, resulting in low diagnosis accuracy. To address this problem, this paper proposes a new diagnosis framework for train bogie bearings based on multi-source data fusion and multi-scale residual network (MF-MSRNet). Firstly, a multi-source data fusion method is designed to extract fault feature information from voiceprint, acoustic emission, and vibration sensor data, effectively extracting the low-dimensional features embedded in high-dimensional nonlinear data and fusing them into RGB images. Secondly, a new dual-scale residual block is presented to learn both profound and shallow features at various scales, thereby capturing bearing fault information in different spatial dimensions and enhancing the capacity to extract fusion features. Finally, to enhance the adaptability of the residual network in noisy scenes, a new denoising module is designed to help the network explore multi-scale features and filter irrelevant information. The experimental outcomes indicate that the classification accuracy of MSRNet in traction motor bearing datasets can reach up to 99.75%, and its comprehensive fault diagnosis performance is the best among all models.
Environment sensing plays an important role for the safe autonomous navigation of intelligent ships. However, the inherent limitations of sensors, such as the low frequency of the automatic identification system (AIS), blind zone of the marine radar, and lack of depth information in visible images, make it difficult to achieve accurate sensing with a single modality of sensor data. To overcome this limitation, we propose a new multi-source data fusion framework and technologies that integrate AIS, radar, and visible data. This framework leverages the complementary strengths of these different types of sensors to enhance sensing performance, especially in real complex scenarios where single-modality data are significantly affected by blind zone and adverse weather conditions. We first design a multi-stage detection and tracking method (named MSTrack). By feeding the historical fusion results back to earlier tracking stages, the proposed method identifies and refines potential missing detections from the layered detection and tracking processes of radar and visible images. Then, a cascade association matching method is proposed to realize the association between multi-source trajectories. It first performs pairwise association in a high-accuracy aligned coordinate system, followed by association in a low-accuracy coordinate system and integrated matching between multi-source data. Through these association operations, the method can effectively reduce the association errors caused by measurement noise and projection system errors. Furthermore, we develop the first multi-source fusion dataset for intelligent vessel (WHUT-MSFVessel), and validate our methods. The experimental results show that our multi-source data fusion methods significantly improve the sensing accuracy and identity consistency of tracking, achieving average MOTA scores of 0.872 and 0.938 on the radar and visible images, respectively, and IDF1 scores of 0.811 and 0.929. Additionally, the fusion accuracy reaches up to 0.9, which can provide vessels with a comprehensive perception of the navigation environment for safer navigation.
No abstract available
To meet the demands for real-time and accurate fault warning of wind turbine gear transmission systems, this study proposes an innovative intelligent warning method based on the integration of digital twin and multi-source data fusion. A digital twin system architecture is developed, comprising a high-precision geometric model and a dynamic mechanism model, enabling real-time interaction and data fusion between the physical transmission system and its virtual model. At the algorithmic level, a CNN-LSTM-Attention fault prediction model is proposed, which innovatively integrates the spatial feature extraction capabilities of a convolutional neural network (CNN), the temporal modeling advantages of long short-term memory (LSTM), and the key information-focusing characteristics of an attention mechanism. Experimental validation shows that this model outperforms traditional methods in prediction accuracy. Specifically, it achieves average improvements of 0.3945, 0.546 and 0.061 in Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), and R-squared (R2) metrics, respectively. Building on the above findings, a monitoring and early warning platform for the wind turbine transmission system was developed, integrating digital twin visualization with intelligent prediction functions. This platform enables a fully intelligent process from data acquisition and status evaluation to fault warning, providing an innovative solution for the predictive maintenance of wind turbines.
No abstract available
Ensuring the sustainable sourcing of soybeans, as mandated by the European Union Deforestation Regulation (EUDR), requires high spatial resolution to trace soybeans back to their origin. Addressing this challenge necessitates integrating multiple analytical approaches, making data fusion a powerful solution. As global soybean demand nearly doubled over the past decade, the industry faces pressing issues like food fraud, deforestation, and climate change. This study evaluates four data fusion strategies-Low-level, Mid-Principal Component Analysis-Random Forest (PCA-RF), Mid-Uniform Manifold Approximation and Projection-Random Forest (UMAP-RF), and High-level fusion-using data from 60 soybean samples from six Brazilian states. Analytical techniques, including stable isotope analysis, elemental profiling, and volatile organic compound characterisation, were employed. High-level data fusion achieved 100 % classification accuracy for the test set, with Mid-UMAP-RF closely following at 99 %, demonstrating data fusion's role in improving traceability and ensuring sustainable agricultural practices.
Vertical deformation and axial strain are crucial indicators for evaluating pipeline performance. Accurate prediction of these indicators provides essential support for early warning and proactive maintenance. To achieve this, a multi-sensor monitoring system was constructed by combining an embedded static leveling instrument with a long-gauge fiber-optic grating sensor system to sequentially monitor vertical deformation and axial strain. A novel multi-task prediction model, HO-CNN-BiGRU-AM-HMTL, was proposed to capture the spatiotemporal correlations between deformation and strain. The sensitivity of the model’s hyperparameters was examined, and the effects of dataset size (DS) and prediction horizon on model performance were analyzed. The model’s performance was evaluated by comparing it with existing single-task learning (STL) and multi-task learning (MTL) models in terms of prediction accuracy and computational efficiency. The results indicated that hyperparameter tuning of the hippopotamus optimization module enhanced the model’s performance. The DS showed an initial improvement followed by a decline in performance, while the prediction horizon demonstrated a gradual decrease, then a sharp drop. Maintaining the DS between 4 and 5 months and the prediction horizon within 7 days is recommended. The HO-CNN-BiGRU-AM-HMTL model outperformed existing STL and MTL models in prediction accuracy, reducing computation time by 70.13%–88.48% compared to STL and improving it by 4.05%–40.54% over MTL. Although there was some computational overhead, the overall runtime of the model remained under 16 s, meeting the requirements for engineering applications. This study integrates pipeline engineering with deep learning techniques to enable advanced prediction of deformation and strain, providing crucial support for risk warning and proactive maintenance.
No abstract available
This paper introduces a novel weighted fusion methodology for grid-level short-term load forecasting that addresses the critical limitations of direct aggregation methods currently used by regional dispatch centers. Traditional approaches accumulate provincial forecasts without considering regional heterogeneity in load characteristics, data quality, and forecasting capabilities. Our methodology implements a comprehensive evaluation index system that quantifies forecast trustworthiness through three key dimensions: forecast reliability, provincial impact, and forecasting complexity. The core innovation lies in our principal component analysis (PCA)-based weighted aggregation mechanism that dynamically adjusts provincial weights according to their evaluated reliability, further enhancing through time-varying weights that adapt to changing load patterns throughout the day. Experimental validation across three representative seasonal periods (moderate temperature, high temperature, and winter conditions) substantiates that our weighted fusion approach consistently outperforms direct aggregation, achieving a 24.67% improvement in overall MAPE (from 3.09% to 2.33%). Performance gains are particularly significant during critical peak periods, with up to 62.6% error reduction under high-temperature conditions. The methodology verifies remarkable adaptability across different temporal scales, seasonal variations, and regional characteristics, consistently maintaining superior performance from ultra-short-term (1 h) to medium-term (168 h) forecasting horizons. Analysis of provincial weight dynamics reveals intelligent redistribution of weights across seasons, with summer months characterized by Jiangsu dominance (0.30–0.35) shifting to increased Anhui contribution (0.30–0.35) during winter. Our approach provides grid dispatch centers with a computationally efficient solution for enhancing the integration of heterogeneous forecasts from diverse regions, leveraging the complementary strengths of individual provincial systems while supporting safer and more economical power system operations without requiring modifications to existing forecasting infrastructure.
Dense sampling of the light field (LF) is essential for various applications, such as virtual reality. However, the collection process is prohibitively expensive due to technological limitations in imaging. Synthesizing novel views from sparse LF data, known as LF Angular Super-Resolution (LFASR), offers an effective solution to this problem. Accurate cross-view interaction is crucial for this task, given the complementary information between LF views. Previous methods, however, suffer from limited reconstruction quality due to inefficient view interaction. To address this, we propose a Multiplane-based Cross-view Interaction Mechanism (MCIM) for robust LFASR. Extensive comparisons with state-of-the-art methods demonstrate that our method achieves superior performance, both visually and quantitatively. Specifically, Drawing inspiration from MultiPlane Images (MPI) in scene modeling, our mechanism incorporates a novel Multiplane Feature Fusion (MPFF) strategy. This strategy facilitates fast and accurate cross-view interaction, enhancing the network’s robustness to scene geometry and suitability for different-baseline LF scenes. Furthermore, to address information redundancy in multiplanes, we leverage the transparency property of MPI and devise a plane selection strategy. Finally, we propose CSTNet, a Cross-Shaped Transformer-based network for LFASR, which employs a cross-shaped self-attention mechanism to enable low-cost training and inference. Experimental results on various angular super-resolution tasks validate that our network achieves state-of-the-art performance on both synthetic and real-world LF scenes.
Cross-view object geo-localization (CVOGL) offers fine-grained geographic information regarding the specified object of interest, in comparison with typical cross-view geo-location at the image level. However, it faces more challenges, primarily due to significant visual appearance changes induced by viewpoint discrepancies and the difficulty of accurately identifying corresponding objects within reference images containing multiple targets. To address these challenges, we propose a novel CVOGL model termed hierarchical adaptive feature learning for geo-localization (HALO). First, we introduce a cascade cross-view feature interaction module that enables effective local-to-global feature fusion across different viewpoints, thereby enhancing the feature representation of objects in the reference image. Second, to mitigate the scale variations and feature distribution discrepancies between the query and reference images, we propose an adaptive feature hierarchization and aggregate module. It extracts more representative global descriptors by adaptively hierarchizing and aggregating features. Finally, utilizing the learned global descriptors, we perform cross-view CL and introduce a hard sample mining strategy (HA-SM) to further enhance the discriminative ability of the network. Through these advancements, we significantly improve the discriminative representations of the query objects at both views. Extensive experiments on the CVOGL dataset demonstrate the effectiveness and robustness of the proposed method. In the CVOGL task of Drone → Satellite, HALO improves 2.36% in acc@0.5 on the testing sets. Similarly, HALO shows an increase of 2.49% in acc@0.5 on the validation set for the CVOGL task of Ground → Satellite. Our codes are publicly available at https://github.com/ZehaoZhang-Uestc/HALO
No abstract available
In the process of stereo super-resolution reconstruction, in addition to the richness of the extracted feature information directly affecting the texture details of the reconstructed image, the texture details of the corresponding pixels between stereo image pairs also have an important impact on the reconstruction accuracy in the process of network learning. Therefore, aiming at the information interaction and stereo consistency of stereo image pairs, a cross-view attention interaction fusion stereo super-resolution algorithm is proposed. Firstly, based on parallax attention mechanism and triple attention mechanism, an attention stereo fusion module is constructed. The attention stereo fusion module is inserted between different levels of two single image super-resolution network branches, and the attention weight is calculated through the cross dimensional interaction of the three branches. It makes full use of the ability of single image super-resolution network to extract single view information and further maintaining the stereo consistency between stereo image pairs. Then, an enhanced cross-view interaction strategy including three fusion methods is proposed. Specifically, the vertical sparse fusion method is used to integrate the interior view information of different levels in the two single image super-resolution sub branches, the horizontal dense fusion method is used to connect the adjacent attention stereo fusion modules and the constraint between stereo image consistency is further strengthened in combination with the feature fusion method. Finally, the experimental results on Flickr 1024, Middlebury and KITTI benchmark datasets show that the proposed algorithm is superior to the existing stereo image super-resolution methods in quantitative measurement and qualitative visual quality while maintaining the stereo consistency of image pairs.
Egocentric 3D Human Pose Estimation (HPE) suffers from severe perspective distortion and self-occlusion, degrading lower-body pose estimation. In face-to-face XR interactions, a third-person view captured by an interacting partner’s front-facing camera provides complementary global context, but the independently moving cameras do not share known cross-view extrinsic parameters or camera odometry. Conventional unweighted Procrustes alignment is sensitive to noisy egocentric joints, which can degrade the estimated rigid transformation. To address this, we propose an odometry-free cross-view fusion framework with Visibility-Guided Alignment (VGA), which incorporates predicted joint visibility as correspondence weights in weighted Procrustes alignment to reduce the influence of unreliable joints during rigid registration. Experiments on a synthetic dataset demonstrate that our method reduces overall MPJPE by 13% compared with the third-person-only baseline.
Traffic flow prediction is a critical task within intelligent transportation systems (ITS). However, the highly nonlinear, dynamically evolving, and strongly spatio-temporal coupled nature of traffic states poses significant challenges for constructing models that are both accurate and robust. Notably, existing approaches often rely on a single static topology or computationally expensive dynamic graph reconstruction for spatial modeling, while lacking mechanisms to jointly capture local temporal trends and global dependencies in spatio-temporal representations. To address these limitations, we propose STIEFormer, an enhanced Transformer model designed for deep spatio-temporal interaction and multi-view fusion. The model leverages a Dynamic SpatioTemporal Propagation Cross-Attention (DSTP-CA) mechanism, which employs lightweight 1 D convolutions to capture local spatio-temporal trends and dynamically models propagation patterns across time and nodes, achieving a deep integration of local trends and global dependencies. Furthermore, a CrossView Fusion Module (CVFM) constructs dynamic semantic graphs to uncover long-range latent similarities and integrates them with static geographic topologies to produce multi-view, consistent node representations, thereby enhancing feature expressiveness and model robustness. Experimental results on the real-world PEMS04 and PEMS08 traffic datasets demonstrate that STIEFormer consistently outperforms existing state-of-theart methods across multiple evaluation metrics, confirming the effectiveness and superiority of the proposed framework.
Multi-View 3D human pose estimation is a fundamental problem in computer vision with applications in human–computer interaction, AR/VR, robotics, and tactile internet, where accurate body motion capture is crucial for immersive and real-time interaction. Traditional approaches often rely on camera calibration at inference, typically through triangulation of 2D detections, which limits deployment in dynamic or unstructured environments. In this work, we propose a novel framework that leverages intermediate features from transformer backbones for multi-level, cross-view feature fusion. A teacher–student supervision strategy exploits camera extrinsics only during training to guide the learning of geometry-aware correspondences. Evaluations on the CMU Panoptic dataset demonstrate that the proposed method achieves state-of-the-art performance among calibration-free approaches.
Vehicle re-identification aims to retrieve images of the same vehicle under different viewpoints, camera perspectives, and environmental conditions from a large-scale image database. However, significant challenges arise due to intra-class variations and inter-class similarities. Moreover, the difficulty in effectively utilizing multi-view information substantially limits further improvements in recognition accuracy. This paper proposes a novel multi-branch network named MGF-Net that enhances vehicle feature representation by jointly optimizing global and fine-grained feature extraction modules. In the global feature extraction module, a dual-dimensional channel–spatial adaptive feature re-weighting mechanism is implemented to amplify responses in vehicle-centered regions, suppress background interference, and improve the robustness of global features. In the fine-grained feature extraction module, a multi-stage feature interaction mechanism is constructed to capture detailed features and spatial semantic associations of key vehicle parts, such as the front grille and hood emblem, thereby avoiding the feature fragmentation issue caused by manual part partitioning. To further improve recognition performance in cross-view scenarios, a multi-view fusion strategy is introduced to dynamically aggregate feature representations from different viewpoints, thereby enhancing the model’s generalization ability. Additionally, both branches are optimized using classification loss and metric loss functions to refine the model from multiple perspectives. Extensive experiments on the challenging VeRi-776 and VehicleID datasets demonstrate that our approach achieves state-of-the-art performance and significantly improves vehicle re-identification accuracy.
Most existing multi-view graph clustering models either seek consistent clustering results from similarity matrices and spectral embeddings respectively or follow direct bidirectional integration of them, which ignores the interaction between them. To make up for this flaw, this paper designs a novel multi-view clustering model that performs Bidirectional Fusion with Cross-view Graph Filter (BF-CGF). To be specific, BF-CGF first learns a consistent graph embedding via performing the interaction between multi-view graphs and spectral embeddings with the perspective of the graph spectral domain and then considers seeking a consistent indicator matrix via the graph cut model from the consistent graph embedding and the similarity matrices. To solve the optimization problem of BF-CGF, we propose an efficient iterative algorithm and provide the corresponding convergence and complexity analyses. Extensive experimental results demonstrate that the proposed BF-CGF outperforms state-of-the-art competitors in most benchmark datasets.
Towards stable human pose estimation from monocular images, there remain two main dilemmas. On the one hand, the different perspectives, i.e., front view, side view, and top view, appear the inconsistent performances due to the depth ambiguity. On the other hand, foot posture plays a significant role in complicated human pose estimation, i.e., dance and sports, and foot-ground interaction, but unfortunately, it is omitted in most general approaches and datasets. In this paper, we first propose the Cross-View Fusion (CVF) module to catch up with better 3D intermediate representation and alleviate the view inconsistency based on the vision transformer encoder. Then the optimization-based method is introduced to reconstruct the foot pose and foot-ground contact for the general multi-view datasets including AIST++ and Human3.6M. Besides, the reversible kinematic topology strategy is innovated to utilize the contact information into the full-body with foot pose regressor. Extensive experiments on the popular benchmarks demonstrate that our method outperforms the state-of-the-art approaches by achieving 40.1mm PA-MPJPE on the 3DPW test set and 43.8mm on the AIST++ test set.
Change captioning is a task that describes changes in image pairs using natural language. This task is more complex than single-image captioning as it requires a comprehensive understanding of each image and the ability to recognize and describe the semantic changes in image pairs. The key challenge lies in making the network generate an accurate and stable change representation under the interference of viewpoint shift. In this paper, we propose a cross-view and multi-step interaction network to generate robust change representation to resist pseudo-change. Specifically, in the intra-image representation learning stage, a cross-view interaction encoder is designed to enhance internal relationships by cross-referencing in image pairs. In the change feature learning stage, a multi-step change perceptron is employed to capture the change semantics from coarse to fine progressively. Then, a fusion module dynamically combines them as a fine-grained change representation. Besides, we propose a backward representation reconstruction module that facilitates the capture of semantic changes, thus improving the quality of captions in a self-supervised manner. Extensive experiments have shown that the method effectively captures real semantic changes under the interference of viewpoint shift and achieves state-of-the-art performance on five public datasets.
Microsurgical anastomosis has become increasingly prevalent in surgical autonomy, requiring accurate and stable control of suturing needles and threads while enhancing the efficiency and safety of microsurgical operations. However, current systems predominantly employ top-down-view microscopes for intraoperative imaging, which are constrained by limited field-of-view and significant occlusion caused by instrument-tissue interactions. To address these challenges, we develop a dual-view vision system for microsurgical anastomosis, integrating both conventional top-down-view microscopes and eye-in-hand cameras mounted on surgical instrument tips. Our approach involves cross-view feature fusion through different schemes to improve microsurgical scene understanding, including surgical action recognition, gripper-object interaction prediction, and instrument pose estimation. Extensive anastomosis datasets are collected on our robotic platform and several experiments are conducted for detailed evaluation of the system performance. Quantitative and qualitative results demonstrate that our dual-view microsurgical system significantly outperforms single-view microscopes in terms of robust visual perception, and cross-view feature fusion improves both the accuracy and precision of anastomosis scene understanding.
Cross-view geo-localization (CVGL) aims to match the correspondence between images of different views captured in the same geographic area. Its key is to mine sufficiently discriminative features in different views to form an invariant representation of the same object location. However, existing methods focus on extracting feature information from a single approach, such as coarse-grained, fine-grained, or intermediate-layer outputs, while ignoring the sensitivity of multiscale feature fusion, such as channel, space, and training strategy. In this article, we propose a new enhanced feature interaction (EFI) and multiscale scene consistency method, named ECSNet. First, we explicitly bridge positive samples across different batches and aggregate features into a global classifier to promote cross-view invariance and enhance scene separability. Second, a tensor-based multiscale attention (TMA) is designed to utilize the attention support to fuse the multiscale features of the input image, the intermediate-layer, and the edges to maintain the view-space details at different granularities. Meanwhile, the EFI leverages a visual state space with residual connections to preserve positional offsets, to preserve local features, and suppresses irrelevant features while selectively zooming in on important details through channel attention. Finally, a regularized curriculum learning (CL) strategy is introduced to CL evaluates the magnitude of positional changes and the difficulty of samples, stabilizing optimization while preventing premature overfitting to noisy high-loss noise pairs. Extensive experiments on University-1652 and SUES-200 demonstrate that the method achieves state-of-the-art performance in drone object localization and drone navigation applications, with an improvement of AP of more than 3% compared to existing methods.
Recommender systems are effective tools to alleviate the challenges posed by information overload, but data sparsity has greatly affected their performance. Knowledge graphs (KG) can be employed to solve this problem because of their rich side information. However, existing KG-enhanced recommendation methods have the following limitations: (1) Noisy interactions: the user-item interactions constructed from implicit feedback are noisy, which is harmful to the user’s true preferences learning; (2) Insufficient fusion between different views: most KG-enhanced recommendation methods ignore the differences between signals from KG and user-item interactions and fail to fuse them effectively. To address these limitations, we propose a novel KG-enhanced recommendation framework named Knowledge Fusion and Denoising with cross-view Contrastive Learning framework (KFDCL). Specifically, we first perform contrastive learning (CL) across KG (knowledge view) and user-item interaction data (collaborative view) and then explicitly prune noisy interactions to alleviate the noisy interactions problem. Additionally, we design a weighted fusion strategy based on the activity of users and items to fuse the signals from different views adaptively. Extensive experiments on three public datasets demonstrate the effectiveness of the proposed KFDCL, notably with an average improvement of 3.79% performance over state-of-the-art methods on Recall@20 and NDCG@20.
No abstract available
Recent learning-based multi-view stereo (MVS) methods have achieved notable progress, yet accurate and complete reconstruction in weakly textured regions remains challenging. To address this issue, we present MGCFI-Net (Multi-Scale Globally Aware Cross-View Feature Interaction Network), which is able to learn multi-scale globally aware representations and explicitly model cross-view feature interactions under supervision. Specifically, MGCFI-Net incorporates a Multi-Scale Hierarchical Perception (MSHP) module that integrates the standard convolution, parallel dilated convolutions, and a Swin Transformer to progressively capture discriminative features with global contextual awareness. For multi-scale feature fusion, we use patch-expanding upsampling to preserve semantics, improving cross-scale integration. To enhance cross-view geometric consistency, a Cross-View Feature Interaction (CVFI) module is introduced, in which epipolar self-attention is applied within each source feature and epipolar cross-attention is performed between each source feature and the reference feature, enabling more reliable reference-source feature matching. Furthermore, we incorporate Cross-View Feature Similarity Supervision (CFSS), which enforces feature-level consistency through geometric warping and improves cross-view alignment and matching robustness under challenging conditions. Extensive evaluations on the DTU, Tanks & Temples, and ETH3D benchmarks demonstrate that MGCFI-Net achieves superior reconstruction quality and establishes new state-of-the-art (SOTA) performance, validating its effectiveness and generalization capability.
Although transformer-based methods excel in stereo image super-resolution, the full potential of the distinctive, complementary information inherent in stereo images has not been fully utilized. We propose a Disparity-Guided Cross-View Transformer (DCT) to extract features across dimensions and views, achieving a more comprehensive feature representation. The proposed method introduces mutual attention within the transformer architecture, establishing the difference between left and right views through cross-view interaction. The proposed algorithm effectively harnesses the complementary information present in stereo image pairs, enhancing the restoration performance. Furthermore, we propose a disparity-guided cross-modal residual fusion module that leverages disparity information as prior knowledge to substantially improve image reconstruction. This module significantly complements the missing information in stereo images, enabling the network to comprehend more effectively and reconstruct the image content with greater accuracy. Extensive experimental results and ablation studies demonstrate the effectiveness of our method.
Accurate prediction of potential drug-drug interactions (DDIs) is vital for ensuring medication safety and efficacy. Existing graph-based methods typically focus on molecular structures but often overlook the complementary semantic information embedded in SMILES (Simplified Molecular Input Line Entry System) representations. To address this gap, we propose CMCL-DDI, a Cross-view Mutual Contrastive Learning framework that jointly leverages pharmacophore-aware molecular graphs and SMILES sequences. Specifically, we encode pharmacophore-based subgraphs to capture functional molecular features and aggregate them into expressive graph-level embeddings. In parallel, SMILES sequences are encoded to preserve sequential drug characteristics. A contrastive learning strategy aligns both views in a shared latent space, facilitating mutual representation enhancement. Furthermore, we design a cross-attention fusion module to integrate heterogeneous features, enabling robust and interpretable DDI prediction. Extensive experiments on benchmark datasets demonstrate that CMCL-DDI consistently outperforms state-of-the-art models, highlighting the effectiveness of cross-view representation learning for DDI prediction. The source codes are available at https://github.com/95LY/CMCL-DDI.
Reliable aerial-ground video-based person re-identification (ReID) remains a challenge due to severe changes in data quality and features, such as viewpoint disparities, resolution drops, and cross-camera appearance inconsistency. This paper presents VM-TAPS, a lightweight and modular extension to the well-known TF-CLIP framework, designed to increase the robustness of ReID, without requiring end-to-end backbone retraining. When compared to its ancestor, VM-TAPS' novelties are five-fold: 1) View-Specific Processing Layers to normalize camera-dependent biases; 2) Scale-Aware Feature Adaptation for resolution-invariant feature fusion; 3) a View-Aware Memory Bank enabling long-range identity context; 4) a Motion Pattern Analyzer capturing temporal dynamics; and (5) Cross-View Interaction Modules that harmonize multi-view feature spaces. Despite adding fewer than two million parameters, VM-TAPS achieves +4.97% Rank-1 and +3.08% mAP gains over TF-CLIP on the challenging AG-VPReID2025 benchmark. At 80m and 120m altitudes, it sets a new performance baseline of 73.68%/75.73% and 69.45%/71.63% (Rank-1/mAP), respectively. All components are trained with frozen CLIP visual encoders in the early stages, enabling efficient and stable convergence. Our results support that the carefully disentanglement of viewpoint, scale, motion and memory factors substantially increases the robustness of cross-view ReID under real-world conditions.
Depression can severely impact an individual's mental health and overall quality of life if not treated promptly. Physiological studies have revealed significant differences in audio and facial expressions between individuals with depression and those without. Multimodal approaches to depression detection, which integrate information from multiple modalities, often exhibit superior performance in comparison to single-modal methods. However, there are still some challenges to hinder effective fusion, including semantic discrepancies and distribution inconsistencises across modalities. To address these issues, we propose a disentangled representation learning framework, designed to decompose homogeneous, heterogeneous, and noise information within multi-modal features, thereby reducing the adverse effects of distribution mismatches. Specifically, the first step involves the utilization of a convolutional neural network and two types of transformer encoder to extract features from audio and facial modalities. Considering the existence of noise signals, feature disentanglement decomposes these features into distinct components, with various regularization losses imposed to constrain the distance relationships among the homogeneous, heterogeneous, and noisy representations. Finally, a cross graph fusion mechanism is introduced to facilitate context-aware fusion across modalities, further enhancing the representation learning and generalization capabilities of the model. Experiments conducted on the D-Vlog and LMVD datasets demonstrate that our method outperforms existing state-of-the-art approaches across key metrics, validating its effectiveness.
Structured and unstructured data are two common data types in the online recruitment process. Traditional resume-job matching algorithms tend to focus on only one of these data types. Effectively integrating both and fully exploiting the hidden information they contain is of great importance. To address this challenge, we propose a novel Hypergraph-based shared Representation Fusion recommendation model, named HrFuse.Specifically, we first learn the representation of structured data through hypergraphs, while leveraging a pre-trained language model to vectorize the unstructured data. Subsequently, we introduce a HrFuse module to disentangle cross-modal shared features from modality-specific features.Finally, we evaluate our model on a real-world global dataset collected from one of the largest job recruitment platforms in China. Experimental results demonstrate that HrFuse outperforms all baseline models across all evaluation metrics. This study highlights the benefits of integrating structured and unstructured data through shared representation fusion.
Multimodal sentiment analysis seeks to utilize various types of signals to identify underlying emotions and sentiments. A key challenge in this field lies in multimodal representation learning, which aims to develop effective methods for integrating multimodal features into cohesive representations. Recent advancements include two notable approaches: one focuses on decomposing multimodal features into modality-invariant and -specific components, while the other emphasizes the use of mutual information to enhance the fusion of modalities. Both strategies have demonstrated effectiveness and yielded remarkable results. In this paper, we propose a novel learning framework that combines the strengths of these two approaches, termed mutual information-based disentangled multimodal representation learning. Our approach involves estimating different types of information during feature extraction and fusion stages. Specifically, we quantitatively assess and adjust the proportions of modality-invariant, -specific, and -complementary information during feature extraction. Subsequently, during fusion, we evaluate the amount of information retained by each modality in the fused representation. We employ mutual information or conditional mutual information to estimate each type of information content. By reconciling the proportions of these different types of information, our approach achieves state-of-the-art performance on popular sentiment analysis benchmarks, including CMU-MOSI and CMU-MOSEI.
Modern medicine increasingly relies on multimodal data, ranging from clinical notes to imaging and genomics, to guide diagnosis and treatment. However, integrating these heterogeneous data sources in a principled and interpretable manner remains a major challenge. We present MODES (Multi-mOdal Disentangled Embedding Space), a representation fusion framework that explicitly separates shared and modality-specific factors of variation, offering a structured latent space for multimodal information that improves both prediction and interpretability. By leveraging pre-trained unimodal foundation models, MODES mitigates the dependency on extensive paired datasets, crucial in data-scarce clinical settings. We introduce a masking strategy that optimizes representation dimensionality by eliminating low-information dimensions, to achieve compact, information-rich representations. Our framework demonstrates superior performance in predicting diagnoses and phenotypes compared to unimodal and conventional fusion models. MODES also enables robust diagnostic inference in missing data scenarios, offering an opportunity toward interpretable and efficient multimodal diagnostics in personalized healthcare.
No abstract available
This paper proposes a new infrared and visible image fusion method based on the densely connected disentangled representation generative adversarial network (DCDR-GAN), which strips the content and the modal features of infrared and visible images through disentangled representation (DR) and fuses them separately. To deal with the mutually exclusive features in infrared and visible images, inject the modal features into the reconstruction of content features through adaptive instance normalization (AdaIN), reducing the interference. To reduce feature loss and ensure the expression of all-level features in the fused image, DCDR-GAN designs the densely connected content encoders and fusion decoder and constructs the multi-scale fusion structures between the enc-dec through long connections. Meanwhile, the content and the modal reconstruction losses are proposed to preserve the information of the source images. Finally, through the two-phase trained model, generate the fused image. The subjective and objective evaluation results of the TNO and INO datasets show that the proposed method has better visual effects and higher index values than other state-of-the-art methods.
No abstract available
In this article, we propose a novel decomposition method by applying disentangled representation for visible and infrared image fusion (DRF). According to the imaging principle, we perform the decomposition depending on the source of information in the visible and infrared images. More concretely, we disentangle the images into the scene- and sensor modality (attribute)-related representations through the corresponding encoders, respectively. In this way, the unique information defined by the attribute-related representation is closer to the information captured by each type of sensor individually. Thus, the problem of inappropriate extraction of unique information can be alleviated. Then, different strategies are applied for the fusion of these different types of representations. Finally, the fused representations are fed into the pretrained generator to generate the fusion result. The qualitative and quantitative experiments on the publicly available TNO and RoadScene data sets demonstrate the comparable performance of our DRF over the state of the art in terms of both visual effect and quantitative metrics.
Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism-based disentangled representation network for medical image fusion, which designs coordinate attention and multimodal attention to extract and strengthen common and exclusive features. First, the common and exclusive features of each modality were obtained by the cross mutual information and adversarial objective methods, respectively. Then, coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, these two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis, and four metrics are used for quantitative comparison. The values of the four metrics demonstrate the effect of the DRCM. Furthermore, the DRCM achieved better results on SCD, Nabf, and MS-SSIM metrics, which indicates that the DRCM achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, it was found that the DRCM outperforms the comparison method.
No abstract available
As the impact of chronic mental disorders increases, multimodal sentiment analysis (MSA) has emerged to improve diagnosis and treatment. In this paper, our approach leverages disentangled representation learning to address modality heterogeneity with self-supervised learning as a guidance. The self-supervised learning is proposed to generate pseudo unimodal labels and guide modality-specific representation learning, preventing the acquisition of meaningless features. Additionally, we also propose a text-centric fusion to effectively mitigate the impacts of noise and redundant information and fuse the acquired disentangled representations into a comprehensive multimodal representation. We evaluate our model on three publicly available benchmark datasets for multimodal sentiment analysis and a privately collected dataset focusing on schizophrenia counseling. The experimental results demonstrate state-of-the-art performance across various metrics on the benchmark datasets, surpassing related works. Furthermore, our learning algorithm shows promising performance in real-world applications, outperforming our previous work and achieving significant progress in schizophrenia assessment.
Complementary and redundant relationships inherently exist between multi-modal medical images captured from the same brain. Fusion processes conducted on intermingled representations can cause information distortion and the loss of discriminative modality information. To fully exploit the interdependency between source images for better feature representation and improve the fusion accuracy, we present the multi-modal brain medical image fusion method in a disentangled pipeline under the deep learning framework. A three-branch auto-encoder with two complementary branches and a redundant branch is designed to extract the exclusive modality features and common structure features from input images. Especially, to promote the disentanglement of complement and redundancy, a complementary group lasso penalty is proposed to constrain the extracted feature maps. Then, based on the disentangled representations, different fusion strategies are adopted for complementary features and redundant features, respectively. The experiments demonstrate the superior performance of the proposed fusion method in terms of structure preservation, visual quality, and running efficiency.
Deep learning-based method for medical image fusion has become a hot topic in recent years. However, they ignore the expression of the most important features in image fusion and only extract the general features for medical image fusion, which will restrict the expression of unique information on the fusion image. To address this restriction, we propose a novel disentangled representation network for medical image fusion with mutual information estimation, which extract the disentangled features of medical image fusion, i.e., the shared and exclusive features between multi-model medical images. In our method, we use the cross mutual information method to obtain the shared features of each modality pair, which enforce the fusion network to achieve the maximum of mutual information estimation for multi-modal medical images. The exclusive features are extracted by the adversarial objective method and it constrains the fusion network with the optimization to the minimum of mutual information estimation between shared and exclusive features. These disentangled features effectively take the interpretative advantages and make the fusion image retaining more details from source images as well as improving the visual quality of fusion image. Our method has achieved better results than several state-of-the-art methods. Both qualitative and quantitative experiments have proved the superiority of our method.
This study addresses the problem of radar maritime target detection in scenarios characterized by low signal-to-clutter ratios (SCRs) and out-of-distribution (OoD). The complex scattering characteristics of sea clutter are difficult to model precisely. Detection methods often exhibit mismatches between their modeled clutter scattering characteristics and the actual sea clutter, resulting in poor detection performance in diverse maritime environments. This study proposes a multifeature fusion detection method, namely, the multifeature fusion detector (MFFD) that employs disentangled representation learning. MFFD is designed to adapt to varying environments and comprehensively utilize information from radar data. Specifically, the MFFD first extracts polarization-Doppler and polarization time–frequency features from raw radar echoes. These features are then encoded into disentangled representational vectors, which include target vectors, sea state classification vectors, and redundant vectors. Finally, the MFFD fuses target vectors using an attention mechanism. During training, MFFD employs a multitask loss function to ensure the accurate disentanglement of representational vectors. Furthermore, a method for filtering target label noise has been introduced to improve the quality of datasets, especially in instances where target occlusion occurs. The experimental results demonstrate that the MFFD achieves superior detection performance compared with several existing detection methods, thereby offering a promising solution for maritime target detection in challenging conditions.
Multiview representation learning has recently attracted significant attention in the machine learning and computer vision community. However, during fusing information from multiple views, existing work often neglect to exploit the complementary information in each view and endow with the interpretability of model. To this end, in this article, we propose a multiview attention fusion information bottleneck network, termed by MVAIBNet. Specifically, MVAIBNet deliberately develops dual-path reconstructions to extract latent embeddings of views, where view-peculiar representations are disentangled from the embeddings using $\beta$-VAE, to help reconstruct each view. Then, to align and fuse view-common representations, an attention fusion unit, namely the multiview channel fusion unit (MVCFU), is presented accordingly. Furthermore, relying on the information bottleneck principle, we integrate the consistency information and specificity information of the views to prompt a compact semantic representation of multiple views with balancing the complementarity and consistency among multiple views flexibly. Extensive experimental results on four real-world datasets show that our algorithm achieves encouraging performance on several evaluation metrics compared to the state-of-the-art methods.
Incremental Multi-view Clustering (IMvC) has received increasing attention due to its ability to handle dynamically arriving views. However, most existing IMvC methods enforce a unified representation to facilitate cross-view knowledge transfer, which inevitably suppresses view-specific characteristics. To overcome this, we propose a novel Anchor-based Disentanglement Framework for Incremental Multi-view Clustering (ADIMC), which introduces disentanglement representation learning to enable conflict-free knowledge transfer under view-incremental scenarios. Specifically, for each newly arrived view, an anchor graph is first learned to extract the latent semantic information. Subsequently, the anchor graph is decomposed into view-consistent and view-specific components to explicitly disentangle the shared semantics and private information of each view. Based on these, knowledge transfer is restricted to the view-consistent components to maintain cross-view coherence, while the view-specific components are preserved to retain the uniqueness of each view. Combined with an efficient iterative optimization algorithm, ADIMC ensures stable and efficient model updating with limited computing resources. Extensive experiments on multiple benchmark datasets validate the superiority and effectiveness of ADIMC over state-of-the-art methods.
Multi-view clustering, as a highly valuable data analysis technique against the backdrop of exponential data growth, usually relies on the principles of consistency and complementarity. However, most existing methods either assume the presence of a consistent latent representation shared or simply fuse the view-specific representations without explicitly separating the shared information from the view-specific information. This often leads to the loss of key complementary information and introduces redundancy or noise. To address these challenges, we propose a Shared-Specific-Disentangled Variational Autoencoder framework called DSPMVC. This framework explicitly separates the view-shared information from the view-specific information via variational inference, which enhances the disentanglement and interpretability of the learned representations. Subsequently, by incorporating part of the useful private information into the clustering process, DSPMVC can fully leverage both the shared information and the complementary information. Experimental evaluations on several popular datasets demonstrate that our approach attains competitive clustering prowess relative to cuttingedge counterparts.
Recently, multi-view multi-label learning has gained significant attention due to its applicability in various domains. However, due to the limitations of data collection and the subjectivity of manual labeling, multi-view multi-label learning often faces both partial views and incomplete labels, substantially impacting the performance of existing classification methods in practical applications. Although existing methods have attempted to address this issue, they struggle to fully exploit the consistency and complementarity of multi-view multi-label data simultaneously. To overcome this limitation, we propose a novel structure-guided decoupled contrastive framework (SGDC). Specifically, to address the limitations of conventional single-encoder paradigms, SGDC innovatively incorporates a decoupled disentanglement mechanism (DDM). By integrating a dual-encoder architecture with mutual information upper bound constraints, DDM decouples multi-view features into view-specific and view-consistent components, which can improve the quality of feature representations. Additionally, the SGDC integrates a structure-guided contrastive (SGC) framework that performs dynamic semantic alignment in consistent feature spaces through global structural modeling, effectively implementing structure-conscious representation learning guided by the multi-view consensus principle. Experimental results on multiple datasets consistently demonstrate the effectiveness of our method in handling complex multi-view multi-label data.
Multi-modal knowledge graph completion (MMKGC) enhances the structural and semantic richness of knowledge graphs by integrating diverse information across modalities. However, existing methods often either overlook the diversity within a single modality or fail to ensure effective cross-modality alignment for entity representation. This leads to suboptimal entity representations, as inconsistent or irrelevant data is treated uniformly. To address these challenges, we propose a unified framework that combines intra-modality multi-view fusion with cross-modality alignment (IMVIA for short). Our approach captures the most relevant information within each modality by leveraging relational context. Simultaneously, we apply information disentanglement and contrastive learning, allowing each modality-specific learner to focus on extracting distinctive features while maintaining consistent training objectives across all modalities. Furthermore, we employ a relation-aware gated decision fusion network to robustly integrate diverse information. Experimental results show that IMVIA significantly outperforms state-of-the-art approaches across multiple benchmark datasets, validating its effectiveness and robustness in MMKGC task.
The real-world data is relatively complex, generally formed by the interaction of different latent factors. Disentanglement of these latent factors can effectively improve the robustness and interpretability of sample representation. However, most existing disentangled multi-view clustering methods focus on the irrelevance of disentangled representations, ignoring the semantic relevance invariance between different latent factors. To address this issue, we propose a disentangled contrastive multi-view clustering via semantic relevance invariance (DMVCS) to learn the disentangled representations and maintain their semantic relevance. Specifically, we first decompose each view into consistent and specific representations by maximizing semantic consistency and minimizing the correlation between multiple views. Meanwhile, to ensure that different disentangled representations have similar semantic relevance, a cross-component semantic relevance alignment module is proposed. Combined with the hierarchical sampling strategy, the learned semantic relevances are aligned progressively in a locally structure-aware manner. Besides, to learn a clustering-friendly unified representation, we propose a multi-hop neighbor contrastive learning to extend the range of positive samples. Comprehensive experiments on ten public multi-view datasets demonstrate that DMVCS outperforms the state-of-the-art clustering methods.
No abstract available
With the increasing adoption of UAV platforms in areas such as public safety and smart cities, Aerial-Ground Person Re-Identification (AGPReID) has emerged as a crucial yet highly challenging task, garnering growing interest from the research community. While existing approaches have leveraged identity attributes and viewpoint disentanglement strategies to improve cross-view matching, their heavy reliance on prior knowledge often compromises model generalization. Furthermore, some methods that explicitly separate viewpoints may unintentionally discard identity-related, view-invariant features, leading to incomplete identity representations. To address these limitations, we propose a CLIP-based View-Consistent Alignment Framework (CVAF) with two training stages. In the first stage, learnable text tokens are employed to represent identity-aware textual descriptions. To promote consistent alignment across varying viewpoints, we introduce a Text Consistency Loss (TCL) that regularizes the stability of text-token interactions with multi-view images. In the second stage, we present a Semantic Filtering Module (SFM) that jointly modulates image patch tokens along spatial and channel dimensions. A text-guided cross-attention mechanism generates spatial attention maps to explicitly emphasize identity-relevant regions, while semantic matching between textual features and visual tokens enables adaptive reweighting of image representations, effectively suppressing background clutter and view-specific noise. Extensive experiments on multiple AGPReID datasets demonstrate that our CVAF outperforms the state-of-the-art methods.
Multi-view learning models the relationships between various observations, and is adept to explore the underlying information of data from multiple perspectives. Since well representation is vital for self-expressive subspace clustering, we propose a method called Multi-View Subspace Clustering with Consistent and view-Specific Latent Factors and Coefficient Matrices (MVSC-CSLFCM) that explores the consensus and complementary information of multiple views, and we also impose suitable constraints on coefficient matrices corresponding to the obtained view-specific and consistent representations, respectively. Finally, an effective optimization algorithm based on augmented lagrangian multiplier is introduced to optimize our proposed MVSC-CSLFCM. Comprehensive experiments on four real-world data sets demonstrate the superiority of our proposed method by comparing with a series of state-of-art subspace algorithms.
No abstract available
As a prominent research topic, multi-view multi-label classification (MvMlC) aims to assign multiple labels to samples by integrating information from various perspectives. However, in real-world scenarios, MvMlC frequently faces the learning challenge of data with missing views and labels, typically resulting from sensor malfunctions, or the costly and time-consuming process of manual annotation. In addition, learning robust representations that are both consistent across views and specific to individual views remains a challenge. To address these issues, we propose a novel double incomplete multi-view multi-label classification framework based on Disentangling Consistent and Specific Information (DCSI). Specifically, we employ a dual-channel encoder with identical architecture but distinct objectives to extract cross-view consistent information and view-specific unique information from all views, respectively. Meanwhile, a view discriminator is constructed to decouple these two types of information, facilitating the extraction of pure consistent and specific information. Moreover, we meticulously design fusion strategies tailored to each representation type. Regarding consistent representations, we propose a dynamic-confidence-aware fusion mechanism that assesses the reliability of each view’s representations in relation to the classification task, enabling the model to prioritize information from trustworthy representations. For specific representations, in light of their complementary rather than redundant property, we suggest treating such representations from each view equally to ensure fairness. Through experimental validation on five datasets, the results demonstrate that our method outperforms existing state-of-the-art methods.
Longitudinal multi-view omics data offer unique insights into the temporal dynamics of individual-level physiology, which provides opportunities to advance personalized healthcare. However, the common occurrence of incomplete views makes extrapolation tasks difficult, and there is a lack of tailored methods for this critical issue. Here, we introduce LEOPARD, an innovative approach specifically designed to complete missing views in multi-timepoint omics data. By disentangling longitudinal omics data into content and temporal representations, LEOPARD transfers the temporal knowledge to the omics-specific content, thereby completing missing views. The effectiveness of LEOPARD is validated on three benchmark datasets constructed with data from the MGH COVID study and the KORA cohort, spanning periods from 3 days to 14 years. Compared to conventional imputation methods, such as missForest, PMM, GLMM, and cGAN, LEOPARD yields the most robust results across the benchmark datasets. LEOPARD-imputed data also achieve the highest agreement with observed data in our analyses for age-associated metabolites detection, estimated glomerular filtration rate-associated proteins identification, and chronic kidney disease prediction. Our work takes the first step toward a generalized treatment of missing views in longitudinal omics data, enabling comprehensive exploration of temporal dynamics and providing valuable insights into personalized healthcare.
Multi-view action recognition aims to identify action categories from given clues. Existing studies ignore the negative influences of fuzzy views between view and action in disentangling, commonly arising the mistaken recognition results. To this end, we regard the observed image as the composition of the view and action components, and give full play to the advantages of multiple views via the adaptive cooperative representation among these two components, forming a Dual-Recommendation Disentanglement Network (DRDN) for multi-view action recognition. Specifically, 1) For the action, we leverage a multi-level Specific Information Recommendation (SIR) to enhance the interaction among intricate activities and views. SIR offers a more comprehensive representation of activities, measuring the trade-off between global and local information. 2) For the view, we utilize a Pyramid Dynamic Recommendation (PDR) to learn a complete and detailed global representation by transferring features from different views. It is explicitly restricted to resist the fuzzy noise influence, focusing on positive knowledge from other views. Our DRDN aims for complete action and view representation, where PDR directly guides action to disentangle with view features and SIR considers mutual exclusivity of view and action clues. Extensive experiments have indicated that the multi-view action recognition method DRDN we proposed achieves state-of-the-art performance over powerful competitors on several standard benchmarks. The code will be available at https://github.com/51cloud/DRDN.
Anchor-based multi-view graph clustering has recently gained popularity as an effective approach for clustering data with multiple views. However, existing methods have limitations in terms of handling inconsistent information and noise across views, resulting in an unreliable consensus representation. In addition, post-processing is needed to obtain final results after anchor graph construction, which negatively affects clustering performance. In this article, we propose a Robust and Consistent Anchor Graph Learning method (RCAGL) for multi-view clustering to address these challenges. RCAGL constructs a consistent anchor graph that captures inter-view commonality and filters out view-specific noise by learning a consistent part and a view-specific part simultaneously. A <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives><mml:math><mml:mi>k</mml:mi></mml:math><inline-graphic xlink:href="liu-ieq1-3364663.gif"/></alternatives></inline-formula>-connectivity constraint is imposed on the consistent anchor graph, leading to a clear graph structure and direct generation of cluster labels without additional post-processing. Experimental results on several benchmark datasets demonstrate the superiority of RCAGL in terms of clustering accuracy, scalability to large-scale data, and robustness to view-specific noise, outperforming advanced multi-view clustering methods.
Multi-view multi-label learning typically suffers from dual data incompleteness due to limitations in feature storage and annotation costs. The interplay of heterogeneous features, numerous labels, and missing information significantly degrades model performance. To tackle the complex yet highly practical challenges, we propose a Theory-Driven Label-Specific Representation (TDLSR) framework. Through constructing the view-specific sample topology and prototype association graph, we develop the proximity-aware imputation mechanism, while deriving class representatives that capture the label correlation semantics. To obtain semantically distinct view representations, we introduce principles of information shift, interaction and orthogonality, which promotes the disentanglement of representation information, and mitigates message distortion and redundancy. Besides, label-semantic-guided feature learning is employed to identify the discriminative shared and specific representations and refine the label preference across views. Moreover, we theoretically investigate the characteristics of representation learning and the generalization performance. Finally, extensive experiments on public datasets and real-world applications validate the effectiveness of TDLSR.
Multi-view region embedding has become an important technique to be pursued due to the increasing availability of diverse urban sensing data. Existing methods typically adopt attention-based fusion or contrastive alignment to integrate multiple data sources such as human mobility and points-of-interest (POIs) into unified region representations. However, these approaches often fail to effectively balance cross-view commonality modeling with the preservation of view-specific characteristics. To address this limitation, we propose ComSRE, a novel region embedding framework that explicitly disentangles shared and distinctive components in multi-view region representations. ComSRE integrates three key modules: (1) a Multi-view Representation Learning module that fuses complementary information across views, (2) a View-to-Commonality Contrastive Alignment module that aligns the representation of each view to a shared commonality anchor to enhance cross-view consistency, and (3) a Multi-view Differential Orthogonality module that isolates distinctive signals unique to each view and promotes their independence through orthogonality constraints. Extensive experiments on three real-world urban datasets demonstrate that ComSRE consistently outperforms state-of-the-art methods across multiple downstream tasks, achieving superior predictive accuracy and representation quality.
Deep incomplete multi-view clustering methods can mine patterns of incomplete multi-view data without labels, gaining great attention in various domains. However, current methods are obsessed with aligning view-specific representations from available samples with complete views to learn view-invariant information for missing data recovery, ignoring fruitful view-complementary information beneficial to data imputations. Meanwhile, such the view-invariant learning often leads to the problem of dominant view dependency that models tend to over-rely on views with clear clustering structures and neglect weaker ones, causing a performance bottleneck. Therefore, a content-style disentanglement guided representation learning is proposed for the incomplete multi-view clustering (CSMVC). Specifically, CSMVC designs a view-specific dual-representation learning architecture that extracts view-invariant content representations and view-specific style representations from self-supervised reconstructions of each view with the guidance of the Hilbert schmidt independence criterion. Then, it performs a content-centered style-modulated representation imputation mechanism to infer missing data via fully utilizing inter-view complementary and consistent information. Meanwhile, it devises a cross-view consensus structure mining strategy to capture the clustering structure with intra-cluster compactness and inter-cluster separability from attention-based fusion representations that adaptively aggregate content representations and style representations of each view. Finally, comprehensive experiments show that CSMVC outperforms state-of-the-art methods.
Hyperspectral image (HSI) and light detection and ranging (LiDAR) data can provide complementary features and have shown great potential for land cover classification. Recently, the joint classification using the HSI and LiDAR data based on deep learning networks (e.g., convolutional neural networks and transformers) have made progress. However, these methods often use the channel or spatial dimension attentions to highlight features, which overlook the interdependencies between these dimensions and face challenges in effectively extracting and fusing diverse features from heterogeneous datasets. To address these challenges, a novel hierarchical cross-modal attention fusion network (HCAFNet) is proposed in this manuscript. First, a hierarchical convolution module is designed to extract diverse features from multisource data and to achieve initial fusion using the octave convolution. Then, a bidirectional feature fusion module is constructed to integrate heterogeneous features within the network. To further enhance the network's feature representation capability, a triplet rotational multihead attention module is designed to capture cross-dimensional dependencies, enabling more effective representation of both channel and spatial information. Experimental results conducted on three public datasets demonstrate that the proposed HCAFNet outperforms other advanced methods.
Background The information between multimodal magnetic resonance imaging (MRI) is complementary. Combining multiple modalities for brain tumor image segmentation can improve segmentation accuracy, which has great significance for disease diagnosis and treatment. However, different degrees of missing modality data often occur in clinical practice, which may lead to serious performance degradation or even failure of brain tumor segmentation methods relying on full-modality sequences to complete the segmentation task. To solve the above problems, this study aimed to design a new deep learning network for incomplete multimodal brain tumor segmentation. Methods We propose a novel cross-modal attention fusion-based deep neural network (CMAF-Net) for incomplete multimodal brain tumor segmentation, which is based on a three-dimensional (3D) U-Net architecture with encoding and decoding structure, a 3D Swin block, and a cross-modal attention fusion (CMAF) block. A convolutional encoder is initially used to extract the specific features from different modalities, and an effective 3D Swin block is constructed to model the long-range dependencies to obtain richer information for brain tumor segmentation. Then, a cross-attention based CMAF module is proposed that can deal with different missing modality situations by fusing features between different modalities to learn the shared representations of the tumor regions. Finally, the fused latent representation is decoded to obtain the final segmentation result. Additionally, channel attention module (CAM) and spatial attention module (SAM) are incorporated into the network to further improve the robustness of the model; the CAM to help focus on important feature channels, and the SAM to learn the importance of different spatial regions. Results Evaluation experiments on the widely-used BraTS 2018 and BraTS 2020 datasets demonstrated the effectiveness of the proposed CMAF-Net which achieved average Dice scores of 87.9%, 81.8%, and 64.3%, as well as Hausdorff distances of 4.21, 5.35, and 4.02 for whole tumor, tumor core, and enhancing tumor on the BraTS 2020 dataset, respectively, outperforming several state-of-the-art segmentation methods in missing modalities situations. Conclusions The experimental results show that the proposed CMAF-Net can achieve accurate brain tumor segmentation in the case of missing modalities with promising application potential.
In large-scale power systems, accurately detecting and diagnosing the type of faults when they occur in the grid is a challenging problem. The classification performance of most existing grid fault diagnosis methods depends on the richness and reliability of the data, in addition, it is difficult to obtain sufficient feature information from unimodal circuit signals. To address these issues, we propose a deep residual convolutional neural network (DRCNN)-based framework for grid fault diagnosis. First, we design a comprehensive information entropy value (CIEV) evaluation metric that combines fuzzy entropy (FuzEn) and mutual approximation entropy (MutEn) to integrate multiple decomposition subsequences. Then, DRCNN and heterogeneous graph transformer (HGT) are constructed for extracting multimodal features and considering modal variability. In addition, to obtain the implicit information of multimodal features and control the degree of their performance, we propose to incorporate the cross-modal attention fusion (CMAF) mechanism in the synthesis framework. We validate the proposed method on the three-phase transmission line dataset and VSB power line dataset with accuracies of 99.4 % and 99.0 %, respectively. The proposed method also achieves superior performance compared to classical and state-of-the-art methods.
Cultural heritage preservation has garnered global attention. Museum artifact classification, a core task, faces challenges related to insufficient multimodal information collaboration and a scarcity of high-quality annotated data. Traditional methods and single-modality deep learning models struggle to achieve both efficiency and accuracy. To address this, this paper proposes a museum artifact classification model (VBG Model) based on cross-modal attention fusion and generative data augmentation. This model constructs an integrated multimodal framework through task-oriented refactoring of the Vision Transformer (ViT), BERT, and a Generative Adversarial Network (GAN). ViT extracts global visual features from artifact images, while BERT mines the historical and cultural semantics of text. A bidirectional interactive attention fusion layer achieves precise feature alignment. The GAN generates diverse samples, forming a closed “generation-feedback-optimization” loop to alleviate data scarcity. Experiments on the MET and MS COCO datasets demonstrate exceptional performance: the VBG Model achieves 92% classification accuracy, 0.85 mAP, and 88% F1 score for the former, while the latter achieves 90% accuracy, 0.83 mAP, and 86% F1 score for the latter. These performance indicators outperform competing models such as ResNet and DenseNet. Ablation experiments confirm that cross-modal fusion and generative data augmentation modules are essential; removing either module results in a 5%-9% drop in accuracy. The current model still has room for improvement in terms of training time and generated image quality. Future work will focus on optimizing performance through lightweight design and multi-scale fusion, enhancing the ability to distinguish similar artifacts and providing technical support for digital artifact management and cultural heritage preservation.
Cognitive impairment is a common health issue among the elderly, posing significant challenges to society and healthcare resources. Early recognition is a critical strategy for preventing or delaying the onset of cognitive impairment, particularly in cognitive rehabilitation. However, achieving convenient and rapid assessment of cognitive impairment remains a significant challenge. This study proposes a multi-query cross-modal attention fusion model based on synchronized multimodal signals for cognitive impairment recognition. First, we constructed a standardized cognitive impairment assessment dataset (EEV-CI) containing synchronized electroencephalography, electrocardiography, and video signals. Then, a frequency-band adaptive physiological signal encoder was designed to address the pseudo-periodic characteristics. Facial Action Units and emotional states were extracted for facial expression analysis. Finally, the multi-query cross-modal attention mechanism was employed to synchronize and fuse multimodal signals along the temporal dimension. Experimental results demonstrate that the proposed model achieved superior performance, with 87.01% accuracy, 78.17% F1-macro score, 86.88% AUC and 57.42% MCC. Furthermore, modality-specific experiments demonstrate the unique contribution of each modality to cognitive impairment recognition. These findings further validate the effectiveness of multimodal fusion in cognitive impairment assessment and offer valuable clinical support for rehabilitation.
Electrocardiogram (ECG) and phonocardiogram (PCG) have emerged as crucial non-invasive and portable diagnostic modalities for early cardiovascular disease (CVD) screening. Despite the individual merits of these signal modalities in CVD detection, significant challenges persist, including insufficient inter-modal interaction and suboptimal weight allocation. To address these critical limitations, we proposed a novel Time-Frequency Cross-Modal Attention Fusion Network (TF-CrossNet) designed for precise early CVD diagnosis. The proposed network employs a dual-path multiscale residual structure to extract key time-frequency domain features from PCG and ECG signals, comprehensively capturing multiscale information. Leveraging the intrinsic electro-mechanical coupling relationship of the heart, a bidirectional mutual enhancement attention module is introduced to capture interactive morphological information between PCG and ECG signals, enabling feature-level signal complementation and enhancement. Furthermore, an adaptive fusion strategy based on Bayesian decision theory is developed, establishing a mapping relationship between confidence levels and loss functions to dynamically optimize modal weight allocation. Validated on the 2016 PhysioNet/CinC dataset, the model achieved exceptional performance metrics: 93.13% accuracy, 97.7% specificity, and 98% area under the curve (AUC). Furthermore, comprehensive noise robustness experiments demonstrate that TF-CrossNet maintains superior performance under various noise conditions, achieving an average robustness index of 94.20% compared to existing methods, validating its practical applicability in clinical environments. The superior effectiveness of the proposed approach in CVD classification, providing a novel technological pathway for non-invasive and precision CVD diagnosis.
Artificial emotional intelligence is a sub-domain of human–computer interaction research that aims to develop deep learning models capable of detecting and interpreting human emotional states through various modalities. A major challenge in this domain is identifying meaningful correlations between heterogeneous modalities—for example, between audio and visual data—due to their distinct temporal and spatial properties. Traditional fusion techniques used in multimodal learning to combine data from different sources often fail to adequately capture meaningful and less computational cross-modal interactions, and struggle to adapt to varying modality reliability. Following a review of the relevant literature, this study adopts an experimental research method to develop and evaluate a mathematical cross-modal fusion model, thereby addressing a gap in the extant research literature. The framework uses the Tucker tensor decomposition to analyse the multi-dimensional array of data into a set of matrices to support the integration of temporal features from audio and spatiotemporal features from visual modalities. A cross-attention mechanism is incorporated to enhance cross-modal interaction, enabling each modality to attend to the relevant information from the other. The efficacy of the model is rigorously evaluated on three publicly available datasets and the results conclusively demonstrate that the proposed fusion technique outperforms conventional fusion methods and several more recent approaches. The findings break new ground in this field of study and will be of interest to researchers and developers in artificial emotional intelligence.
No abstract available
No abstract available
The proliferation of hateful memes on social media platforms presents significant challenges for content moderation, as these multimodal artifacts leverage the interplay between visual and textual elements to convey implicit offensive meanings. Existing detection approaches suffer from either inadequate cross-modal understanding or prohibitive computational costs unsuitable for real-world deployment. This paper proposes CMAFN-Lite (Cross-Modal Attention Fusion Network-Lite), a lightweight yet effective framework that achieves competitive hate speech detection performance while reducing model complexity by approximately 70%. Our approach employs bidirectional cross-modal attention mechanisms to capture semantic dependencies between image and text modalities, utilizing DistilBERT and MobileNetV3-Small as efficient feature extractors. Extensive experiments on the Hateful Memes dataset demonstrate that CMAFN-Lite achieves an F1-score of 0.599 and accuracy of 73.2% while maintaining only 28.6M parameters and accelerating training speed by 3.5× compared to heavyweight baselines. The proposed model achieves a favorable trade-off between performance and efficiency, making it suitable for deployment in resource-constrained environments.
The rapid growth of diverse information formats, such as text and images on social media, has made multimodal rumor detection a crucial area of research. Existing models often struggle to capture the complex interactions between multimodal features, as they typically focus on within-modality connections. This limitation constrains their ability to leverage complementary strengths across modalities, reducing their effectiveness in comprehensive rumor detection. To address this challenge, we propose a Hierarchical Cross-Modal Attention Fusion Network (HiCAN), designed to enhance rumor detection through multilevel integration of textual and visual features. HiCAN includes an image description generation module that employs BLIP to generate descriptive text from images, enriching the model’s textual context. It further introduces a local feature enhancement module to identify key regions in images, mitigating the impact of irrelevant content. Finally, a hierarchical cross-modal attention fusion module captures and synthesizes correlations between multimodal features. Experimental results on two benchmark datasets demonstrate that HiCAN achieves superior performance compared to existing state-of-the-art methods, effectively integrating text and visual information to advance multimodal rumor detection.
No abstract available
The emergence of AI-generated information poses a big threat to the validity of information. Synthetic media has become more realistic than ever with the development of the large language models and diffusion-inspired generators, such as GPT-4 and DALL•E. The social media ecosystems also play a significant role in this dilemma because false information becomes viral within networks. Current detection methods tend to concentrate on either unimodal structure or ensemble systems and do not encode the semantic connections between text/visual information. Our research paper is a proposal of a new framework, CIVA-Net (Caption-Image Visual Attention Network), which uses cross-modal attention to learn joint representations on caption-image pairs in order to classify AI-generated content (AIGC). We are using RoBERTa and the text encoder and Vision Transformer (ViT-B/16) as the image encoder. Our central gift is that of a bidirectional cross-attention mechanism that facilitates the information flow between modalities: image features attend to textual tokens, and the text representations attend to visual patches. The architecture is able to capture semantic-visual anomalies that describe misinformation generated by AI. We tested CIVA-Net on the dataset of MiRAGeNews, which consists of 15,000 pairs of captions and images of various generators and news outlets. Our model has 89% accuracy and 96% recision, which is a major improvement over the unimodal baselines as well as the competing multimodal models. The representations learned are clearly separated in the feature space, which confirms the usefulness of cross-modal fusion. Even though domain generalization issues, the detection of complex synthetic material, and related concerns have still not been completely addressed, our findings demonstrate that the hybrid textual-visual modeling of attention-based fusion is an effective tool for AI-generated content detection.
Real-time intent recognition can enhance the timeliness of support in AI-assisted programming environments. Code snapshots alone are often insufficient because they capture editing outcomes rather than the interaction process. This paper presents NeuroMouse-CodeNet, a lightweight dual-stream framework that integrates mouse micro-interaction dynamics with local code semantics for fine-grained programming intent recognition. The model combines stacked 1D-CNN layers for cursor trajectories, a partially fine-tuned CodeBERT with AST cues for code representation, and a cross-modal attention module for multimodal alignment. Experiments on a synchronized dataset of about 180,000 multimodal segments from 50 participants across 12 programming tasks, evaluated under a strict participant-disjoint protocol, show that the proposed method achieves 91.5% accuracy and 91.3% Macro-F1, with an average inference latency of 16.2 ms. The results support its use in low-latency intent-aware programming assistance.
No abstract available
A Multi-Modal Attention Fusion Framework for Road Connectivity Enhancement in Remote Sensing Imagery
Ensuring the structural continuity and completeness of road networks in high-resolution remote sensing imagery remains a major challenge for current deep learning methods, especially under conditions of occlusion caused by vegetation, buildings, or shadows. To address this, we propose a novel post-processing enhancement framework that improves the connectivity and accuracy of initial road extraction results produced by any segmentation model. The method employs a dual-stream encoder architecture, which jointly processes RGB images and preliminary road masks to obtain complementary spatial and semantic information. A core component is the MAF (Multi-Modal Attention Fusion) module, designed to capture fine-grained, long-range, and cross-scale dependencies between image and mask features. This fusion leads to the restoration of fragmented road segments, the suppression of noise, and overall improvement in road completeness. Experiments on benchmark datasets (DeepGlobe and Massachusetts) demonstrate substantial gains in precision, recall, F1-score, and mIoU, confirming the framework’s effectiveness and generalization ability in real-world scenarios.
Color–thermal (RGB-T) urban scene parsing has recently attracted widespread interest. However, most existing approaches to RGB-T urban scene parsing do not deeply explore the information complementarity between RGB-T features. In this study, we propose a cross-modal attention-cascaded fusion network (CACFNet) that fully exploits cross-modality. In our design, a cross-modal attention fusion module mines complementary information from two modalities. Subsequently, a cascaded fusion module decodes the multi-level features in an up-bottom manner. Noting that each pixel is labeled with the category of the region to which it belongs, we present a region-based module that explores the relationship between pixel and region. Moreover, in contrast to previous methods that employ only the cross-entropy loss to penalize pixel-wise predictions, we propose an additional loss to learn pixel–pixel relationships. Extensive experiments on two datasets demonstrate that the proposed CACFNet achieves state-of-the-art performance in RGB-T urban scene parsing.
Multimodal Sentiment Analysis (MSA) is the technology of intelligently recognizing and assessing human sentiments using various data forms such as text, image, and audio. Despite current mainstream methods have made significant progress, MSA still faces the following issues: 1) most current methods train models based on pre-extracted features, lacking a sufficient understanding of sentiment diversity in multimodal data and may even lead to the loss of critical information in the raw data; and 2) textual modality, which possesses high-level semantic features, should typically dominate the fusion process, yet current methods fail to fully leverage this characteristic to balance modality information. To address the aforementioned issues, we propose a novel Multimodal Sentiment Analysis framework using Multimodal-Prefixed and Cross-Modal Attention (DB-MPCA). For the first issue, DB-MPCA employs multimodal raw data for pre-training, which not only allows for in-depth exploration of multimodal information but also significantly enhances the model’s learning capabilities and generalization, while reducing the substantial costs associated with manual annotation. Regarding the second issue, DB-MPCA introduces two prefix encoders designed to convert acoustic and visual features into prefix tokens. These tokens are then embedded into a pre-trained language model, where they are encoded together with textual tokens. Through this approach, DB-MPCA effectively learns cross-modal attention while maintaining the dominance of the textual modality, thereby optimizing the fusion of modalities. Comprehensive experiments conducted on the widely utilized dataset (CMU-MOSI) demonstrate the effectiveness of our model, highlighting its superiority over baseline models.
Anomaly detection in system software traditionally relies on single-modal algorithms that analyze either discrete log events or continuous performance metrics in isolation, potentially missing complex anomalies that manifest across both modalities. We present a novel deep learning framework that leverages cross-modal attention mechanisms to jointly model log sequences and performance metrics for enhanced anomaly detection. Our method proposes Long Short-Term Memory (LSTM) networks to capture temporal dependencies in log event sequences and Temporal Convolutional Networks (TCNs) to model performance metric time series. The core innovation lies in our Cross-Modal Attention Mechanism, which dynamically weighs log events and metric features based on inter-modal relationships, enabling the detection of subtle anomalies that require contextual information from both data sources. Unlike conventional multi-modal fusion techniques that merely concatenate features, our attention mechanism explicitly models the dependencies between log patterns and metric behaviors, allowing the network to focus on relevant log events during metric anomalies and vice versa. We conduct comprehensive experiments on public datasets including HDFS and BGL logs paired with cloud computing performance metrics, as well as real-world cloud environments. Our method achieves significant improvements over single-modal baselines, with F1-scores increasing by 12.3% on average across datasets. Ablation studies confirm the effectiveness of the cross-modal attention mechanism, while real-time deployment experiments using Apache Flink demonstrate practical applicability with sub-second latency. The proposed framework addresses a critical gap in system software monitoring by providing a principled approach to multi-modal anomaly detection that scales to enterprise-level deployments.
Accurate identification of mental stress states is fundamental to cognitive human–machine systems. Multimodal physiological signals offer a rich basis for characterizing stress-related responses, yet existing methods insufficiently model cross-modal interactions, limiting both recognition performance and physiological interpretability. To address this issue, we propose a multimodal representation learning framework that captures the coordination between central and autonomic nervous system activities under stress. An EEG–ECG response analysis module is first constructed to reveal the dynamic coupling between brain activity and autonomic regulation, providing a biologically grounded basis for stress assessment. Building on this basis, a cross-modality representation learning network with a dual-branch architecture extracts shared and modality-specific features, and a multi-task learning strategy jointly optimizes classification and feature disentanglement to improve generalization across stress levels. Experiments on a multimodal mental stress dataset demonstrate that the proposed framework achieves 87.69% classification accuracy and enhanced physiological interpretability, confirming its effectiveness and robustness in modeling heart–brain interaction for mental stress recognition.
Multimodal Sentiment Analysis (MSA) has gained wide attention in many fields in recent years. However, the problem of heterogeneity and redundant information among different signals seriously affects the extraction and fusion of sentiment features. To address this challenge, we propose a Multilevel Representational Disentanglement Framework (MRDF) to achieve effective modality fusion and produce refined joint multimodal representations. Specifically, we design a refined semantic decomposition module for learning task-shared representations and modality-exclusive representations by crossmodal translations and task semantic reconstruction. Furthermore, we propose a contrastive learning-based distribution alignment mechanism and an adversarial learning-based distribution alignment strategy to utilize contrastive adversarial learning paradigms to further align the disentangled task-shared representations Experimental results show that the MRDF framework significantly outperforms existing state-of-the-art methods on the MOSI and MOSEI benchmarks.
Deformable registration of multimodal medical images constitutes a fundamental task in many medical image analysis applications, particularly in the diagnosis and treatment of liver cancer where different modality images are frequently employed. Despite the emergence of various learning-based registration models, multimodal registration remains a challenging task due to pronounced intensity differences and significant tissue deformations. To address these challenges, we approach the problem from two perspectives-mitigating modality discrepancies and enhancing the model's capacity to handle large deformations. We propose a multimodal deformable registration model based on adaptive recursive inference and representation disentanglement (ARDMR). Specifically, to enhance the model's ability to learn features from different modalities, we introduce a modality representation disentanglement method, incorporating Multi-layer Contrastive Loss (MCL) to enforce the learning of modality-invariant features. To address the challenge of large complex deformations, we propose a Multi-Scale Feature Registration module (MSFR), which integrates features and deformation fields from different scales during the registration process. Finally, to further exploit the registration potential of the trained model, we propose an adaptive recursive inference strategy. This strategy automatically determines the optimal recursive registration scale and the number of iterations based on the image's real-time registration performance. We conducted experiments on multimodal datasets collected from multiple medical centers and compared our method with several state-of-the-art registration models. The results indicate that our proposed ARDMR model outperforms others in both qualitative and quantitative evaluations. Specifically, compared with the baseline VoxelMorph model, ARDMR improves the DSC metric by 2.5%-5% across different datasets. Furthermore, tests on distribution-shifted data demonstrate that our model exhibits the best robustness and generalization. Code is available at https://github.com/ybby2020/ARDMR_MIA.
Multimodal recommender systems (MRSs) exploit diverse content sources (e.g., text, images) to enhance recommendation accuracy. However, they still face two fundamental challenges: (1) e ff ectively capturing users’ diverse interests, and (2) filtering out noisy signals from heterogeneous modalities. To address these issues, we propose DMIMRec, a framework for Disentan-gled Multi-Interest Modeling in multimodal recommendation. On the item side, we construct modality-specific item–item graphs and introduce a reconstruction-di ff erence guided pruning strategy that evaluates each edge’s usefulness by checking how much the reconstruction quality changes when that edge is removed, thereby discarding connections that contribute little or introduce noise. On the user side, we perform clustering over interacted items to initialize multi-interest prototypes, then apply a triple disentanglement module that separates user representations into interest-invariant, e ff ective interest-specific, and ine ff ective interest-specific components, ensuring clearer semantic boundaries and avoiding representation collapse. Experiments on three bench-mark datasets demonstrate that our methods achieve better performance on different metrics.
Adaptive radiation therapy (ART) aims to deliver radiotherapy accurately and precisely in the presence of anatomical changes, in which the synthesis of computed tomography (CT) from cone-beam CT (CBCT) is an important step. However, because of serious motion artifacts, CBCT-to-CT synthesis remains a challenging task for breast-cancer ART. Existing synthesis methods usually ignore motion artifacts, thereby limiting their performance on chest CBCT images. In this paper, we decompose CBCT-to-CT synthesis into artifact reduction and intensity correction, and we introduce breath-hold CBCT images to guide them. To achieve superior synthesis performance, we propose a multimodal unsupervised representation disentanglement (MURD) learning framework that disentangles the content, style, and artifact representations from CBCT and CT images in the latent space. MURD can synthesize different forms of images using the recombination of disentangled representations. Also, we propose a multipath consistency loss to improve structural consistency in synthesis and a multidomain generator to improve synthesis performance. Experiments on our breast-cancer dataset show that MURD achieves impressive performance with a mean absolute error of 55.23±9.94 HU, a structural similarity index measurement of 0.721±0.042, and a peak signal-to-noise ratio of 28.26±1.93 dB in synthetic CT. The results show that compared to state-of-the-art unsupervised synthesis methods, our method produces better synthetic CT images in terms of both accuracy and visual quality.
Cross-domain recommendation has received considerable attention as a method for improving recommendation accuracy in target domains with sparse data by leveraging knowledge from source domains. Recent research has focused on separating user preferences into domain-common and domain-specific representations. However, many existing methods rely solely on user-item interaction information, which limits their ability to effectively separate shared and unique components. Furthermore, approaches that separate representations within a domain and then adjust their relationships afterward may introduce information distortion and fail to capture users’ essential common preferences. Moreover, representation separation methods using contrastive learning rely on relative distance optimization, which cannot directly suppress semantic redundancy and may lead to inappropriate knowledge transfer. In this paper, we propose a new cross-domain recommendation approach that integrates multimodal features and introduces a cosine similarity-based regularization term to suppress the correlation between domain-common and domain-specific representations. This ensures that each representation maintains semantically distinct information. Experimental results on the Amazon dataset demonstrate that the proposed method outperforms state-of-the-art cross-domain recommendation techniques.
B-mode ultrasound (BUS) is widely used in breast cancer diagnosis, while the emerging super-resolution ultrasound (SRUS) provides microvascular information with high spatial resolution, which has shown great potential in improving breast cancer diagnosis. However, as a new ultrasound modality, its diagnosis remains highly dependent on the clinical experience of sonologists, highlighting the need for reliable computer-aided diagnosis (CAD) approaches. In this work, a novel dual-branch network with a Channel-Wise Joint Disentanglement Representation Learning (CW-JDRL) method is proposed for the multimodal ultrasound-based CAD of breast cancer, where one branch processes BUS and the other analyzes multimodal SRUS data. The CW-JDRL is implemented on the SRUS branch by grouping the final-layer network channels to capture both common and specific properties. It consists of two modules, namely Gradient-guided Disentanglement (GD) module and Gramian-based Contrastive Learning Disentanglement (GCLD) module. The former disentangles with gradient guidance to encourage consistency among common channels and distinctiveness among specific ones, and the latter disentangles common and specific representations by integrating them into a unified contrastive objective. Extensive experiments on a multicenter SRUS dataset demonstrate that the proposed dual-branch network with CW-JDRL achieves superior performance over the compared algorithms and maintains robust generalizability to external data. It suggests not only the effectiveness of SRUS for diagnosis of breast cancer, but also the potential of the proposed CAD model in clinical practice. The codes are publicly available at https://github.com/Zyh-AIUltra/CW-JDRL.
Early scoliosis screening enables timely intervention and may prevent disease progression. However, gait-based screening remains challenging: abnormalities are often subtle, labeled data are limited, and multimodal analysis must handle cross-modal discrepancy and missing modalities. To address these challenges, we propose Uncertainty-Aware Multimodal Gait Representation Learning (UMGRL), a cross-modal gait representation learning framework that transfers knowledge from large-scale gait data to scoliosis screening. To learn transferable representations robust to missing modalities, we develop a teacher-student paradigm. A shared latent projector and disentanglement module decompose multimodal features into modality-invariant geometry and modality-specific attributes, enabling cross-modal alignment while preserving complementary information. Modality dropout and representation prediction further allow the student to recover task-ready representations of missing modalities from available inputs. For downstream screening, we employ Dynamic Time Warping (DTW) to partition videos into phase-consistent bags for temporal alignment, followed by Perceiver IO-based fusion to capture long-range dependencies across modalities and gait cycles. To enhance sensitivity to borderline cases near the 10° Cobb threshold, we introduce a borderline-focused branch. Finally, we incorporate Evidential Deep Learning (EDL) for uncertainty-aware prediction and reliable triage. Experiments on Scoliosis1K demonstrate superior performance, particularly under incomplete-modality settings and on clinically ambiguous cases. These results suggest that UMGRL provides a promising path toward scalable, radiation-free scoliosis screening.
Multimodal representation learning faces significant challenges due to the heterogeneous nature of different modalities, which leads to feature distribution gaps and hinders effective fusion. To address the feature alignment problem in multimodal sentiment analysis, this paper proposes a Disentanglement MultiTask Learning framework in Hyperbolic Space (DMLIH). The model disentangles unimodal representations through an encoder-decoder structure and aligns them in hyperbolic space to capture cross-modal similarity and differences. A multitask learning strategy is introduced to jointly model modality consistency and modality-specific characteristics, while a weight adjustment mechanism balances the learning progress among subtasks. Experiments conducted on the MOSI and MOSEI datasets demonstrate that the proposed method achieves F1 scores of 86.4% and 87.1%, and accuracies of 86.3% and 86.4%, respectively, outperforming several existing approaches. The results verify the effectiveness of the proposed framework for multimodal sentiment analysis.
Multimodal recommendation, which utilizes rich multimodal information to learn user preferences, has attracted significant attention. Most works focus on designing powerful encoders for extracting multimodal features, and simply aggregate the learned features together to make prediction. Consequently, they have a limited capacity to learn the inter-modality knowledge including the modality-shared and modality-unique knowledge. In fact, learning the modality-shared knowledge enables us to align cross-modality data for fusing heterogeneous modality features. Learning the modality-unique knowledge is equally important when recommendation tasks only involve a small amount of shared features and the necessary information is contained within specific modality. In this article, we propose Contrastive Modality-Disentangled Learning (CMDL) to overcome this critical limitation. CMDL exactly captures the inter-modality knowledge by achieving modality disentanglement. Specifically, CMDL first disentangles the initial representation into the modality-invariant and modality-specific representations. Afterwards, CMDL introduces a novel manner of contrastive learning to approximate the MI upper bounds for achieving disentanglement regularization. Building upon the proposed regularization, CMDL encourages the modality-invariant and modality-specific representations to capture the modality-shared and modality-unique knowledge respectively and to be statistically independent to each other. Empirically, extensive experiments are conducted on benchmark datasets, demonstrating the superior performance of CMDL compared with strong multimodal recommenders.
3D reconstruction is a pivotal technology that recreates three-dimensional structures from two-dimensional representations, facilitating AI's understanding and interaction with the real world. However, existing methods pose challenges from two perspectives, i.e., a lack of understanding of the semantics behind 3D representations and the excessive reliance on extra priors to achieve 3D control. To address these challenges, we propose an interpretable 3D reconstruction framework, dubbed NaviNeRF++, to discover and identify the underlying 3D semantics by integrating multimodal large language models (MLLMs) and Neural Radiance Fields (NeRF). The model achieves fine-grained 3D disentanglement from the perspective of disentangled representation learning (DRL), while preserving high-quality and view-consistent 3D reconstruction. Specifically, the framework consists of three key components: i) a lightweight 2D perception module designed to derive an orthogonal and well-disentangled latent space with knowledge distilled from a pre-trained DRL model; ii) a NeRF-based 3D navigation module dedicated to finding semantic factors in the learned latent space, while concurrently enabling high-quality 3D reconstruction; and iii) an attribute identification module that identifies textual concepts of semantic factors by leveraging the commonsense knowledge of MLLMs. To our knowledge, this work is the first to achieve interpretable 3D reconstruction and fine-grained 3D disentanglement in an unsupervised manner. Empirical results further demonstrate its superior performance compared to off-the-shelf solutions.
Multimodal sentiment analysis (MSA) aims to comprehend human sentiment by integrating heterogeneous multisource information on social media. The challenge of MSA lies in representation learning and multimodal fusion. Existing disentangled methods rely on global information while neglecting fine-grained critical cues, amplifying representation distribution bias in disentangled spaces. Meanwhile, current fusion approaches heavily depend on text modality while disregarding informative cues from other modalities, exacerbating the contribution imbalance across modalities. To tackle these limitations, we propose a multi-task framework that collaborates with fine-grained disentanglement and modal-agnostic fusion. It extracts sentiment attributes at a fine granularity based on the multi-head property of the Transformer, and dynamically exploits the contribution of each modality while enhancing the reliability of cross-modal representations through fuzzy logic. We evaluate FDMA on two renowned multimodal benchmark datasets, CMU-MOSI and CMU-MOSEI. Experiments show that our method surpasses state-of-the-art methods on CMU-MOSI and CMU-MOSEI datasets. Further analysis demonstrates that our model also exhibits superior reliability and stability.
Immersive digital human interaction requires robust multimodal representation learning across visual, acoustic, motion, and linguistic streams. However, existing methods often suffer from modality heterogeneity, temporal inconsistency, and noise-sensitive feature fusion. To address these issues, this paper proposes Temporal-Contextual Semantic-Disentangled Interaction Alignment (TC-SDIA), a cross-modal framework that decomposes multimodal features into shared semantic and modality-private spaces, models temporal interaction dependencies, and performs reliability-aware adaptive fusion. Experimental results show that TC-SDIA consistently outperforms LSTM-AE and Isolation Forest under scenario shifts, parameter variations, and increasing multimodal disturbance, achieving stronger F1-scores and more stable interaction state prediction. These results indicate that semantic disentanglement and temporal-contextual alignment are effective for preserving robust digital human interaction understanding. TC-SDIA provides a practical representation learning framework for immersive human–AI interaction systems, while future work will further explore larger-scale datasets, real-time deployment, and more fine-grained interaction behavior generation.
Cross-domain recommender systems (CDRS) have become an important research direction for addressing sparsity, cold-start, and fragmented user-behavior problems that cannot be solved effectively within a single domain. In recent years, the field has expanded rapidly from early neural transfer models into a broad landscape that includes graph-based methods, sequential recommendation, adversarial alignment, contrastive learning, disentanglement, multimodal transfer, federated recommendation, and emerging LLM-assisted approaches. This paper presents a comparative review of modern algorithmic strategies in CDRS published between 2018 and 2025. [21, 23, 32, 54] Unlike reviews that summarize the literature only chronologically, this study organizes prior work by algorithm family and compares how different methods perform cross-domain transfer, what assumptions they rely on, what types of datasets and user signals they use, and where their major strengths and limitations lie. The review integrates representative journal and conference studies from major venues and presents a taxonomy of modern CDRS methods, cross-family comparison tables, a method-evolution timeline, a dataset-and-evaluation pipeline view, and a research-gap map. [57, 59] The comparative analysis shows that graph and knowledge-graph methods are particularly strong in structurally rich and sparse settings, sequential methods are especially effective for temporally evolving behavior, contrastive and disentanglement-based approaches improve robustness and negative-transfer control, multimodal methods expand transfer under weak overlap through richer semantic information, and federated methods address growing privacy and deployment requirements. The review also finds that no single family is universally optimal, and that future progress is likely to depend on hybrid, condition-aware models that combine selective transfer, semantic enrichment, robustness, and distributed learning. [21, 23, 32, 54] The paper concludes by identifying major open challenges, including weak overlap, domain heterogeneity, benchmark inconsistency, scalability, explainability, and privacy-aware deployment. It argues that the next stage of CDRS research will be shaped by more reliable selective-transfer mechanisms, multimodal and foundation-model integration, continual adaptation, and stronger evaluation standards for real-world recommendation environments. [35, 53, 55, 56]
No abstract available
Multimodal affective analysis benefits from combining textual, acoustic, and visual cues, yet many methods implicitly mix modality-invariant information with modality-specific factors, which can reduce robustness when modalities vary in reliability. We propose DiMoE, a representation-first framework that integrates feature disentanglement with sparse Mixture of Experts (MoE) fusion. DiMoE first decomposes each modality into shared (modality-invariant) and private (modality-specific) representations. It then fuses these factors using top-k sparse MoE routing, enabling input-adaptive expert selection. We study two routing strategies with a shared expert pool, a joint router that operates on combined shared and private factors, and separate routers that process shared and private factors independently. We evaluate DiMoE on five widely used benchmarks spanning sentiment intensity prediction and emotion classification, covering both trimodal and bimodal settings. Across benchmarks, DiMoE consistently achieves strong performance against recent competitive baselines, indicating that explicit modality disentanglement combined with Mixture of Experts fusion yields effective multimodal affective representations.
The combination of electroencephalography (EEG) signals and eye movements offers robust emotion recognition, but the inconvenience of acquiring EEG signals significantly limits its practical applications. Recent studies have focused on leveraging complementary multimodal information during training to enhance single-modality (eye movement) performance at inference. However, existing methods typically adopt a single-stage training paradigm that jointly optimizes multimodal representation learning and feature reconstruction, resulting in ambiguous reconstruction objectives and entangled feature representations. To address these issues, we propose a three-stage progressive training paradigm involving pre-training, reconstruction compensation, and fine-tuning, which effectively balances feature disentanglement and reconstruction. Specifically, the pre-training stage firstly employs a cross-attention-based multimodal fusion strategy to obtain a robust multimodal model. Subsequently, the reconstruction compensation stage utilizes single eye movement modality and Variational Autoencoder (VAE) to reconstruct the multimodal representations. Finally, the fine-tuning stage applies knowledge distillation and the reconstructed multimodal features to refine the eye movement-based model. Experiments conducted on SEED-IV and SEED-V demonstrate the effectiveness and superiority of our proposed three-stage training paradigm over relevant state-of-the-art methods in cross-subject emotion recognition.
Existing single-modality hyperspectral image compression methods often lack explicit structural constraints in the latent space. Most methods rely on pixel-level rate–distortion objectives. Spatial–spectral consistency with physical meaning becomes difficult to preserve under high compression ratios. Multimodal representation learning often depends on empirical modeling. Theoretical guidance remains limited. To address these issues, an information-theoretic constrained framework for multimodal hyperspectral image compression is proposed. Auxiliary modality information provides structural guidance for the latent space. Latent representations are explicitly disentangled into modality-specific and shared components. Mutual information-based regularization is introduced to suppress statistical dependence. The regularization is integrated into the rate–distortion optimization framework. In addition, a best neighbor matching mechanism is introduced. Cross-modal spatial features are aligned. Structural consistency in the latent space is further enhanced. Experiments on mixed multimodal hyperspectral datasets demonstrate superior performance over existing single-modality compression methods. Improvements are observed in both rate–distortion performance and spectral consistency.
Multimodal spatiotemporal data (MST) consists of multiple simultaneous spatiotemporal modalities that interact with each other in a dynamic manner. Due to the complexity of MST and the recent desire for the explainability of artificial intelligent systems, disentangled representation learning for MST (DisentMST) has become a significant task, which aims to learn disentangled representations that can expose the underlying spatial semantics, temporal dynamic patterns, and inter-modality interaction modes of the complex MST. One limitation of existing approaches is that they might fail to tolerate the real-world incomplete MST data, where missing information might break the cross-modal spatiotemporal dynamics and bring noise and ambiguity to the learning process. Another limitation is that no existing work systematically reveals the structure of different types of disentangled information. To tackle the two limitations, we define a novel two-level hierarchically structured disentanglement task for MST, which reveals informative and structured disentangled representations for MST as well as digests the real-world MST with incompleteness. We propose a new framework, BiDisentMST, which leverages Gaussian Processes and Graph Factorization on the latent space to achieve our purposes. The experimental results demonstrate the effectiveness of our proposed framework compared with baselines with respect to disentanglement and imputation results.
No abstract available
Neural networks continue to face challenges in effectively extracting information from the high-dimensional and sparse range-azimuth-Doppler (RAD) tensor. While multi-view architectures using three-axis projections of the RAD tensor improve efficiency and maintain accuracy, existing designs suffer from inefficient inter-view information interaction. To address this issue, this letter proposes a new multi-view information fusion viewpoint: instead of directly blending feature maps, the orthogonal relationships among the RAD coordinate axes are leveraged to guide feature decomposition across views, thereby disentangling the mixed semantics introduced during view compression. Based on this idea, a new module, termed Cross-View Guided Disentangler (CVGD), is introduced to enable inter-view information interaction in multi-view architectures. Extensive experiments on the public CARRADA dataset demonstrate that networks incorporating the proposed module achieve performance comparable to state-of-the-art methods, while utilizing 50% fewer parameters and attaining a 6-fold increase in inference speed. Additionally, promising results are also observed on RADIal, achieving near-SOTA performance with higher efficiency.
No abstract available
Multi-view clustering (MVC) strives to reveal the hidden correlations and potential distribution of data from multiple views. However, most existing methods separate feature extraction and clustering processes, relying heavily on pre-learning and post-processing, leading to information loss and suboptimal clustering results. Therefore, we propose a novel embedded multi-view clustering model EMVCTM via collaborative tensor subspace representation and adaptive multi-graph fusion. The tensor self-representation framework is designed to capture global structural information. Symmetry constraints are relaxed synchronously to adaptively learn view-specific affinity graphs and dynamically generate fusion graph. The near-global optimal clustering result is derived through an efficient embedded clustering framework with activated view interactions, providing a more interpretable explanation for cluster division. Experimental results on various real-world datasets confirm that EMVCTM outperforms existing state-of-the-art techniques.
No abstract available
Incomplete multi-view clustering based on kernel graph fusion tensor under self-expression framework
No abstract available
No abstract available
No abstract available
Abstract. Extracting effective scene features is important for remote sensing image classification. Generally, the multi-view features contain information of consistency and complementarity, and efficient integration of them is helpful to enhance the performance of remote sensing image classification. Although some recent methods are able to achieve promising results, they lack analysis of the inherent relevance of multiple-view features. Thus, we present a multi-view fusion optimization method via low-rank tensor decomposition. First, the Laplacian matrix is constructed by utilizing K-nearest neighbors to generate a set of low-dimensional eigenvalues. Second, a third-order tensor is built by combining the multiple-view Laplacian features, which are factorized into many components with rank 1 using the canonical polyadic decomposition. Finally, the alternating optimization model is reconstructed by utilizing the relationship among fibers and slices of a tensor to generate optimal low-dimensional embedding features. Experiments of classification on three remote sensing image data sets AID, WHU, and UCMerced are constructed. The results of the experiments show that the new proposed method achieves better performance than others.
In recent years, recommendation systems based on graph neural networks GNN have achieved remarkable success. Despite their effectiveness, GNN-based methods are often affected by noisy interac tions in user-item data. Consequently, several approaches have adopted graph contrastive learning GCL to address this challenge. However, most existing GCL approaches construct contrastive views from the user-item graph, without explicitly leveraging high-order relational in formation i.e., user-user and item-item relationships . Moreover, they often adopt a uniform perspective on user-item connections, neglecting the diversity of user interests.To address these limitations, we present a graph contrastive recommendation model that incorporates an adaptive multi-view fusion strategy,named AdaFCL. Specifically,to more explic itly exploit high-order information, we design an adaptive fusion mod ule that fuses edge weights derived from both the user-item interaction graph and the high-order collaborative graph i.e., user-user and item item graph .Then this fusion module introduces a learnable generator based on GCN and GAT to generate low-noise contrastive views, as an alternative to traditional random perturbations. Furthermore, we de sign a interest aggregation module to embed users’ personalized prefer ences into the representation learning process.Extensive experiments on three public benchmark datasets demonstrate the superiority of AdaFCL. Compared to the strongest baselines , our model improves performance by up to 9.27 for NDCG@20 and 8.67 for Recall@20.
No abstract available
Abstract Motivation Accurately identifying the drug–target interactions (DTIs) is one of the crucial steps in the drug discovery and drug repositioning process. Currently, many computational-based models have already been proposed for DTI prediction and achieved some significant improvement. However, these approaches pay little attention to fuse the multi-view similarity networks related to drugs and targets in an appropriate way. Besides, how to fully incorporate the known interaction relationships to accurately represent drugs and targets is not well investigated. Therefore, there is still a need to improve the accuracy of DTI prediction models. Results In this study, we propose a novel approach that employs Multi-view similarity network fusion strategy and deep Interactive attention mechanism to predict Drug–Target Interactions (MIDTI). First, MIDTI constructs multi-view similarity networks of drugs and targets with their diverse information and integrates these similarity networks effectively in an unsupervised manner. Then, MIDTI obtains the embeddings of drugs and targets from multi-type networks simultaneously. After that, MIDTI adopts the deep interactive attention mechanism to further learn their discriminative embeddings comprehensively with the known DTI relationships. Finally, we feed the learned representations of drugs and targets to the multilayer perceptron model and predict the underlying interactions. Extensive results indicate that MIDTI significantly outperforms other baseline methods on the DTI prediction task. The results of the ablation experiments also confirm the effectiveness of the attention mechanism in the multi-view similarity network fusion strategy and the deep interactive attention mechanism. Availability and implementation https://github.com/XuLew/MIDTI.
Drug-Drug Interaction Prediction (DDI) is critical for ensuring safe medication. However, existing methods often rely on local structures or limited single-view molecular information, making them insufficient to capture long-range dependencies and multi-scale chemical semantics. To address these limitations, we propose MVCADF - a relation-aware Graph Transformer framework with multi-view consistency alignment and dynamic fusion for DDIs. Beyond leveraging molecular and biomedical knowledge graphs, MVCADF integrates motif fragment graphs, element-level view, and ECFP fingerprints, thus enabling comprehensive representations at the atom, fragment, element, fingerprint, and semantic levels. We design a multi-view consistency alignment strategy to project different views into a shared anchor space, which mitigates inter-view distributional shift and enhances representational coherence. This strategy is followed by a dynamic fusion module that adaptively integrates multi-view features based on their discriminative significance. Additionally, the relation- and distance-aware attention mechanism further facilitates modeling of long-range interactions within heterogeneous graphs. Extensive experiments show that MVCADF outperforms the best baseline by 3% - 5% across multiple public datasets, offering a robust and holistic solution for DDIs prediction.
Graph collaborative filtering has achieved great success in capturing users' preferences over items. Despite effectiveness, graph neural network (GNN)-based methods suffer from data sparsity in real scenarios. Recently, contrastive learning (CL) has been used to address the problem of data sparsity. However, most CL-based methods only leverage the original user-item interaction graph to construct the CL task, lacking the explicit exploitation of the higher-order information (i.e., user-user and item-item relationships). Even for the CL-based method that uses the higher-order information, the reception field of the higher-order information is fixed and regardless of the difference between nodes. In this paper, we propose a novel adaptive multi-view fusion contrastive learning framework, named AdaMCL, for graph collaborative filtering. To exploit the higher-order information more accurately, we propose an adaptive fusion strategy to fuse the embeddings learned from the user-item and user-user graphs. Moreover, we propose a multi-view fusion contrastive learning paradigm to construct effective CL tasks. Besides, to alleviate the noisy information caused by aggregating higher-order neighbors, we propose a layer-level CL task. Extensive experimental results reveal that AdaMCL is effective and outperforms existing collaborative filtering models significantly.
In many computer vision applications, one image can be represented by multiple heterogeneous features from different views, most of them commonly locate in high-dimensional space. These features can reflect different characteristics of one same object, they contain compatible and complementary information among each other. How to construct an uniform low-dimensional embedding features which represent useful information of multi-view features is still an important and urgent issue to be solved. Therefore, we propose a multi-view fusion method via tensor learning and gradient descent (MvF-TG) in this paper. MvF-TG reconstructs a lowdimensional mapping subspace of each object by utilizing its k nearest neighbors, which preserves the underlying neighborhood structure of the original local manifold. The new method can effectively exploit the spatial correlation information from the multi-view features by tensor learning. Furthermore, the method constructs a gradient descent optimization model to generate the better unified low dimensional embedding. The proposed method is compared with several single-view and multi-view dimensional reduction methods in these indicators of P, R, MAP and F-measure. In the retrieval experiments, the P values of the newmethod respectively are 86.80%, 52.00%, 68.56% and 78.80% on datasets of Corel1k, Corel5k, Corel10k and Holidays. In the classification experiments, the mean accuracies of it respectively are 47.94% and 87.58% on datasets of Caltech101 and Coil. These values are higher than those obtained by other comparison methods, various evaluations based on the applications of image classification and retrieval demonstrates the effectiveness of our proposed method on multi-view feature fusion dimension reduction.
Multi-view unsupervised feature selection (MV-UFS) has recently gained significant attention as an effective technique for handling high-dimensional multi-view data. Despite recent progress, three flaws are present in most existing MV-UFS methods. First, they predominantly pay attention to the feature dimensionality while neglecting sample size considerations, typically resulting in cubic time complexity concerning sample size (i.e., $\mathcal {O}(n^{3})$) that hinders them in real-world applications. Second, existing methods primarily focus on the relationships between pairwise views, failing to leverage the complex interactions among multiple views effectively. Third, they generally overlook crucial graph semantic information during graph fusion. An ingenious MV-UFS method is presented in this paper, named Joint Information Interaction and Semantic Fusion for Multi-View Unsupervised Feature Selection (JIISF), to overcome these limitations. Specifically, JIISF devises a multi-view double reduction learning framework that projects original data into a low-dimensional embedding space and dynamically picks out representative anchors, eliminating redundant information while constructing compact view-specific anchor graphs. Meanwhile, JIISF presents a cross-view information interaction scheme, which enables comprehensive information interaction among multiple views rather than being limited to pairwise views. A semantic alignment fusion strategy is presented to learn a diverse, consistent anchor graph with an explicit semantic alignment module, which can preserve diversity while maintaining consistency. Finally, an alternating optimization algorithm is devised to resolve the proposed JIISF, and experimental results on real-life datasets validate its advantage compared with some state-of-the-art UFS and MV-UFS methods.
LiDAR semantic segmentation is a key task in advanced autonomous driving systems. Projection-based methods exhibit real-time potential due to their efficiency, but suffer from inevitable 3D information loss and rely on time-consuming post-processing, limiting overall performance. To address this, we propose MFINet, a real-time semantic segmentation network based on multi-view fusion and 2D-3D interaction enhancement. It adopts a three-branch architecture that integrates 3D Point View (3D-PV), 2D Bird’s Eye View (2D-BEV) and 2D Range View (2D-RV) to make full use of 2D and 3D representation. From 3D to 2D, we design a 3D Point Feature Projector (3DPFP), which injects 3D features into the 2D BEV and RV pseudo-images to retain effective 3D information. From 2D to 3D, a Feature Enhancement (FE) module is designed to leverage the advantages of 2D information in extracting geometric and semantic features. We also introduce a 2D-3D Fusion Head (FH) to aggregate point features from multiple views. Besides, we incorporate a Multi-Scale Dilated Attention (MSDA) module with a sliding window strategy to enhance feature discrimination. Extensive experiments on the SemanticKITTI and NuScenes benchmarks demonstrate that MFINet outperforms existing methods on the SemanticKITTI, NuScenes val set and achieves competitive results on the NuScenes test set.
Existing drug–target interaction (DTI) prediction methods still face challenges caused by sparse interaction data, complex multi-source relationships, and imbalanced information contributions among different nodes. In this study, we propose NAFF-DTI, a node-level adaptive feature fusion network based on multi-view graphs. The model uniformly represents drug similarity, target similarity, and known drug–target interactions as multiple relational views, and learns node representations through graph encoding and cross-view representation learning. To more effectively utilize heterogeneous relational information, NAFF-DTI introduces cross-view feature discrepancy modeling and a node-level adaptive fusion mechanism to dynamically adjust the contribution of different views according to node structural characteristics. Experimental results show that NAFF-DTI achieves the best AUC and AUPR on all five benchmark datasets. Compared with the strongest baseline for each dataset and metric, NAFF-DTI achieves average relative improvements of 3.81% in AUC and 3.23% in AUPR. It can also improve the utilization of multi-source information, maintain relatively stable prediction under different data distributions, and prioritize biologically plausible candidate drug–target associations from the unannotated candidate space. These results indicate that NAFF-DTI can provide computational support for DTI candidate prioritization and repurposing-oriented hypothesis generation.
Recently, Multi-View Graph Clustering (MVGC) methods have achieved significant progress, leading to their wide adoption in various applications. However, most MVGC methods merely pursue consistent information by simply fusing multi-view graphs, ignoring the cross-view interactions among them, which limits the ceiling of their performance. To make up for this deficiency, we design a credible cross-view graph enhancement module to explore the credible topological structure, while accomplishing cross-view interactions, to boost clustering performance in multi-view graph scenarios. Besides, we reconsider the graph clustering task from the perspective of graph signal processing. From this novel perspective, we adapt the high-order Graph Trend Filter to reveal the inhomogeneities in graph smoothness levels and further consider the brand-new local preference in MVGC, which provides theoretical guidance for graph clustering. Building on these insights, we propose the Enhanced Graph Trend Filter Clustering (EGTFC) method and present an effective algorithm accompanied by corresponding theoretical analyses to tackle the optimization problem inherent in EGTFC. Finally, substantial experimental results on twelve benchmark datasets demonstrate the effectiveness of our proposals and the superiority over thirteen state-of-the-art MVGC methods.