Shapley Additive exPlanations/CNN/Neural Network

Artificial intelligence (AI) and machine learning (ML) models have become essential tools used in many critical systems to make significant decisions; the decisions taken by these models need to be trusted and explained on many occasions. On the other hand, the performance of different ML and AI models varies with the same used dataset. Sometimes, developers have tried to use multiple models before deciding which model should be used without understanding the reasons behind this variance in performance. Explainable artificial intelligence (XAI) models have presented an explanation for the models’ performance based on highlighting the features that the model considered necessary while making the decision. This work presents an analytical approach to studying the density functions for intrusion detection dataset features. The study explains how and why these features are essential during the XAI process. We aim, in this study, to explain XAI behavior to add an extra layer of explainability. The density function analysis presented in this paper adds a deeper understanding of the importance of features in different AI models. Specifically, we present a method to explain the results of SHAP (Shapley additive explanations) for different machine learning models based on the feature data’s KDE (kernel density estimation) plots. We also survey the specifications of dataset features that can perform better for convolutional neural networks (CNN) based models.

No abstract

No abstract

This paper presents a novel approach that merges a lightweight parallel depth-wise separable convolutional neural network (LPDCNN) with a ridge regression extreme learning machine (Ridge-ELM) for precise classification of three lung cancer types alongside normal lung tissue (adenocarcinoma, large cell carcinoma, normal, and squamous cell carcinoma) using CT images. The proposed methodology combines contrast-limited adaptive histogram equalization (CLAHE) and Gaussian blur to enhance image quality, reduce noise, and improve visual clarity. The LPDCNN extracts discriminant features while minimizing computational complexity (0.53 million parameters and 9 layers). The Ridge-ELM model was developed to enhance classification performance, replacing the traditional pseudoinverse in the ELM approach. Through comprehensive evaluation against state-of-the-art models, the framework achieves remarkable average recall and accuracy values of 98.25 ± 1.031 % and 98.40 ± 0.822 %, respectively, through rigorous five-fold cross-validation for four-class classifications. In binary classifications, outstanding results are obtained with recall and accuracy values of 99.70 ± 0.671 % and 99.70 ± 0.447 %%, respectively. Notably, the framework exhibits exceptional efficiency, with a testing time of only 0.003 s. Additionally, integrating the SHAP (Shapley Additive Explanations) in the proposed framework enhances Explain-ability, providing insights into decision-making and boosting confidence in real-world lung cancer diagnoses.

No abstract

Uveal melanoma (UM) is a severe intraocular cancer in adults aged 50-80, often originating from choroidal nevus, a common intraocular tumour. This transformation can lead to vision loss, metastasis, and even death. Early prediction of UM can reduce the risk of death. In this study, we employed transfer learning techniques and four convolutional neural network (CNN)-based architectures to detect UM and enhance the interpretation of diagnostic results. To accomplish this, we manually gathered 854 RGB fundus images from two distinct datasets, representing the right and left eyes of 854 unique patients (427 lesions and 427 non-lesions). Preprocessing steps, such as image conversion, resizing, and data augmentation, were performed before training and validating the classification results. We utilized InceptionV3, Xception, DenseNet121, and DenseNet169 pre-trained models to improve the generalizability and performance of our results, evaluating each architecture on an external validation set. Addressing the issue of interpretability in deep learning (DL) models to minimize the blackbox problem, we employed the SHapley Additive exPlanations (SHAP) analysis approach to identify regions of an eye image that contribute most to the prediction of choroidal nevus (CN). The performance results of the DL models revealed that DenseNet169 achieved the highest accuracy 89%, and lowest loss value 0.65%, for the binary classification of CN. The SHAP findings demonstrate that this method can serve as a tool for interpreting classification results by providing additional context information about individual sample images and facilitating a more comprehensive evaluation of binary classification in CN.

No abstract

This study investigates the efficacy of Explainable Artificial Intelligence (XAI) methods, specifically Gradient-weighted Class Activation Mapping (Grad-CAM) and Shapley Additive Explanations (SHAP), in the feature selection process for national demand forecasting. Utilising a multi-headed Convolutional Neural Network (CNN), both XAI methods exhibit capabilities in enhancing forecasting accuracy and model efficiency by identifying and eliminating irrelevant features. Comparative analysis revealed Grad-CAM’s exceptional computational efficiency in high-dimensional applications and SHAP’s superior ability in revealing features that degrade forecast accuracy. However, limitations are found in both methods, with Grad-CAM including features that decrease model stability, and SHAP inaccurately ranking significant features. Future research should focus on refining these XAI methods to overcome these limitations and further probe into other XAI methods’ applicability within the time-series forecasting domain. This study underscores the potential of XAI in improving load forecasting, which can contribute significantly to the development of more interpretative, accurate and efficient forecasting models.

Crop yield forecasting depends on many interactive factors, including crop genotype, weather, soil, and management practices. This study analyzes the performance of machine learning and deep learning methods for winter wheat yield prediction using an extensive dataset of weather, soil, and crop phenology variables in 271 counties across Germany from 1999 to 2019. We proposed a Convolutional Neural Network (CNN) model, which uses a 1-dimensional convolution operation to capture the time dependencies of environmental variables. We used eight supervised machine learning models as baselines and evaluated their predictive performance using RMSE, MAE, and correlation coefficient metrics to benchmark the yield prediction results. Our findings suggested that nonlinear models such as the proposed CNN, Deep Neural Network (DNN), and XGBoost were more effective in understanding the relationship between the crop yield and input data compared to the linear models. Our proposed CNN model outperformed all other baseline models used for winter wheat yield prediction (7 to 14% lower RMSE, 3 to 15% lower MAE, and 4 to 50% higher correlation coefficient than the best performing baseline across test data). We aggregated soil moisture and meteorological features at the weekly resolution to address the seasonality of the data. We also moved beyond prediction and interpreted the outputs of our proposed CNN model using SHAP and force plots which provided key insights in explaining the yield prediction results (importance of variables by time). We found DUL, wind speed at week ten, and radiation amount at week seven as the most critical features in winter wheat yield prediction.

This study prognoses the remaining useful life of a turbofan engine using a deep learning model, which is essential for the health management of an engine. The proposed deep learning model affords a significantly improved accuracy by organizing networks with a one-dimensional convolutional neural network, long short-term memory, and bidirectional long short-term memory. In particular, this paper investigates two practical and crucial issues in applying the deep learning model for system prognosis. The first is the requirement of numerous sensors for different components, i.e., the curse of dimensionality. Second, the deep neural network cannot identify the problematic component of the turbofan engine due to its "black box" property. This study thus employs dimensionality reduction and Shapley additive explanation (SHAP) techniques. Dimensionality reduction in the model reduces the complexity and prevents overfitting, while maintaining high accuracy. SHAP analyzes and visualizes the black box to identify the sensors. The experimental results demonstrate the high accuracy and efficiency of the proposed model with dimensionality reduction and show that SHAP enhances the explainability in a conventional deep learning model for system prognosis.

Abstract. The use of complex models such as deep neural networks has yielded large improvements in predictive tasks in many fields including digital soil mapping. One of the concerns about using these models is that they are perceived as black boxes with low interpretability. In this paper we introduce the use of game theory, specifically Shapley additive explanations (SHAP) values, in order to interpret a digital soil mapping model. SHAP values represent the contribution of a covariate to the final model predictions. We applied this method to a multi-task convolutional neural network trained to predict soil organic carbon in Chile. The results show the contribution of each covariate to the model predictions in three different contexts: (a) at a local level, showing the contribution of the various covariates for a single prediction; (b) a global understanding of the covariate contribution; and (c) a spatial interpretation of their contributions. The latter constitutes a novel application of SHAP values and also the first detailed analysis of a model in a spatial context. The analysis of a SOC (soil organic carbon) model in Chile corroborated that the model is capturing sensible relationships between SOC and rainfall, temperature, elevation, slope, and topographic wetness index. The results agree with commonly reported relationships, highlighting environmental thresholds that coincide with significant areas within the study area. This contribution addresses the limitations of the current interpretation of models in digital soil mapping, especially in a spatial context. We believe that SHAP values are a valuable tool that should be included within the DSM (digital soil mapping) framework, since they address the important concerns regarding the interpretability of more complex models. The model interpretation is a crucial step that could lead to generating new knowledge to improve our understanding of soils.

Cancer research has seen explosive development exploring deep learning (DL) techniques for analysing magnetic resonance imaging (MRI) images for predicting brain tumours. We have observed a substantial gap in explanation, interpretability, and high accuracy for DL models. Consequently, we propose an explanation-driven DL model by utilising a convolutional neural network (CNN), local interpretable model-agnostic explanation (LIME), and Shapley additive explanation (SHAP) for the prediction of discrete subtypes of brain tumours (meningioma, glioma, and pituitary) using an MRI image dataset. Unlike previous models, our model used a dual-input CNN approach to prevail over the classification challenge with images of inferior quality in terms of noise and metal artifacts by adding Gaussian noise. Our CNN training results reveal 94.64% accuracy as compared to other state-of-the-art methods. We used SHAP to ensure consistency and local accuracy for interpretation as Shapley values examine all future predictions applying all possible combinations of inputs. In contrast, LIME constructs sparse linear models around each prediction to illustrate how the model operates in the immediate area. Our emphasis for this study is interpretability and high accuracy, which is critical for realising disparities in predictive performance, helpful in developing trust, and essential in integration into clinical practice. The proposed method has a vast clinical application that could potentially be used for mass screening in resource-constraint countries.

Floods are natural hazards that lead to devastating financial losses and large displacements of people. Flood susceptibility maps can improve mitigation measures according to the specific conditions of a study area. The design of flood susceptibility maps has been enhanced through use of hybrid machine learning and deep learning models. Although these models have achieved better accuracy than traditional models, they are not widely used by stakeholders due to their black-box nature. In this study, we propose the application of an explainable artificial intelligence (XAI) model that incorporates the Shapley additive explanation (SHAP) model to interpret the outcomes of convolutional neural network (CNN) deep learning models, and analyze the impact of variables on flood susceptibility mapping. This study was conducted in Jinju Province, South Korea, which has a long history of flood events. Model performance was evaluated using the area under the receiver operating characteristic curve (AUROC), which showed a prediction accuracy of 88.4%. SHAP plots showed that land use and various soil attributes significantly affected flood susceptibility in the study area. In light of these findings, we recommend the use of XAI-based models in future flood susceptibility mapping studies to improve interpretations of model outcomes, and build trust among stakeholders during the flood-related decision-making process.

Malaria is a major global health concern in low-resource countries. Artificial intelligence (AI), particularly convolutional neural networks (CNNs), has become an effective approach for the automatic identification of malaria in blood smear images. However, the lack of interpretability in AI models limits their widespread adoption in clinical practice. This research introduces an innovative approach combining CNNs for malaria detection with two explainable AI (XAI) methods: SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). This dual method improves the transparency of the network's decision-making procedure, ensuring that both medical experts and patients can understand the reasoning behind predictions. Our model achieved a 97.8% accuracy in detecting malaria, highlighting its effectiveness and reliability. Through SHAP and LIME, we provide visual explanations of model decisions, making AI-driven tools more interpretable and reliable.

Landslides are among the most devastating natural hazards, severely impacting human lives and damaging property and infrastructure. Landslide susceptibility maps, which help to identify which regions in a given area are at greater risk of a landslide occurring, are a key tool for effective mitigation. Research in this field has grown immensely, ranging from quantitative to deterministic approaches, with a recent surge in machine learning (ML)-based computational models. The development of ML models, in particular, has undergone a meteoritic rise in the last decade, contributing to the successful development of accurate susceptibility maps. However, despite their success, these models are rarely used by stakeholders owing to their “black box” nature. Hence, it is crucial to explain the results, thus providing greater transparency for the use of such models. To address this gap, the present work introduces the use of an ML-based explainable algorithm, SHapley Additive exPlanations (SHAP), for landslide susceptibility modeling. A convolutional neural network model was used conducted in the CheongJu region in South Korea. A total of 519 landslide locations were examined with 16 landslide-affected variables, of which 70% was used for training and 30% for testing, and the model achieved an accuracy of 89%. Further, the comparison was performed using Support Vector Machine mode, which achieved an accuracy of 84%. The SHAP plots showed variations in feature interactions for both landslide and non-landslide locations, thus providing more clarity as to how the model achieves a specific result. The SHAP dependence plots explained the relationship between altitude and slope, showing a negative relationship with altitude and a positive relationship with slope. This is the first use of an explainable ML model in landslide susceptibility modeling, and we argue that future works should include aspects of explainability to open up the possibility of developing a transferable artificial intelligence model.

No abstract

Acute intestinal ischemia is a life-threatening condition. The current gold standard, with evaluation based on visual and tactile sensation, has low specificity. In this study, we explore the feasibility of using machine learning models on images of the intestine, to assess small intestinal viability. A digital microscope was used to acquire images of the jejunum in 10 pigs. Ischemic segments were created by local clamping (approximately 30 cm in width) of small arteries and veins in the mesentery and reperfusion was initiated by releasing the clamps. A series of images were acquired once an hour on the surface of each of the segments. The convolutional neural network (CNN) has previously been used to classify medical images, while knowledge is lacking whether CNNs have potential to classify ischemia-reperfusion injury on the small intestine. We compared how different deep learning models perform for this task. Moreover, the Shapley additive explanations (SHAP) method within explainable artificial intelligence (AI) was used to identify features that the model utilizes as important in classification of different ischemic injury degrees. To be able to assess to what extent we can trust our deep learning model decisions is critical in a clinical setting. A probabilistic model Bayesian CNN was implemented to estimate the model uncertainty which provides a confidence measure of our model decisions.

No abstract

To purpose of this paper was to assess the feasibility of volumetric breast density estimations on MRI without segmentations accompanied with an explainability step. A total of 615 patients with breast cancer were included for volumetric breast density estimation. A 3-dimensional regression convolutional neural network (CNN) was used to estimate the volumetric breast density. Patients were split in training (N = 400), validation (N = 50), and hold-out test set (N = 165). Hyperparameters were optimized using Neural Network Intelligence and augmentations consisted of translations and rotations. The estimated densities were evaluated to the ground truth using Spearman's correlation and Bland-Altman plots. The output of the CNN was visually analyzed using SHapley Additive exPlanations (SHAP). Spearman's correlation between estimated and ground truth density was ρ = 0.81 (N = 165, P < 0.001) in the hold-out test set. The estimated density had a median bias of 0.70% (95% limits of agreement = - 6.8% to 5.0%) to the ground truth. SHAP showed that in correct density estimations, the algorithm based its decision on fibroglandular and fatty tissue. In incorrect estimations, other structures such as the pectoral muscle or the heart were included. To conclude, it is feasible to automatically estimate volumetric breast density on MRI without segmentations, and to provide accompanying explanations.

No abstract

Success in machine learning has led to a wealth of Artificial Intelligence (AI) systems. A great deal of attention is currently being set on the development of advanced Machine Learning (ML)-based solutions for a variety of automated predictions and classification tasks in a wide array of industries. However, such automated applications may introduce bias in results, making it risky to use these ML models in security-and privacy-sensitive domains. The prediction should be accurate and models have to be interpretable/explainable to understand how they work. In this research, we conduct an empirical evaluation of two major explainer/interpretable methods called LIME and SHAP on two datasets using deep learning models, including Artificial Neural Network (ANN) and Convolutional Neural Network (CNN). The results demonstrated that SHAP performs slightly better than LIME in terms of Identity, Stability, and Separability from two different datasets (Breast Cancer Wisconsin (Diagnostic) and NIH Chest X-Ray) that we used.

There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced interpretations. In this study, we explore a more machine-centric strategy for quantifying the performance of explainability methods on deep neural networks via the notion of decision-making impact analysis. We introduce two quantitative performance metrics: i) Impact Score, which assesses the percentage of critical factors with either strong confidence reduction impact or decision changing impact, and ii) Impact Coverage, which assesses the percentage coverage of adversarially impacted factors in the input. A comprehensive analysis using this approach was conducted on several state-of-the-art explainability methods (LIME, SHAP, Expected Gradients, GSInquire) on a ResNet-50 deep convolutional neural network using a subset of ImageNet for the task of image classification. Experimental results show that the critical regions identified by LIME within the tested images had the lowest impact on the decision-making process of the network (~38%), with progressive increase in decision-making impact for SHAP (~44%), Expected Gradients (~51%), and GSInquire (~76%). While by no means perfect, the hope is that the proposed machine-centric strategy helps push the conversation forward towards better metrics for evaluating explainability methods and improve trust in deep neural networks.

Abstract The limited number of nitrogen dioxide (NO 2 ) surface measurements calls for the development of highly accurate approaches to estimating surface NO 2 concentrations. In this study, we leverage a new satellite instrument, the TROPOspheric Monitoring Instrument (TROPOMI), along with other predictor variables, to estimate daily surface NO 2 concentrations over Texas in 2019. We use the deep convolutional neural network (Deep‐CNN), an advanced deep learning algorithm, to obtain estimates and achieve a correlation coefficient ( R ) of 0.91, an index of agreement (IOA) of 0.95, and a mean absolute bias (MAB) of 1.75 ppb in surface NO 2 estimation. Additionally, we leverage a novel approach, shapley additive explanations (SHAP), to describe how Deep‐CNN understands each predictor variable. The SHAP results show that the Deep‐CNN model has an advanced understanding of the data set, revealing that TROPOMI closely captures levels of NO 2 . In addition, we show the superiority of our Deep‐CNN model at estimating surface NO 2 over other well‐known machine learning and regression models in the field, including the support vector machines (SVM), random forest (RF), and multiple linear regression (MLR). Although SVM and RF show strong capabilities at estimating surface NO 2 concentrations, their accuracy is inferior to that of the Deep‐CNN model, ranking second and third in model accuracy in this study. The MLR, however, shows a poor ability at NO 2 estimation and ranks last among all models. Testing the impact of sample size on model performance, we also show that, compared to other models, Deep‐CNN needs more samples to trigger its strength at surface NO 2 estimation.

Deep neural networks can be used to distinguish partial discharge (PD) signals despite their complexity. This study analyzes the appropriateness of interpreting phase-resolved partial discharge (PRPD) signals using a convolutional neural network (CNN) through the Shapley additive explanation (SHAP) method. The generated PRPD signals were accumulated by applying AC voltage to four types of electrodes with a polyethylene sheet, followed by their conversion into scattered images to construct a classification model, CNN. The SHAP values for each pixel in the test images were then calculated. The result indicated that the pixels around the 0 V line retained high absolute SHAP values in every label, and the average of the summation of absolute SHAP values over all labels and all test images, which indicates the weight of each pixel, shows a similar tendency. Additionally, insight tests of the two CNN models were conducted, and the results showed that some structural defects could be detected by visualizing the SHAP values for each pixel. Finally, the verification of parameter-and-data vulnerability showed that SHAP has sufficient endurance against some types of instability in the data and model. Although the SHAP method lacks a perfect causal model because of its origin, the results imply that in appropriate use cases, weights on classifications of PD signals could be described by SHAP’s interpretability.

Recently, machine learning (ML) and deep learning (DL) models based on artificial intelligence (AI) have emerged as fast and reliable tools for predicting water quality index (WQI) in various regions worldwide. In this study, we propose a novel stacking framework based on DL models for WQI prediction, employing a convolutional neural network (CNN) model. Additionally, we introduce explainable AI (XAI) through XGBoost-based SHAP (SHapley Additive exPlanations) values to gain valuable insights that can enhance decision-making strategies in water management. Our findings demonstrate that the stacking model achieves the highest accuracy in WQI prediction (R2: 0.99, MAPE: 15.99%), outperforming the CNN model (R2: 0.90, MAPE: 58.97%). Although the CNN model shows a relatively high R2 value, other statistical measures indicate that it is actually the worst-performing model among the five tested. This discrepancy may be attributed to the limited training data available for the CNN model. Furthermore, the application of explainable AI (XAI) techniques, specifically XGBoost-based SHAP values, allows us to gain deep insights into the models and extract valuable information for water management purposes. The SHAP values and interaction plot reveal that elevated levels of total dissolved solids (TDS), zinc, and electrical conductivity (EC) are the primary drivers of poor water quality. These parameters exhibit a nonlinear relationship with the water quality index, implying that even minor increases in their concentrations can significantly impact water quality. Overall, this study presents a comprehensive and integrated approach to water management, emphasizing the need for collaborative efforts among all stakeholders to mitigate pollution levels and uphold water quality. By leveraging AI and XAI, our proposed framework not only provides a powerful tool for accurate WQI prediction but also offers deep insights into the models, enabling informed decision-making in water management strategies.

Convolutional neural networks (CNNs) have been originally used for computer vision tasks, such as image classification. While several digital soil mapping studies have been assessing these deep learning algorithms for the prediction of soil properties, their potential for soil classification has not been explored yet. Moreover, the use of deep learning and neural networks in general has often raised concerns because of their presumed low interpretability (i.e., the black box pitfall). However, a recent and fast-developing sub-field of Artificial Intelligence (AI) called explainable AI (XAI) aims to clarify complex models such as CNNs in a systematic and interpretable manner. For example, it is possible to apply model-agnostic interpretation methods to extract interpretations from any machine learning model. In particular, SHAP (SHapley Additive exPlanations) is a method to explain individual predictions: SHAP values represent the contribution of a covariate to the final model predictions. The present study aimed at, first, evaluating the use of CNNs for the classification of potential acid sulfate soils located in the wetland areas of Jutland, Denmark (c. 6,500 km 2 ), and second and most importantly, applying a model-agnostic interpretation method on the resulting CNN model. About 5,900 soil observations and 14 environmental covariates, including a digital elevation model and derived terrain attributes, were utilized as input data. The selected CNN model yielded slightly higher prediction accuracy than the random forest models which were using original or scaled covariates. These results can be explained by the use of a common variable selection method, namely recursive feature elimination, which was based on random forest and thus optimized the selection for this method. Notably, the SHAP method results enabled to clarify the CNN model predictions, in particular through the spatial interpretation of the most important covariates, which constitutes a crucial development for digital soil mapping.

No abstract

Deepfake technology uses auto-encoders and generative adversarial networks to replace or artificially construct fine-tuned faces, emotions, and sounds. Although there have been significant advancements in the identification of particular fake images, a reliable counterfeit face detector is still lacking, making it difficult to identify fake photos in situations with further compression, blurring, scaling, etc. Deep learning models resolve the research gap to correctly recognize phony images, whose objectionable content might encourage fraudulent activity and cause major problems. To reduce the gap and enlarge the fields of view of the network, we propose a dual input convolutional neural network (DICNN) model with ten-fold cross validation with an average training accuracy of 99.36 ± 0.62, a test accuracy of 99.08 ± 0.64, and a validation accuracy of 99.30 ± 0.94. Additionally, we used 'SHapley Additive exPlanations (SHAP) ' as explainable AI (XAI) Shapely values to explain the results and interoperability visually by imposing the model into SHAP. The proposed model holds significant importance for being accepted by forensics and security experts because of its distinctive features and considerably higher accuracy than state-of-the-art methods.

In recent years, a significant amount of research has focused on analyzing the effectiveness of machine learning (ML) models for malware detection. These approaches have ranged from methods such as decision trees and clustering to more complex approaches like support vector machine (SVM) and deep neural networks. In particular, neural networks have proven to be very effective in detecting complex and advanced malware. This, however, comes with a caveat. Neural networks are notoriously complex. Therefore, the decisions that they make are often just accepted without questioning why the model made that specific decision. The black box characteristic of neural networks has challenged researchers to explore methods to explain black-box models such as SVM and neural networks and their decision-making process. Transparency and explainability give the experts and malware analysts assurance and trustworthiness about the ML models’ decisions. In addition, it helps in generating comprehensive reports that can be used to enhance cyber threat intelligence sharing. As such, this much needed analysis drives our work in this paper to explore the explainability and interpretability of ML models in the field of online malware detection. In this paper, we used the Shapley Additive exPlanations (SHAP) explainability technique to achieve efficient performance in interpreting the outcome of different ML models such as SVM Linear, SVM-RBF (Radial Basis Function), Random Forest (RF), Feed-Forward Neural Net (FFNN), and Convolutional Neural Network (CNN) models trained on an online malware dataset. To explain the output of these models, explainability techniques such as KernalSHAP, TreeSHAP, and DeepSHAP are applied to the obtained results.

Geopolymer concrete is a sustainable and eco-friendly substitute for traditional OPC (Ordinary Portland Cement) based concrete, as it reduces greenhouse gas emissions. With various supplementary cementitious materials, the compressive strength of geopolymer concrete should be accurately predicted. Recent studies have applied deep learning techniques to predict the compressive strength of geopolymer concrete yet its hidden decision-making criteria diminish the end-users’ trust in predictions. To bridge this gap, the authors first developed three deep learning models: an artificial neural network (ANN), a deep neural network (DNN), and a 1D convolution neural network (CNN) to predict the compressive strength of slag ash-based geopolymer concrete. The performance indices for accuracy revealed that the DNN model outperforms the other two models. Subsequently, Shapley additive explanations (SHAP) were used to explain the best-performed deep learning model, DNN, and its compressive strength predictions. SHAP exhibited how the importance of each feature and its relationship contributes to the compressive strength prediction of the DNN model. Finally, the authors developed a novel DNN-based open-source software interface to predict the mix design proportions for a given target compressive strength (using inverse modeling technique) for slag ash-based geopolymer concrete. Additionally, the software calculates the Global Warming Potential (kg CO2 equivalent) for each mix design to select the mix designs with low greenhouse emissions.

Traditionally, sleep monitoring has been performed in hospital or clinic environments, requiring complex and expensive equipment set-up and expert scoring. Wearable devices increasingly provide a viable alternative for sleep monitoring and are able to collect movement and heart rate (HR) data. In this work, we present a set of algorithms for sleep-wake and sleep-stage classification based upon actigraphy and cardiac sensing amongst 1,743 participants. We devise movement and cardiac features that could be extracted from research-grade wearable sensors and derive models and evaluate their performance in the largest open-access dataset for human sleep science. Our results demonstrated that neural network models outperform traditional machine learning methods and heuristic models for both sleep-wake and sleep-stage classification. Convolutional neural networks (CNNs) and long-short term memory (LSTM) networks were the best performers for sleep-wake and sleep-stage classification, respectively. Using SHAP (SHapley Additive exPlanation) with Random Forest we identified that frequency features from cardiac sensors are critical to sleep-stage classification. Finally, we introduced an ensemble-based approach to sleep-stage classification, which outperformed all other baselines, achieving an accuracy of 78.2% and F1 score of 69.8% on the classification task for three sleep stages. Together, this work represents the first systematic multimodal evaluation of sleep-wake and sleep-stage classification in a large, diverse population. Alongside the presentation of an accurate sleep-stage classification approach, the results highlight multimodal wearable sensing approaches as scalable methods for accurate sleep-classification, providing guidance on optimal algorithm deployment for automated sleep assessment. The code used in this study can be found online at: https://github.com/bzhai/multimodal_sleep_stage_benchmark.git

No abstract

This paper proposes a deep-stacked neural network to prognose the remaining useful life of the turbofan engines and analyze results using explainable artificial intelligence. The proposed model prognoses the remaining useful life of the turbofan engines accurately by properly stacking a one-dimensional convolutional neural network (1D-CNN), long short-term memory (LSTM), and bidirectional LSTM algorithms. This model also uses dropout technique and residual network to enhance the prediction accuracy. The Explainable artificial intelligence analyzes the prognostic results of RUL. Using SHAP (SHapely Addictive exPlanation), this model can analyze features that have significantly influenced RUL prediction. The SHAP force plot and decision plot can help decision-makers of the turbofan engine properly manage the maintenance by showing the influence of sensors.

We developed a method to apply artificial neural networks (ANNs) for predicting time-series pharmacokinetics (PKs), and an interpretable the ANN-PK model, which can explain the evidence of prediction by applying Shapley additive explanations (SHAP). A previous population PK (PopPK) model of cyclosporin A was used as the comparison model. The patients' data were used for the ANN-PK model input, and the output by ANN was the clearance (CL). The estimated CL value from the ANN were substituted into the one-compartment with one-order absorption model, the concentrations were calculated, and the parameters of ANN were updated by the back-propagation method. Kernel SHAP was applied to the trained model and the SHAP value of each input was calculated. The root mean squared error for the PopPK model and the ANN-PK model were 41.1 and 31.0 ng/ml, respectively. The goodness of fit plots for the ANN-PK model represented more convergence to y = x compared with that for the PopPK model, with good model performance for the ANN-PK model. The most influential factors on CL output were age and body weight from the evaluation using Kernel SHAP, and these factors were incorporated into the PopPK model as the significant covariates of CL. The ANN-PK model could handle time-series data and showed higher prediction accuracy then the conventional PopPK model, and the scientific validity for the model could be evaluated by applying SHAP. Study Highlights WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC? A black-box property of an artificial neural network (ANN) decreases the scientific confidence of the model, and making it difficult to utilize the ANN in the medical field. Moreover, difficulty in handling the time-series data is a significant problem for applying the ANN for pharmacometrics study. WHAT QUESTION DID THIS STUDY ADDRESS? How can we apply the ANN for predicting the time-series pharmacokinetics (PKs) , and confirm the scientific validity of the ANN model? WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE? Using the ANN in combination with a conventional compartment (ANN-PK) model enabled to handle the time-series PK data, and the predicting performance of the model was higher than that of the population PK model. Furthermore, we could evaluate the scientific validity of the ANN model by applying the Shapley additive explanations. HOW MIGHT THIS CHANGE DRUG DISCOVERY, DEVELOPMENT, AND/OR THERAPEUTICS? We expect that our study will contribute to develop the interpretable ANN model, which can predict the time-series PKs, drug efficacies, and side effects with high prediction performance.

No abstract

No abstract

Abstract Predicting and understanding travellers’ mode choices is crucial to developing urban transportation systems and formulating traffic demand management strategies. Machine learning (ML) methods have been widely used as promising alternatives to traditional discrete choice models owing to their high prediction accuracy. However, a significant body of ML methods, especially the branch of neural networks, is constrained by overfitting and a lack of model interpretability. This study employs a neural network with feature selection for predicting travel mode choices and Shapley additive explanations (SHAP) analysis for model interpretation. A dataset collected in Chengdu, China was used for experimentation. The results reveal that the neural network achieves commendable prediction performance, with a 12% improvement over the traditional multinomial logit model. Also, feature selection using a combined result from two embedded methods can alleviate the overfitting tendency of the neural network, while establishing a more robust model against redundant or unnecessary variables. Additionally, the SHAP analysis identifies factors such as travel expenditure, age, driving experience, number of cars owned, individual monthly income, and trip purpose as significant features in our dataset. The heterogeneity of mode choice behaviour is significant among demographic groups, including different age, car ownership, and income levels.

Difficulties in interpreting machine learning (ML) models and their predictions limit the practical applicability of and confidence in ML in pharmaceutical research. There is a need for agnostic approaches aiding in the interpretation of ML models regardless of their complexity that is also applicable to deep neural network (DNN) architectures and model ensembles. To these ends, the SHapley Additive exPlanations (SHAP) methodology has recently been introduced. The SHAP approach enables the identification and prioritization of features that determine compound classification and activity prediction using any ML model. Herein, we further extend the evaluation of the SHAP methodology by investigating a variant for exact calculation of Shapley values for decision tree methods and systematically compare this variant in compound activity and potency value predictions with the model-independent SHAP method. Moreover, new applications of the SHAP analysis approach are presented including interpretation of DNN models for the generation of multi-target activity profiles and ensemble regression models for potency prediction.

No abstract

Predicting pilots' mental states is a critical challenge in aviation safety and performance, with electroencephalogram data offering a promising avenue for detection. However, the interpretability of machine learning and deep learning models, which are often used for such tasks, remains a significant issue. This study aims to address these challenges by developing an interpretable model to detect four mental states-channelised attention, diverted attention, startle/surprise, and normal state-in pilots using EEG data. The methodology involves training a convolutional neural network on power spectral density features of EEG data from 17 pilots. The model's interpretability is enhanced via the use of SHapley Additive exPlanations values, which identify the top 10 most influential features for each mental state. The results demonstrate high performance in all metrics, with an average accuracy of 96%, a precision of 96%, a recall of 94%, and an F1 score of 95%. An examination of the effects of mental states on EEG frequency bands further elucidates the neural mechanisms underlying these states. The innovative nature of this study lies in its combination of high-performance model development, improved interpretability, and in-depth analysis of the neural correlates of mental states. This approach not only addresses the critical need for effective and interpretable mental state detection in aviation but also contributes to our understanding of the neural underpinnings of these states. This study thus represents a significant advancement in the field of EEG-based mental state detection.

Abstract In tunnel construction, efficiently predicting the energy usage of tunnel boring machines (TBMs) is critical for optimizing operations and reducing costs. This research proposes a novel method for predicting the specific energy of micro slurry tunnel boring machines (MSTBMs) using an explainable neural network (xNN) that leverages operator-monitored data. The xNN model provides transparency and interpretability by integrating the Shapley additive explanation (SHAP) technique, enabling tunneling engineers and operators to gain valuable insights into the prediction process. Extensive data from MSTBM umbrella pipe support excavation are the foundation for training, testing, and unseen data in the xNN model. The specific energy formula derived from the operational parameters of the MSTBM defines the dependent variable for the xNN model. The test dataset evaluates the model’s performance with an R ² of 98.7%, an MSE of 2.40, and an MAE of 0.003, demonstrating its accuracy and reliability. Ten percent of the dataset was reserved as unseen data to assess the model’s generalization capabilities. Upon evaluation, the model achieved an R 2 value of 89%, an MAE of 0.01, and a root mean squared error (RMSE) of 0.01. The xNN empowers operators to optimize operational parameters and promote more efficient and sustainable tunneling practices by identifying influential factors affecting energy consumption through its interpretable nature. This research has significant implications for the future of underground construction, paving the way for improved resource management.

No abstract

Credit card fraud is a significant challenge for financial security, with traditional detection systems often lacking accuracy and interpretability. Current methods fall short of capturing complex fraud patterns. This research evaluates the effectiveness of deep neural networks in fraud detection, including Convolutional Neural Networks (CNN), Long Short-Term Memory networks (LSTM), Recurrent Neural Networks (RNN), Multilayer Perceptrons (MLP), and Deep Belief Networks (DBN). We employ a comprehensive dataset of up to 284,000 transactions for training and further incorporate two external datasets consisting of up to 1.3 million transactions to perform rigorous testing. According to our experiments, the CNN model outperformed LSTM, RNN, MLP and DBN by achieving a remarkable accuracy of up to 99%. Furthermore, we employ Adam optimiser to enhance model performance and SHAP (SHapley Additive exPlanations) analysis to improve the interpretability of the best-performing classifier and to gain insights into feature importance and model decisions. The proposed approach outperforms existing methods, combining high accuracy with model interpretability, and contributes to advancing fraud detection for financial security.

In recent years, machine learning-based intrusion detection systems (IDSs) have proven to be effective; especially, deep neural networks improve the detection rates of intrusion detection models. However, as models become more and more complex, people can hardly get the explanations behind their decisions. At the same time, most of the works about model interpretation focuses on other fields like computer vision, natural language processing, and biology. This leads to the fact that in practical use, cybersecurity experts can hardly optimize their decisions according to the judgments of the model. To solve these issues, a framework is proposed in this paper to give an explanation for IDSs. This framework uses SHapley Additive exPlanations (SHAP), and combines local and global explanations to improve the interpretation of IDSs. The local explanations give the reasons why the model makes certain decisions on the specific input. The global explanations give the important features extracted from IDSs, present the relationships between the feature values and different types of attacks. At the same time, the interpretations between two different classifiers, one-vs-all classifier and multiclass classifier, are compared. NSL-KDD dataset is used to test the feasibility of the framework. The framework proposed in this paper leads to improve the transparency of any IDS, and helps the cybersecurity staff have a better understanding of IDSs' judgments. Furthermore, the different interpretations between different kinds of classifiers can also help security experts better design the structures of the IDSs. More importantly, this work is unique in the intrusion detection field, presenting the first use of the SHAP method to give explanations for IDSs.

This Research has focused on optimizing metallurgical processes by integrating Artificial Neural Networks with Shapley additive explanation modeling. This approach helps understand the intricate relationships and mechanisms underlying chemical processes. Neural networks provide accurate predictions based on input parameter interactions, while Shapley values identify the relative importance of each input variable and offer detailed explanations for model predictions, enhancing transparency and interpretability. In this study on roasting and leaching processes for sodium dichromate formation, the neural network - Shapley modeling framework was employed. The goal was to uncover the intricate interplay between input variables and sodium dichromate formation, providing valuable insights for process optimization and prediction. Key factors such as temperature, roasting time, reaction time, and sulfuric acid concentration were optimized in relation to the efficacy of sodium dichromate formation under different settings. The suggested neural networks model predicted optimal yields for combined roasting and leaching settings. The optimum conditions included a roasting temperature of 1046.26 °C, roasting time of 2.7 h, Cr: NaCl ratio of 1.5, leaching time of 41 min at a temperature of 40 °C, and sulfuric acid concentration of 12M. Global sensitivity analysis revealed that the yields of different metals were directly influenced by the temperature during roasting, concentration of sulfuric acid, Cr:NaCl ratio, roasting time, leaching temperature, and leaching time. These parameters were ranked in terms of sensitivity coefficients, indicating their relative importance.

No abstract

The prediction and analysis of traffic crashes in expressway tunnels plays a pivotal role in enhancing tunnel safety. A modified convolutional neural network (M-CNN) for tunnel traffic crash prediction was developed in this study. The synthetic minority over-sampling technique was used to address the issue of imbalanced crash data. Based on the prediction results, sections of high risk in tunnels were identified and Shapley additive explanations (Shap) were used to enhance the interpretability of the M-CNN. The results showed that the prediction accuracy of the M-CNN is high (74.62%) and surpassed the accuracy of baseline models (convolutional neural network, back-propagation neural network, random forest, long short-term memory and support vector machine). The tunnel entrance and exit sections were identified as risk zones. In addition, driver’s operation, tunnel grade and vehicle speed were found to have the greatest impact on rear-end crashes, sideswipe crashes and hitting guardrail crashes, respectively. This study also revealed intricate interaction effects between the variables and the skidding resistance index, with this index exhibiting a negative correlation with crash risk. The research findings have significant implications for the future implementation of machine learning models in crash studies, with practical applications for reducing crash rates.

Internet of Things (IoT) is an emerging paradigm that is turning and revolutionizing worldwide cities into smart cities. However, this emergence is accompanied with several cybersecurity concerns due mainly to the data sharing and constant connectivity of IoT networks. To address this problem, multiple Intrusion Detection Systems (IDSs) have been designed as security mechanisms, which showed their efficiency in mitigating several IoT-related attacks, especially when using deep learning (DL) algorithms. Indeed, Deep Neural Networks (DNNs) significantly improve the detection rate of IoT-related intrusions. However, DL-based models are becoming more and more complex, and their decisions are hardly interpreted by users, especially companies&#x2019; executive staff and cybersecurity experts. Hence, the corresponding users cannot neither understand and trust DL models decisions, nor optimize their decisions (users) based on DL models outputs. To overcome these limits, Explainable Artificial Intelligence (XAI) is an emerging paradigm of Artificial Intelligence (AI), that provides a set of techniques to help interpreting and understanding predictions made by DL models. Thus, XAI enables to explain the decisions of DL-based IDSs to make them interpretable by cybersecurity experts In this paper, we design a new XAI-based framework to give explanations to any critical DL-based decisions for IoT-related IDSs. Our framework relies on a novel IDS for IoT networks, that we also develop by leveraging deep neural network, to detect IoT-related intrusions. In addition, our framework uses three main XAI techniques (i.e., RuleFit, Local Interpretable Model-Agnostic Explanations (LIME), and SHapley Additive exPlanations (SHAP)), on top of our DNN-based model. Our framework can provide both local and global explanations to optimize the interpretation of DL-based decisions. The local explanations target a single/particular DL output, while global explanations focus on deducing the most important features that have conducted to each made decision (e.g., intrusion detection). Thus, our proposed framework introduces more transparency and trust between the decisions made by our DL-based IDS model and cybersecurity experts. Both NSL-KDD and UNSW-NB15 datasets are used to validate the feasibility of our XAI framework. The experimental results show the efficiency of our framework to improve the interpretability of the IoT IDS against well-known IoT attacks, and help the cybersecurity experts get a better understanding of IDS decisions.

No abstract

No abstract

Vehicle insurance claim fraud presents a major challenge in the insurance industry, leading to financial losses and increased premiums for policyholders. Traditional fraud detection methods, such as rule-based systems and manual claim assessment, struggle to handle the complexity and growing volume of fraudulent claims. With the advancement of Machine Learning (ML), models such as Artificial Neural Networks (ANNs) have significantly improved fraud detection accuracy. However, a key limitation of existing ML-based methods is their lack of interpretability, making it difficult for insurers to justify fraud detection decisions. To address this issue, this study proposes an interpretable fraud detection framework based on an ANN integrated with Shapley Additive Explanations (SHAP). The framework involves preprocessing insurance claim data, training an ANN for fraud prediction, and applying SHAP to analyze feature importance and provide interpretability. Experimental results demonstrate that the proposed model achieves high accuracy in fraud detection while offering insights into influential features affecting claim decisions. The findings highlight the importance of incorporating explainability into ML-based fraud detection, ensuring transparency and trustworthiness in the insurance industry.

No abstract

Adverse effects induced by drug-drug interactions may result in early termination of drug development or even withdrawal of drugs from the market, and many drug-drug interactions are caused by the inhibition of cytochrome P450 (CYP450) enzymes. Therefore, the accurate prediction of the inhibition capability of a given compound against a specific CYP450 isoform is highly desirable. In this study, three ensemble learning methods, including random forest, gradient boosting decision tree, and eXtreme gradient boosting (XGBoost), and two deep learning methods, including deep neural networks and convolutional neural networks, were used to develop classification models to discriminate inhibitors and noninhibitors for five major CYP450 isoforms (1A2, 2C9, 2C19, 2D6, and 3A4). The results demonstrate that the ensemble learning models generally give better predictions than the deep learning models for the external test sets. Among all of the models, the XGBoost models achieve the best classification capability (average prediction accuracy of 90.4%) for the test sets, which even outperform the previously reported model developed by the multitask deep autoencoder neural network (88.5%). The Shapley additive explanation method was then used to interpret the models and analyze the misclassified molecules. The important molecular descriptors given by our models are consistent with the structural preferences for inhibitors of different CYP450 isoforms, which may provide valuable clues to detect potential drug-drug interactions in the early stage of drug discovery.

State-of-the-art deep neural networks (DNNs) are highly effective in solving many complex real-world problems. However, these models are vulnerable to adversarial perturbation attacks, and despite the plethora of research in this domain, to this day, adversaries still have the upper hand in the cat and mouse game of adversarial example generation methods vs. detection and prevention methods. In this research, we present a novel detection method that uses Shapley Additive Explanations (SHAP) values computed for the internal layers of a DNN classifier to discriminate between normal and adversarial inputs. We evaluate our method by building an extensive dataset of adversarial examples over the popular CIFAR-10 and MNIST datasets, and training a neural network-based detector to distinguish between normal and adversarial inputs. We evaluate our detector against adversarial examples generated by diverse state-of-the-art attacks and demonstrate its high detection accuracy and strong generalization ability to adversarial inputs generated with different attack methods.

Anticancer peptides constitute one of the most promising therapeutic agents for combating common human cancers. Using wet experiments to verify whether a peptide displays anticancer characteristics is time-consuming and costly. Hence, in this study, we proposed a computational method named identify anticancer peptides via deep representation learning features (iACP-DRLF) using light gradient boosting machine algorithm and deep representation learning features. Two kinds of sequence embedding technologies were used, namely soft symmetric alignment embedding and unified representation (UniRep) embedding, both of which involved deep neural network models based on long short-term memory networks and their derived networks. The results showed that the use of deep representation learning features greatly improved the capability of the models to discriminate anticancer peptides from other peptides. Also, UMAP (uniform manifold approximation and projection for dimension reduction) and SHAP (shapley additive explanations) analysis proved that UniRep have an advantage over other features for anticancer peptide identification. The python script and pretrained models could be downloaded from https://github.com/zhibinlv/iACP-DRLF or from http://public.aibiochem.net/iACP-DRLF/.

This paper presents the use of two popular explainability tools called Local Interpretable Model-Agnostic Explanations (LIME) and Shapley Additive exPlanations (SHAP) to explain the predictions made by a trained deep neural network. The deep neural network used in this work is trained on the UCI Breast Cancer Wisconsin dataset. The neural network is used to classify the masses found in patients as benign or malignant based on 30 features that describe the mass. LIME and SHAP are then used to explain the individual predictions made by the trained neural network model. The explanations provide further insights into the relationship between the input features and the predictions. SHAP methodology additionally provides a more holistic view of the effect of the inputs on the output predictions. The results also present the commonalities between the insights gained using LIME and SHAP. Although this paper focuses on the use of deep neural networks trained on UCI Breast Cancer Wisconsin dataset, the methodology can be applied to other neural networks and architectures trained on other applications. The deep neural network trained in this work provides a high level of accuracy. Analyzing the model using LIME and SHAP adds the much desired benefit of providing explanations for the recommendations made by the trained model.

This study aims to develop and evaluate an obesity classification model using an Artificial Neural Network (ANN) combined with Explainable Artificial Intelligence (XAI) techniques based on SHAP (SHapley Additive exPlanations). The model was trained and tested using two different optimizers, Adaptive Moment Estimation (Adam) and Stochastic Gradient Descent (SGD), across multiple train-test ratios and epoch variations. The experimental results indicate that the Adam optimizer consistently outperformed SGD in terms of accuracy, loss value, and stability of evaluation metrics. The best performance was achieved with a 90:10 train-test ratio at 100 epochs, yielding an accuracy of 94.74%, a loss of 0.1899, precision, recall, and an f1-score of 0.95. To improve interpretability, SHAP was applied to identify the most influential features in the classification process. The analysis revealed that features such as Weight, Height, Gender, and Age significantly contribute to the model's predictions. Based on the SHAP interpretation, feature selection was conducted using the top nine features with the highest SHAP values. Retraining the ANN with these selected features resulted in improved performance, achieving 98.56% accuracy, a loss of 0.0638, and a precision, recall, and F1-score of 0.99 . These findings demonstrate that integrating XAI with ANN not only enhances transparency and interpretability but also boosts classification performance and computational efficiency. This approach shows strong potential for supporting decision-making in healthcare, particularly for early detection and intervention in cases related to obesity.

With the development of data mining, machine learning offers opportunities to improve discrimination by analyzing complex interactions among massive variables. To test the ability of machine learning algorithms for predicting risk of type 2 diabetes mellitus (T2DM) in a rural Chinese population, we focus on a total of 36,652 eligible participants from the Henan Rural Cohort Study. Risk assessment models for T2DM were developed using six machine learning algorithms, including logistic regression (LR), classification and regression tree (CART), artificial neural networks (ANN), support vector machine (SVM), random forest (RF) and gradient boosting machine (GBM). The model performance was measured in an area under the receiver operating characteristic curve, sensitivity, specificity, positive predictive value, negative predictive value and area under precision recall curve. The importance of variables was identified based on each classifier and the shapley additive explanations approach. Using all available variables, all models for predicting risk of T2DM demonstrated strong predictive performance, with AUCs ranging between 0.811 and 0.872 using laboratory data and from 0.767 to 0.817 without laboratory data. Among them, the GBM model performed best (AUC: 0.872 with laboratory data and 0.817 without laboratory data). Performance of models plateaued when introduced 30 variables to each model except CART model. Among the top-10 variables across all methods were sweet flavor, urine glucose, age, heart rate, creatinine, waist circumference, uric acid, pulse pressure, insulin, and hypertension. New important risk factors (urinary indicators, sweet flavor) were not found in previous risk prediction methods, but determined by machine learning in our study. Through the results, machine learning methods showed competence in predicting risk of T2DM, leading to greater insights on disease risk factors with no priori assumption of causality.

Urban vegetation mapping is critical in many applications, i.e., preserving biodiversity, maintaining ecological balance, and minimizing the urban heat island effect. It is still challenging to extract accurate vegetation covers from aerial imagery using traditional classification approaches, because urban vegetation categories have complex spatial structures and similar spectral properties. Deep neural networks (DNNs) have shown a significant improvement in remote sensing image classification outcomes during the last few years. These methods are promising in this domain, yet unreliable for various reasons, such as the use of irrelevant descriptor features in the building of the models and lack of quality in the labeled image. Explainable AI (XAI) can help us gain insight into these limits and, as a result, adjust the training dataset and model as needed. Thus, in this work, we explain how an explanation model called Shapley additive explanations (SHAP) can be utilized for interpreting the output of the DNN model that is designed for classifying vegetation covers. We want to not only produce high-quality vegetation maps, but also rank the input parameters and select appropriate features for classification. Therefore, we test our method on vegetation mapping from aerial imagery based on spectral and textural features. Texture features can help overcome the limitations of poor spectral resolution in aerial imagery for vegetation mapping. The model was capable of obtaining an overall accuracy (OA) of 94.44% for vegetation cover mapping. The conclusions derived from SHAP plots demonstrate the high contribution of features, such as Hue, Brightness, GLCM_Dissimilarity, GLCM_Homogeneity, and GLCM_Mean to the output of the proposed model for vegetation mapping. Therefore, the study indicates that existing vegetation mapping strategies based only on spectral characteristics are insufficient to appropriately classify vegetation covers.

Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.

Analysis of traffic crash and associated data provides insights and assists with identification of cause-and-effect relationships with crash probabilities and outcomes. This study utilized eight years of police-reported Nebraska crash data using a deep neural network (DNN) to model crash injury severity outcomes. Prediction performances and model interpretability were examined. The developed DNN excelled in prediction accuracy, precision, and recall but was computationally intensive compared with a baseline multinomial logistic regression model. While the lack of interpretability power of deep learning models limits their usage, the adoption of SHapley Additive exPlanation (SHAP) values was an improvement. Conclusions drawn from the DNN model are generally consistent with the estimated baseline model. For instance, the variable total number of pedestrians was found significant in both scenarios of the multinomial logit model indicating a strong relationship with more severe crash injury outcomes. It was also found important in all three sets of parameters in DNN. SHAP values also allow in-depth analysis of prediction results on a single observation, such as the variable crash type (same direction sideswipe) contributing to classifying a single observation as property damage only. These findings are beneficial for making more informed transportation safety-related decisions.

No abstract

No abstract

Anthropometry is a Greek word that consists of the two words "Anthropo" meaning human species and "metery" meaning measurement. It is a science that deals with the size of the body including the dimensions of different parts, the field of motion and the strength of the muscles of the body. Specific individual dimensions such as heights, widths, depths, distances, environments and curvatures are usually measured. In this article, we investigate the anthropometric characteristics of patients with chronic diseases (diabetes, hypertension, cardiovascular disease, heart attacks and strokes) and find the factors affecting these diseases and the extent of the impact of each to make the necessary planning. We have focused on cohort studies for 10047 qualified participants from Ravansar County. Machine learning provides opportunities to improve discrimination through the analysis of complex interactions between broad variables. Among the chronic diseases in this cohort study, we have used three deep neural network models for diagnosis and prognosis of the risk of type 2 diabetes mellitus (T2DM) as a case study. Usually in Artificial Intelligence for medicine tasks, Imbalanced data is an important issue in learning and ignoring that leads to false evaluation results. Also, the accuracy evaluation criterion was not appropriate for this task, because a simple model that is labeling all samples negatively has high accuracy. So, the evaluation criteria of precession, recall, AUC, and AUPRC were considered. Then, the importance of variables in general was examined to determine which features are more important in the risk of T2DM. Finally, personality feature was added, in which individual feature importance was examined. Performing by Shapley Values, the model is tuned for each patient so that it can be used for prognosis of T2DM risk for that patient. In this paper, we have focused and implemented a full pipeline of Data Creation, Data Preprocessing, Handling Imbalanced Data, Deep Learning model, true Evaluation method, Feature Importance and Individual Feature Importance. Through the results, the pipeline demonstrated competence in improving the Diagnosis and Prognosis the risk of T2DM with personalization capability.

No abstract

This study collected and analyzed data to predict particulate matter (PM) concentrations in Korea at regular intervals. Automated synoptic observation system data, real-time atmospheric observation data from AirKorea, and Geostationary Korea Multipurpose Satellite – 2A data were used. We also used deep learning, which is useful for PM predictions. The deep learning model used a neural network (NN) to predict concentrations of PM with a diameter less than 2.5 μm (PM2.5) and PM with a diameter less than 10 μm (PM10). To illustrate the results of the NN model, we calculated the Shapley value using eXplanable Artificial Intelligence (XAI) in the SHapley Additive exPlanations (SHAP) library. The difference in the analysis according to the diameter of aerosols was explained. To analyze the contribution of features for each grid, the SHAP values were normalized. The normalized SHAP values were clustered and represented visually. PM2.5 and PM10 were classified into four clusters. The next day's PM2.5 and PM10 predictions were both heavily influenced by weather variables in the western region, and air quality data were more influential in the inland region. Unlike PM2.5, the next day's PM10 prediction in the southern region was affected to a greater degree by the wind.

We investigate the effect of including application knowledge about a robotic system states’ causal relations when generating explanations of deep neural network policies. To this end, we compare two methods from explainable artificial intelligence, KernelSHAP, and causal SHAP, on a deep neural network trained using deep reinforcement learning on the task of controlling a lever using a robotic manipulator. A primary disadvantage of KernelSHAP is that its explanations represent only the features’ direct effects on a model’s output, not considering the indirect effects a feature can have on the output by affecting other features. Causal SHAP uses a partial causal ordering to alter KernelSHAP’s sampling procedure to incorporate these indirect effects. This partial causal ordering defines the causal relations between the features, and we specify this using application knowledge about the lever control task. We show that enabling an explanation method to account for indirect effects and incorporating some application knowledge can lead to explanations that better agree with human intuition. This is especially favorable for a real-world robotics task, where there is considerable causality at play, and in addition, the required application knowledge is often handily available.

No abstract

As image-based deep learning becomes pervasive on every device, from cell phones to smart watches, there is a growing need to develop methods that continually learn from data while minimizing memory footprint and power consumption. While memory replay techniques have shown exceptional promise for this task of continual learning, the best method for selecting which buffered images to replay is still an open question. In this paper, we specifically focus on the online class-incremental setting where a model needs to learn new classes continually from an online data stream. To this end, we contribute a novel Adversarial Shapley value scoring method that scores memory data samples according to their ability to preserve latent decision boundaries for previously observed classes (to maintain learning stability and avoid forgetting) while interfering with latent decision boundaries of current classes being learned (to encourage plasticity and optimal learning of new class boundaries). Overall, we observe that our proposed ASER method provides competitive or improved performance compared to state-of-the-art replay-based continual learning methods on a variety of datasets.

This paper performs a comprehensive evaluation of Smartwatch in-air signature classification based on multiple deep learning models. We leverage the Shapley value in dimension-wise feature selection to provide the in-air signature community with the most and least dominant dimension regarding the accuracy of in-air signature classification. Our experiment results highlight InceptionTime as the top-performing model, achieving an accuracy of 97.73%. Through our Shapley Value analysis, among all the sensors embedded in the Smartwatch, we find that the y dimension of the gyroscope and the z dimension of the gyroscope contribute the most to classification accuracy with 12.57% and 12.51% respectively, while the x dimension of the accelerometer produces the least contribution with 8.71%.

The early and accurate detection of the onset of acute myocardial infarction (AMI) is imperative for the timely provision of medical intervention and the reduction of its mortality rate. Machine learning techniques have demonstrated great potential in aiding disease diagnosis. In this paper, we present a framework to predict the onset of AMI using 713,447 extracted ECG samples and associated auxiliary data from the longitudinal and comprehensive ECG-ViEW II database, previously unexplored in the field of machine learning in healthcare. The framework is realized with two deep learning models, a convolutional neural network (CNN) and a recurrent neural network (RNN), and a decision-tree based model, XGBoost. Synthetic minority oversampling technique (SMOTE) was utilized to address class imbalance. High prediction accuracy of 89.9%, 84.6%, 97.5% and ROC curve areas of 90.7%, 82.9%, 96.5% have been achieved for the best CNN, RNN, and XGBoost models, respectively. Shapley values were utilized to identify the features that contributed most to the classification decision with XGBoost, demonstrating the high impact of auxiliary inputs such as age and sex. This paper demonstrates the promising application of explainable machine learning in the field of cardiovascular disease prediction.

State-of-the-art machine learning (ML) artificial intelligence methods are increasingly leveraged in clinical predictive modeling to provide clinical decision support systems to physicians. Modern ML approaches such as artificial neural networks (ANNs) and tree boosting often perform better than more traditional methods like logistic regression. On the other hand, these modern methods yield a limited understanding of the resulting predictions. However, in the medical domain, understanding of applied models is essential, in particular, when informing clinical decision support. Thus, in recent years, interpretability methods for modern ML methods have emerged to potentially allow explainable predictions paired with high performance. To our knowledge, we present in this work the first explainability comparison of two modern ML methods, tree boosting and multilayer perceptrons (MLPs), to traditional logistic regression methods using a stroke outcome prediction paradigm. Here, we used clinical features to predict a dichotomized 90 days post-stroke modified Rankin Scale (mRS) score. For interpretability, we evaluated clinical features' importance with regard to predictions using deep Taylor decomposition for MLP, Shapley values for tree boosting and model coefficients for logistic regression. With regard to performance as measured by Area under the Curve (AUC) values on the test dataset, all models performed comparably: Logistic regression AUCs were 0.83, 0.83, 0.81 for three different regularization schemes; tree boosting AUC was 0.81; MLP AUC was 0.83. Importantly, the interpretability analysis demonstrated consistent results across models by rating age and stroke severity consecutively amongst the most important predictive features. For less important features, some differences were observed between the methods. Our analysis suggests that modern machine learning methods can provide explainability which is compatible with domain knowledge interpretation and traditional method rankings. Future work should focus on replication of these findings in other datasets and further testing of different explainability methods.

Abstract. Soil moisture plays a crucial role in the hydrological cycle, but accurately predicting soil moisture presents challenges due to the nonlinearity of soil water transport and the variability of boundary conditions. Deep learning has emerged as a promising approach for simulating soil moisture dynamics. In this study, we explore 10 different network structures to uncover their data utilization mechanisms and to maximize the potential of deep learning for soil moisture prediction, including three basic feature extractors and seven diverse hybrid structures, six of which are applied to soil moisture prediction for the first time. We compare the predictive abilities and computational costs of the models across different soil textures and depths systematically. Furthermore, we exploit the interpretability of the models to gain insights into their workings and attempt to advance our understanding of deep learning in soil moisture dynamics. For soil moisture forecasting, our results demonstrate that the temporal modeling capability of long short-term memory (LSTM) is well suited. Furthermore, the improved accuracy achieved by feature attention LSTM (FA-LSTM) and the generative-adversarial-network-based LSTM (GAN-LSTM), along with the Shapley (SHAP) additive explanations analysis, help us discover the effectiveness of attention mechanisms and the benefits of adversarial training in feature extraction. These findings provide effective network design principles. The Shapley values also reveal varying data leveraging approaches among different models. The t-distributed stochastic neighbor embedding (t-SNE) visualization illustrates differences in encoded features across models. In summary, our comprehensive study provides insights into soil moisture prediction and highlights the importance of the appropriate model design for specific soil moisture prediction tasks. We also hope this work serves as a reference for deep learning studies in other hydrology problems. The codes of 3 machine learning and 10 deep learning models are open source.

The ability to explain in understandable terms, why a machine learning model makes a certain prediction is becoming immensely important, as it ensures trust and transparency in the decision process of the model. Complex models, such as ensemble or deep learning models, are hard to interpret. Various methods have been proposed that deal with this matter. Shapley values provide accurate explanations, as they assign each feature an importance value for a particular prediction. However, the exponential complexity of their calculation is dealt efficiently only in decision tree-based models. Another method is surrogate models, which emulate a black-box model's behavior and provide explanations effortlessly, since they are constructed to be interpretable. Surrogate models are model-agnostic, but they produce only approximate explanations, which cannot always be trusted. We propose a method that combines these two approaches, so that we can take advantage of the model-agnostic part of the surrogate models, as well as the explanatory power of the Shapley values. We introduce a new metric, Top <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">j</sub> Similarity, that measures the similitude of two given explanations, produced by Shapley values, in order to evaluate our work. Finally, we recommend ways on how this method could be improved further.

Polycystic Ovary Syndrome (PCOS) is a complex disorder predominantly defined by biochemical hyperandrogenism, oligomenorrhea, anovulation, and in some cases, the presence of ovarian microcysts. This endocrinopathy inhibits ovarian follicle development causing symptoms like obesity, acne, infertility, and hirsutism. Artificial Intelligence (AI) has revolutionized healthcare, contributing remarkably to science and engineering domains. Therefore, we have demonstrated an AI approach using heterogeneous Machine Learning (ML) and Deep Learning (DL) classifiers to predict PCOS among fertile patients. We used an Open-source dataset of 541 patients from Kerala, India. Among all the classifiers, the final multi-stack of ML models performed best with accuracy, precision, recall, and F1-score of 98%, 97%, 98%, and 98%. Explainable AI (XAI) techniques make model predictions understandable, interpretable, and trustworthy. Hence, we have utilized XAI techniques such as SHAP (SHapley Additive Values), LIME (Local Interpretable Model Explainer), ELI5, Qlattice, and feature importance with Random Forest for explaining tree-based classifiers. The motivation of this study is to accurately detect PCOS in patients while simultaneously proposing an automated screening architecture with explainable machine learning tools to assist medical professionals in decision-making.

No abstract

To explore a robust tool for advancing digital breeding practices through an artificial intelligence-driven phenotype prediction expert system, we undertook a thorough analysis of 11 non-linear regression models. Our investigation specifically emphasized the significance of Support Vector Regression (SVR) and SHapley Additive exPlanations (SHAP) in predicting soybean branching. By using branching data (phenotype) of 1918 soybean accessions and 42 k SNP (Single Nucleotide Polymorphism) polymorphic data (genotype), this study systematically compared 11 non-linear regression AI models, including four deep learning models (DBN (deep belief network) regression, ANN (artificial neural network) regression, Autoencoders regression, and MLP (multilayer perceptron) regression) and seven machine learning models (e.g., SVR (support vector regression), XGBoost (eXtreme Gradient Boosting) regression, Random Forest regression, LightGBM regression, GPs (Gaussian processes) regression, Decision Tree regression, and Polynomial regression). After being evaluated by four valuation metrics: R² (R-squared), MAE (Mean Absolute Error), MSE (Mean Squared Error), and MAPE (Mean Absolute Percentage Error), it was found that the SVR, Polynomial Regression, DBN, and Autoencoder outperformed other models and could obtain a better prediction accuracy when they were used for phenotype prediction. In the assessment of deep learning approaches, we exemplified the SVR model, conducting analyses on feature importance and gene ontology (GO) enrichment to provide comprehensive support. After comprehensively comparing four feature importance algorithms, no notable distinction was observed in the feature importance ranking scores across the four algorithms, namely Variable Ranking, Permutation, SHAP, and Correlation Matrix, but the SHAP value could provide rich information on genes with negative contributions, and SHAP importance was chosen for feature selection. The results of this study offer valuable insights into AI-mediated plant breeding, addressing challenges faced by traditional breeding programs. The method developed has broad applicability in phenotype prediction, minor QTL (quantitative trait loci) mining, and plant smart-breeding systems, contributing significantly to the advancement of AI-based breeding practices and transitioning from experience-based to data-based breeding.

No abstract

Deep eutectic solvents (DESs) are emerging as novel green solvents for the processes of mass transport and heat transfer, in which the viscosity of DESs is important for their industrial applications. However, for DESs, the measurement of viscosity is time-consuming, and there are many factors influencing the viscosity, which impedes their wider application. This study aims to develop a data-driven model which could accurately and rapidly predict the viscosity of diverse DESs at different temperatures, and furthermore boost the design and screening of novel DESs. In this work, we collected 107 DESs with 994 experimental values of viscosity from published works. Given the significant effect of water on viscosity, the water content of each collected DES was labeled. The Morgan fingerprint was first employed as a feature to describe the chemical environment of DESs. And four machine learning algorithms were used to train models: support vector regression (SVR), random forest (RF), neural network (NN), and extreme gradient boosting (XGBoost), and XGBoost showed the best predictive performance. In combination with the powerful interpretation method SHapley Additive exPlanation (SHAP), we further revealed the positive or negative effect of features on viscosity. Overall, this work provides a machine learning model which could predict viscosity precisely and facilitate the design and application of DESs.

Kidney abnormality is one of the major concerns in modern society, and it affects millions of people around the world. To diagnose different abnormalities in human kidneys, a narrow-beam x-ray imaging procedure, computed tomography, is used, which creates cross-sectional slices of the kidneys. Several deep-learning models have been successfully applied to computer tomography images for classification and segmentation purposes. However, it has been difficult for clinicians to interpret the model’s specific decisions and, thus, creating a “black box” system. Additionally, it has been difficult to integrate complex deep-learning models for internet-of-medical-things devices due to demanding training parameters and memory-resource cost. To overcome these issues, this study proposed (1) a lightweight customized convolutional neural network to detect kidney cysts, stones, and tumors and (2) understandable AI Shapely values based on the Shapley additive explanation and predictive results based on the local interpretable model-agnostic explanations to illustrate the deep-learning model. The proposed CNN model performed better than other state-of-the-art methods and obtained an accuracy of 99.52 ± 0.84% for K = 10-fold of stratified sampling. With improved results and better interpretive power, the proposed work provides clinicians with conclusive and understandable results.

No abstract

Machine learning (ML) and Deep Learning (DL) methods are being adopted rapidly, especially in computer network security, such as fraud detection, network anomaly detection, intrusion detection, and much more. However, the lack of transparency of ML and DL based models is a major obstacle to their implementation and criticized due to its black-box nature, even with such tremendous results. Explainable Artificial Intelligence (XAI) is a promising area that can improve the trustworthiness of these models by giving explanations and interpreting its output. If the internal working of the ML and DL based models is understandable, then it can further help to improve its performance. The objective of this paper is to show that how XAI can be used to interpret the results of the DL model, the autoencoder in this case. And, based on the interpretation, we improved its performance for computer network anomaly detection. The kernel SHAP method, which is based on the shapley values, is used as a novel feature selection technique. This method is used to identify only those features that are actually causing the anomalous behaviour of the set of attack/anomaly instances. Later, these feature sets are used to train and validate the autoencoderbut on benign data only. Finally, the built SHAP_Model outperformed the other two models proposed based on the feature selection method. This whole experiment is conducted on the subset of the latest CICIDS2017 network dataset. The overall accuracy and AUC of SHAP_Model is 94% and 0.969, respectively.

The application of artificial intelligence (AI) system is more and more extensive, using the explainable AI (XAI) technology to explain why machine learning (ML) models make certain predictions as important as the accuracy of the predictions, because it ensures the trust and transparency in the model decision-making process. For deep reinforcement learning (DRL) model, although some outstanding progress based on DRL has been made in many fields, it is difficult to explain and cannot be used in safety related occasions. Especially in power system, for the power system emergency control based on DRL, how to provide an intuitive and reliable XAI technology is urgent and necessary. The Shapley additive explanations (SHAP) method has been adopted to provide a reasonable interpretable model for an open-source platform named Reinforcement Learning for Grid Control (RLGC). Through a series of summary plots, force plots and probability of SHAP value, the under-voltage load shedding of power system based on DRL can be interpreted much easier and clearer. More importantly, this work is unique in the power system field, presenting the first use of the SHAP method and the probability of SHAP value to give explanations for emergency control based on DRL in power system.

Machine learning (ML) models are becoming increasingly complex. In fact, a sophisticated model (XGBoost boosting or deep learning) generally leads to more accurate predictions than a simple model (linear regression or decision tree). There is therefore a trade-off between the performance of a model and its interpretability: what a model gains in performance, it loses in interpretability (and vice versa), where interpretability is the ability for a human to understand the reasons for a model's decision. However, explaining the predictions made by machine learning models aims at computing and interpreting the importance of features. To achieve this, game theory has recently gained attention for better understanding the similarity between group members. In this paper, we use SHAP (SHapley Additive exPlanations), which is a method based on cooperative game theory, to analyze and evaluate the properties of each group. More importantly, we rely k-means PCA and Light gbm classifier to improve the data preparation before grouping the features into multiple clusters. The simulation results prove the importance of shapley value in creating an accurate and meaningful representation of each cluster.

Machine learning (ML) models have been shown to predict the presence of clinical factors from medical imaging with remarkable accuracy. However, these complex models can be difficult to interpret and are often criticized as “black boxes”. Prediction models that provide no insight into how their predictions are obtained are difficult to trust for making important clinical decisions, such as medical diagnoses or treatment. Explainable machine learning (XML) methods, such as Shapley values, have made it possible to explain the behavior of ML algorithms and to identify which predictors contribute most to a prediction. Incorporating XML methods into medical software tools has the potential to increase trust in ML-powered predictions and aid physicians in making medical decisions. Specifically, in the field of medical imaging analysis the most used methods for explaining deep learning-based model predictions are saliency maps that highlight important areas of an image. However, they do not provide a straightforward interpretation of which qualities of an image area are important. Here, we describe a novel pipeline for XML imaging that uses radiomics data and Shapley values as tools to explain outcome predictions from complex prediction models built with medical imaging with well-defined predictors. We present a visualization of XML imaging results in a clinician-focused dashboard that can be generalized to various settings. We demonstrate the use of this workflow for developing and explaining a prediction model using MRI data from glioma patients to predict a genetic mutation.

No abstract

No abstract

Understanding the flow of information in Deep Neural Networks (DNNs) is a challenging problem that has gain increasing attention over the last few years.While several methods have been proposed to explain network predictions, there have been only a few attempts to compare them from a theoretical perspective.What is more, no exhaustive empirical comparison has been performed in the past.In this work, we analyze four gradient-based attribution methods and formally prove conditions of equivalence and approximation between them.By reformulating two of these methods, we construct a unified framework which enables a direct comparison, as well as an easier implementation.Finally, we propose a novel evaluation metric, called Sensitivity-n and test the gradient-based attribution methods alongside with a simple perturbation-based attribution method on several datasets in the domains of image and text classification, using various network architectures.

Acute critical illness is often preceded by deterioration of routinely measured clinical parameters, e.g., blood pressure and heart rate. Early clinical prediction is typically based on manually calculated screening metrics that simply weigh these parameters, such as early warning scores (EWS). The predictive performance of EWSs yields a tradeoff between sensitivity and specificity that can lead to negative outcomes for the patient. Previous work on electronic health records (EHR) trained artificial intelligence (AI) systems offers promising results with high levels of predictive performance in relation to the early, real-time prediction of acute critical illness. However, without insight into the complex decisions by such system, clinical translation is hindered. Here, we present an explainable AI early warning score (xAI-EWS) system for early detection of acute critical illness. xAI-EWS potentiates clinical translation by accompanying a prediction with information on the EHR data explaining it.

No abstract

Local feature attribution methods are increasingly used to explain complex machine learning models. However, current methods are limited because they are extremely expensive to compute or are not capable of explaining a distributed series of models where each model is owned by a separate institution. The latter is particularly important because it often arises in finance where explanations are mandated. Here, we present Generalized DeepSHAP (G-DeepSHAP), a tractable method to propagate local feature attributions through complex series of models based on a connection to the Shapley value. We evaluate G-DeepSHAP across biological, health, and financial datasets to show that it provides equally salient explanations an order of magnitude faster than existing model-agnostic attribution techniques and demonstrate its use in an important distributed series of models setting.

An interpretable deep learning framework for land use and land cover classification (LULC) in remote sensing using SHAP is introduced. It utilizes a compact CNN model for the classification of satellite images and then feeds the results to a SHAP deep explainer so as to strengthen the classification results. The proposed framework is applied to Sentinel-2 satellite images containing 27000 images of pixel size 64 × 64 and operates on three-band combinations, reducing the model’s input data by 77% considering that 13 channels are available, while at the same time investigating on how different spectrum bands affect predictions on the dataset’s classes. Experimental results on the EuroSAT dataset demonstrate the CNN’s accurate classification with an overall accuracy of 94.72%, whereas the classification accuracy on three-band combinations on each of the dataset’s classes highlights its improvement when compared to standard approaches with larger number of trainable parameters. The SHAP explainable results of the proposed framework shield the network’s predictions by showing correlation values that are relevant to the predicted class, thereby improving the classifications occurring in urban and rural areas with different land uses in the same scene.

Concrete mix design plays a pivotal role in ensuring the mechanical performance, durability, and sustainability of construction projects. However, the nonlinear interactions among the mix components challenge traditional approaches in predicting compressive strength and optimizing proportions. This study presents a two-stage hybrid framework that integrates deep learning with reinforcement learning to overcome these limitations. First, a Convolutional Neural Network–Long Short-Term Memory (CNN–LSTM) model was developed to capture spatial–temporal patterns from a dataset of 1030 historical concrete samples. The extracted features were enhanced using an eXtreme Gradient Boosting (XGBoost) meta-model to improve generalizability and noise resistance. Then, a Dueling Double Deep Q-Network (Dueling DDQN) agent was used to iteratively identify optimal mix ratios that maximize the predicted compressive strength. The proposed framework outperformed ten benchmark models, achieving an MAE of 2.97, RMSE of 4.08, and R2 of 0.94. Feature attribution methods—including SHapley Additive exPlanations (SHAP), Elasticity-Based Feature Importance (EFI), and Permutation Feature Importance (PFI)—highlighted the dominant influence of cement content and curing age, as well as revealing non-intuitive effects such as the compensatory role of superplasticizers in low-water mixtures. These findings demonstrate the potential of the proposed approach to support intelligent concrete mix design and real-time optimization in smart construction environments.

No abstract

Recent developments in machine learning have introduced models that approach human performance at the cost of increased architectural complexity. Efforts to make the rationales behind the models' predictions transparent have inspired an abundance of new explainability techniques. Provided with an already trained model, they compute saliency scores for the words of an input instance. However, there exists no definitive guide on (i) how to choose such a technique given a particular application task and model architecture, and (ii) the benefits and drawbacks of using each such technique. In this paper, we develop a comprehensive list of diagnostic properties for evaluating existing explainability techniques. We then employ the proposed list to compare a set of diverse explainability techniques on downstream text classification tasks and neural network architectures. We also compare the saliency scores assigned by the explainability techniques with human annotations of salient input regions to find relations between a model's performance and the agreement of its rationales with human ones. Overall, we find that the gradient-based explanations perform best across tasks and model architectures, and we present further insights into the properties of the reviewed explainability techniques.

No abstract

No abstract

No abstract

Automated fibre layup techniques are widely used in the aviation sector for the efficient production of composite components. However, the required manual inspection can take up to 50 % of the manufacturing time. The automated classification of fibre layup defects with Neural Networks potentially increases the inspection efficiency. However, the machine decision-making processes of such classifiers are difficult to verify. Hence, we present an approach for analysing the classification procedure of fibre layup defects. Therefore, we comprehensively evaluate 20 Explainable Artificial Intelligence methods from the literature. Accordingly, the techniques Smoothed Integrated Gradients, Guided Gradient Class Activation Mapping and DeepSHAP are applied to a Convolutional Neural Network classifier. These methods analyse the neural activations and robustness of a classifier for an unknown and manipulated input data. Our investigations show that especially Smoothed Integrated Gradients and DeepSHAP are well suited for the visualisation of such classifications. Additionally, maximum-sensitivity and infidelity calculations confirm this behaviour. In future, customers and developers could apply the presented methods for the certification of their inspection systems.

The rapid rise of non-communicable diseases (NCDs) becomes one of the serious health issues and the leading cause of death worldwide. In recent years, artificial intelligence-based systems have been developed to assist clinicians in decision-making to reduce morbidity and mortality. However, a common drawback of these modern studies is related to explanations of their output. In other words, understanding the inner logic behind the predictions is hidden to the end-user. Thus, clinicians struggle to interpret these models because of their black-box nature, and hence they are not acceptable in the medical practice. To address this problem, we have proposed a Deep Shapley Additive Explanations (DeepSHAP) based deep neural network framework equipped with a feature selection technique for NCDs prediction and explanation among the population in the United States. Our proposed framework comprises three components: First, representative features are done based on the elastic net-based embedded feature selection technique; second a deep neural network classifier is tuned with the hyper-parameters and used to train the model with the selected feature subset; third, two kinds of model explanation are provided by the DeepSHAP approach. Herein, (I) explaining the risk factors that affected the model’s prediction from the population-based perspective; (II) aiming to explain a single instance from the human-centered perspective. The experimental results indicated that the proposed model outperforms various state-of-the-art models. In addition, the proposed model can improve the medical understanding of NCDs diagnosis by providing general insights into the changes in disease risk at the global and local levels. Consequently, DeepSHAP based explainable deep learning framework contributes not only to the medical decision support systems but also can provide to real-world needs in other domains.

A recent trend in IR has been the usage of neural networks to learn retrieval models for text based adhoc search. While various approaches and architectures have yielded significantly better performance than traditional retrieval models such as BM25, it is still difficult to understand exactly why a document is relevant to a query. In the ML community several approaches for explaining decisions made by deep neural networks have been proposed -- including DeepSHAP which modifies the DeepLift algorithm to estimate the relative importance (shapley values) of input features for a given decision by comparing the activations in the network for a given image against the activations caused by a reference input. In image classification, the reference input tends to be a plain black image. While DeepSHAP has been well studied for image classification tasks, it remains to be seen how we can adapt it to explain the output of Neural Retrieval Models (NRMs). In particular, what is a good "black" image in the context of IR? In this paper we explored various reference input document construction techniques. Additionally, we compared the explanations generated by DeepSHAP to LIME (a model agnostic approach) and found that the explanations differ considerably. Our study raises concerns regarding the robustness and accuracy of explanations produced for NRMs. With this paper we aim to shed light on interesting problems surrounding interpretability in NRMs and highlight areas of future work.

Abstract Many Machine Learning (ML) systems, especially deep neural networks, are fundamentally regarded as black boxes since it is difficult to fully grasp how they function once they have been trained. Here, we tackle the issue of the interpretability of a high‐accuracy ML model created to model the flux of Earth's radiation belt electrons. The Outer RadIation belt Electron Neural net (ORIENT) model uses only solar wind conditions and geomagnetic indices as input features. Using the Deep SHAPley additive explanations (DeepSHAP) method, for the first time, we show that the “black box” ORIENT model can be successfully explained. Two significant electron flux enhancement events observed by Van Allen Probes during the storm interval of 17–18 March 2013 and non‐storm interval of 19–20 September 2013 are investigated using the DeepSHAP method. The results show that the feature importance calculated from the purely data‐driven ORIENT model identifies physically meaningful behavior consistent with current physical understanding. This work not only demonstrates that the physics of the radiation belt was captured in the training of our previous model, but that this method can also be applied generally to other similar models to better explain the results and to potentially discover new physical mechanisms.

This work aims at making the prediction process of neural network-based turbulence models more transparent. Due to its black-box ingredients, the model’s predictions cannot be anticipated. Therefore, this paper is concerned with the quantification of each feature’s importance for the prediction of trained and fixed NNs, which is one possibly type of explanation for opaque models. Two conceptually different attribution methods, namely permutation feature importance and DeepSHAP, are chosen in order to assess global, regional and local feature importance. The neuralSST turbulence model, which serves as an example, will be investigated in greater detail. While the global importance scores provide a quick and reliable way to detect irrelevant features and may thus be used for feature selection, only the (semi-)local analysis provides meaningful and trustworthy interpretations of the model. In fact, the local importance scores suggest that hypotheses with a common high-level influence on the turbulence model, e.g. adjusting the net production of turbulent kinetic energy or the Reynolds stress anisotropy, are similarly affected by local mean flow structures such as attached boundary layers, free shear layers or recirculation zones.

Automated fibre layup techniques are commonly used composite manufacturing processes in the aviation sector and require a manual visual inspection. Neural Network classification of defects has the potential to automate this visual inspection, however, the machine decision-making processes are hard to verify. Thus, we present an approach for visualising Convolutional Neural Network (CNN) based classifications of manufacturing defects and quantifying its robustness. Our investigations have shown that especially Smoothed Integrated Gradients and DeepSHAP are particularly well suited for the visualisation of CNN classifications. The Smoothed Integrated Gradients technique also reveals advantages in robustness when evaluating degraded input images.

The prediction and identification of key factors in road traffic accidents are crucial for accident prevention, yet previous studies have often examined these aspects separately. To comprehensively assess the risk level of road traffic accidents and their key determinants, this paper proposes a comprehensive forecasting and analysis framework that offers a novel perspective for identifying key risk factors from a modeling standpoint compared to existing methods. The CNN-BiLSTM-Attention model was developed for predicting the risk value of road accidents, and DeepSHAP was employed to interpret the model and extract the key factors contributing to traffic accidents. This deep learning framework combines convolutional neural networks (CNN) and Bi-directional long short-term memory (BiLSTM), while incorporating a spatial-temporal local attention mechanism to enhance its capability in capturing spatiotemporal features. Through analysis and experimentation on real-world datasets, our model demonstrates superior accuracy in predicting traffic accident risk compared to the benchmark model, achieving a Mean Absolute Error (MAE) of 0.2475 on the UK dataset and 0.2683 on the US dataset. The results obtained from DeepSHAP were found to be more rational and informative in identifying key factors of different severity levels using four methods. To verify the rationality and stability of obtaining these key factors, the first 15 factors were reintegrated into the prediction model, resulting in almost unchanged accuracy and reduced model iteration time. By improving the influential factors, road traffic accidents can be effectively mitigated.

No abstract

We show new connections between adversarial learning and explainability for deep neural networks (DNNs). One form of explanation of the output of a neural network model in terms of its input features, is a vector of feature-attributions. Two desirable characteristics of an attribution-based explanation are: (1) $\textit{sparseness}$: the attributions of irrelevant or weakly relevant features should be negligible, thus resulting in $\textit{concise}$ explanations in terms of the significant features, and (2) $\textit{stability}$: it should not vary significantly within a small local neighborhood of the input. Our first contribution is a theoretical exploration of how these two properties (when using attributions based on Integrated Gradients, or IG) are related to adversarial training, for a class of 1-layer networks (which includes logistic regression models for binary and multi-class classification); for these networks we show that (a) adversarial training using an $\ell_\infty$-bounded adversary produces models with sparse attribution vectors, and (b) natural model-training while encouraging stable explanations (via an extra term in the loss function), is equivalent to adversarial training. Our second contribution is an empirical verification of phenomenon (a), which we show, somewhat surprisingly, occurs $\textit{not only}$ $\textit{in 1-layer networks}$, $\textit{but also DNNs}$ $\textit{trained on }$ $\textit{standard image datasets}$, and extends beyond IG-based attributions, to those based on DeepSHAP: adversarial training with $\ell_\infty$-bounded perturbations yields significantly sparser attribution vectors, with little degradation in performance on natural test data, compared to natural training. Moreover, the sparseness of the attribution vectors is significantly better than that achievable via $\ell_1$-regularized natural training.

Post-transcriptionally RNA modifications, also known as the epitranscriptome, play crucial roles in the regulation of gene expression during development. Recently, deep learning (DL) has been employed for RNA modification site prediction and has shown promising results. However, due to the lack of relevant studies, it is unclear which DL architecture is best suited for some pyrimidine modifications, such as 5-methyluridine (m⁵U). To fill this knowledge gap, we first performed a comparative evaluation of various commonly used DL models for epigenetic studies with the help of autoBioSeqpy. We identified optimal architectural variations for m⁵U site classification, optimizing the layer depth and neuron width. Second, we used this knowledge to develop Deepm5U, an improved convolutional-recurrent neural network that accurately predicts m⁵U sites from RNA sequences. We successfully applied Deepm5U to transcriptomewide m⁵U profiling data across different sequencing technologies and cell types. Third, we showed that the techniques for interpreting deep neural networks, including LayerUMAP and DeepSHAP, can provide important insights into the internal operation and behavior of models. Overall, we offered practical guidance for the development, benchmark, and analysis of deep learning models when designing new algorithms for RNA modifications.

Accurate prediction of wind power generation in regions characterised by complex terrain is a critical gap in renewable energy research. To address this challenge, the present study articulates a novel methodological framework using Convolutional Neural Networks (CNNs) to improve wind power forecasting in such geographically diverse areas. The core research question is to investigate the extent to which terrain complexity affects forecast accuracy. To this end, DeepSHAP—an advanced interpretability technique—is used to dissect the CNN model and identify the most significant features of the weather forecast grid that have the greatest impact on forecast accuracy. Our results show a clear correlation between certain topographical features and forecast accuracy, demonstrating that complex terrain features are an important part of the forecasting process. The study’s findings support the hypothesis that a detailed understanding of terrain features, facilitated by model interpretability, is essential for improving wind energy forecasts. Consequently, this research addresses an important gap by clarifying the influence of complex terrain on wind energy forecasting and provides a strategic pathway for more efficient use of wind resources, thereby supporting the wider adoption of wind energy as a sustainable energy source, even in regions with complex terrain.

Background There is a need for diagnostic tests of early PD diagnosis. A subset of AI known as deep learning (DL) has shown great promise in diagnostic medical imaging, sometimes outperforming radiolo- gists by detecting patterns invisible to the human eye. Using DL, we explored whether such changes are detectable on routine PD MRI scans. Methods We trained a convolutional neural network to classify 138 PD and 60 control brain MRI images acquired from the Parkinson’s Progression Marker Initiative (PPMI) database. Models were assessed using k-fold cross-validation. We used Deep SHapley Additive exPlanations (DeepSHAP) to visualise the contri- bution of individual pixels to the model’s prediction. Results A combined dataset of axial T2 and proton density MRI images was classified with 79% accuracy and an area under the curve (AUC) of 0.86. Respectively T2 and proton density models classified cases with 81/84% accuracy and AUC of 0.83/0.88. DeepSHAP heat maps demonstrated predominant interest in midbrain slices. Conclusion Our models exhibited good diagnostic performance and demonstrated interest in PD relevant brain regions. We will validate this model in a large dataset of routinely collected NHS MRI scans, many of which predate onset of motor symptoms.

Deep neural networks have demonstrated exceptional performance breakthroughs in the field of document image classification; yet, there has been limited research in the field that delves into the explainability of these models. In this paper, we present a comprehensive study in which we analyze 9 different explainability methods across 10 different state-of-the-art document classification models and 2 popular benchmark datasets, RVL-CDIP and Tobacco3482, making three major contributions. First, through an exhaustive qualitative and quantitative analysis of various explainability approaches, we demonstrate that the majority of them perform poorly in generating useful explanations for document images. Only two techniques, Occlusion and DeepSHAP, provide relatively faithful explanations, with DeepSHAP additionally offering better interpretability while Occlusion falls short in this regard. Second, to pinpoint the features most crucial to the models’ predictions, we present an approach for generating counterfactual explanations, the analysis of which reveals that many document classification models can be highly susceptible to minor perturbations in the input. Additionally, it suggests that these models may easily fall victim to biases in the document data, ultimately relying on seemingly irrelevant features to make their decisions. Specifically, on the RVL-CDIP dataset, we show that 25-50% of the overall model predictions and up to 60% of predictions for some classes were strongly dependent on these irrelevant features. Lastly, our analysis reveals that the popular document benchmark datasets, RVL-CDIP and Tobacco3482, are inherently biased, with document identification (ID) numbers of specific styles consistently appearing in certain document regions. If unaddressed, this bias allows the models to predict document classes solely by looking at the ID numbers and prevents them from learning more complex document features. Overall, by unveiling the strengths and weaknesses of various explainability methods, document datasets and deep learning models, our work presents a major step towards creating more transparent and robust AI-powered document image classification systems.

No abstract

Automated essay scoring (AES) is a compelling topic in Learning Analytics for the primary reason that recent advances in AI find it as a good testbed to explore artificial supplementation of human creativity. However, a vast swath of research tackles AES only holistically; few have even developed AES models at the rubric level, the very first layer of explanation underlying the prediction of holistic scores. Consequently, the AES black box has remained impenetrable. Although several algorithms from Explainable Artificial Intelligence have recently been published, no research has yet investigated the role that these explanation models can play in a) discovering the decision-making process that drives AES, b) fine-tuning predictive models to improve generalizability and interpretability, and c) providing personalized, formative, and fine-grained feedback to students during the writing process. Building on previous studies where models were trained to predict both the holistic and rubric scores of essays, using the Automated Student Assessment Prize's essay datasets, this study focuses on predicting the quality of the writing style of Grade-7 essays and exposes the decision processes that lead to these predictions. In doing so, it evaluates the impact of deep learning (multi-layer perceptron neural networks) on the performance of AES. It has been found that the effect of deep learning can be best viewed when assessing the trustworthiness of explanation models. As more hidden layers were added to the neural network, the descriptive accuracy increased by about 10%. This study shows that faster (up to three orders of magnitude) SHAP implementations are as accurate as the slower model-agnostic one. It leverages the state-of-the-art in natural language processing, applying feature selection on a pool of 1592 linguistic indices that measure aspects of text cohesion, lexical diversity, lexical sophistication, and syntactic sophistication and complexity. In addition to the list of most globally important features, this study reports a) a list of features that are important for a specific essay (locally), b) a range of values for each feature that contribute to higher or lower rubric scores, and c) a model that allows to quantify the impact of the implementation of formative feedback.

No abstract

In the neurological intensive care setting implementing tier-based therapy, osmotic therapy serves as an intermediate approach for reducing intracranial pressure (ICP). However, discussions regarding quantitative and specific criteria for the maintenance and transition of treatment remain limited. This study proposes an explainable neural network forecasting of physiological responses after osmotic therapy administration for the purpose of treatment guidance. Recordings of arterial blood pressure and intracranial pressure were obtained from 107 patients with traumatic brain injury. To maintain adequate intracranial pressure, arterial blood pressure, and cerebral perfusion pressure, 20 % mannitol (100 ml; no. infusion, 3,571) and 11.3 % hypertonic saline (40 ml; no. infusion, 574) were administered. The LTSF-NLinear model was used to predict the target signal response for the subsequent 1 h. Six prediction models were derived by combining two agents (mannitol or hypertonic saline) and three target signals (mean arterial blood pressure, mean intracranial pressure and cerebral perfusion pressure). To validate the model’s robustness while considering patient heterogeneity, group (5)-fold cross-validation was implemented. To assess the predictive ability of the proposed model for adverse clinical events, binary classification of hypotension, intracranial hypertension, and cerebral hypoperfusion was performed. The overall function of the model was elucidated using linear weight visualization. Additionally, the DeepSHAP application allows for a detailed investigation of the prediction process for a single case. The relationship between response to osmotic therapy and patient outcomes was statistically significant. The proposed models achieved reasonable performance, with R2 score > 0.8 for physiological signal predictions and accuracy > 0.9 for event predictions. Through global explanation, the recent values of mean arterial blood pressure, mean intracranial pressure, and cerebral perfusion pressure of input signals significantly influenced physiological responses after osmotic treatment. In the context of local explanation (a single case), it was possible to determine how physiological state influenced the response. The proposed model is expected to provide objective and quantitative information on osmotherapy in neurological intensive care environments, enabling guidance for alternative and aggressive treatments. Thus, it can potentially aid in enhancing the prognosis of patients with traumatic brain injury and improving the clinical workflow. Moreover, the global and local explanations of the proposed model can provide valuable insights for future researchers.

Electroencephalography (EEG)-based auditory attention detection (AAD) offers a non-invasive way to enhance hearing aids, but conventional methods rely on too many electrodes, limiting wearability and comfort. This paper presents SHAP-AAD, a two-stage framework that combines DeepSHAP-based channel selection with a lightweight temporal convolutional network (TCN) for efficient AAD using fewer channels.DeepSHAP, an explainable AI technique, is applied to a Convolutional Neural Network (CNN) trained on topographic alpha-power maps to rank channel importance, and the top-k EEG channels are used to train a compact TCN. Experiments on the DTU dataset show that using 32 channels yields comparable accuracy to the full 64-channel setup (79.21% vs. 81.06%) on average. In some cases, even 8 channels can deliver satisfactory accuracy. These results demonstrate the effectiveness of SHAP-AAD in reducing complexity while preserving high detection performance.

In this diagnostic study, an automated ROP screening platform was able to identify and classify multidimensional pathologic lesions in the retinal images. This platform may be able to assist routine ROP screening in general and children hospitals.

Substantial progress in spoofing and deepfake detection has been made in recent years. Nonetheless, the community has yet to make notable inroads in providing an explanation for how a classifier produces its output. The dominance of black box spoofing detection solutions is at further odds with the drive toward trustworthy, explainable artificial intelligence. This paper describes our use of SHapley Additive exPlanations (SHAP) to gain new insights in spoofing detection. We demonstrate use of the tool in revealing unexpected classifier behaviour, the artefacts that contribute most to classifier outputs and differences in the behaviour of competing spoofing detection models. The tool is both efficient and flexible, being readily applicable to a host of different architecture models in addition to related, different applications. All results reported in the paper are reproducible using open-source software.

Data innovation has advanced rapidly in recent years, and the network media has undergone several problematic changes. Places where consumers can express their thoughts through messages, photos, and notes, such as Facebook, Twitter, and Instagram, are gaining popularity. Unfortunately, it has become a place of toxic, insults, cyberbullying, and mysterious dangers. There is a lot of research here, but none has found a sufficient level of accuracy. This paper proposes a Convolutional Neural Network with Long Short-Term Memory (CNN-LSTM) and Natural Language Processing (NLP) fusion strategy that characterizes malicious and non-malicious remarks with a word embedding technique at an initial stage. And this model can categorize any voice data into six levels of classification. Furthermore, the processed dataset is applied to two traditional Machine Learning Algorithms (Random Forest and Extra Tress Algorithm) with an estimator (Logistic Regression) and interprets these algorithms with an Explainable AI (XAI)-SHAP. In the final step, two classifiers and the estimator are ensembled with Stacking Classifier, which is better than any previous activity.

Mulberry leaves feed Bombyx mori silkworms to generate silk thread. Diseases that affect mulberry leaves have reduced crop and silk yields in sericulture, which produces 90% of the world's raw silk. Manual leaf disease identification is tedious and error-prone. Computer vision can categorize leaf diseases early and overcome the challenges of manual identification. No mulberry leaf deep learning (DL) models have been reported. Therefore, in this study, two types of leaf diseases: leaf rust and leaf spot, with disease-free leaves, were collected from two regions of Bangladesh. Sericulture experts annotated the leaf images. The images were pre-processed, and 6,000 synthetic images were generated using typical image augmentation methods from the original 764 training images. Additional 218 and 109 images were employed for testing and validation respectively. In addition, a unique lightweight parallel depth-wise separable CNN model, PDS-CNN was developed by applying depth-wise separable convolutional layers to reduce parameters, layers, and size while boosting classification performance. Finally, the explainable capability of PDS-CNN is obtained through the use of SHapley Additive exPlanations (SHAP) evaluated by a sericulture specialist. The proposed PDS-CNN outperforms well-known deep transfer learning models, achieving an optimistic accuracy of 95.05 ± 2.86% for three-class classifications and 96.06 ± 3.01% for binary classifications with only 0.53 million parameters, 8 layers, and a size of 6.3 megabytes. Furthermore, when compared with other well-known transfer models, the proposed model identified mulberry leaf diseases with higher accuracy, fewer factors, fewer layers, and lower overall size. The visually expressive SHAP explanation images validate the models' findings aligning with the predictions made the sericulture specialist. Based on these findings, it is possible to conclude that the explainable AI (XAI)-based PDS-CNN can provide sericulture specialists with an effective tool for accurately categorizing mulberry leaves.

The latest Deep Learning (DL) models for detection and classification have achieved an unprecedented performance over classical machine learning algorithms. However, DL models are black-box methods hard to debug, interpret, and certify. DL alone cannot provide explanations that can be validated by a non technical audience such as end-users or domain experts. In contrast, symbolic AI systems that convert concepts into rules or symbols – such as knowledge graphs – are easier to explain. However, they present lower generalization and scaling capabilities. A very important challenge is to fuse DL representations with expert knowledge. One way to address this challenge, as well as the performance-explainability trade-off is by leveraging the best of both streams without obviating domain expert knowledge. In this paper, we tackle such problem by considering the symbolic knowledge is expressed in form of a domain expert knowledge graph. We present the eXplainable Neural-symbolic learning (X-NeSyL) methodology, designed to learn both symbolic and deep representations, together with an explainability metric to assess the level of alignment of machine and human expert explanations. The ultimate objective is to fuse DL representations with expert domain knowledge during the learning process so it serves as a sound basis for explainability. In particular, X-NeSyL methodology involves the concrete use of two notions of explanation, both at inference and training time respectively: (1) EXPLANet: Expert-aligned eXplainable Part-based cLAssifier NETwork Architecture, a compositional convolutional neural network that makes use of symbolic representations, and (2) SHAP-Backprop, an explainable AI-informed training procedure that corrects and guides the DL process to align with such symbolic representations in form of knowledge graphs. We showcase X-NeSyL methodology using MonuMAI dataset for monument facade image classification, and demonstrate that with our approach, it is possible to improve explainability at the same time as performance.

No abstract

No abstract

The demand for food delivery services (FDSs) during the COVID-19 crisis has been fuelled by consumers who prefer to order meals online and have it delivered to their door than to wait at a restaurant. Since many restaurants moved online and joined FDSs such as Uber Eats, Menulog, and Deliveroo, customer reviews on internet platforms have become a valuable source of information about a company's performance. FDS organisations strive to collect customer complaints and effectively utilise the information to identify improvements needed to enhance customer satisfaction. However, only a few customer opinions are addressed because of the large amount of customer feedback data and lack of customer service consultants. Organisations can use artificial intelligence (AI) instead of relying on customer service experts and find solutions on their own to save money as opposed to reading each review. Based on the literature, deep learning (DL) methods have shown remarkable results in obtaining better accuracy when working with large datasets in other domains, but lack explainability in their model. Rapid research on explainable AI (XAI) to explain predictions made by opaque models looks promising but remains to be explored in the FDS domain. This study conducted a sentiment analysis by comparing simple and hybrid DL techniques (LSTM, Bi-LSTM, Bi-GRU-LSTM-CNN) in the FDS domain and explained the predictions using SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME). The DL models were trained and tested on the customer review dataset extracted from the ProductReview website. Results showed that the LSTM, Bi-LSTM and Bi-GRU-LSTM-CNN models achieved an accuracy of 96.07%, 95.85% and 96.33%, respectively. The model should exhibit fewer false negatives because FDS organisations aim to identify and address each and every customer complaint. The LSTM model was chosen over the other two DL models, Bi-LSTM and Bi-GRU-LSTM-CNN, due to its lower rate of false negatives. XAI techniques, such as SHAP and LIME, revealed the feature contribution of the words used towards positive and negative sentiments, which were used to validate the model.

Detection of diabetic retinopathy (DR) as early as possible is vital in mitigating the complicated issues associated with the disease. Recent advances in artificial intelligence (AI), particularly deep learning (DL) techniques, have led to appreciable increase in the accuracy of predicting various disease classes. However, the challenge of AI models is the difficulty in providing insights into how and why a model arrives in attaining decision-making to facilitate trust and adoption in clinical settings. Therefore, this study aimed to enhance the detection rate of DR and explain the significant regions on the image for the model's overall performance. This study utilised Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) networks, Simple Recurrent Neural Networks (SRNN), and XGBoost in an ensemble model (EM). Specifically, Shapley Additive exPlanations (SHAP), a popular Explainable Artificial Intelligence (XAI) technique was utilised to identify and provide insights to which parts of the images features that contribute to the model's overall performance. After a series of experiments using the APTOS 2019 eye pack dataset collected from the Kaggle repository to evaluate the performance of CNN, LSTM, SRNN, and XGBoost. The EM outperformed all the other models with 95.63% accuracy, 97.79% precision, 93.64% recall rate, 98.79% F1-score and 97.75% AUC score. Also, SHAP analysis revealed significant regions on the image that influenced predictions, thus showing how important interpretability was for the model. The results imply that the ensemble DL, particularly with XGBoost, enhances the detection of DR, thereby improving the efficiency of screening tests and supporting personalised treatment plans in clinical practice through integrating these advanced models with XAI tools, creating trust towards automated diagnostic systems.

No abstract

The primary objective of this research is to create a reliable technique to determine whether a patient has glioma, a specific kind of brain tumour, by examining various diagnostic markers, using a variety of machine learning as well as deep learning approaches, and involving XAI (explainable artificial intelligence) methods. Through the integration of patient data, including medical records, genetic profiles, algorithms using machine learning have the ability to predict how each individual will react to different medical interventions. To guarantee regulatory compliance and inspire confidence in AI-driven healthcare solutions, XAI is incorporated. Machine learning methods employed in this study includes Random Forest, decision trees, logistic regression, KNN, Adaboost, SVM, Catboost, LGBM classifier, and Xgboost whereas the deep learning methods include ANN and CNN. Four alternative XAI strategies, including SHAP, Eli5, LIME, and QLattice algorithm, are employed to comprehend the predictions of the model. The Xgboost, a ML model achieved accuracy, precision, recall, f1 score, and AUC of 88%, 82%, 94%, 88%, and 92%, respectively. The best characteristics according to XAI techniques are IDH1, Age at diagnosis, PIK3CA, ATRX, PTEN, CIC, EGFR and TP53. By applying data analytic techniques, the objective is to provide healthcare professionals with practical tool that enhances their capacity for decision-making, enhances resource management, and ultimately raises the bar for patient care. Medical experts can customise treatments and improve patient outcomes by taking into account patient’s particular characteristics. XAI provides justifications to foster faith amongst patients and medical professionals who must rely on AI-assisted diagnosis and treatment recommendations.

No abstract

Although the detection procedure has been shown to be highly effective, there are several obstacles to overcome in the usage of AI-assisted cancer cell detection in clinical settings. These issues stem mostly from the failure to identify the underlying processes. Because AI-assisted diagnosis does not offer a clear decision-making process, doctors are dubious about it. In this instance, the advent of Explainable Artificial Intelligence (XAI), which offers explanations for prediction models, solves the AI black box issue. The SHapley Additive exPlanations (SHAP) approach, which results in the interpretation of model predictions, is the main emphasis of this work. The intermediate layer in this study was a hybrid model made up of three Convolutional Neural Networks (CNNs) (InceptionV3, InceptionResNetV2, and VGG16) that combined their predictions. The KvasirV2 dataset, which comprises pathological symptoms associated to cancer, was used to train the model. Our combined model yielded an accuracy of 93.17% and an F1 score of 97%. After training the combined model, we use SHAP to analyze images from these three groups to provide an explanation of the decision that affects the model prediction.

No abstract

Despite significant advancements in brain tumor classification, many existing models suffer from complex structures that make them difficult to interpret. This complexity can hinder the transparency of the decision-making process, causing models to rely on irrelevant features or normal soft tissues. Besides, these models often include additional layers and parameters, which further complicate the classification process. Our work addresses these limitations by introducing a novel methodology that combines Explainable AI (XAI) techniques with a Convolutional Neural Network (CNN) architecture. The major contribution of this paper is ensuring that the model focuses on the most relevant features for tumor detection and classification, while simultaneously reducing complexity, by minimizing the number of layers. This approach enhances the model's transparency and robustness, giving clear insights into its decision-making process through XAI techniques such as Gradient-weighted Class Activation Mapping (Grad-Cam), Shapley Additive explanations (Shap), and Local Interpretable Model-agnostic Explanations (LIME). Additionally, the approach demonstrates better performance, achieving 99% accuracy on seen data and 95% on unseen data, highlighting its generalizability and reliability. This balance of simplicity, interpretability, and high accuracy represents a significant advancement in the classification of brain tumor.

No abstract

No abstract

The rising prevalence of gastrointestinal (GI) tract disorders worldwide highlights the urgent need for precise diagnosis, as these diseases greatly affect human life and contribute to high mortality rates. Fast identification, accurate classification, and efficient treatment approaches are essential for addressing this critical health issue. Common side effects include abdominal pain, bloating, and discomfort, which can be chronic and debilitating. Nausea and vomiting are also frequent, leading to difficulties in maintaining adequate nutrition and hydration. The current study intends to develop a deep learning (DL)-based approach that automatically classifies GI tract diseases. For the first time, a GastroVision dataset with 8000 images of 27 different GI diseases was utilized in this work to design a computer-aided diagnosis (CAD) system. This study presents a novel lightweight feature extractor with a compact size and minimum number of layers named Parallel Depthwise Separable Convolutional Neural Network (PD-CNN) and a Pearson Correlation Coefficient (PCC) as the feature selector. Furthermore, a robust classifier named the Ensemble Extreme Learning Machine (EELM), combined with pseudo inverse ELM (ELM) and L1 Regularized ELM (RELM), has been proposed to identify diseases more precisely. A hybrid preprocessing technique, including scaling, normalization, and image enhancement techniques such as erosion, CLAHE, sharpening, and Gaussian filtering, are employed to enhance image representation and improve classification performance. The proposed approach consists of twenty-four layers and only 0.815 million parameters with a 9.79 MB model size. The proposed PD-CNN-PCC-EELM extracts essential features, reduces computational overhead, and achieves excellent classification performance on multiclass GI images. The PD-CNN-PCC-EELM achieved the highest precision, recall, f1, accuracy, ROC-AUC, and AUC-PR values of 88.12 ± 0.332 %, 87.75 ± 0.348 %, 87.12 ± 0.324 %, 87.75 %, 98.89 %, and 92 %, respectively, while maintaining a minimum testing time of 0.000001 s. A comparative study utilizes 10-fold cross-validation, ablation study and various state-of-the-art (SOTA) transfer learning (TL) models as feature extractors. Then, the PCC and EELM are integrated with TL to generate predictions, notably in terms of performance and real-time processing capability; the proposed model significantly outperforms the other models. Moreover, various explainable AI (XAI) methods, such as SHAP (Shapley Additive Explanations), heatmap, guided heatmap, Grad-Cam (Gradient-weighted Class Activation Mapping), guided Grad-CAM, and guided Saliency mapping, have been employed to explore the interpretability and decision-making capability of the proposed model. Therefore, the model provides practical intelligence for increasing confidence in diagnosing GI diseases in real-world scenarios.

The implementation of AI assisted cancer detection systems in clinical environments has faced numerous hurdles, mainly because of the restricted explainability of their elemental mechanisms, even though such detection systems have proven to be highly effective. Medical practitioners are skeptical about adopting AI assisted diagnoses as due to the latter's inability to be transparent about decision making processes. In this respect, explainable artificial intelligence (XAI) has emerged to provide explanations for model predictions, thereby overcoming the computational black box problem associated with AI systems. In this particular research, the focal point has been the exploration of the Shapley additive explanations (SHAP) and local interpretable model-agnostic explanations (LIME) approaches which enable model prediction explanations. This study used an ensemble model consisting of three convolutional neural networks(CNN): InceptionV3, InceptionResNetV2 and VGG16, which was based on averaging techniques and by combining their respective predictions. These models were trained on the Kvasir dataset, which consists of pathological findings related to gastrointestinal cancer. An accuracy of 96.89% and F1-scores of 96.877% were attained by our ensemble model. Following the training of the ensemble model, we employed SHAP and LIME to analyze images from the three classes, aiming to provide explanations regarding the deterministic features influencing the model's predictions. The results obtained from this analysis demonstrated a positive and encouraging advancement in the exploration of XAI approaches, specifically in the context of gastrointestinal cancer detection within the healthcare domain.

Optical Coherence Tomography (OCT) is an imperative symptomatic tool empowering the diagnosis of retinal diseases and anomalies. The manual decision towards those anomalies by specialists is the norm, but its labor-intensive nature calls for more proficient strategies. Consequently, the study recommends employing a Convolutional Neural Network (CNN) for the classification of OCT images derived from the OCT dataset into distinct categories, including Choroidal NeoVascularization (CNV), Diabetic Macular Edema (DME), Drusen, and Normal. The average k-fold (k = 10) training accuracy, test accuracy, validation accuracy, training loss, test loss, and validation loss values of the proposed model are 96.33%, 94.29%, 94.12%, 0.1073, 0.2002, and 0.1927, respectively. Fast Gradient Sign Method (FGSM) is employed to introduce non-random noise aligned with the cost function's data gradient, with varying epsilon values scaling the noise, and the model correctly handles all noise levels below 0.1 epsilon. Explainable AI algorithms: Local Interpretable Model-Agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP) are utilized to provide human interpretable explanations approximating the behaviour of the model within the region of a particular retinal image. Additionally, two supplementary datasets, namely, COVID-19 and Kidney Stone, are assimilated to enhance the model's robustness and versatility, resulting in a level of precision comparable to state-of-the-art methodologies. Incorporating a lightweight CNN model with 983,716 parameters, 2.37×108 floating point operations per second (FLOPs) and leveraging explainable AI strategies, this study contributes to efficient OCT-based diagnosis, underscores its potential in advancing medical diagnostics, and offers assistance in the Internet-of-Medical-Things.

Early detection of plant nutrient deficiency is crucial for agricultural productivity. This study investigated the performance and interpretability of Convolutional Neural Networks (CNNs) for this task. Using the rice and banana datasets, we compared three CNN architectures (CNN, VGG-16, Inception-V3). Inception-V3 achieved the highest accuracy (93% for rice and banana), but simpler models such as VGG-16 might be easier to understand. To address this trade-off, we employed Explainable AI (XAI) techniques (SHAP and Grad-CAM) to gain insights into model decision-making. This study emphasises the importance of both accuracy and interpretability in agricultural AI and demonstrates the value of XAI for building trust in these models.

No abstract

No abstract

No abstract

Lung-related diseases continue to be a leading cause of global mortality. Timely and precise diagnosis is crucial to save lives, but the availability of testing equipment remains a challenge, often coupled with issues of reliability. Recent research has highlighted the potential of Chest X-ray (CXR) images in identifying various lung diseases, including COVID-19, fibrosis, pneumonia, and more. In this comprehensive study, four publicly accessible datasets have been combined to create a robust dataset comprising 6650 CXR images, categorized into seven distinct disease groups. To effectively distinguish between normal and six different lung-related diseases (namely, bacterial pneumonia, COVID-19, fibrosis, lung opacity, tuberculosis, and viral pneumonia), a Deep Learning (DL) architecture called a Multi-Scale Convolutional Neural Network (MS-CNN) is introduced. The model is adapted to classify multiple numbers of lung disease classes, which is considered to be a persistent challenge in the field. While prior studies have demonstrated high accuracy in binary and limited-class scenarios, the proposed framework maintains this accuracy across a diverse range of lung conditions. The innovative model harnesses the power of combining predictions from multiple feature maps at different resolution scales, significantly enhancing disease classification accuracy. The approach aims to shorten testing duration compared to the state-of-the-art models, offering a potential solution toward expediting medical interventions for patients with lung-related diseases and integrating explainable AI (XAI) for enhancing prediction capability. The results demonstrated an impressive accuracy of 96.05%, with average values for precision, recall, F1-score, and AUC at 0.97, 0.95, 0.95, and 0.94, respectively, for the seven-class classification. The model exhibited exceptional performance across multi-class classifications, achieving accuracy rates of 100%, 99.65%, 99.21%, 98.67%, and 97.47% for two, three, four, five, and six-class scenarios, respectively. The novel approach not only surpasses many pre-existing state-of-the-art (SOTA) methodologies but also sets a new standard for the diagnosis of lung-affected diseases using multi-class CXR data. Furthermore, the integration of XAI techniques such as SHAP and Grad-CAM enhanced the transparency and interpretability of the model’s predictions. The findings hold immense promise for accelerating and improving the accuracy and confidence of diagnostic decisions in the field of lung disease identification.

No abstract

No abstract

The advent of sixth-generation (6G) networks promises ultra-high bandwidth, reliable connectivity, and ultra-low latency, enabling large-scale IoT deployment. Network slicing is central to these capabilities, but conventional deep learning approaches often suffer from privacy risks, high computational cost, and poor energy efficiency. To address these challenges, this work proposes a federated and explainable AI framework for energy-efficient IoT slicing in 6G. Federated learning enables collaborative training without sharing raw data, preserving privacy and reducing communication overhead. A transformer-based model captures complex traffic patterns, while a hybrid swarm-intelligence optimizer balances throughput, latency, and energy consumption. SHAP-based explainability enhances transparency in slice allocation. Experiments on real and simulated traffic confirm superior performance, with the proposed framework achieving 98.42% accuracy compared to CNN+LSTM (92.67%) and HHO-CNN+LSTM (95.12%). These results demonstrate a scalable, privacy-preserving, and sustainable solution for future 6G IoT ecosystems.

No abstract

Abstract In this paper, we adapt computational design approaches, widely used by the engineering design community, to address the unique challenges associated with mission design using RTS games. Specifically, we present a modeling approach that combines experimental design techniques, meta-modeling using convolutional neural networks (CNNs), uncertainty quantification, and explainable AI (XAI). We illustrate the approach using an open-source real-time strategy (RTS) game called microRTS. The modeling approach consists of microRTS player agents (bots), design of experiments that arranges games between identical agents with asymmetric initial conditions, and an AI infused layer comprising CNNs, XAI, and uncertainty analysis through Monte Carlo Dropout Network analysis that allows analysis of game balance. A sample balanced game and corresponding predictions and SHapley Additive exPlanations (SHAP) are presented in this study. Three additional perturbations were introduced to this balanced gameplay and the observations about important features of the game using SHAP are presented. Our results show that this analysis can successfully predict probability of win for self-play microRTS games, as well as capture uncertainty in predictions that can be used to guide additional data collection to improve the model, or refine the game balance measure.

Abstract Money laundering has been a global issue for decades. The ever-changing technology landscape, digital channels, and regulations make it increasingly difficult. Financial institutions use rule-based systems to detect suspicious money laundering transactions. However, it suffers from large false positives (FPs) that lead to operational efforts or misses on true positives (TPs) that increase the compliance risk. This paper presents a study of convolutional neural network (CNN) to predict money laundering and employs SHapley Additive exPlanations (SHAP) explainable artificial intelligence (AI) method to explain the CNN predictions. The results highlight the role of CNN in detecting suspicious transactions with high accuracy and SHAP’s role in bringing out the rationale of deep learning predictions.

The healthcare industry has been revolutionized by the convergence of Artificial Intelligence of Medical Things (AIoMT), allowing advanced data-driven solutions to improve healthcare systems. With the increasing complexity of Artificial Intelligence (AI) models, the need for Explainable Artificial Intelligence (XAI) techniques become paramount, particularly in the medical domain, where transparent and interpretable decision-making becomes crucial. Therefore, in this work, we leverage a custom XAI framework, incorporating techniques such as Local Interpretable Model-Agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), and Gradient-weighted Class Activation Mapping (Grad-Cam), explicitly designed for the domain of AIoMT. The proposed framework enhances the effectiveness of strategic healthcare methods and aims to instill trust and promote understanding in AI-driven medical applications. Moreover, we utilize a majority voting technique that aggregates predictions from multiple convolutional neural networks (CNNs) and leverages their collective intelligence to make robust and accurate decisions in the healthcare system. Building upon this decision-making process, we apply the XAI framework to brain tumor detection as a use case demon strating accurate and transparent diagnosis. Evaluation results underscore the exceptional performance of the XAI framework, achieving high precision, recall, and F1 scores with a training accuracy of 99% and a validation accuracy of 98%. Combining advanced XAI techniques with ensemble-based deep-learning (DL) methodologies allows for precise and reliable brain tumor diagnoses as an application of AIoMT.

Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predictions of a real-world bank account takeover fraud detection RNN model, and draw key insights from its explanations: i) the model identifies important features and events aligned with what fraud analysts consider cues for account takeover; ii) positive predicted sequences can be pruned to only 10% of the original length, as older events have residual attribution values; iii) the most recent input event of positive predictions only contributes on average to 41% of the model's score; iv) notably high attribution to client's age, upheld on higher false positive rates for older clients.

No abstract

Artificial intelligence (AI) has the potential to revolutionize healthcare by automating the detection and classification of events and anomalies. In the scope of this work, events and anomalies are abnormalities in the patient’s data, where the former are due to a medical condition, such as a seizure or a fall, and the latter are erroneous data due to faults or malicious attacks. AI-based event and anomaly detection (EAD) and their classification can improve patient outcomes by identifying problems earlier, enabling more timely interventions while minimizing false alarms caused by anomalies. Moreover, the advancement of Medical Internet of Things (MIoT), or wearable devices, and their high processing capabilities facilitated the gathering, AI-based processing, and transmission of data, which enabled remote patient monitoring, and personalized and predictive healthcare. However, it is fundamental in healthcare to ensure the explainability of AI systems, meaning that they can provide understandable and transparent reasoning for their decisions. This article proposes an online EAD approach using a lightweight autoencoder (AE) on the MIoT. The detected abnormality is explained using KernelSHAP, an explainable AI (XAI) technique, where the explanation of the abnormality is used, by an artificial neural network (ANN), to classify it into an event or anomaly. Intensive simulations are conducted using the Medical Information Mart for Intensive Care (MIMIC) data set for various physiological data. Results showed the robustness of the proposed approach in the detection and classification of events, regardless of the percentage of the present anomalies.

With the continuous modernization of water plants, malicious, often state-sponsored attacks continue to create havoc in such critical realms. Motivated by this, this paper proposes an unsupervised data-driven approach to support cyber forensics in such unique setups. Specifically, the proposed approach aims at inferring and attributing cyber attacks using sensor readings and actuators states. The approach operates using attack-free data, which is attractive towards cyber forensics of such systems, where attack-related empirical data is rarely widely available due to security and privacy reasons. The proposed method also provides the capability to track and identify the attacked assets for prioritization purposes. The proposed approach exploits Bidirectional Generative Adversarial Networks (BiGAN) to fingerprint the behavior of the system under regular operation. It employs a combination of Recurrent Neural Network (RNN) and Convolutional Neural Networks (CNN) as a basis of its design components. The Energy Distance (ED) and Maximum Mean Discrepancy (MMD) are used to evaluate how firmly the model has learned the system's behavior. The approach also leverages the l1-norm distance between unseen data and corresponding reconstruction to estimate the irregularity score representing cyber attacks. The relative importance of the obtained residual error for each sensor/actuator is put forward to attribute the attacked assets. To this end, we independently employ a regression tree technique, a game-theoretic concept known as Shapley values, and a model-wise approach, the KernelSHAP, as residual loss to identify the relation of each asset to the inferred anomaly. The results are then amalgamated to pinpoint the attacked asset. Empirical evaluations using data collected in a testbed representing a small-scale water treatment plant uncovered 32 out of the 36 cyber incidents; exceeding the performance of state-of-the-art. We also show that the proposed approach identifies the exploited sensors/actuators with more than 8–15% accuracy improvement over current available works. We postulate and stress the fact that such proposed methods significantly contributes towards the forensics of critical infrastructure.

While the field of object detection has seen remarkable performance gains since the incorporation of Deep Neural Networks (DNNs), a significant drawback in DNN detection algorithms is that they lack transparency, making their behaviour somewhat unpredictable. Without transparency, employing DNNs for on-board object detection on Uncrewed Aerial Vehicles (UAVs) could have massive societal and safety consequences. In this paper, we propose adopting a proven explainer, KernelSHAP, to provide visual explanations for bounding boxes produced by detection algorithms intended for on-board UAVs. Our explainer can identify the important parts of an image that assisted a given detection or contributed to a specific failure mode. We evaluate our explainer&#x2019;s discriminative ability on aerial imagery through a pointing metric and an automatic deletion/insertion metric. We further assess our explainer by intentionally introducing a bias to the dataset for it to detect and using that bias to simulate failure modes that can then be discovered using our explainer.

Despite increasing interest in using Artificial Intelligence (AI) and Machine Learning (ML) models for drug development, effectively interpreting their predictions remains a challenge, which limits their impact on clinical decisions. We address this issue by providing a practical guide to SHapley Additive exPlanations (SHAP), a popular feature-based interpretability method, which can be seamlessly integrated into supervised ML models to gain a deeper understanding of their predictions, thereby enhancing their transparency and trustworthiness. This tutorial focuses on the application of SHAP analysis to standard ML black-box models for regression and classification problems. We provide an overview of various visualization plots and their interpretation, available software for implementing SHAP, and highlight best practices, as well as special considerations, when dealing with binary endpoints and time-series models. To enhance the reader's understanding for the method, we also apply it to inherently explainable regression models. Finally, we discuss the limitations and ongoing advancements aimed at tackling the current drawbacks of the method.

Neural networks are ubiquitous in applied machine learning for education. Their pervasive success in predictive performance comes alongside a severe weakness, the lack of explainability of their decisions, especially relevant in human-centric fields. We implement five state-of-the-art methodologies for explaining black-box machine learning models (LIME, PermutationSHAP, KernelSHAP, DiCE, CEM) and examine the strengths of each approach on the downstream task of student performance prediction for five massive open online courses. Our experiments demonstrate that the families of explainers do not agree with each other on feature importance for the same Bidirectional LSTM models with the same representative set of students. We use Principal Component Analysis, Jensen-Shannon distance, and Spearman's rank-order correlation to quantitatively cross-examine explanations across methods and courses. Furthermore, we validate explainer performance across curriculum-based prerequisite relationships. Our results come to the concerning conclusion that the choice of explainer is an important decision and is in fact paramount to the interpretation of the predictive results, even more so than the course the model is trained on. Source code and models are released at http://github.com/epfl-ml4ed/evaluating-explainers.

Graph neural networks (GNNs) have been widely applied in software-defined network (SDN) to enhance network modeling and performance forecasting. However, the black-box nature of deep learning makes GNNs difficult to interpret, hindering their broad use and the application of GNN-based SDN systems in engineering. In this paper, we propose a novel interpretation framework named GEEK-Explainer, designed to efficiently provide instance-level interpretation of GNNs in SDN. Specifically, we introduce a KernelSHAP-based scoring module to generate intuitive and human-friendly explanations for each performance prediction. To address conflicts in computation cost, we propose a soft discrete mask matrix that identifies a critical set of important nodes. Extensive experiments demonstrate that the RouteNet model can effectively learn the relationships among features, which can provide a better understanding of the prediction process with less computation cost. These findings improve the transparency and robustness of the model and promote the application of GNN-based SDN systems in engineering practice.

SHAP explanations are a popular feature-attribution mechanism for explainable AI. They use game-theoretic notions to measure the influence of individual features on the prediction of a machine learning model. Despite a lot of recent interest from both academia and industry, it is not known whether SHAP explanations of common machine learning models can be computed efficiently. In this paper, we establish the complexity of computing the SHAP explanation in three important settings. First, we consider fully-factorized data distributions, and show that the complexity of computing the SHAP explanation is the same as the complexity of computing the expected value of the model. This fully-factorized setting is often used to simplify the SHAP computation, yet our results show that the computation can be intractable for commonly used models such as logistic regression. Going beyond fully-factorized distributions, we show that computing SHAP explanations is already intractable for a very simple setting: computing SHAP explanations of trivial classifiers over naive Bayes distributions. Finally, we show that even computing SHAP over the empirical distribution is #P-hard.

<title>Abstract</title> As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of if and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we introduce and study the disagreement problem in explainable machine learning. More specifically, we formalize the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how do practitioners resolve these disagreements. To this end, we first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and eight different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that (1) state-of-the-art explanation methods often disagree in terms of the explanations they output, and (2) machine learning practitioners often employ ad hoc heuristics when resolving such disagreements. These findings suggest that practitioners may be relying on misleading explanations when making consequential decisions. They also underscore the importance of developing principled frameworks for effectively evaluating and comparing explanations output by various state-of-the-art methods.

Over the last few years, the Shapley value, a solution concept from cooperative game theory, has found numerous applications in machine learning. In this paper, we first discuss fundamental concepts of cooperative game theory and axiomatic properties of the Shapley value. Then we give an overview of the most important applications of the Shapley value in machine learning: feature selection, explainability, multi-agent reinforcement learning, ensemble pruning, and data valuation. We examine the most crucial limitations of the Shapley value and point out directions for future research.

LambdaMART has been shown to outperform neural network models on tabular Learning-to-Rank (LTR) tasks. Similar to the neural network models, LambdaMART is considered a black-box model due to the complexity of the logic behind its predictions. Explanation techniques can help us understand these models. Our study investigates the faithfulness of point-wise explanation techniques when explaining LambdaMART models. Our analysis includes LTR-specific explanation techniques, such as LIRME and EXS, as well as explanation techniques that are not adapted to LTR use cases, such as LIME, KernelSHAP, and LPI. The explanation techniques are evaluated using several measures: Consistency, Fidelity, (In)fidelity, Validity, Completeness, and Feature Frequency (FF) Similarity. Three LTR benchmark datasets are used in the investigation: LETOR 4 (MQ2008), Microsoft Bing Search (MSLR-WEB10K), and Yahoo! LTR challenge dataset. Our empirical results demonstrate the challenges of accurately explaining LambdaMART: no single explanation technique is consistently faithful across all our evaluation measures and datasets. Furthermore, our results show that LTR-based explanation techniques are not consistently better than their non-LTR-based counterparts across the evaluation measures. Specifically, the LTR-based explanation techniques consistently are the most faithful with respect to (In)fidelity, whereas the non-LTR-specific approaches are shown to frequently provide the most faithful explanations with respect to Validity, Completeness, and FF Similarity.

Abstract Post-hoc interpretability methods are critical tools to explain neural-network results. Several post-hoc methods have emerged in recent years but they produce different results when applied to a given task, raising the question of which method is the most suitable to provide accurate post-hoc interpretability. To understand the performance of each method, quantitative evaluation of interpretability methods is essential; however, currently available frameworks have several drawbacks that hinder the adoption of post-hoc interpretability methods, especially in high-risk sectors. In this work we propose a framework with quantitative metrics to assess the performance of existing post-hoc interpretability methods, particularly in time-series classification. We show that several drawbacks identified in the literature are addressed, namely, the dependence on human judgement, retraining and the shift in the data distribution when occluding samples. We also design a synthetic dataset with known discriminative features and tunable complexity. The proposed methodology and quantitative metrics can be used to understand the reliability of interpretability methods results obtained in practical applications. In turn, they can be embedded within operational workflows in critical fields that require accurate interpretability results for, example, regulatory policies.

Although the effectiveness of machine learning (ML) for machine diagnosis has been widely established, the interpretation of the diagnosis outcomes is still an open issue. Machine learning models behave as black boxes; therefore, the contribution given by each of the selected features to the diagnosis is not transparent to the user. This work is aimed at investigating the capabilities of the SHapley Additive exPlanation (SHAP) to identify the most important features for fault detection and classification in condition monitoring programs for rotating machinery. The authors analyse the case of medium-sized bearings of industrial interest. Namely, vibration data were collected for different health states from the test rig for industrial bearings available at the Mechanical Engineering Laboratory of Politecnico di Torino. The Support Vector Machine (SVM) and k-Nearest Neighbour (kNN) diagnosis models are explained by means of the SHAP. Accuracies higher than 98.5% are achieved for both the models using the SHAP as a criterion for feature selection. It is found that the skewness and the shape factor of the vibration signal have the greatest impact on the models’ outcomes.

The growing field of explainable Artificial Intelligence (xAI) has given rise to a multitude of techniques and methodologies, yet this expansion has created a growing gap between existing xAI approaches and their practical application. This poses a considerable obstacle for data scientists striving to identify the optimal xAI technique for their needs. To address this problem, our study presents a customized decision support framework to aid data scientists in choosing a suitable xAI approach for their use-case. Drawing from a literature survey and insights from interviews with five experienced data scientists, we introduce a decision tree based on the trade-offs inherent in various xAI approaches, guiding the selection between six commonly used xAI tools. Our work critically examines six prevalent ante-hoc and post-hoc xAI methods, assessing their applicability in real-world contexts through expert interviews. The aim is to equip data scientists and policymakers with the capacity to select xAI methods that not only demystify the decision-making process, but also enrich user understanding and interpretation, ultimately advancing the application of xAI in practical settings.

Multi-view learning is a machine learning app0roach aiming to exploit the knowledge retrieved from data, represented by multiple feature subsets known as views. Co-training is considered the most representative form of multi-view learning, a very effective semi-supervised classification algorithm for building highly accurate and robust predictive models. Even though it has been implemented in various scientific fields, it has not adequately used in educational data mining and learning analytics, since the hypothesis about the existence of two feature views cannot be easily implemented. Some notable studies have emerged recently dealing with semi-supervised classification tasks, such as student performance or student dropout prediction, while semi-supervised regression is uncharted territory. Therefore, the present study attempts to implement a semi-regression algorithm for predicting the grades of undergraduate students in the final exams of a one-year online course, which exploits three independent and naturally formed feature views, since they are derived from different sources. Moreover, we examine a well-established framework for interpreting the acquired results regarding their contribution to the final outcome per student/instance. To this purpose, a plethora of experiments is conducted based on data offered by the Hellenic Open University and representative machine learning algorithms. The experimental results demonstrate that the early prognosis of students at risk of failure can be accurately achieved compared to supervised models, even for a small amount of initially collected data from the first two semesters. The robustness of the applying semi-supervised regression scheme along with supervised learners and the investigation of features’ reasoning could highly benefit the educational domain.

Anomaly-based In-Vehicle Intrusion Detection System (IV-IDS) is one of the protection mechanisms to detect cyber attacks on automotive vehicles. Using artificial intelligence (AI) for anomaly detection to thwart cyber attacks is promising but suffers from generating false alarms and making decisions that are hard to interpret. Consequently, this issue leads to uncertainty and distrust towards such IDS design unless it can explain its behavior, e.g., by using eXplainable AI (XAI). In this paper, we consider the XAI-powered design of such an IV-IDS using CAN bus data from a public dataset, named “Survival”. Novel features are engineered, and a Deep Neural Network (DNN) is trained over the dataset. A visualization-based explanation, “VisExp”, is created to explain the behavior of the AI-based IV-IDS, which is evaluated by experts in a survey, in relation to a rule-based explanation. Our results show that experts’ trust in the AI-based IV-IDS is significantly increased when they are provided with VisExp (more so than the rule-based explanation). These findings confirm the effect, and by extension the need, of explainability in automated systems, and VisExp, being a source of increased explainability, shows promise in helping involved parties gain trust in such systems.

No abstract

Predicting student performance as early as possible and analysing to which extent initial student behaviour could lead to failure or success is critical in introductory programming (CS1) courses, for allowing prompt intervention in a move towards alleviating their high failure rate. However, in CS1 performance prediction, there is a serious lack of studies that interpret the predictive model’s decisions. In this sense, we designed a long-term study using very fine-grained log-data of 2056 students, collected from the first two weeks of CS1 courses. We extract features that measure how students deal with deadlines, how they fix errors, how much time they spend programming, and so forth. Subsequently, we construct a predictive model that achieved cutting-edge results with area under the curve (AUC) of.89, and an average accuracy of 81.3%. To allow an effective intervention and to facilitate human-AI collaboration towards prescriptive analytics, we, for the first time, to the best of our knowledge, go a step further than the prediction itself and leverage this field by proposing an approach to explaining our predictive model decisions individually and collectively using a game-theory based framework (SHAP), (Lundberg <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , 2020) that allows interpreting our black-box non-linear model linearly. In other words, we explain the feature effects, clearly by visualising and analysing individual predictions, the overall importance of features, and identification of typical prediction paths. This method can be further applied to other emerging competitive models, as the CS1 prediction field progresses, ensuring transparency of the process for key stakeholders: administrators, teachers, and students.

No abstract

Generating explanations for neural networks has become crucial for their applications in real-world with respect to reliability and trustworthiness. In natural language processing, existing methods usually provide important features which are words or phrases selected from an input text as an explanation, but ignore the interactions between them. It poses challenges for humans to interpret an explanation and connect it to model prediction. In this work, we build hierarchical explanations by detecting feature interactions. Such explanations visualize how words and phrases are combined at different levels of the hierarchy, which can help users understand the decision-making of blackbox models. The proposed method is evaluated with three neural text classifiers (LSTM, CNN, and BERT) on two benchmark datasets, via both automatic and human evaluations. Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models and interpretable to humans.

No abstract

Deep neural networks obtain state-of-the-art performance on a series of tasks. However, they are easily fooled by adding a small adversarial perturbation to the input. The perturbation is often imperceptible to humans on image data. We observe a significant difference in feature attributions between adversarially crafted examples and original examples. Based on this observation, we introduce a new framework to detect adversarial examples through thresholding a scale estimate of feature attribution scores. Furthermore, we extend our method to include multi-layer feature attributions in order to tackle attacks that have mixed confidence levels. As demonstrated in extensive experiments, our method achieves superior performances in distinguishing adversarial examples from popular attack methods on a variety of real data sets compared to state-of-the-art detection methods. In particular, our method is able to detect adversarial examples of mixed confidence levels, and transfer between different attacking methods. We also show that our method achieves competitive performance even when the attacker has complete access to the detector.

In this work, we have provided a new design for explainable AI used in stress prediction based on physiological measurements. Based on the report, users and medical practitioners can determine what biological features have the most impact on the prediction of stress in addition to any health-related abnormalities. The effectiveness of the explainable AI report was evaluated using a quantitative and a qualitative assessment. The stress prediction accuracy was shown to be comparable to state-of-the-art. The contributions of each physiological signal to the stress prediction was shown to correlate with ground truth. In addition to these quantitative evaluations, a qualitative survey with psychiatrists confirmed the confidence and effectiveness of the explanation report in the stress made by the AI system. Future work includes the addition of more explanatory features related to other emotional states of the patient, such as sadness, relaxation, anxiousness, or happiness.

Understanding the predictions made by machine learning (ML) models and their potential biases remains a challenging and labor-intensive task that depends on the application, the dataset, and the specific model. We present Amazon SageMaker Clarify, an explainability feature for Amazon SageMaker that launched in December 2020, providing insights into data and ML models by identifying biases and explaining predictions. It is deeply integrated into Amazon SageMaker, a fully managed service that enables data scientists and developers to build, train, and deploy ML models at any scale. Clarify supports bias detection and feature importance computation across the ML lifecycle, during data preparation, model evaluation, and post-deployment monitoring. We outline the desiderata derived from customer input, the modular architecture, and the methodology for bias and explanation computations. Further, we describe the technical challenges encountered and the tradeoffs we had to make. For illustration, we discuss two customer use cases. We present our deployment results including qualitative customer feedback and a quantitative evaluation. Finally, we summarize lessons learned, and discuss best practices for the successful adoption of fairness and explanation tools in practice.

No abstract

Abstract Practitioners increasingly use machine learning (ML) models, yet models have become more complex and harder to understand. To understand complex models, researchers have proposed techniques to explain model predictions. However, practitioners struggle to use explainability methods because they do not know which explanation to choose and how to interpret the explanation. Here we address the challenge of using explainability methods by proposing TalkToModel: an interactive dialogue system that explains ML models through natural language conversations. TalkToModel consists of three components: an adaptive dialogue engine that interprets natural language and generates meaningful responses; an execution component that constructs the explanations used in the conversation; and a conversational interface. In real-world evaluations, 73% of healthcare workers agreed they would use TalkToModel over existing systems for understanding a disease prediction model, and 85% of ML professionals agreed TalkToModel was easier to use, demonstrating that TalkToModel is highly effective for model explainability.

The Shapley value, which is arguably the most popular approach for assigning a meaningful contribution value to players in a cooperative game, has recently been used intensively in explainable artificial intelligence. Its meaningfulness is due to axiomatic properties that only the Shapley value satisfies, which, however, comes at the expense of an exact computation growing exponentially with the number of agents. Accordingly, a number of works are devoted to the efficient approximation of the Shapley value, most of them revolve around the notion of an agent's marginal contribution. In this paper, we propose with SVARM and Stratified SVARM two parameter-free and domain-independent approximation algorithms based on a representation of the Shapley value detached from the notion of marginal contribution. We prove unmatched theoretical guarantees regarding their approximation quality and provide empirical results including synthetic games as well as common explainability use cases comparing ourselves with state-of-the-art methods.

In this study, we proposed a data-driven approach to the condition monitoring of the marine engine. Although several unsupervised methods in the maritime industry have existed, the common limitation was the interpretation of the anomaly; they do not explain why the model classifies specific data instances as an anomaly. This study combines explainable AI techniques with anomaly detection algorithm to overcome the limitation above. As an explainable AI method, this study adopts Shapley Additive exPlanations (SHAP), which is theoretically solid and compatible with any kind of machine learning algorithm. SHAP enables us to measure the marginal contribution of each sensor variable to an anomaly. Thus, one can easily specify which sensor is responsible for the specific anomaly. To illustrate our framework, the actual sensor stream obtained from the cargo vessel collected over 10 months was analyzed. In this analysis, we performed hierarchical clustering analysis with transformed SHAP values to interpret and group common anomaly patterns. We showed that anomaly interpretation and segmentation using SHAP value provides more useful interpretation compared to the case without using SHAP value.

Currently, artificial intelligence technologies are rapidly entering many areas of our society. It should be said that currently several reforms are being implemented in the Republic of Uzbekistan for the development of artificial intelligence technologies. In particular, the decision of the President of the Republic of Uzbekistan "On measures to create conditions for the rapid introduction of artificial intelligence technologies" dated February 17, 2021, No. PQ-4996 defines the main tasks in this regard. One of the most pressing issues studied by modern researchers is the issues related to the topic of digitalization of the educational process. In this article, the stages of development of artificial intelligence, and its application in the field of modern education are presented. The potential of artificial intelligence in education is also considered. At the same time, the processes of intellectual education are modeled mathematically. Algorithms of intellectual educational processes are proposed using CCN, and SHAP algorithms for organizing intellectual educational processes.Also, issues of ensuring the immutability and confidentiality of student evaluation results on distance education platforms, a strategy for ensuring the security of student test and control data using blockchain technology are presented.

Despite a sea of interpretability methods that can produce plausible explanations, the field has also empirically seen many failure cases of such methods. In light of these results, it remains unclear for practitioners how to use these methods and choose between them in a principled way. In this paper, we show that for moderately rich model classes (easily satisfied by neural networks), any feature attribution method that is complete and linear-for example, Integrated Gradients and Shapley Additive Explanations (SHAP)-can provably fail to improve on random guessing for inferring model behavior. Our results apply to common end-tasks such as characterizing local model behavior, identifying spurious features, and algorithmic recourse. One takeaway from our work is the importance of concretely defining end-tasks: Once such an end-task is defined, a simple and direct approach of repeated model evaluations can outperform many other complex feature attribution methods.

Failure detection is an important part of failure management, and network operators encounter serious consequences when operating under failure conditions. Machine learning (ML) is widely applied in the failure management of optical networks, where neural networks (NNs) have particularly attracted considerable attention and become the most extensively applied algorithm among all MLs. However, the black-box nature of NN makes it difficult to interpret or analyze why and how NNs work during execution. In this paper, we propose a cause-aware failure detection scheme for optical transport network (OTN) boards, adopting the interpretable extreme gradient boosting (XGBoost) algorithm. According to the feature importance ranking by XGBoost, the high-relevance features with the equipment failure are found. Then, SHapley Additive exPlanations (SHAP) is applied to solve the inconsistency of feature attribution under three common global feature importance measurement parameters of XGBoost, and can obtain a consistent feature attribution by calculating the contribution (SHAP value) of each input feature to detection result of XGBoost. Based on the feature importance ranking of SHAP values, the features most related to two types of OTN board failures are confirmed, enabling the identification of failure causes. Moreover, we evaluate the failure detection performance for two types of OTN boards, in which the practical data are balanced and unbalanced respectively. Experimental results show that the F1 score of the two types of OTN boards based on the proposed scheme is higher than 98%, and the most relevant features of the two types of board failures are confirmed based on SHAP value, which are the average and maximum values of the environment temperature, respectively.

No abstract

No abstract

No abstract

Classification and regression problems can be challenging when the relevant input features are diluted in noisy datasets, in particular when the sample size is limited. Traditional Feature Selection (FS) methods address this issue by relying on some assumptions such as the linear or additive relationship between features. Recently, a proliferation of Deep Learning (DL) models has emerged to tackle both FS and prediction at the same time, allowing non-linear modeling of the selected features. In this study, we systematically assess the performance of DL-based feature selection methods on synthetic datasets of varying complexity, and benchmark their efficacy in uncovering non-linear relationships between features. We also use the same settings to benchmark the reliability of gradient-based feature attribution techniques for Neural Networks (NNs), such as Saliency Maps (SM). A quantitative evaluation of the reliability of these approaches is currently missing. Our analysis indicates that even simple synthetic datasets can significantly challenge most of the DL-based FS and SM methods, while Random Forests, TreeShap, mRMR and LassoNet are the best performing FS methods. Our conclusion is that when quantifying the relevance of a few non linearly-entangled predictive features diluted in a large number of irrelevant noisy variables, DL-based FS and SM interpretation methods are still far from being reliable.

The use of Shap scores has become widespread in Explainable AI. However, their computation is in general intractable, in particular when done with a black-box classifier, such as neural network. Recent research has unveiled classes of open-box Boolean Circuit classifiers for which Shap can be computed efficiently. We show how to transform binary neural networks into those circuits for efficient Shap computation.We use logic-based knowledge compilation techniques. The performance gain is huge, as we show in the light of our experiments.

Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.

Many methods to explain black-box models, whether local or global, are additive. In this paper, we study global additive explanations for non-additive models, focusing on four explanation methods: partial dependence, Shapley explanations adapted to a global setting, distilled additive explanations, and gradient-based explanations. We show that different explanation methods characterize non-additive components in a black-box model's prediction function in different ways. We use the concepts of main and total effects to anchor additive explanations, and quantitatively evaluate additive and non-additive explanations. Even though distilled explanations are generally the most accurate additive explanations, non-additive explanations such as tree explanations that explicitly model non-additive components tend to be even more accurate. Despite this, our user study showed that machine learning practitioners were better able to leverage additive explanations for various tasks. These considerations should be taken into account when considering which explanation to trust and use to explain black-box models.

Shapley values have become one of the most popular feature attribution explanation methods. However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding due to its exponential time complexity and preclude model regularization based on Shapley explanations during training. Thus, we propose to incorporate Shapley values themselves as latent representations in deep models thereby making Shapley explanations first-class citizens in the modeling paradigm. This intrinsic explanation approach enables layer-wise explanations, explanation regularization of the model during training, and fast explanation computation at test time. We define the Shapley transform that transforms the input into a Shapley representation given a specific function. We operationalize the Shapley transform as a neural network module and construct both shallow and deep networks, called ShapNets, by composing Shapley modules. We prove that our Shallow ShapNets compute the exact Shapley values and our Deep ShapNets maintain the missingness and accuracy properties of Shapley values. We demonstrate on synthetic and real-world datasets that our ShapNets enable layer-wise Shapley explanations, novel Shapley regularizations during training, and fast computation while maintaining reasonable performance. Code is available at https://github.com/inouye-lab/ShapleyExplanationNetworks.

The Shapley value is one of the most widely used measures of feature importance partly as it measures a feature's average effect on a model's prediction. We introduce joint Shapley values, which directly extend Shapley's axioms and intuitions: joint Shapley values measure a set of features' average contribution to a model's prediction. We prove the uniqueness of joint Shapley values, for any order of explanation. Results for games show that joint Shapley values present different insights from existing interaction indices, which assess the effect of a feature within a set of features. The joint Shapley values provide intuitive results in ML attribution problems. With binary features, we present a presence-adjusted global value that is more consistent with local intuitions than the usual approach.

What is the value of an individual model in an ensemble of binary classifiers? We answer this question by introducing a class of transferable utility cooperative games called \textit{ensemble games}. In machine learning ensembles, pre-trained models cooperate to make classification decisions. To quantify the importance of models in these ensemble games, we define \textit{Troupe} -- an efficient algorithm which allocates payoffs based on approximate Shapley values of the classifiers. We argue that the Shapley value of models in these games is an effective decision metric for choosing a high performing subset of models from the ensemble. Our analytical findings prove that our Shapley value estimation scheme is precise and scalable; its performance increases with size of the dataset and ensemble. Empirical results on real world graph classification tasks demonstrate that our algorithm produces high quality estimates of the Shapley value. We find that Shapley values can be utilized for ensemble pruning, and that adversarial models receive a low valuation. Complex classifiers are frequently found to be responsible for both correct and incorrect classification decisions.

We propose a novel Shapley value approach to help address neural networks' interpretability and "vanishing gradient" problems. Our method is based on an accurate analytical approximation to the Shapley value of a neuron with ReLU activation. This analytical approximation admits a linear propagation of relevance across neural network layers, resulting in a simple, fast and sensible interpretation of neural networks' decision making process. We then derived a globally continuous and non-vanishing Shapley gradient, which can replace the conventional gradient in training neural network layers with ReLU activation, and leading to better training performance. We further derived a Shapley Activation (SA) function, which is a close approximation to ReLU but features the Shapley gradient. The SA is easy to implement in existing machine learning frameworks. Numerical tests show that SA consistently outperforms ReLU in training convergence, accuracy and stability.

Self-interpreting neural networks have garnered significant interest in research. Existing works in this domain often (1) lack a solid theoretical foundation ensuring genuine interpretability or (2) compromise model expressiveness. In response, we formulate a generic Additive Self-Attribution (ASA) framework. Observing the absence of Shapley value in Additive Self-Attribution, we propose Shapley Additive Self-Attributing Neural Network (SASANet), with theoretical guarantees for the self-attribution value equal to the output's Shapley values. Specifically, SASANet uses a marginal contribution-based sequential schema and internal distillation-based training strategies to model meaningful outputs for any number of features, resulting in un-approximated meaningful value function. Our experimental results indicate SASANet surpasses existing self-attributing models in performance and rivals black-box models. Moreover, SASANet is shown more precise and efficient than post-hoc methods in interpreting its own predictions.

Explainable artificial intelligence (XAI) holds immense significance in enhancing the deep neural network's transparency and credibility, particularly in some risky and high-cost scenarios, like synthetic aperture radar (SAR). Shapley is a game-based explanation technique with robust mathematical foundations. However, Shapley assumes that model's features are independent, rendering Shapley explanation invalid for high dimensional models. This study introduces a manifold-based Shapley method by projecting high-dimensional features into low-dimensional manifold features and subsequently obtaining Fusion-Shap, which aims at (1) addressing the issue of erroneous explanations encountered by traditional Shap; (2) resolving the challenge of interpretability that traditional Shap faces in complex scenarios.

Reinforcement learning has achieved remarkable success in complex decision-making environments, yet its lack of transparency limits its deployment in practice, especially in safety-critical settings. Shapley values from cooperative game theory provide a principled framework for explaining reinforcement learning; however, the computational cost of Shapley explanations is an obstacle to their use. We introduce FastSVERL, a scalable method for explaining reinforcement learning by approximating Shapley values. FastSVERL is designed to handle the unique challenges of reinforcement learning, including temporal dependencies across multi-step trajectories, learning from off-policy data, and adapting to evolving agent behaviours in real time. FastSVERL introduces a practical, scalable approach for principled and rigorous interpretability in reinforcement learning.

The SHAP (short for Shapley additive explanation) framework has become an essential tool for attributing importance to variables in predictive tasks. In model-agnostic settings, SHAP uses the concept of Shapley values from cooperative game theory to fairly allocate credit to the features in a vector $X$ based on their contribution to an outcome $Y$. While the explanations offered by SHAP are local by nature, learners often need global measures of feature importance in order to improve model explainability and perform feature selection. The most common approach for converting these local explanations into global ones is to compute either the mean absolute SHAP or mean squared SHAP. However, despite their ubiquity, there do not exist approaches for performing statistical inference on these quantities. In this paper, we take a semi-parametric approach for calibrating confidence in estimates of the $p$th powers of Shapley additive explanations. We show that, by treating the SHAP curve as a nuisance function that must be estimated from data, one can reliably construct asymptotically normal estimates of the $p$th powers of SHAP. When $p \geq 2$, we show a de-biased estimator that combines U-statistics with Neyman orthogonal scores for functionals of nested regressions is asymptotically normal. When $1 \leq p < 2$ (and the hence target parameter is not twice differentiable), we construct de-biased U-statistics for a smoothed alternative. In particular, we show how to carefully tune the temperature parameter of the smoothing function in order to obtain inference for the true, unsmoothed $p$th power. We complement these results by presenting a Neyman orthogonal loss that can be used to learn the SHAP curve via empirical risk minimization and discussing excess risk guarantees for commonly used function classes.

Graph Neural Networks (GNNs) achieve significant performance for various learning tasks on geometric data due to the incorporation of graph structure into the learning of node representations, which renders their comprehension challenging. In this paper, we first propose a unified framework satisfied by most existing GNN explainers. Then, we introduce GraphSVX, a post hoc local model-agnostic explanation method specifically designed for GNNs. GraphSVX is a decomposition technique that captures the "fair" contribution of each feature and node towards the explained prediction by constructing a surrogate model on a perturbed dataset. It extends to graphs and ultimately provides as explanation the Shapley Values from game theory. Experiments on real-world and synthetic datasets demonstrate that GraphSVX achieves state-of-the-art performance compared to baseline models while presenting core theoretical and human-centric properties.

Shapley Values (SV) are widely used in explainable AI, but their estimation and interpretation can be challenging, leading to inaccurate inferences and explanations. As a starting point, we remind an invariance principle for SV and derive the correct approach for computing the SV of categorical variables that are particularly sensitive to the encoding used. In the case of tree-based models, we introduce two estimators of Shapley Values that exploit the tree structure efficiently and are more accurate than state-of-the-art methods. Simulations and comparisons are performed with state-of-the-art algorithms and show the practical gain of our approach. Finally, we discuss the limitations of Shapley Values as a local explanation. These methods are available as a Python package.

In the classical context, the cooperative game theory concept of the Shapley value has been adapted for post hoc explanations of machine learning models. However, this approach does not easily translate to eXplainable Quantum ML (XQML). Finding Shapley values can be highly computationally complex. We propose quantum algorithms which can extract Shapley values within some confidence interval. Our results perform in polynomial time. We demonstrate the validity of each approach under specific examples of cooperative voting games.

Measuring contributions is a classical problem in cooperative game theory where the Shapley value is the most well-known solution concept. In this paper, we establish the convergence property of the Shapley value in parametric Bayesian learning games where players perform a Bayesian inference using their combined data, and the posterior-prior KL divergence is used as the characteristic function. We show that for any two players, under some regularity conditions, their difference in Shapley value converges in probability to the difference in Shapley value of a limiting game whose characteristic function is proportional to the log-determinant of the joint Fisher information. As an application, we present an online collaborative learning framework that is asymptotically Shapley-fair. Our result enables this to be achieved without any costly computations of posterior-prior KL divergences. Only a consistent estimator of the Fisher information is needed. The effectiveness of our framework is demonstrated with experiments using real-world data.

Despite their ubiquitous use, Shapley value feature attributions can be misleading due to feature interaction in both model and data. We propose an alternative attribution approach, Shapley Sets, which awards value to sets of features. Shapley Sets decomposes the underlying model into non-separable variable groups using a recursive function decomposition algorithm with log linear complexity in the number of variables. Shapley Sets attributes to each non-separable variable group their combined value for a particular prediction. We show that Shapley Sets is equivalent to the Shapley value over the transformed feature set and thus benefits from the same axioms of fairness. Shapley Sets is value function agnostic and we show theoretically and experimentally how Shapley Sets avoids pitfalls associated with Shapley value based alternatives and are particularly advantageous for data types with complex dependency structure.

Shapley value is a popular approach for measuring the influence of individual features. While Shapley feature attribution is built upon desiderata from game theory, some of its constraints may be less natural in certain machine learning settings, leading to unintuitive model interpretation. In particular, the Shapley value uses the same weight for all marginal contributions -- i.e. it gives the same importance when a large number of other features are given versus when a small number of other features are given. This property can be problematic if larger feature sets are more or less informative than smaller feature sets. Our work performs a rigorous analysis of the potential limitations of Shapley feature attribution. We identify simple settings where the Shapley value is mathematically suboptimal by assigning larger attributions for less influential features. Motivated by this observation, we propose WeightedSHAP, which generalizes the Shapley value and learns which marginal contributions to focus directly from data. On several real-world datasets, we demonstrate that the influential features identified by WeightedSHAP are better able to recapitulate the model's predictions compared to the features identified by the Shapley value.

In this article, we provide an axiomatic characterization of feature attribution for multi-output predictors within the Shapley framework. While SHAP explanations are routinely computed independently for each output coordinate, the theoretical necessity of this practice has remained unclear. By extending the classical Shapley axioms to vector-valued cooperative games, we establish a rigidity theorem showing that any attribution rule satisfying efficiency, symmetry, dummy player, and additivity must necessarily decompose component-wise across outputs. Consequently, any joint-output attribution rule must relax at least one of the classical Shapley axioms. This result identifies a previously unformalized structural constraint in Shapley-based interpretability, clarifying the precise scope of fairness-consistent explanations in multi-output learning. Numerical experiments on a biomedical benchmark illustrate that multi-output models can yield computational savings in training and deployment, while producing SHAP explanations that remain fully consistent with the component-wise structure imposed by the Shapley axioms.

Numerous works propose post-hoc, model-agnostic explanations for learning to rank, focusing on ordering entities by their relevance to a query through feature attribution methods. However, these attributions often weakly correlate or contradict each other, confusing end users. We adopt an axiomatic game-theoretic approach, popular in the feature attribution community, to identify a set of fundamental axioms that every ranking-based feature attribution method should satisfy. We then introduce Rank-SHAP, extending classical Shapley values to ranking. We evaluate the RankSHAP framework through extensive experiments on two datasets, multiple ranking methods and evaluation metrics. Additionally, a user study confirms RankSHAP's alignment with human intuition. We also perform an axiomatic analysis of existing rank attribution algorithms to determine their compliance with our proposed axioms. Ultimately, our aim is to equip practitioners with a set of axiomatically backed feature attribution methods for studying IR ranking models, that ensure generality as well as consistency.

The performance of cardiac arrhythmia detection with electrocardiograms(ECGs) has been considerably improved since the introduction of deep learning models. In practice, the high performance alone is not sufficient and a proper explanation is also required. Recently, researchers have started adopting feature attribution methods to address this requirement, but it has been unclear which of the methods are appropriate for ECG. In this work, we identify and customize three evaluation metrics for feature attribution methods based on the characteristics of ECG: localization score, pointing game, and degradation score. Using the three evaluation metrics, we evaluate and analyze eleven widely-used feature attribution methods. We find that some of the feature attribution methods are much more adequate for explaining ECG, where Grad-CAM outperforms the second-best method by a large margin.

Shapley value is a widely used tool in explainable artificial intelligence (XAI), as it provides a principled way to attribute contributions of input features to model outputs. However, estimation of Shapley value requires capturing conditional dependencies among all feature combinations, which poses significant challenges in complex data environments. In this article, EmSHAP (Energy-based model for Shapley value estimation), an accurate Shapley value estimation method, is proposed to estimate the expectation of Shapley contribution function under the arbitrary subset of features given the rest. By utilizing the ability of energy-based model (EBM) to model complex distributions, EmSHAP provides an effective solution for estimating the required conditional probabilities. To further improve estimation accuracy, a GRU (Gated Recurrent Unit)-coupled partition function estimation method is introduced. The GRU network captures long-term dependencies with a lightweight parameterization and maps input features into a latent space to mitigate the influence of feature ordering. Additionally, a dynamic masking mechanism is incorporated to further enhance the robustness and accuracy by progressively increasing the masking rate. Theoretical analysis on the error bound as well as application to four case studies verified the higher accuracy and better scalability of EmSHAP in contrast to competitive methods.

We consider the performance of a least-squares regression model, as judged by out-of-sample $R^2$. Shapley values give a fair attribution of the performance of a model to its input features, taking into account interdependencies between features. Evaluating the Shapley values exactly requires solving a number of regression problems that is exponential in the number of features, so a Monte Carlo-type approximation is typically used. We focus on the special case of least-squares regression models, where several tricks can be used to compute and evaluate regression models efficiently. These tricks give a substantial speed up, allowing many more Monte Carlo samples to be evaluated, achieving better accuracy. We refer to our method as least-squares Shapley performance attribution (LS-SPA), and describe our open-source implementation.

We consider an investment process that includes a number of features, each of which can be active or inactive. Our goal is to attribute or decompose an achieved performance to each of these features, plus a baseline value. There are many ways to do this, which lead to potentially different attributions in any specific case. We argue that a specific attribution method due to Shapley is the preferred method, and discuss methods that can be used to compute this attribution exactly, or when that is not practical, approximately.

The Shapley value has become popular in the Explainable AI (XAI) literature, thanks, to a large extent, to a solid theoretical foundation, including four "favourable and fair" axioms for attribution in transferable utility games. The Shapley value is provably the only solution concept satisfying these axioms. In this paper, we introduce the Shapley value and draw attention to its recent uses as a feature selection tool. We call into question this use of the Shapley value, using simple, abstract "toy" counterexamples to illustrate that the axioms may work against the goals of feature selection. From this, we develop a number of insights that are then investigated in concrete simulation settings, with a variety of Shapley value formulations, including SHapley Additive exPlanations (SHAP) and Shapley Additive Global importancE (SAGE).

Large language models (LLMs) demonstrate strong capabilities in in-context learning, but verifying the correctness of their generated responses remains a challenge. Prior work has explored attribution at the sentence level, but these methods fall short when users seek attribution for specific keywords within the response, such as numbers, years, or names. To address this limitation, we propose TokenShapley, a novel token-level attribution method that combines Shapley value-based data attribution with KNN-based retrieval techniques inspired by recent advances in KNN-augmented LLMs. By leveraging a precomputed datastore for contextual retrieval and computing Shapley values to quantify token importance, TokenShapley provides a fine-grained data attribution approach. Extensive evaluations on four benchmarks show that TokenShapley outperforms state-of-the-art baselines in token-level attribution, achieving an 11-23% improvement in accuracy.

This paper re-examines the Shapley value methods for attribution analysis in the area of online advertising. As a credit allocation solution in cooperative game theory, Shapley value method directly quantifies the contribution of online advertising inputs to the advertising key performance indicator (KPI) across multiple channels. We simplify its calculation by developing an alternative mathematical formulation. The new formula significantly improves the computational efficiency and therefore extends the scope of applicability. Based on the simplified formula, we further develop the ordered Shapley value method. The proposed method is able to take into account the order of channels visited by users. We claim that it provides a more comprehensive insight by evaluating the attribution of channels at different stages of user conversion journeys. The proposed approaches are illustrated using a real-world online advertising campaign dataset.

Mitigating the dependence on spurious correlations present in the training dataset is a quickly emerging and important topic of deep learning. Recent approaches include priors on the feature attribution of a deep neural network (DNN) into the training process to reduce the dependence on unwanted features. However, until now one needed to trade off high-quality attributions, satisfying desirable axioms, against the time required to compute them. This in turn either led to long training times or ineffective attribution priors. In this work, we break this trade-off by considering a special class of efficiently axiomatically attributable DNNs for which an axiomatic feature attribution can be computed with only a single forward/backward pass. We formally prove that nonnegatively homogeneous DNNs, here termed $\mathcal{X}$-DNNs, are efficiently axiomatically attributable and show that they can be effortlessly constructed from a wide range of regular DNNs by simply removing the bias term of each layer. Various experiments demonstrate the advantages of $\mathcal{X}$-DNNs, beating state-of-the-art generic attribution methods on regular DNNs for training with attribution priors.

Explainable artificial intelligence promises to yield insights into relevant features, thereby enabling humans to examine and scrutinize machine learning models or even facilitating scientific discovery. Considering the widespread technique of Shapley values, we find that purely data-driven operationalization of multivariate feature importance is unsuitable for such purposes. Even for simple problems with two features, spurious associations due to collider bias and suppression arise from considering one feature only in the observational context of the other, which can lead to misinterpretations. Causal knowledge about the data-generating process is required to identify and correct such misleading feature attributions. We propose cc-Shapley (causal context Shapley), an interventional modification of conventional observational Shapley values leveraging knowledge of the data's causal structure, thereby analyzing the relevance of a feature in the causal context of the remaining features. We show theoretically that this eradicates spurious association induced by collider bias. We compare the behavior of Shapley and cc-Shapley values on various, synthetic, and real-world datasets. We observe nullification or reversal of associations compared to univariate feature importance when moving from observational to cc-Shapley.

Model interpretability is one of the most intriguing problems in most of the Machine Learning models, particularly for those that are mathematically sophisticated. Computing Shapley Values are arguably the best approach so far to find the importance of each feature in a model, at the row level. In other words, Shapley values represent the importance of a feature for a particular row, especially for Classification or Regression problems. One of the biggest limitations of Shapley vales is that, Shapley value calculations assume all the features are uncorrelated (independent of each other), this assumption is often incorrect. To address this problem, we present a unified framework to calculate Shapley values with correlated features. To be more specific, we do an adjustment (Matrix formulation) of the features while calculating Independent Shapley values for the rows. Moreover, we have given a Mathematical proof against the said adjustments. With these adjustments, Shapley values (Importance) for the features become independent of the correlations existing between them. We have also enhanced this adjustment concept for more than features. As the Shapley values are additive, to calculate combined effect of two features, we just have to add their individual Shapley values. This is again not right if one or more of the features (used in the combination) are correlated with the other features (not in the combination). We have addressed this problem also by extending the correlation adjustment for one feature to multiple features in the said combination for which Shapley values are determined. Our implementation of this method proves that our method is computationally efficient also, compared to original Shapley method.

In anomaly detection, the degree of irregularity is often summarized as a real-valued anomaly score. We address the problem of attributing such anomaly scores to input features for interpreting the results of anomaly detection. We particularly investigate the use of the Shapley value for attributing anomaly scores of semi-supervised detection methods. We propose a characteristic function specifically designed for attributing anomaly scores. The idea is to approximate the absence of some features by locally minimizing the anomaly score with regard to the to-be-absent features. We examine the applicability of the proposed characteristic function and other general approaches for interpreting anomaly scores on multiple datasets and multiple anomaly detection methods. The results indicate the potential utility of the attribution methods including the proposed one.

The coefficient of determination, the $R^2$, is often used to measure the variance explained by an affine combination of multiple explanatory covariates. An attribution of this explanatory contribution to each of the individual covariates is often sought in order to draw inference regarding the importance of each covariate with respect to the response phenomenon. A recent method for ascertaining such an attribution is via the game theoretic Shapley value decomposition of the coefficient of determination. Such a decomposition has the desirable efficiency, monotonicity, and equal treatment properties. Under a weak assumption that the joint distribution is pseudo-elliptical, we obtain the asymptotic normality of the Shapley values. We then utilize this result in order to construct confidence intervals and hypothesis tests regarding such quantities. Monte Carlo studies regarding our results are provided. We found that our asymptotic confidence intervals are computationally superior to competing bootstrap methods and are able to improve upon the performance of such intervals. In an expository application to Australian real estate price modelling, we employ Shapley value confidence intervals to identify significant differences between the explanatory contributions of covariates, between models, which otherwise share approximately the same $R^2$ value. These different models are based on real estate data from the same periods in 2019 and 2020, the latter covering the early stages of the arrival of the novel coronavirus, COVID-19.

We share observations and challenges from an ongoing effort to implement Explainable AI (XAI) in a domain-specific workflow for cybersecurity analysts. Specifically, we briefly describe a preliminary case study on the use of XAI for source code classification, where accurate assessment and timeliness are paramount. We find that the outputs of state-of-the-art saliency explanation techniques (e.g., SHAP or LIME) are lost in translation when interpreted by people with little AI expertise, despite these techniques being marketed for non-technical users. Moreover, we find that popular XAI techniques offer fewer insights for real-time human-AI workflows when they are post hoc and too localized in their explanations. Instead, we observe that cyber analysts need higher-level, easy-to-digest explanations that can offer as little disruption as possible to their workflows. We outline unaddressed gaps in practical and effective XAI, then touch on how emerging technologies like Large Language Models (LLMs) could mitigate these existing obstacles.

With the advances in computationally efficient artificial Intelligence (AI) techniques and their numerous applications in our everyday life, there is a pressing need to understand the computational details hidden in black box AI techniques such as most popular machine learning and deep learning techniques; through more detailed explanations. The origin of explainable AI (xAI) is coined from these challenges and recently gained more attention by the researchers by adding explainability comprehensively in traditional AI systems. This leads to develop an appropriate framework for successful applications of xAI in real life scenarios with respect to innovations, risk mitigation, ethical issues and logical values to the users. In this book chapter, an in-depth analysis of several xAI frameworks and methods including LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) are provided. Random Forest Classifier as black box AI is used on a publicly available Diabetes symptoms dataset with LIME and SHAP for better interpretations. The results obtained are interesting in terms of transparency, valid and trustworthiness in diabetes disease prediction.

We introduce a novel metric for measuring semantic continuity in Explainable AI methods and machine learning models. We posit that for models to be truly interpretable and trustworthy, similar inputs should yield similar explanations, reflecting a consistent semantic understanding. By leveraging XAI techniques, we assess semantic continuity in the task of image recognition. We conduct experiments to observe how incremental changes in input affect the explanations provided by different XAI methods. Through this approach, we aim to evaluate the models' capability to generalize and abstract semantic concepts accurately and to evaluate different XAI methods in correctly capturing the model behaviour. This paper contributes to the broader discourse on AI interpretability by proposing a quantitative measure for semantic continuity for XAI methods, offering insights into the models' and explainers' internal reasoning processes, and promoting more reliable and transparent AI systems.

Explainable Artificial Intelligence (XAI) is essential for building advanced machine learning-powered applications, especially in critical domains such as medical diagnostics or autonomous driving. Legal, business, and ethical requirements motivate using effective XAI, but the increasing number of different methods makes it challenging to pick the right ones. Further, as explanations are highly context-dependent, measuring the effectiveness of XAI methods without users can only reveal a limited amount of information, excluding human factors such as the ability to understand it. We propose to evaluate XAI methods via the user's ability to successfully perform a proxy task, designed such that a good performance is an indicator for the explanation to provide helpful information. In other words, we address the helpfulness of XAI for human decision-making. Further, a user study on state-of-the-art methods was conducted, showing differences in their ability to generate trust and skepticism and the ability to judge the rightfulness of an AI decision correctly. Based on the results, we highly recommend using and extending this approach for more objective-based human-centered user studies to measure XAI performance in an end-to-end fashion.

Product retrieval systems have served as the main entry for customers to discover and purchase products online. With increasing concerns on the transparency and accountability of AI systems, studies on explainable information retrieval has received more and more attention in the research community. Interestingly, in the domain of e-commerce, despite the extensive studies on explainable product recommendation, the studies of explainable product search is still in an early stage. In this paper, we study how to construct effective explainable product search by comparing model-agnostic explanation paradigms with model-intrinsic paradigms and analyzing the important factors that determine the performance of product search explanations. We propose an explainable product search model with model-intrinsic interpretability and conduct crowdsourcing to compare it with the state-of-the-art explainable product search model with model-agnostic interpretability. We observe that both paradigms have their own advantages and the effectiveness of search explanations on different properties are affected by different factors. For example, explanation fidelity is more important for user's overall satisfaction on the system while explanation novelty may be more useful in attracting user purchases. These findings could have important implications for the future studies and design of explainable product search engines.