Open research questions in Explainable Artificial Intelligence (XAI)

What the literature leaves open

• While explainable AI (XAI) has largely been framed as a mechanism for transparency and accountability, its potential to be used as a developmental learning support for end users remains underexplored.

  Explainable artificial intelligence as AI literacy scaffolding · 2026 · DOI

• 21405479 ISSN : 3139-1478 Future Scope Future research may focus on validating the framework using large-scale real-world datasets collected from hospitals, pediatric clinics, and child development centers.

  A Proposed Explainable AI Framework for Early Detection of Developmental Risks Associated with Excessive Screen Exposure in Toddlers · 2026 · DOI

• It has argued that treating AI outputs as final decisions is insufficient in environments where consistency, accountability, and traceability are essential.

  Integrating Generative AI into Enterprise Software Architectures: From Data Pipelines to Decision Intelligence Systems · 2026 · DOI

• involvement, MATERIALS AND METHODS Anticipating which supervised model will optimally fit the data a priori is challenging; this principle is referred to as the No Free Lunch theorem. Algorithms with superior theoretical predictive capabilities may sometimes fail to elucidate the links between input and output variables. Consequently, in light of the performance disparities among various algorithms, five methods will be employed: Extreme Gradient Boosting (XGBoost), Generalised Linear Model – Elastic Net Regularisation (GLMNET), SVM with linear kernel, Decision Trees (DT), and Linear Discriminant Analysis (LDA).

  Actionable Learning Analytics: Predicting University Performance Levels with Interpretable Machine Learning · 2026 · DOI

• Although CLARIFIES+ encourages explicit goal setting and bounded assistance, learners may still over-rely on AI support if guidance remains insufficiently monitored.

  The CLARIFIES+ Framework: A Contextualized Prompting Approach for Generative Artificial Intelligence · 2026 · DOI

• By integrating philosophical analysis with current developments in medical AI, the paper outlines principles for designing XAI systems that offer explanations that are not only epistemically robust but also aligned with the epistemic and practical requirements of clinical decision-making, shaping ongoing debates in medical XAI toward underexplored conceptual foundations.

  Scientific Explanations in Health Sciences: Causality, Trust, and Epistemic Adequacy · 2026

• Our approach is inspired by dual-process theories of human cognition, integrating a fast-thinking System I module for generating and verifying solutions in familiar tasks, with a slow-thinking System II module that iteratively refines predictions using self-play reinforcement learning, even when task-specific data is limited.

  Reasoning in machine vision by learning fast and slow thinking · 2026 · DOI

• 6 Gap in Literature Despite extensive research across AI, finance, and control systems, a unified framework that: • integrates risk, irreversibility, and entropy, • enforces deterministic decision validation, • provides an executive-level authority model, remains absent.

  **Chief Decision & Capital Officer (CDCO): A Governance-First Framework for Decision Authority Infrastructure in AI-Driven Systems** · 2026 · DOI

• ROI annotation dependency. The ROI overlap component Ri of ExpiScore requires pathologist-annotated masks, which are not available for the BreaKHis dataset. In the primary experiments reported here, the weight wr was set to zero and redistributed proportionally among the remaining components (effective weights: wc = 0.50, ws = 0.3125, wk = 0.1875).

  A quantitative framework for explainability assessment in medical imaging · 2026 · DOI

• ArgLLM-App is a prototype that can be extended in several direc- tions. The system could be adjusted to support QBAF depths higher than 2 while avoiding cognitive overload of users. It could also be enhanced to enable the upload of documents in formats other than PDF. Other ways to compute base confidence may also be useful [9]. The current realisation allows the use of only base LLMs from Ope- nAI, and only single LLMs. We envisage the support of LLMs from other providers, and multi-agent variants where different agents rely upon different LLMs, in the spirit of [4]. We are currently allow- ing debate of a single binary decision: multiple decisions/question answering would also be useful. Further, it would be interesting to fully integrate RAG [5] into ArgLLM-App, allowing our agents to find relevant sources autonomously and extract relevant argu- ments therefrom, as in [4]. Finally, we accommodate interactions with single users: multiple users interacting concurrently with the system and each other may bring additional value. ACKNOWLEDGEMENTS This research was partially supported by ERC under the EU’s Hori- zon 2020 research and innovation programme (grant agreement No. 101020934, ADIX). We thank all members of the Computational Logic and Argumentation group at Imperial College London, and in particular Antonio Rago and Gabriel Freedman, for their sugges- tions and feedback on earlier versions of this work. We also thank Pranay Padavala for his early contributions. REFERENCES [1] Leila Amgoud and Jonathan Ben-Naim. 2017. Evaluation of Arguments in Weighted Bipolar Graphs. In ECSQARU 2017. Springer. https://doi.org/10.1007/978-3-319- 61581-3_3 [2] Pietro Baroni, Antonio Rago, and Francesca Toni. 2018. How Many Properties Do We Need for Gradual Argumentation?. In AAAI 2018. AAAI Press. https: //doi.org/10.1609/aaai.v32i1.11544 [3] Gabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin, Antonio Rago, and Francesca Toni. 2025. Argumentative Large Language Models for Explain- able and Contestable Claim Verification. In AAAI 2025. AAAI Press. https: //doi.org/10.1609/AAAI.V39I14.33637 [4] Deniz Gorur, Antonio Rago, and Francesca Toni. 2026. Retrieval- and Argumentation-Enhanced Multi-Agent LLMs for Judgmental Forecasting. In AA- MAS 2026. International Foundation for Autonomous Agents and Multiagent Systems / ACM. https://doi.org/10.65109/SNBR1486 [5] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Se- bastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS 2020. https://proceedings.neurips.cc/ paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html [6] Nico Potyka. 2018. Continuous Dynamical Systems for Weighted Bipolar Argu- mentation. In KR 2018. AAAI Press. https://aaai.org/ocs/index.php/KR/KR18/ paper/view/17985 [7] Antonio Rago, Francesca Toni, Marco Aurisicchio, and Pietro Baroni. 2016. Discontinuity-Free Decision Support with Quantitative Argumentation Debates. In KR 2016. AAAI Press. http://www.aaai.org/ocs/index.php/KR/KR16/paper/ view/12874 [8] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain- of-Thought Prompting Elicits Reasoning in Large Language Models.

  ArgLLM-App: An Interactive System for Argumentative Reasoning with Large Language Models · 2026 · DOI

• 3Note that while there is some work on evaluating human-AI interactions [13, 34, 35], adapting these to contestable argumentation systems is a big open challenge.

  Argumentative Human-AI Decision-Making: Toward AI Agents That Reason With Us, Not For Us · 2026 · DOI

• While inspired by dual-process theories of moral cognition [32, 36], our framework abstracts away critical psychological features like emotion, development, and social context. Future iterations could enhance real-world validity by integrating these mechanisms, building on the current version’s robust foundation for modelling moral reasoning in AI. Further, the differences between the trolley and footbridge dilemma extend beyond the personal/impersonal distinction and includes causality and responsibility factors [12, 15]. While MoralityGym models the causal norm ‘personal action caused harm’, future work could extend it to other causal norms such as those involving responsibility or counterfactuals. Finally, Morality Chains assume a strict ordering of norms. While this simplifies scalarisation, it limits the representation of ‘tragic dilemmas’ where conflicting norms hold equal force. 8 CONCLUSION We introduced morality chains and MoralityGym, a framework and benchmark grounded in moral philosophy and dual-process theories of moral psychology. By modelling moral norms as hierarchically ranked deontic constraints, our approach allows agents to be evaluated not just by what they accomplish, but by how they act when moral trade-offs arise. Empirical results show that existing Safe RL methods often fail under such conditions, revealing a critical gap between current capabilities and the demands of ethical decision-making. MoralityGym addresses this by offering a testbed for developing agents that can reason through moral structure, reflect normative priorities, and ultimately behave in ways that are more aligned with human values. ACKNOWLEDGMENTS Computations were performed using infrastructure provided by the Mathematical Sciences Support unit at the University of the Witwatersrand and the Centre for High Performance Computing of South Africa. V.W. received funding from the Oppenheimer Memorial Trust Award (OMT Ref.2150701). REFERENCES David Abel, James MacGlashan, and Michael L Littman. 2016. Reinforcement Learning as a Framework for Ethical Decision Making.. In AAAI workshop: AI, ethics, and society, Vol. 16. Phoenix, AZ. Abdelrahman Abubshait and Eva Wiese. 2017. You look human, but act like a machine: Agent appearance and behavior modulate different aspects of human– robot interaction. Frontiers in Psychology 8 (2017), 1393. https://doi.org/10.3389/ fpsyg.2017.01393 Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. In International conference on machine learning. PMLR, 22–31. C Fred Alford. 2001. Whistleblowers: Broken lives and organizational power. Cornell University Press. Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Eitan Altman. 1998.

  MoralityGym: A Benchmark for Evaluating Hierarchical Moral Alignment in Sequential Decision-Making Agents · 2026 · DOI

• We discuss the contribution, limitations of our current approach and potential future research directions: • Attack Scope: This work is limited to MEA that exploit CFs. Other types of privacy attacks, such as MIA and inversion, are not examined in this paper and therefore fall outside the scope of our current analysis. A future direction is to evaluate whether the proposed mitigation strategy remains effective against this broader class of privacy attacks. • Privacy-Performance Trade-off: Integrating DP into the CF generator mitigates MEA, but it also introduces limitations including proximity, by degrading explanation quality, prediction gain, and plausibility in some cases. As future work, a more systematic study of the privacy-utility trade-off is needed, including evaluating a wider range of privacy budgets and optimization settings for the DP-based CF generator to better balance protection and explanation quality. • Focus on Deep Learning Applications: This work focuses on MEA for DNNs, where knowledge distillation is particularly effective due to the expressive capacity of DNNs and their ability to learn rich data representations. This focus limits the applicability of our findings to other model families, such as traditional baselines including tree-based or ensemble Frontiers in Artificial Intelligence 15 frontiersin.org Ezzeddine et al. 10.3389/frai.2026.1746910 methods. Since KD proved effective in the DNN setting, a natural future direction is to explore how KD-based extraction strategies can be adapted to non-DNN algorithms and whether similar performance gains can be achieved.

  Exploiting explanations for model extraction via knowledge distillation and mitigation with private counterfactuals · 2026 · DOI

• The results should be interpreted as preliminary evidence, not final proof. The current PoC has several limitations: single model execution, possible evaluation bias, limited sample size, prompt sensitivity, limited domain scope, and the absence of independent external review. These limitations do not invalidate the PoC. They define the next validation steps. The correct interpretation is that HNS-36 Structural Feedback provides preliminary evidence of improving structural stability in AI reasoning while preserving intention alignment. It does not prove universal AI alignment. It does not prove that hallucination is eliminated. It does not prove that HNS will work equally across all models, languages, domains, or deployment conditions. The next phase should test whether the observed effect replicates under independent conditions.

  EVA-HNS: A Structural Full-Stack Operating System for AI Alignment — Architecture and 50-Turn Evidence · 2026 · DOI

• Future research should focus on conducting empirical studies to validate the proposed relationships between XAI and consumer trust, particularly through experimental and longitudinal designs. Additionally, further investigation is needed to develop standardized metrics for evaluating explainability and to explore the role of moderating factors such as cultural differences, user experience, and industry context.

  Explainable Artificial Intelligence (XAI) in Personalized Marketing: A Systematic Literature Review of Algorithms, Interpretability Techniques, and Consumer Trust Implications · 2026 · DOI

• This section states what the protocol does not do and what we do not know about it. We have been explicit about uncertainty throughout the paper; this section collects the most important limitations into one place so readers can calibrate how much weight to put on what we have described. 11.1 We Cannot Demonstrate the Protocol Works The strongest limitation is the hardest to acknowledge: we cannot show that the protocol produces better results than unstructured work. The claims we have made about what the protocol catches and prevents are based on practitioner experience in one investigation. We have not run controlled comparisons, we do not have a baseline to measure against, and our ability to evaluate our own work is itself subject to the biases the protocol is supposed to reduce. Specifically, we cannot rule out: - That the investigation would have progressed similarly without the protocol, and the protocol is descriptive of what we did rather than causal in what we accomplished. - That the protocol’s specific components are not what produces the benefits we attribute to it, and some other feature of our work is what actually matters. - That the benefits we believe we see are themselves a product of the confirmation bias the protocol does not fully prevent. - That the protocol works for our investigation specifically and does not generalize to other problems, other teams, or other researchers. The honest epistemic position is that the protocol is a working pattern that we have found useful. Whether it is genuinely better than alternatives, and whether it transfers to other contexts, are open questions that would require evidence beyond what we have. 11.2 Single-Investigation Evidence Everything we have reported comes from one investigation: MFT. The problem has specific characteristics (speculative, theoretical, mathematical, exploratory) that match the protocol’s design. We do not know how the protocol would perform on problems with different characteristics: well-established rather than speculative, empirical rather than theoretical, applied rather than exploratory, bounded rather than open-ended. Our three field-instantiation sketches in Section 10 are not evidence of transferability; they are extrapolations. The protocol might transfer to the fields we sketched, or might fail in ways we cannot predict. Until practitioners in other fields actually try it and report back, the protocol’s generality is an open question. Even within theoretical physics, our evidence is limited. One investigation, one team, one human participant, one set of AI nodes. The protocol could be specific to this configuration in ways we cannot see from inside it. 11.3 Human Review as the Bottleneck The protocol relies on a human direction-setter to exercise judgment that the AI participants cannot. This is a feature—we argued in Section 3 that direction-setting should remain human—but it is also a limitation.

  Structured Truth-Finding with AI-Augmented Teams: A Multi-Node Epistemic Protocol Preserving the Scientific Method Under Capacity Scaling · 2026 · DOI

• 55 11.1 We Cannot Demonstrate the Protocol Works...................... 55 11.2 Single-Investigation Evidence............................... 55 11.3 Human Review as the Bottleneck............................. 55 11.4 AI Capability Dependence................................. 56 11.5 The Protocol Does Not Make Investigations Correct.................. 56 11.6 The Protocol Adds Overhead............................... 56 11.7 We Cannot Fully Specify the Protocol.......................... 57 11.8 Specific Unknowns..................................... 57 11.9 What This Paper Can and Cannot Do for Readers................... 57 11.10What This Section Has and Has Not Established....................

  Structured Truth-Finding with AI-Augmented Teams: A Multi-Node Epistemic Protocol Preserving the Scientific Method Under Capacity Scaling · 2026 · DOI

• GraphAware - graph learning with traditional machine learning. In this study, we introduced GraphAware, a framework that allows customizing aggregations over neigh- borhoods of a different order to train classifiers on the aggregated features for the desired use case. Thereby, we enable traditional machine learning algorithms like logistic regres- sion to analyze graph-structured data. We showed that GraphAware can analyze graph structures under transductive and inductive settings with competitive or even higher performance compared to GNNs (Graph Convolutional Networks [8], ChebyNets [6], and Graph Attention Networks [9]). Increased interpretability. Furthermore, since GraphAware uses simple machine learning models for the classification, returned results are highly interpretable, e.g., by Walke et al. Discover Artificial Intelligence (2026) 6:345 Page 13 of 15 using packages like SHAP [21]. We showed the feature importance of trained classifi- ers exemplified on the Cora dataset using averaged SHAP values to show the compat- ibility of GraphAware with SHAP [21]. This increased interpretability allows users and developers to better understand the trained models, increase trust in trained models, and facilitate decision-making based on trained models. Additionally, local interpreta- tions (i.e., evaluating the contribution of features on a single prediction) with SHAP [21] can be easily applied to each individual classifier (or averaged SHAP values over all clas- sifiers) to increase the interpretability of GraphAware for individual instances. Future directions. Nevertheless, there are some promising directions for future studies: •Scalability: First, future evaluations could further benchmark GraphAware’s scalabil- ity and generalizability on other datasets, such as the Open Graph Benchmarks [30]. •Other graph learning tasks: Additionally, GraphAware only supports node level tasks (classification or regression) and not link-level or (sub-)graph-level tasks. Aggre- gating features of node pairs (link-level) or communities of nodes (graph level) could extend the capabilities of GraphAware to execute link-level (e.g., link predictions) and graph-level (e.g., graph classification) tasks. •Heterogeneous graphs: Another limitation is the missing support for heterogeneous graphs (e.g., IMDb, MIMIC IV [31]). Customized aggregations for different meta-paths (e.g., author-writes-paper) or classifiers for each meta-path could enable GraphAware to exploit heterogeneous graph structures. •Edge features: Finally, edge features are currently not supported. A customized aggregation of node and edge features to obtain new “messages” for each respective neighbor could further extend GraphAware’s functionalities.

  GraphAware: Interpretable machine learning on graphs · 2026 · DOI

• SHAP-based interpretations revealed group-level poverty mechanisms, but individual-level prediction transparency for heterogeneous effects (e.g., income improvements reducing poverty risk only for certain demographic groups) has not been systematically mapped; conditional SHAP analysis across household types, education levels, and employment statuses is needed for targeted social policy interventions.

  Utilizing Machine Learning and Explainable AI for Assessing Income Poverty Risk: Evidence from EU-SILC 2023 in Slovakia · 2026 · DOI

• The paper found that Random Forest achieved highest F1-score and accuracy but lowest recall, while Logistic Regression achieved highest recall; the trade-off between precision and recall for identifying at-risk poverty populations in imbalanced datasets requires investigation of ensemble methods and threshold optimization strategies specific to poverty detection tasks.

  Utilizing Machine Learning and Explainable AI for Assessing Income Poverty Risk: Evidence from EU-SILC 2023 in Slovakia · 2026 · DOI

• The analysis was limited to EU-SILC 2023 Slovakia data (11,834 individuals); validation of the ML and XAI models across multiple EU-SILC years (2004-2023 historical data available) and different EU member states is needed to assess generalizability and temporal stability of the identified at-risk-of-poverty (AROP) mechanisms.

  Utilizing Machine Learning and Explainable AI for Assessing Income Poverty Risk: Evidence from EU-SILC 2023 in Slovakia · 2026 · DOI

• The paper acknowledges that nonlinear and interaction effects between poverty risk factors (e.g., combination of low education and specific household type) have not been thoroughly investigated; future work must systematically model these complex interaction mechanisms to reveal more detailed poverty formation mechanisms beyond the current additive feature contributions.

  Utilizing Machine Learning and Explainable AI for Assessing Income Poverty Risk: Evidence from EU-SILC 2023 in Slovakia · 2026 · DOI

• The paper discusses automated interpretation of blood culture gram stains using deep convolutional neural networks (reference 34), but does not specify how errors in gram stain interpretation by these models are detected, communicated to clinicians, or how confidence scores or uncertainty quantification methods are integrated into the explainable AI framework for clinical decision support.

  Uncloaking the black-box: the need for explainable artificial intelligence in clinical microbiology and infectious diseases applications · 2026 · DOI

• Reference 40 addresses fairness aspects of clinical prediction models, but the paper lacks specific guidance on detecting and mitigating bias in machine learning models for antimicrobial resistance prediction, sepsis diagnosis, or infectious disease detection across disparate patient populations, geographic regions, and resource-limited clinical laboratory settings.

  Uncloaking the black-box: the need for explainable artificial intelligence in clinical microbiology and infectious diseases applications · 2026 · DOI

• The paper references discovery of antibiotics with explainable deep learning (reference 17), but does not specify validation protocols for ensuring that explanations of deep learning predictions align with known microbial biology, resistance mechanisms, and pharmacological properties of discovered compounds before clinical translation.

  Uncloaking the black-box: the need for explainable artificial intelligence in clinical microbiology and infectious diseases applications · 2026 · DOI

Most-cited papers in Explainable Artificial Intelligence (XAI)

• Explaining individual predictions when features are dependent: More accurate approximations to Shapley values · Artificial Intelligence · 2021 · 735 citations
• A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME · Advanced Intelligent Systems · 2024 · 651 citations
• What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research · Artificial Intelligence · 2021 · 531 citations
• Explainability for Large Language Models: A Survey · ACM Transactions on Intelligent Systems and Technology · 2024 · 504 citations
• Artificial intelligence and illusions of understanding in scientific research · Nature · 2024 · 499 citations
• Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions · Information Fusion · 2024 · 490 citations
• Trust in AI: progress, challenges, and future directions · Humanities and Social Sciences Communications · 2024 · 357 citations
• Using Explainable Artificial Intelligence to Improve Process Quality: Evidence from Semiconductor Manufacturing · Management Science · 2021 · 262 citations
• Knowledge graphs as tools for explainable machine learning: A survey · Artificial Intelligence · 2021 · 247 citations
• Relation between prognostics predictor evaluation metrics and local interpretability SHAP values · Artificial Intelligence · 2022 · 247 citations

Most recent work

• SΔϕ-62 — World Model Kernel: Observed Trace, Inference, UMR, Binding Status, and Revision Path Protocols (v1.1, AI-Readable Kernel Package) · Zenodo (CERN European Organization for Nuclear Research) · 2026
• SΔϕ-62 — World Model Kernel: Trace–UMR–Binding Protocol for AI Inference · Zenodo (CERN European Organization for Nuclear Research) · 2026
• Reasoning Under Load · 01: Claude Opus 4.8 — An Independent Reasoning-Integrity Evaluation · Zenodo (CERN European Organization for Nuclear Research) · 2026
• Explainable Artificial Intelligence (AI) for Medical Imaging: A Framework for Bridging the AI Trust Gap · American Journal of Roentgenology · 2026
• In Defense of Post Hoc Explanations in Medical AI · The Hastings Center Report · 2026
• Explainable artificial intelligence for cross domain evaluation of predictive models in multi-disease diagnosis · Discover Computing · 2026
• Emergent, not Immanent: A Baradian Reading of Explainable AI · 2026
• WEO Methodology Rationale: Empirical Derivation and Calibration Justification for AI Infrastructure Coordination Analysis · Zenodo (CERN European Organization for Nuclear Research) · 2026
• Evaluating General-Purpose AI with Psychometrics · Communications of the ACM · 2026
• The physics behind ML-based quark-gluon taggers · SciPost Physics · 2026

Related topics in Computer Science