Open research questions in Explainable Artificial Intelligence (XAI)

What the literature leaves open

• The need for manual concept annotation is a major bottleneck in classical Concept-Bottleneck Models. Automated concept discovery methods have been proposed, but their effectiveness and interpretability are unknown. The study aims to fill this gap by evaluating the effectiveness of automated concept discovery methods for fine-grained fungal classification.

  Concept-Bottleneck Models with Expert-Discovered Versus Automatically-Discovered Concepts: A Comparative Study on Fine-Grained Fungal Classification · 2026 · DOI

• One challenge is the difficulty of identifying and visualizing computational mechanisms underlying language model capacities. Another challenge is the lack of stability in LLM representations, as evidenced by contextualization errors. The paper also notes the challenge of deciding whether to classify LLM representations as intentions, given their differences from human intentions.

  Intention-like representations in language models? · 2026 · DOI

• One challenge is the difficulty of understanding why AI/ML systems make certain decisions. Another challenge is the need to provide meaningful action guidance. A third challenge is the requirement to provide normative reasons for AI/ML decisions.

  Justifiability and AI: putting explainability in its place · 2026 · DOI

• The gap in current post hoc explainability algorithms is their inability to provide meaningful action guidance. The gap in current approaches to trust in AI/ML systems is their reliance on understanding. The paper argues that justifiability can fill this gap.

  Justifiability and AI: putting explainability in its place · 2026 · DOI

• It highlights the difficulty of ensuring that explanations are designed in a manner that fosters user trust and adaptability. The paper also notes the challenge of balancing technical transparency with the need for narrative authority and interpretive autonomy.

  Towards XAI as identity protection · 2026 · DOI

• Existing feature explanation methods have limitations in terms of efficiency and quality. There is a need for a novel approach to compute succinct and conformant explanations.

  Best of both worlds: contextual feature explanation · 2026 · DOI

• The need to balance accuracy and interpretability in image classification models. The challenge of providing insights into the model's decision-making process, particularly in high-stakes applications. The limitation of prior work in generalizing to novel domains such as ImageNet.

  InfoDisent: Explainability of image classification models by information disentanglement · 2026 · DOI

• The need for a hybrid approach that combines the strengths of post-hoc and inherently interpretable methods. The lack of transparency in neural networks, particularly in high-stakes applications. The limitation of prior work in providing insights into the model's decision-making process.

  InfoDisent: Explainability of image classification models by information disentanglement · 2026 · DOI

• The lack of consideration of relative complexity or similarity of domains in benchmarks. The difficulty of computing the minimum description length (MDL) of a system. The need to generalize testing to include more sub-tests without biasing the benchmark.

  The Artificial Intelligence Quotient (AIQ): Measuring Machine Intelligence Based on Multi-Domain Complexity and Similarity · 2025 · DOI

• The modest performance prevents EBM from being used as a clinical decision support tool, when applied to limited data.

  Hypothesis Generation via Interpretable Machine Learning: A Case Study on Risk Factors for Postradiation Therapy Lung Cancer Recurrence · 2026 · DOI

• EBM yielded competitive performances and more interpretable insights compared with logistic regression and random forest but was not immune from generalizability challenges arising from limited data.

  Hypothesis Generation via Interpretable Machine Learning: A Case Study on Risk Factors for Postradiation Therapy Lung Cancer Recurrence · 2026 · DOI

• Artificial intelligence (AI) systems increasingly make decisions that require human trust, yet little is known about how users evaluate these decisions when moral reasoning is involved.

  Aligning explanations with human values: context-sensitive trust calibration in AI decision systems · 2026 · DOI

• This suggests that lack of explainability can indeed be a major limitation for the application of ML models.

  Explainability Versus Accuracy of Machine Learning Models: The Role of Task Uncertainty and Need for Interaction with the Machine Learning Model · 2025 · DOI

• Beyond confirming the established observables, our analysis suggests combinations of known observables that may be useful for future tagging studies. An example is rλ to isolate radial jet structure while remaining decorrelated from multiplicity. Combinations involving fragmentation entropy or charge-related observables show how ParticleNet uses information not captured by the leading substructure observables.

  The physics behind ML-based quark-gluon taggers · 2026 · DOI

• The lack of transparency and interpretability in AI-driven marketing automation systems. The need for XAI frameworks that can enhance model transparency without compromising predictive accuracy.

  Designing Explainable AI Based Marketing Automation Architectures for Healthcare and Financial Applications · 2026 · DOI

• The superior performance of SVM and LR across all datasets can be attributed to their robustness against overfitting in low-dimensional feature spaces and their capability to maintain stable margins under limited data conditions.

  Explainable artificial intelligence for cross domain evaluation of predictive models in multi-disease diagnosis · 2026 · DOI

• Future research should focus on refining hybrid models that balance the computational the efficiency interpretability and fairness of ML algorithms.

  Advancements in Machine Learning Algorithms for Big Data Analytics · 2026 · DOI

• Future research should focus on developing hybrid approaches that combine strengths of multiple XAI methods, particularly for multisource datasets like WheatGrain, presents a promising research direction for improving explanation quality.

  Comparative analysis of interpretable artificial intelligence methods · 2026 · DOI

• Spurious correlations and bias are perpetuated when training data lack diversity. Explanation instability occurs when slight changes in inputs yield inconsistent saliency maps.

  Explainable ensembles and transformer‑centric learning in vision and beyond: Methods, validation and deployment assurance · 2026 · DOI

• Despite the increasing amount of proposed methods in research, only a few implementations exist, whose interfaces and requirements vary widely.

  counterfactuals : An R Package for Counterfactual Explanation Methods · 2025 · DOI

• Deciding whether to classify these representations as intentions will require careful consideration of the costs and pay-offs of extending the concept beyond the paradigm human…

  Intention-like representations in language models? · 2026 · DOI

• The performance is often limited in complex real-world settings, where heterogeneous representations, synonyms, and noisy data make it difficult to design comprehensive rules, - PINE explanations can fail to capture the discriminative difference…

  PINE: Extracting Correlated Token Pairs for Explainable Entity Matching · 2026 · DOI

• Improving training data or model design to capture discriminative differences between tokens, - Exploring the use of PINE in other domains…

  PINE: Extracting Correlated Token Pairs for Explainable Entity Matching · 2026 · DOI

• A full token-level study, including the attention-mediated interactions through which information is distributed across positions - Causal mediation analysis - Extending the analysis to broader datasets, additional architectures, and more diverse…

  SYNAPSE: Framework for neuron analysis and perturbation in sequence encoding · 2026 · DOI

• Developing a more effective approach to examine existing domains and integrate them into a measure without having to change their implementation, - Creating a standard framework or testing system that can be widely adopted by researchers in the field to quantify a system's…

  The Artificial Intelligence Quotient (AIQ): Measuring Machine Intelligence Based on Multi-Domain Complexity and Similarity · 2025 · DOI

Most-cited papers in Explainable Artificial Intelligence (XAI)

• From local explanations to global understanding with explainable AI for trees · Nature Machine Intelligence · 2020 · 10,419 citations
• Explanation in artificial intelligence: Insights from the social sciences · Artificial Intelligence · 2018 · 4,019 citations
• A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI · IEEE Transactions on Neural Networks and Learning Systems · 2020 · 2,233 citations
• Explainability for artificial intelligence in healthcare: a multidisciplinary perspective · BMC Medical Informatics and Decision Making · 2020 · 2,114 citations
• Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence · Information Fusion · 2023 · 1,623 citations
• Machine Learning: An Applied Econometric Approach · The Journal of Economic Perspectives · 2017 · 1,603 citations
• TabNet: Attentive Interpretable Tabular Learning · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · 2021 · 1,596 citations
• The false hope of current approaches to explainable artificial intelligence in health care · The Lancet Digital Health · 2021 · 1,559 citations
• Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning · Perspectives on Psychological Science · 2017 · 1,535 citations
• The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI · International Journal of Human-Computer Studies · 2020 · 1,503 citations

Most recent work

• Towards trustworthy AI in Industry 5.0: Ante-hoc interpretability with deep learning · Computers & Industrial Engineering · 2026
• SΔϕ-62 — World Model Kernel: Observed Trace, Inference, UMR, Binding Status, and Revision Path Protocols (v1.1, AI-Readable Kernel Package) · Zenodo (CERN European Organization for Nuclear Research) · 2026
• A comprehensive review of explainable artificial intelligence in healthcare methods, evaluation, and clinical integration · iScience · 2026
• SΔϕ-62 — World Model Kernel: Trace–UMR–Binding Protocol for AI Inference · Zenodo (CERN European Organization for Nuclear Research) · 2026
• Concept-based Explainable Artificial Intelligence: A Survey · ACM Computing Surveys · 2026
• Explainable artificial intelligence for molecular design in pharmaceutical research · Chemical Science · 2026
• Smart and transparent grid stability prediction for efficient energy management using explainable AI · Energy Strategy Reviews · 2026
• Explainable AI-Driven Quality and Condition Monitoring in Smart Manufacturing · Sensors · 2026
• Uncertainty explanation of artificial intelligence models by SHAP · Knowledge-Based Systems · 2026
• Shedding Light on Explainable AI: Insights, Challenges, and the Future of Infrastructure Management · ISPRS International Journal of Geo-Information · 2026

Related topics in Computer Science