The need for integrating 'black box' models into research and clinical workflows

The gap

Evidence profile

Sourced from the future work and conclusions and inline gaps and limitations and stated research gap of the source papers, classified as general, drawn from work published between 2024 and 2026, spanning 6 journals. Those papers have been cited 97 times in total.

Supporting evidence — 6 representative gaps

• Uncloaking the black-box: the need for explainable artificial intelligence in clinical microbiology and infectious diseases applications (2026) · Frontiers in Public Health · doi

  Transparency mandated by regulatory frameworks such as GDPR (EU) and Health Insurance Portability and Accountability Act (HIPAA, US) highlights the importance of AI explainability in healthcare data management. Balancing interpretability and predictive performance become increasingly complex as AI models evolve but remain pertinent for clinical acceptance and patient safety (73). Ensuring robustness (consistency despite data variability) and fairness (equitable model performance across diverse populations) is vital for user trust and reliability. Tailored explanations for clinicians, patients, and healthcare administrators through intuitive and pri- vacy-preserving interfaces can enhance AI usability and acceptance significantly. A strategic shift towards inherently transparent, inte- grated models embedded directly within clinical workflows can reduce skepticism, support informed clinical decisions, and ulti- mately enhance patient outcomes. Such ethically sound AI frame- works require interdisciplinary collaboration involving clinicians, data scientists, ethicists, and regulatory bodies. Five steps are recommended to implement XAI: (i) Curate. Create structured, high-quality datasets, e.g., MALDI-TOF spectra linked to resistance phenotypes. (ii) Clarify. Use interpretable models, e.g., SHAP with gradient-boosted trees for predicting anti- microbial resistance (e.g., against carbapenems). (iii) Validate. Prospectively test models in clinical routines, e.g., AI-driven early sepsis diagnosis in emergency units. (iv) Train. Educate specialists with interactive XAI workshops, e.g., interpreting SHAP visuals for MDR tuberculosis. (v) Comply. Align transparently with ethical and regulatory frameworks, e.g., IVDR and GDPR for trustworthy clini- cal integration. Open data initiatives are essential for equitable AI and can enable researchers, particularly those in low- and middle-income countries, to effectively develop and validate AI models. Encouraging inclusive international collaboration and data sharing can address global health disparities by ensuring AI solutions accurately repre- sent diverse populations. Furthermore, structured education pro- grams promoting AI literacy among healthcare professionals will be crucial for successful AI integration into clinical practice and public health systems.

  generalfuture work
  Keywords: models clinical regulatory health healthcare frameworks gdpr performance acceptance patient ensuring equitable diverse populations clinicians
• From Promising Capabilities to Pervasive Bias: Assessing Large Language Models for Emergency Department Triage (2026) · Journal of Healthcare Informatics Research · doi

  While a full exploration of clinical integration is beyond the scope of this work, we hope our study serves as a foundational step, by addressing two critical pil- lars of clinical AI, predictive accuracy and bias, and provides a springboard for future research that brings these systems closer to safe, real-world deployment.

  generalconclusions
  Keywords: clinical full exploration integration beyond scope hope serves foundational step addressing critical lars predictive accuracy
• Determinants of Trust and Reliance on Artificial Intelligence in Clinical Settings: A Study on Physicians’ Use of Large Language Model in Psychiatric Symptom Assessment (2026) · Psychiatry Investigation · doi

  The role of explainability and transparency in foster- ing trust in AI healthcare systems: a systematic literature review, open issues and potential solutions. The switch rate alone was insufficient, as participants were instructed to maximize assessment accuracy by referencing LLM outputs, not simply to accept or reject them.

  generalinline gaps
  Keywords: role explainability transparency foster trust healthcare systems systematic literature review open issues potential solutions switch
• Graph neural networks for multi-scale gene-drug interaction modeling in cancer: applications, challenges, and therapeutic opportunities for breast cancer drug resistance (2026) · Frontiers in Oncology · doi

  Looking ahead, future research should focus on three priorities: (i) establishing standardized experimental validation pipelines for computational predictions, (ii) developing shared benchmark re- sources that integrate multi-omics, drug response, and resistance data, and (iii) designing clinically interpretable and generalizable GNN models capable of generating mechanistically testable hy- potheses across diverse patient cohorts. Third, significant methodological challenges including limited data availability, dataset heterogeneity, poor model generalizability to unseen data, and inadequate interpretability continue to constrain clinical adoption and must be systematically addressed through standardized benchmarking and explainable AI approaches.

  generalconclusions
  Keywords: standardized looking ahead future focus three priorities establishing experimental validation pipelines computational predictions developing shared
• Explainable Artificial Intelligence for Drug Discovery and Development: A Comprehensive Survey (2024) · IEEE Access · cited 97× · doi

  S. Dhanorkar, C. T. Wolf, K. Qian, A. Xu, L. Popa, and Y. Li, ‘‘Who needs to know what, when?: Broadening the explainable ai (xai) design space by looking at explanations across the ai lifecycle,’’ in Designing Interactive Systems Conference 2021, 2021, pp. 1591–1602. A. Adadi and M. Berrada, ‘‘Explainable ai for healthcare: from black box to interpretable models,’’ in Embedded Systems and Artificial Intelligence: Proceedings of ESAI 2019, Fez, Morocco. Springer, 2020, pp. 327–337. Y. Zhang, P. Tiňo, A. Leonardis, and K. Tang, ‘‘A survey on neural network interpretability,’’ IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 5, no. 5, pp. 726–742, 2021.

  generallimitationsevidence 5/5
  Keywords: intelligence artificial ieee explainable systems based drug interpretable neural transactions information publication access deep machine
• Recent Progress and Challenges of Digital Health and Bioengineering (2026) · Applied Sciences · doi

  The paper identifies the need for integrating 'black box' models into research and clinical workflows. The paper highlights the need for ensuring data privacy and mitigating biases in AI algorithms. The paper discusses the regulatory and liability issues associated with the use of AI in digital health and bioengineering.

  generalstated research gapevidence 5/5
  Keywords: paper identifies need integrating black box models research

Questions about this gap