Challenges in the use of AI in safety-critical domains, such as the military

The gap

Evidence profile

Sourced from the future work and recommendations and stated challenges of the source papers, classified as general, spanning 3 journals. Those papers have been cited 1 times in total.

Supporting evidence — 3 representative gaps

• ARTIFICIAL INTELLIGENCE: IMPLICATIONS ON MILITARY DECISION MAKING (2026) · Journal of Defense Resources Management · doi

  from study. the Empirical investigation of human- AI teaming in live or simulated test military whether the theoretical automation bias risks identifi ed here manifest consistently under operational stress time pressure. Comparative and analysis of how major military powers are institutionalizing AI in doctrine — and whether divergent interoperability approaches create challenges risks within alliances — represents an urgent strategic research priority.

  generalfuture work
  Keywords: military whether risks empirical investigation human teaming live simulated test theoretical automation bias identi here
• Beyond Automation: The Ethical, Doctrinal, and Operational Challenges of Human-AI Collaboration in Military Decision-Making (2026) · De Securitate et Defensione O Bezpieczeństwie i Obronności · doi

  As artificial intelligence systems assume increasingly critical roles in military operations, designing effective human-AI teams becomes a key strategic objective. Beyond just technol- ogy, the success of these teams depends on organizational vision, ethical alignment, clear op- erations, and consistent doctrine. Human-AI interaction is not just a technical interface; it’s a socio-technical relationship marked by mutual adaptation, shared agency, and institutional accountability. A widely supported principle in the emerging doctrine of human-AI collaboration is maintaining meaningful human control over the system. This concept emphasizes that humans must hold ultimate authority over critical decisions, particularly those involving the use of lethal force. Military organizations seek to implement this principle through systems such as human- in-the-loop and human-on-the-loop, which enable humans to approve or override machine rec- ommendations. For example, systems such as the U.S. Navy’s Aegis Combat System provide rapid threat assessment and targeting but require human confirmation before firing35. These frameworks aim to strike a balance between ethical and legal accountability while leveraging AI’s rapid processing power. However, during rapid-fire scenarios, such as missile interception or cyber defense, the time needed for human confirmation often becomes impractical. In such cases, delays caused by human decision-making could jeopardize missions or endanger forces. Consequently, insisting on human intervention at all times may limit the capabilities of auton- omous systems, especially where quick action is crucial and machines outperform humans. The debate over meaningful human control also raises a deeper philosophical question about responsibility in technologically mediated warfare. As AI systems gain more autonomy, traditional boundaries of individual accountability start to blur. In complex multi-agent 34 Z. Zahedi, S. Kambhampati, Human-AI Symbiosis: A Survey of Current Approaches, arXiv 2021, https://arxiv.org/abs/2103.09990 (26.06.2025). 35 Lockheed Martin, Aegis Combat System, 27.05.2025, https://www.lockheedmartin.com/en-us/products/aegis- combat-system.html (24.06.2025). © 2025 UWS 2(11) 2025 DESECURITATE.UWS.EDU.PL 152 operations, decision-making becomes more distributed across human-machine networks. This decentralization challenges classical legal frameworks for assigning liability in cases of mis- conduct or unintended harm. The idea that a commander or operator can be entirely held re- sponsible for an AI system’s actions, especially one governed by probabilistic inference, emer- gent behavior, or opaque neural architectures, requires new models of shared responsibility and institutional accountability. These legal and doctrinal changes must

  generalrecommendations
  Keywords: human systems system accountability becomes humans aegis combat rapid legal critical military operations teams ethical
• Stop Saying "AI" (2026) · Philosophy & Technology · cited 1× · doi

  The paper identifies challenges in the use of AI in safety-critical domains, such as the military. The paper highlights the challenges of autonomous weapon systems, including the potential for mistakes and the need for clear lines of responsibility. The paper argues that the use of AI in target nomination and recommendation raises significant ethical and legal challenges.

  generalstated challengesevidence 5/5
  Keywords: paper identifies challenges use safety-critical domains military highlights

Questions about this gap