Open research questions in Software Testing and Debugging Techniques

What the literature leaves open

• In this paper, we proposed SMART , a novel automated testing framework that syn- ergizes code coverage with gameplay intent. By leveraging LLMs to interpret AST differences and decomposing them into semantic subgoals, SMART constructs a context-aware hybrid reward system. This system guides RL agents to not only ful- fill the functional requirements of new game updates but also to actively explore and cover the underlying structural modifications. Our experiment evaluation on Over- cooked and Minecraft demonstrates that SMART significantly outperforms traditional RL and curiosity-driven baselines, achieving an extremely high coverage of modified code branches while maintaining high task success rates. In future work, we plan to expand SMART in three directions. First, we aim to implement a closed-loop refinement mechanism to mitigate the impact of LLM hallucinations in one-off subgoals and rewards. Specifically, we plan to introduce an iterative cycle where the agent’s failure trajectories or low-coverage reports are fed back to the LLM, allowing it to dynamically debug and refine its generated semantic rewards and anchor mappings, thereby improving robustness against ambiguous code changes. Second, we aim to address the challenge of non-code updates by incorporat- ing multi-modal analysis, allowing the framework to test changes in game assets, UI layouts, and data tables that do not manifest in the AST. Finally, acknowledging the trade-off between RL exploration and deep defensive logic coverage, we will explore hybridizing SMART with symbolic execution or fuzz testing.

  Synergizing code coverage and gameplay intent: Coverage-aware game playtesting with LLM-guided reinforcement learning · 2026 · DOI

• should focus on improving its exploration capabil‑ ity. Hybridizing it with global optimization tech‑ niques such as simulated annealing, genetic algo‑ rithm, or Lévy (cid:976)light could help mitigate premature convergence and improve search diversity. Addition‑ ally, extending its application to variable‑strength and input–output relationship testing could enhance its adaptability. Exploring integration with other meta‑ heuristic strategies may further re(cid:976)ine its ef(cid:976)iciency, making SCSO a more competitive approach for t‑way test suite generation.

  Implementation of Sand Cat Swarm Optimization for Uniform T-Way Test Suite Generation · 2026 · DOI

• Future work should focus on quantitative analysis of runtime overhead, including per-test execution time, latency introduced by guardrail mecha- nisms, and scalability in CI/CD environments with large and frequently executed test suites. In this context, future research might explore hybrid agent configurations in which smaller, fine-tuned LLMs specialise in well-defined subtasks (e.

  Reliable execution of natural language test cases for GUI applications using LLM agents · 2026 · DOI

• Initial implementation of standard PIT MT with a reduced set of mutation operators showed mixed results, as the stock feedback failed to meet student needs.

  Improving students’ testing skill through mutation testing with modified feedback and training · 2026 · DOI

• , HTTP status codes, schema conformance), which are insufficient for detecting semantic faults, business logic violations, and state-dependent inconsistencies.

  MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs · 2026

• We presented a fault taxonomy, a lightweight hybrid static–semantic verifier (including a gold-free metamorphic leakage test), and a fault-injection benchmark for LLM-generated data-pipeline transformations. The verifier catches data-flow and temporal faults with perfect precision; an LLM-judge complements it on semantic target leakage but over-flags clean code and misses granularity faults. Future work: a combined verifier; detectors for value-level grain and mild data-quality faults; in-the-wild LLM- generated transformations; and broader pipeline and dialect coverage.

  Verifying LLM-Generated Data-Pipeline Transformations: A Hybrid Static–Semantic Approach for Leakage and Data-Quality Faults · 2026 · DOI

• This report is a working paper, and its claims should be weighed accordingly. The sample is two builds, by one author, in one week; the seventy-percent transfer figure is a structural estimate from those builds, not a measured ratio across a corpus. The boolean-test finding, that the descriptively written gate required repeated defect cycles while the test-written gate converged immediately, is consistent with a real effect but equally consistent with ordinary learning between a first and second build; only further aimings by other builders can separate the two. Independent execution of the translation worksheet has so far been performed by AI models rather than human strangers; it surfaced and confirmed real template defects, but the gold-standard test, a person with no shared context building a working gate from the worksheet alone, has not yet been run. Live autonomous operation is demonstrated on a single repository. Finally, this report contains no systematic survey of related work in automated code review or agent governance; it documents a pattern and its evidence, and makes no priority claims beyond what the repositories timestamp.

  One Gate, Aimed Twice: A Reusable Pattern for Deterministic AI Governance · 2026 · DOI

• In this work, we present ComPass, a novel automated patch correctness prediction technique based on contrastive learning and data argumentation. ComPass is built on top of the pre- trained model BERT by a self-supervised contrastive objective, thus learning code representa- tion from large-scale unlabeled code snippets. ComPass then utilizes code transformation rules to generate semantic-preserving patches, which are used to fine-tune BERT with a binary clas- sifier jointly. We conduct experiments on Defects4J datasets and show that ComPass signifi- cantly outperforms state-of-the-art APCA techniques in terms of accuracy, precision, recall, and F1-score. We further demonstrate that ComPass can be integrated with different models and still achieves optimal prediction performance in a cross-project scenario. In the future, we will conduct more extensive experiments to further explore the applica- bility and effectiveness of ComPass with more powerful PLMs, benchmarks, and evaluation settings. Author Contributions Quanjun Zhang: Conceptualization, Methodology, Writing - original draft, Investi- gation. Ye Shang: Software, Data curation, Writing - review & editing. Haichuan Hu: Software, Formal analysis, Writing - review & editing. Chunrong Fang: Conceptualization, Writing review & editing, Valida- tion. Zhenyu Chen: Methodology, Writing review & editing, Supervision. Liang Xiao: Methodology, Writing review & editing, Supervision. Funding This work is supported partially by the Frontier Technologies R&D Program of Jiangsu (BF2024070), National Natural Science Foundation of China (U24A20337, 62372228), Natural Science Foundation of Jiangsu Province (BK20251458) and Fundamental Research Funds for the Central Universi- ties (AE89991/463). 1 3Empirical Software Engineering (2026) 31:157 157 Page 26 of 30 Data Availibility Statement Our code and dataset are available on the repository h t t p s : / / g i t h u b . c o m / i S E n g L a b / C o m P a s s .

  ComPass: Contrastive Learning for Automated Patch Correctness Assessment in Program Repair · 2026 · DOI

• RQ2: To what degree does the mitigation of enhance automated test generation? (i) Type Tracing has a positive influence on the resulting coverage and can improve upon developer-provided type hints, yielding additional gains in branch coverage. (ii) Subprocess-Execution allows automated testing of up to 56.5% more modules and has discovered 213 unique crash causes, revealing 32 previously unknown faults. (iii) Using API Constraints achieves up to 63.9% higher code coverage for modules that impose constraints. Table I shows the achieved coverage gains, which indicate that further limitations remain to be overcome. Planned Work. We will use the insights from RQ1 not only to understand limitations of test generation, but also to develop approaches to overcome those limitations. Evaluating these newly developed approaches will then allow us to fully answer RQ2. Whenever we overcome a limitation, we enhance the underlying testing tool with the new approach, broadening the applicability of automated test generation. Evaluation. To answer RQ1, we will create a taxonomy of limitations. For RQ2, we evaluate the effectiveness of our approaches by comparing them against the default configurations. We measure and compare branch coverage, coverage over time, mutation score, bug coverage, and fault detection capability. We use the new dataset and existing datasets, as baselines. As test generation is non-deterministic, we run each generation multiple times and report the mean values. We use the Mann- Whitney U-test (α = 0.05) to compare configurations across modules and the Vargha and Delaney effect size ˆA12 to quantify overall differences. These metrics do not assume a specific data distribution, are in line with recommendations for assessing randomised algorithms, and were used in previous work,,, [11]–,. ACKNOWLEDGMENT I am incredibly grateful to Prof. Dr. Gordon Fraser, the supervisor of this thesis, for his unwavering support. REFERENCES G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in Proc. ESEC/FSE. ACM, 2011, pp. 416–419. S. Lukasczyk and G. Fraser, “Pynguin: Automated unit test generation for Python,” in Proc. ICSE Companion, 2022, pp. 168–172. A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained Large Language Models and mutation testing,” Inf. Softw. Technol., vol. 171, p. 107468, 2024. C. Yang, J. Chen, B. Lin, J. Zhou, and Z. Wang, “Enhancing LLM-based test generation for hard-to-cover branches via program analysis,” CoRR, vol. abs/2404.04966, 2024. J. A. Pizzorno and E. D. Berger, “CoverUp: Effective high coverage test generation for python,” Proc. ACM Softw. Eng., vol. 2, no. FSE, pp. 2897–2919, 2025. C. Lemieux, J. P. Inala, S. K. Lahiri, and S.

  Understanding and Overcoming Limitations of Automated Test Generation · 2026 · DOI

• Enhancements I. MOTIVATION The scale and complexity of software render manual testing an expensive and error-prone task. Search-Based Software Testing (SBST) has emerged to generate tests automatically [1], [2]. Using meta-heuristic search algorithms, SBST tools explore the input space of programs to maximise code coverage. Recently, Large Language Models (LLMs) have been integrated into SBST tools to improve their performance [3]–[9]. Despite these advancements, state-of-the-art tools and their extensions do not achieve 100% coverage. In fact, the achieved coverage is heavily influenced by the subject under test. For example, the coverage values achieved by PYNGUIN [2], [10], a test generation tool for Python, vary considerably across different datasets (see Table I): While CODAMOSA [6]—one of many extensions of PYNGUIN—achieved 91.8% coverage on a general dataset, performance on machine learning libraries [11] or projects using C-extensions [12] is lower (38.6% and 38.7%, respectively). Even if test generation tools achieve high coverage, there is room for improvement regarding other metrics, such as mutation score, bug coverage, and bug-finding capabilities [4], [13], [14]. Achieving high-quality test suites across diverse datasets has many open challenges. We have laid the first foundations by overcoming known limitations of test generation (see Fig. 1), such as missing type information [14], instability of test generation [12], and missing information on input constraints [11]. The goal of this thesis is to understand and overcome limitations of search-based and LLM-based test generation. We will use Python and PYNGUIN to evaluate our approaches, which are designed to generalise to other languages and tools. II. TEST GENERATION Current research in test generation focuses on four categories: 1. Property-Based Testing tools, such as HYPOTHESIS [15], [16], allow developers to define properties and strategies for data generation, but they remain semi-automated. 2. Symbolic Execution explores program paths systematically using symbolic inputs [17] and has been successfully applied via Dynamic Symbolic Execution [7], [18], though it often struggles with the path explosion problem. 3. Search-Based Software Testing (SBST) applies evolution- ary algorithms, such as DynaMOSA [19], to optimise test cases according to a defined fitness function [20]–[22].

  Understanding and Overcoming Limitations of Automated Test Generation · 2026 · DOI

• TransFuzz's bug transfer mechanism relies on API similarity matching to select target APIs for test migration, but the paper lacks analysis of which bug categories transfer successfully across API boundaries versus those that fail (code migration failures). Characterizing the transferability landscape of different silent bug types in deep learning libraries would enable more targeted bug transfer strategies.

  LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

• The repeated validation analysis (Fig. 11b) examines LLM stability across multiple checks per prompt for silent bug classification, but the paper does not specify how many repeated checks are empirically necessary to reach stable predictions or how performance degrades with incomplete API documentation across different deep learning libraries. Determining optimal repetition thresholds for reliable silent bug confirmation is needed.

  LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

• The window size ablation study for Bug Transfer-Guided Test Generation (Fig. 11a) identifies a plateau effect at size 10 where marginal bug discovery gains diminish beyond this threshold. However, the generalizability of this optimal window size across different cross-library transfer scenarios (PyTorch→TensorFlow, PyTorch→JAX) and varying API similarity distributions remains untested, limiting the scalability of batch testing strategy recommendations.

  LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

• LLM performance on LLM-Powered Self-Validation shows significant model-dependent variation in precision-recall trade-offs (Table 6): DeepSeek R1 achieves 76.92% precision but only 47.62% recall, while GPT-4.1 mini achieves 90.48% recall but only 65.52% precision. The strictness parameter governing validation behavior across different LLM models requires systematic investigation to develop model-agnostic calibration strategies for silent bug validation in deep learning fuzzing.

  LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

• The failure case analysis reveals that TransFuzz cannot distinguish compiler-level optimization artifacts from genuine functional bugs in gradient computation, as demonstrated by the torch.compile flex_attention case where numerical differences from compiler optimizations were misclassified. Developing domain-specific oracles that incorporate expert-level reasoning about compiler internals and floating-point precision behavior is needed for silent bug fuzzing in compiled deep learning operations.

  LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

• TransFuzz's oracle validation achieves 71.42% precision due to three specific failure modes: (1) oracle design errors causing logical flaws in bug classification, (2) code migration failures that fail to preserve bug characteristics across different PyTorch APIs, and (3) insufficient API documentation leading to LLM misjudgment. These sources of false positives (28.58% rate) require targeted improvements in oracle construction and API information enrichment for silent bug detection in deep learning libraries.

  LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

• While the paper identifies that missing imports, wrong generic bounds, and misuse of reactive forms are common failure modes in ArkTS code generation, it does not quantify how frequently each error type appears by task category (algorithmic vs. API usage vs. UI design) or provide specific metrics for error reduction from instruction templates versus example-driven prompting.

  ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

• The compiler-guided repair process only uses the first compiler diagnostic to generate a minimal patch; the paper does not explore how multiple sequential repair iterations or prioritization of different error types (bracket balance vs. unresolved symbols vs. type mismatches) would affect Pass@1 and generation time trade-offs.

  ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

• The analysis shows that API usage and UI design tasks remain harder than algorithmic tasks in ArkTS code generation, but the paper provides no investigation into which specific API categories (e.g., layout APIs, event handling, state management) contribute most to generation failures, limiting targeted improvements.

  ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

• The paper plans a second wave of evaluation adding mobile-specific APIs and UI lifecycle tasks from ArkTS repositories, but does not define the specific number of tasks, selection criteria, or how these new items will be distributed across easy/medium/hard difficulty levels to maintain comparability with the current 300-task EMH benchmark.

  ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

• The EMH ArkTS benchmark identified that models perform significantly worse on hard prompts requiring stable state handling and ArkTS-specific constructs, yet the paper does not specify concrete methods to augment training data or fine-tune light adapters specifically for reactive form state updates and lifecycle management in component-based languages.

  ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

• The pilot study on compiler-guided repair for ArkTS code generation was conducted on only 30 held-out tasks with a single model (DeepSeek-R1); validation across multiple LLMs and a substantially larger task set is needed to determine whether the two-step self-repair process generalizes beyond this limited sample and whether deeper agent workflows with more planning steps provide additional gains.

  ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

• However, alignment is highly representation-sensitive: lexical similarity alone is insufficient; full diffs provide the most stable basis for judging report-patch correspondence; and structured summaries trade surface similarity for predicate-level salience.

  Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches · 2026

• As these agents become more prevalent, ensuring the code they generate is adequately tested, by existing tests or by tests the agents write, is critical to preventing regressions, yet little is known about testing in agentic PRs.

  Test Coverage Analysis of Agentic Pull Requests · 2026

• Moreover, PanicKiller has successfully resolved 28 open issues related to panics, all of which have been confirmed and merged by developers.

  PanicFI: An Infrastructure for Fixing Panic Bugs in Real-World Rust Programs · 2026 · DOI

Most-cited papers in Software Testing and Debugging Techniques

• Software Testing With Large Language Models: Survey, Landscape, and Vision · IEEE Transactions on Software Engineering · 2024 · 382 citations
• Large Language Model guided Protocol Fuzzing · 2024 · 184 citations
• Fuzz4All: Universal Fuzzing with Large Language Models · 2024 · 166 citations
• Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT · 2024 · 121 citations
• AutoCodeRover: Autonomous Program Improvement · 2024 · 111 citations
• Large Language Models for Test-Free Fault Localization · 2024 · 107 citations
• ChatUniTest: A Framework for LLM-Based Test Generation · 2024 · 103 citations
• A Robustness Test Protocol for Applied QCA: Theory and R Software Application · Sociological Methods & Research · 2021 · 101 citations
• Evaluating and Improving ChatGPT for Unit Test Generation · Proceedings of the ACM on software engineering. · 2024 · 98 citations
• Effective test generation using pre-trained Large Language Models and mutation testing · Information and Software Technology · 2024 · 95 citations

Most recent work

• DeepVerifier: Learning to Update Test Sequences for Coverage-Guided Verification · ACM Transactions on Design Automation of Electronic Systems · 2026
• An empirical study of testing practices in open source AI agent frameworks and agentic applications · Empirical Software Engineering · 2026
• DBugScribe: Automatic Database Bug Reproduction from Community Reports · Proceedings of the ACM on Management of Data · 2026
• ArkTS code generation: A comprehensive evaluation with large language models · Empirical Software Engineering · 2026
• LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · Proceedings of the ACM on Programming Languages · 2026
• TestForgeAI - AI-Powered Test Case Generation System · International Scientific Journal of Engineering and Management · 2026
• Large Language Models for Automated Software Test Generation · Zenodo (CERN European Organization for Nuclear Research) · 2026
• Enhancing Agile QA Through Intelligent Exploratory Testing Approaches · International Journal of Creative and Open Research in Engineering and Management · 2026
• Defect Leakage Analysis in Software Testing · International Journal of Creative and Open Research in Engineering and Management · 2026
• DepMigrateJS: LLM-Based Dependency Migration for JavaScript Using Structured API Diffs and Test Failure Feedback · Zenodo (CERN European Organization for Nuclear Research) · 2026

Related topics in Computer Science