Science AI Journal
Computer Science · Research topic

Open research questions in Software Testing and Debugging Techniques

168 unresolved questions extracted from the limitations and future-work sections of 634 Software Testing and Debugging Techniques papers in our library. Each links back to the study that raised it.

What the literature leaves open

  • External Validity The generalizability of our approach is limited by the dataset’s scope, which consists of 50 real-world ESCMs from industry sources.

    Supporting System Testing with a Multi-Agent LLM-based Framework for Knowledge Graph Extraction: A Case Study with Ethernet Switch Systems · 2026

  • Developing reduction metrics that better reflect human debugging effort is an open problem in the field, and one to which we believe DRReduce could contribute, since its dependency-graph-based design enables structurally-aware manipulation of the reduced program rather than only size-based optimization. In several bugs, such as clang-22382, we observed that developers provided reduced LLVM IR programs rather than reduced C source code, suggesting that IR-level reduction is already a practical option when source-level reduction is insufficient.

    DRReduce: Enhancing Syntax-Guided Program Reduction with Dependency Reconstruction · 2026

  • The constrained random generation problem is a key challenge in property-based testing. Prior approaches to synthesis for recursive functions are complex and not expressive. There is a need for an efficient and effective solution to the constrained random generation problem.

    The Search for Constrained Random Generators · 2026 · DOI

  • Compilation and runtime errors caused by the hallucination of LLMs. Lack of testing and coverage feedback information restricting the increase of code coverage. Repetitive suppression problem causing invalid LLM-based repair and generation attempts.

    TestART: Improving LLM-based Unit <u>Test</u> ing via Co-evolution of <u>A</u> utomated Generation and <u>R</u> epair I <u>t</u> eration · 2026 · DOI

  • Evaluating the effectiveness of TestART on more datasets and comparing it with other state-of-the-art methods. Investigating the application of TestART in other domains, such as integration testing and system testing. Improving the efficiency of TestART, reducing the time cost and monetary cost.

    TestART: Improving LLM-based Unit <u>Test</u> ing via Co-evolution of <u>A</u> utomated Generation and <u>R</u> epair I <u>t</u> eration · 2026 · DOI

  • However, areas such as model-based testing, exploratory testing, and testing under extreme conditions remain underexplored.

    Mapping the landscape of deep learning in software testing: a bibliometric analysis · 2025 · DOI

  • Exploring the impact of fine tuning and prompt augmentation on ArkTS code generation. Developing more effective methods for improving ArkTS code generation with large language models.

    ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

  • Current large language models struggle with ArkTS code generation due to gaps in knowledge and handling of framework lifecycles. There is a need for a systematic evaluation of ArkTS code generation with large language models.

    ArkTS code generation: A comprehensive evaluation with large language models · 2026 · DOI

  • The lack of effective test programs and corresponding oracles for silent bug detection. The limitations of existing DL fuzzing techniques in detecting silent bugs.

    LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

  • TransFuzz's bug transfer mechanism relies on API similarity matching to select target APIs for test migration, but the paper lacks analysis of which bug categories transfer successfully across API boundaries versus those that fail (code migration failures). Characterizing the transferability landscape of different silent bug types in deep learning libraries would enable more targeted bug transfer strategies.

    LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer · 2026 · DOI

  • Generalization to other languages and domains. Investigation of the cost-effectiveness of using Large Language Models for automated software test generation.

    Large Language Models for Automated Software Test Generation · 2026 · DOI

  • Existing automated test generation tools have limitations in terms of branch coverage and bug detection. Large Language Models have not been fully explored for automated software test generation.

    Large Language Models for Automated Software Test Generation · 2026 · DOI

  • Exploratory testing is often criticized for its lack of structure, repeatability, and measurable outcomes. Challenges remain regarding model accuracy, data quality, and integration into existing workflows. Limited testing time, evolving requirements, and reduced documentation are challenges in Agile environments.

    Enhancing Agile QA Through Intelligent Exploratory Testing Approaches · 2026 · DOI

  • The development of a comprehensive framework that integrates intelligent techniques and provides measurable outcomes. The investigation of the application of AI in software testing to improve defect detection efficiency.

    Enhancing Agile QA Through Intelligent Exploratory Testing Approaches · 2026 · DOI

  • The study highlights future opportunities for improving software testing through Artificial Intelligence (AI) and Machine Learning (ML). Predictive defect analysis and intelligent automation frameworks can be explored in future research.

    Defect Leakage Analysis in Software Testing · 2026 · DOI

  • The growing complexity of software systems makes it difficult to detect every defect before release. Many software companies continue to face production issues even after completing multiple testing phases. The study identifies the need for effective testing strategies to reduce defect leakage.

    Defect Leakage Analysis in Software Testing · 2026 · DOI

  • Python’s dynamic typing makes precise call-graph construction difficult. Python’s eager importing mechanism renders file-level dependency analysis overly conservative. The lack of effective regression test selection techniques for Python is a significant challenge.

    Names Are All You Need: Effective and Safe Regression Test Selection for Python · 2026 · DOI

  • To improve test generation tools to achieve higher coverage and better metrics. To apply the failure taxonomy to improve test generation tools. To use Large Language Models to support the process of manually labelling test generation failures.

    Understanding and Overcoming Limitations of Automated Test Generation · 2026 · DOI

  • RQ2: To what degree does the mitigation of enhance automated test generation? (i) Type Tracing has a positive influence on the resulting coverage and can improve upon developer-provided type hints, yielding additional gains in branch coverage. (ii) Subprocess-Execution allows automated testing of up to 56.5% more modules and has discovered 213 unique crash causes, revealing 32 previously unknown faults. (iii) Using API Constraints achieves up to 63.9% higher code coverage for modules that impose constraints. Table I shows the achieved coverage gains, which indicate that further limitations remain to be overcome. Planned Work. We will use the insights from RQ1 not only to understand limitations of test generation, but also to develop approaches to overcome those limitations. Evaluating these newly developed approaches will then allow us to fully answer RQ2. Whenever we overcome a limitation, we enhance the underlying testing tool with the new approach, broadening the applicability of automated test generation. Evaluation. To answer RQ1, we will create a taxonomy of limitations. For RQ2, we evaluate the effectiveness of our approaches by comparing them against the default configurations. We measure and compare branch coverage, coverage over time, mutation score, bug coverage, and fault detection capability. We use the new dataset and existing datasets, as baselines. As test generation is non-deterministic, we run each generation multiple times and report the mean values. We use the Mann- Whitney U-test (α = 0.05) to compare configurations across modules and the Vargha and Delaney effect size ˆA12 to quantify overall differences. These metrics do not assume a specific data distribution, are in line with recommendations for assessing randomised algorithms, and were used in previous work,,, [11]–,. ACKNOWLEDGMENT I am incredibly grateful to Prof. Dr. Gordon Fraser, the supervisor of this thesis, for his unwavering support. REFERENCES G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” in Proc. ESEC/FSE. ACM, 2011, pp. 416–419. S. Lukasczyk and G. Fraser, “Pynguin: Automated unit test generation for Python,” in Proc. ICSE Companion, 2022, pp. 168–172. A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained Large Language Models and mutation testing,” Inf. Softw. Technol., vol. 171, p. 107468, 2024. C. Yang, J. Chen, B. Lin, J. Zhou, and Z. Wang, “Enhancing LLM-based test generation for hard-to-cover branches via program analysis,” CoRR, vol. abs/2404.04966, 2024. J. A. Pizzorno and E. D. Berger, “CoverUp: Effective high coverage test generation for python,” Proc. ACM Softw. Eng., vol. 2, no. FSE, pp. 2897–2919, 2025. C. Lemieux, J. P. Inala, S. K. Lahiri, and S.

    Understanding and Overcoming Limitations of Automated Test Generation · 2026 · DOI

  • Generalizing the approach to more complex programs with nested or multiple loops and memory manipulation. Applying the framework to various domains that use the guess-and-check paradigm.

    Guiding LLM-based Loop Invariant Synthesis via Feedback on Local Reasoning Errors · 2026

  • The lack of a framework that provides constructive feedback to an LLM in the guess-and-check paradigm. The need for a novel approach to guiding an LLM by formally verifying its local reasoning.

    Guiding LLM-based Loop Invariant Synthesis via Feedback on Local Reasoning Errors · 2026

  • ProcCtrlBench has several limitations, including an ontology that is not exhaustive. Some defect classes are only partially observable from execution traces. The current instantiation depends on process-level annotation, and the scale of this annotation remains limited.

    ProcBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents · 2026

  • Additional validation is needed before extending strong conclusions to other agent domains. The current study focuses on coding agents, and future research should explore the applicability of ProcBench to other domains.

    ProcBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents · 2026

  • The lack of iterative refinement capabilities in LLMs for automated code generation. The need for a framework that combines solution generation and specialized debugging.

    A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback · 2026

  • The complexity of software engineering agents. The variability in behavioral effects across different agent configurations. The need for large-scale studies to analyze the behavioral effects of different agent configurations.

    Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents · 2026

Most-cited papers in Software Testing and Debugging Techniques

Most recent work

Find a gap in your own Software Testing and Debugging Techniques sub-topic

This page shows what the Software Testing and Debugging Techniques literature already flags as unresolved. To narrow it to your specific question, run the guided finder — it searches the gap library on demand and checks candidates against 250M+ OpenAlex works.

Open the Research Gap Finder →

Related topics in Computer Science

Online Learning and Analytics
1,106 open questions
AI in Service Interactions
861 open questions
Quantum Computing Algorithms and Architecture
645 open questions
Teaching and Learning Programming
623 open questions
Digital literacy in education
617 open questions
Music Technology and Sound Studies
576 open questions
Virtual Reality Applications and Impacts
496 open questions
Topic Modeling
476 open questions
168 open questions have been extracted from the limitations and future-work passages of 634 Software Testing and Debugging Techniques papers in our library. Each one below links back to the study that raised it, so you can read the original claim in context.

Tools for your next paper

Gaps & Literature SearchFind open questions in any field — 120,000+ open gaps mined from our own library.Pre-CheckIs your paper ready? A Tier 1–5 acceptance probability in seconds.Journal FinderA ranked shortlist of target journals from a 17,500-venue index.Duplicate Publication CheckerCheck prior-publication & salami-slicing risk across 8 sources.Citation GeneratorFreeA DOI or title → APA, MLA, Vancouver, Chicago, BibTeX, RIS. No signup.Graphical Abstract MakerFreeTurn your findings into a colorblind-safe graphical abstract.

Compare the category — Honest roundups of the AI research tools, ours listed alongside the alternatives.

Command palette

Jump anywhere, run any action.