Open research questions in Software Engineering Research
248 unresolved questions extracted from the limitations and future-work sections of 695 Software Engineering Research papers in our library. Each links back to the study that raised it.
What the literature leaves open
We select test samples from 17 of these programming languages for our evaluation (the remaining 2 programming languages could not be assessed due to Docker corruption).
Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning · 2026Observation(s) Theme 1: Retrieval Quality and Pipeline Architecture Dense retrieval recall bounds pipeline performance R1: Prioritize retrieval enhancement O2, O3, O5
BLAgent: Agentic RAG for File-Level Bug Localization · 2026Based on the observations (O★) from our study of agentic RAG for bug localization and its impact on program repair, we provide targeted recommendations (R★) organized by three overarching themes: (1) Retrieval Quality and Pipeline Architecture, (2) Hierarchical Localization and Fine-Grained Reasoning, and (3) Practical Utility. Each recommendation is grounded in specific findings and paired with actionable implementation strategies. Table 13 summarizes the relationships among our findings, recommendations, and supporting observations. 6.1 Theme 1. Retrieval Quality and Pipeline Architecture Recommendation (R1). Dense retrieval recall fundamentally bounds a RAG or agentic RAG pipeline—when the correct file is absent from the candidates provided to the pipeline, no subsequent reasoning can compensate. Instead of scaling 38 Md Afif Al Mamun and Gias Uddin to larger models, practitioners may focus on improving embedding quality and retrieval mechanisms. Path-aware, code-structured chunking yields 20.4% improvement over text-based chunking, and incorporating relative file paths adds another 16.9% gain to retrieval accuracy in our experiments. Similar approaches may be explored for code-aware embeddings that preserve syntactic boundaries and augment chunks with hierarchical repository context (module/file paths) to enhance semantic recall at the file level. Recommendation (R2). Retrieval pipelines should treat bug reports as multi-perspective queries rather than single text inputs. Refining queries has been found to be useful in RAG pipelines [6, 26]. Similarly, we also showed how transformations that disentangle structural (syntactic) and behavioral (semantic) aspects of a bug report allow the system to retrieve complementary code regions. This dual-channel formulation promotes balanced recall and precision, ensuring robust localization even when individual query views are incomplete. Similar approaches should be explored for better retrieval. Recommendation (R3). While RAG pipelines improve over dense retrieval, agentic reranking allows for further improvement in file-level localization. Such pipelines can generalize to function-level with minimal adaptation, and cascading them with graph-guided function-level agents represents a promising direction toward fully hierarchical, cost-efficient localization for LLM-based patch generation. Recommendation (R4). We discussed in Section 3.6 that models with substantially different sizes can still yield competitive performance within an agentic pipeline when the right context is provided to the agent. This observation enables a practical approach for cost-conscious organizations: adopt tiered LLM utilization where medium-sized models (e.g., Qwen3-32B) handle localization through agentic reasoning, reserving larger or proprietary models (e.g., GPT-4) only for computationally demanding downstream tasks like patch generation, where semantic complexity may justify the additional expen
BLAgent: Agentic RAG for File-Level Bug Localization · 2026Investigating the application of RevAgent to other code review scenarios. Comparing RevAgent with other state-of-the-art models. Exploring the potential of using other machine learning techniques to improve the performance of RevAgent. Analyzing the trade-off between performance and efficiency in RevAgent.
Existing approaches rely on a single model to identify various issues within the code, limiting their ability to handle complex scenarios. The lack of a framework that can handle the diverse, issue-specific nature of code changes. The need for a framework that can generate accurate, readable, and context-aware review comments.
Establishing accurate traceability between documentation and code presents significant challenges. LLMs have strict output token limits and struggle to maintain focus when processing everything at once. The paper identifies the need for careful tool design and human oversight to effectively use LLMs for trace discovery.
Investigating the impact of context-window management on trace discovery, - Examining the effect of surrounding documentation on trace discovery, - Developing more effective prompting approaches for LLMs
Further evaluation of Socrates across different domains and settings, - Investigation of the applicability of Socrates to other areas of program synthesis
Most existing approaches elicit supervision through labeled examples, which are often error-prone and may fail to capture user intent. Manual input labeling can be cumbersome and error-prone in structure-rich domains. Most existing methods assume a fixed set of inputs over which the synthesized program will be evaluated and only guarantee correctness over those.
Existing software metrics are not useful for answering practical questions, - Software metrics were developed without understanding metrology, - There is a lack of comprehensive measurement models that estimate a construct using multiple metrics
Traditional penetration testing methods can be inefficient and labor-intensive, - Scaling traditional methods to meet the complexity and rapid evolution of modern IT systems is challenging, - Current evaluation frameworks for cybersecurity models do not adequately measure a model's ability to discover novel vulnerabilities, - Developing standardized automated benchmarks remains a significant challenge
Traditional cybersecurity Q&A benchmarks do not provide a realistic and rigorous standard for evaluating LLM's technical knowledge, reasoning capabilities, and practical utility in dynamic penetration testing scenarios. Current evaluation frameworks for cybersecurity models primarily focus on predefined scenarios and known attack patterns. Developing standardized, automated benchmarks that reflect real-world conditions remains a significant challenge for advancing the evaluation of cybersecurity LLMs.
This divergence highlights a key insight: relying on a single reasoning paradigm is insufficient for detecting the complex and diverse vulnerabilities that arise in real-world scenarios.
Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detection · 2026Little is known about cross-ecosystem packages, especially regarding how they are structured.
Cross-Ecosystem Packages As Multilingual: Prevalence, Architecture, and Health · 2026However, existing benchmarks typically focus on final functional correctness or confine instruction-following evaluation to single-turn, general chat or simple code generation scenarios, leaving instruction-following in multi-turn agentic coding underexplored.
MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding · 2026Text-to-Time Series Generation (Text-to-TS) provides a promising paradigm for synthesizing time series from natural language, enabling scenario-specific generation when real observations are scarce or costly to acquire.
CodeTS: Verifiable Text-to-Time Series Generation via Executable Code · 2026These results suggest that explicit attribution alone is insufficient; durable improvement emerges when attribution is integrated with repair selection and persistent retest-conditioned update.
RESKILL: Explicit Failure Attribution and Structured Repair for Interactive Language Agents · 2026Despite their importance, scarce research systematically reports their value and provides constructive guidelines for programmers.
Test-time refinement aims to improve generated programs through additional inference, but its value after an initial candidate has been produced remains unclear.
the lack of systematic collation in LLM-based code migration research, - the lack of technical reliability and evaluation testing dimensions
Specifically, future work should focus on developing verification methods independent of complete test suites, constructing architecture-aware models that can understand project-level dependencies, and incorporating non-functional goals such as efficiency and security into migration constraints. However, the research also reveals the common limitations of current code migration technologies, mainly including over-reliance on existing test suites, research granularity mostly limited to the function level with a lack of project-level architectural awareness, and general neglect of non-functional attributes such as code performance and security.
Developing more robust and consistent LLMs. Improving the recall rate for Type-4 clone detection. Exploring new technical ways for code clone detection.
The lack of systematic surveys on LLM-based clone detection. The limited generalization ability of traditional fine-tuned models.
AI systems remain limited in their ability to support complex architectural reasoning. AI-generated code may not reflect unique coding conventions and architectural constraints used within organizations. Evaluating AI-generated code requires assessing maintainability, security practices, and architectural consistency.
The Metamorphosis of Code: How Artificial Intelligence is Reshaping Software Architecture · 2026 · DOIFuture research should focus on improving the ability of AI systems to support complex architectural reasoning. Developing more robust testing methodologies is necessary to address the fragility of AI-generated test code.
The Metamorphosis of Code: How Artificial Intelligence is Reshaping Software Architecture · 2026 · DOI
Most-cited papers in Software Engineering Research
- Large Language Models for Software Engineering: A Systematic Literature Review · ACM Transactions on Software Engineering and Methodology · 2024 · 664 citations
- Programmers use slices when debugging · Communications of the ACM · 1982 · 538 citations
- A few billion lines of code later · Communications of the ACM · 2010 · 501 citations
- Top 10 list [software development] · Computer · 2001 · 487 citations
- Large Language Models for Software Engineering: Survey and Open Problems · 2023 · 442 citations
- Grounded Copilot: How Programmers Interact with Code-Generating Models · Proceedings of the ACM on Programming Languages · 2023 · 398 citations
- Successful combination of database search and snowballing for identification of primary studies in systematic literature studies · Information and Software Technology · 2022 · 396 citations
- Program comprehension during software maintenance and evolution · Computer · 1995 · 391 citations
- Automated Program Repair in the Era of Large Pre-trained Language Models · 2023 · 365 citations
- Using an LLM to Help With Code Understanding · 2024 · 283 citations
Most recent work
- On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub · ACM Transactions on Software Engineering and Methodology · 2026
- Generative Monoculture: Model Collapse in Code as Systemic Vulnerability (EA-UMBML-MONOCULTURE-01 v1.1) · Zenodo (CERN European Organization for Nuclear Research) · 2026
- LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead · ACM Transactions on Software Engineering and Methodology · 2026
- M2CVD: Enhancing Vulnerability Understanding through Multi-Model Collaboration for Code Vulnerability Detection · ACM Transactions on Software Engineering and Methodology · 2026
- Visual Argument Structure Tool (VAST) Version 1.0 · Meta-Psychology · 2026
- How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach · ACM Transactions on Software Engineering and Methodology · 2026
- Helping LLMs improve code generation using feedback from testing and static analysis · Discover Artificial Intelligence · 2026
- A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models · ACM Transactions on Software Engineering and Methodology · 2026
- Abstract Syntax Tree for Programming Language Understanding and Representation · ACM Transactions on Software Engineering and Methodology · 2026
- Generative Monoculture: Model Collapse in Code as Systemic Vulnerability (EA-UMBML-MONOCULTURE-01 v1.0) · Zenodo (CERN European Organization for Nuclear Research) · 2026
Find a gap in your own Software Engineering Research sub-topic
This page shows what the Software Engineering Research literature already flags as unresolved. To narrow it to your specific question, run the guided finder — it searches the gap library on demand and checks candidates against 250M+ OpenAlex works.
Open the Research Gap Finder →