Open research questions in Software Engineering Research

What the literature leaves open

• In the same way, regarding the implications for researchers, a possible future work could explore the development of autonomous systems capa- ble of performing continuous refactoring in real-time as developers work on their codebases. Finally, although our investigation focused exclusively on the zero-shot prompt, future work could explore alternative prompting strate- gies to assess whether they enable LLMs to provide more accurate judgments. Our findings are limited to the context of a single programming language, Java. Considering the limited information provided, LLMs might start making as- sumptions, leading to the reported hallucinations.

  Refactoring with LLMs: Bridging human expertise and machine understanding · 2026 · DOI

• Future work could explore finer-grained subcategories of complementary PTM relationships based on architecture, modality, and functional role to identify potential connections between these dimensions. Large language model supply chain: Open problems from the security perspective, in: Proceedings of the 34th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. These findings highlight the gaps in understanding downstream adoption and reuse of PTM that we work to address in this work. This approach fills a critical gap in the literature by focusing exclusively on PTM reuse, rather than general ML model usage, offering actionable insights for improving re- producibility, maintainability, and design of ML-enabled systems.

  Software dependencies 2.0: An empirical study of reuse and integration of pre-trained models in open-source projects · 2026 · DOI

• Defining concern boundaries that align with human intuition remains an open problem, and we believe it warrants future investigation through approaches such as reinforcement learning from human feedback, where developer judgments can be incorporated as reward signals to guide the grouping strategy. Large language models for software engineering: Survey and open problems. These findings indicate that the multi-agent architecture of ColaUntangle not only improves untangling accuracy but also produces human-comprehensible justifications, addressing a key limitation of prior black-box approaches to commit untangling.

  LLM-Driven Collaborative Model for Untangling Commits via Explicit and Implicit Dependency Reasoning · 2026 · DOI

• While a few user studies have examined the broader impact of AI assistants on software vulnerabilities, the use of security APIs remains unexplored from a developer-centered perspective.

  Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Study · 2026

• By mapping existing studies, identifying open challenges, and proposing future research directions, this review aims to guide researchers and practitioners in advancing the development of reliable, generalizable, and practically applicable LLM-based vulnerability detection systems.

  A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models · 2026 · DOI

• Implication for Researchers Our findings suggest that keyword-based filtering alone is insufficient for robust ATD identification, although it remains valuable as an initial triaging mechanism that reduces the manual inspection space…

  Reducing labeling effort in architecture technical debt detection through active learning and explainable AI · 2026 · DOI

• In this study, we introduced a new approach to annotating and detecting ATD in Jira issue tracking systems by combining keyword-based methods with active learning. This work was motivated by the scarcity of labeled ATD instances, which makes it difficult to train effective machine learning models and hinders progress in automated technical debt detection. Our findings indicate that keyword-based filtering, although limited in recall, can identify only 21 to 33% of True-ATD items but can significantly reduce the manual labeling workload, filtering out up to 85% of Non-ATD issues. To address the limitations of keyword approaches, we applied active learning strategies that focus annotation efforts on the most informative samples. This enabled our BERT-based classifier to achieve an F1 score of 0.72 while only requiring half of the available labeled data. Our comparative analysis revealed that the Breaking Ties query strategy delivers the best performance in the early annotation stages, while Contrastive Active Learning is more effective for larger annotation budgets. By combining lightweight keyword filtering with active learning, we propose a practical and cost-efficient method for building robust ATD datasets and detection models. To improve the interpretability of model predictions, we integrated XAI techniques, such as LIME and SHAP, to enhance their transparency. To assess the effectiveness of these methods, we conducted an expert evaluation involving ten experts. The results demonstrated that both LIME and SHAP significantly improved practitioners’ understanding and trust in automated ATD classification results. Notably, most experts preferred LIME over SHAP, citing its more intuitive and accessible explanations. However, the evaluation also indicated that further customization of these methods may be beneficial to better address the specific needs of architectural contexts. To improve reproducibility and support future research, we have made our dataset publicly available, which includes a new set of 1,100 ATD items. The workflow we propose offers a scalable solution for identifying technical debt in large software projects, helping teams focus on issues with high architectural impact. Looking ahead, future research should investigate ATD identification across multiple sources, expand to additional software domains and artifact types, and explore integrating ATD detection into real-time development workflows. Further studies on human-centered evaluation of XAI will also be important for advancing technical debt management in practice.

  Reducing labeling effort in architecture technical debt detection through active learning and explainable AI · 2026 · DOI

• This benchmark evaluates the latest available version of each library from their main or master repository branch. Any metrics tracking score changes from the most recent commit are this analysis. For example, ignored as to while retained, duplicated_lines_density new_duplicated_lines_density dropped.

  An Empirical Study of Automated Machine Learning Python Libraries Using Source Code Analysis · 2026 · DOI

• Generative AI coding agents are entering the open-source supply chain, yet their diverse and often invisible traces leave their prevalence poorly understood.

  Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories · 2026

• tion, 2024. GitHub, Inc., “Collaborating with Pull Requests,” [Online]. https://docs.github.com/en/pull- GitHub Documenta- Available: requests/ collaborating-with-pull-requests Vercel, Inc., “Next.js Documentation—The React Framework for the Web,” Next.js Official [Online]. Available: Documentation, 2024.

  Gen-AI Code Reviewer: An AI-Assisted Pull Request Review Platform for Faster First-Pass Software Quality Analysis · 2026 · DOI

• This study provides evidence that, within the specific experimental configuration of frozen- T5 sentence embeddings and the Eclipse Bugzilla FCS-based labeling, representation choice and feature utilization strategy (complete vectors vs. cosine distance) are associated with differences in model performance descriptive metrics. Across the tested configurations in this frozen-T5 context, hybrid lexical-semantic representations and strong TF-IDF baselines showed higher descriptive performance than isolated frozen sentence-level T5 embeddings in the reported Wilcoxon comparisons; however, T5-related contrasts should be interpreted cautiously because repeated T5 values across preprocessing variants reduce effective independence. Within this particular setup and dataset, domain-specific lexical features demonstrated competitive performance. We emphasize that these findings are specific to the Eclipse Bugzilla dataset, FCS-based ground truth, frozen T5 configuration, and the experimental design employed. Conclusions about the broader role of semantic context-awareness in BR similarity prediction cannot be drawn from evaluation of a single semantic model architecture and would require systematic comparison across multiple semantic encoders and fine-tuning strategies. From a practical perspective, two decisions were consistently relevant: preserving complete vectors (instead of reducing them to cosine distance only) and aligning balancing strategy with recommendation goals (high-confidence top-1 vs broader top-K retrieval). These findings provide an actionable baseline for practitioners and a clear reference point for future comparisons with fine-tuned or cross-encoder architectures. This work contributes an empirical comparison of TF-IDF, frozen sentence-level T5 embeddings, and a hybrid approach for FCS-based BR relatedness prediction in Eclipse Bugzilla, accompanied by empirical descriptive evidence that complete vectors are associated with higher metrics than cosine-distance-only inputs in this specific task and dataset configuration. The structured ablation presented here clarifies when hybridization shows competitive performance and when TF-IDF remains preferable, and provides a reproducible pipeline for data retrieval, labeling, vectorization, normalization, and evaluation that can be adapted to similar bug-tracking datasets. Beyond the current findings, several promising research directions emerge naturally from this work. Investigating the generalizability of the hybrid approach across different software projects and bug tracking systems would assess its robustness and adaptability in distinct 1 3Software Quality Journal (2026) 34:34 34 Page 30 of 33 development contexts.

  Lexical and semantic representations for similar bug report detection: A TF-IDF and frozen T5 comparative study · 2026 · DOI

• Despite the evidences we raised in our study, some limitations should be acknowledged: 1 3Software Quality Journal (2026) 34:34 34 Page 28 of 33 ● Construct Validity and Semantic Mismatch: A potential threat to our study is the use of FCS as a proxy for ground truth. While FCS provides an objective and scalable labeling for large datasets, it may not always capture human-perceived semantic similarity. For example, bugs may share similar root causes and be semantically related yet be fixed in different components, resulting in low file overlap. Conversely, certain bugs may share infrastructure files without addressing the same user-facing issue. This construct-validity mismatch between FCS labels and semantic relatedness is a fundamental limitation of this proxy-based approach, though it is consistent with established literature in technical similarity for automated bug triaging. ● Scalability and Computational Cost: Training the models explored in this study can be computationally expensive, particularly due to the broad and extensive experimental design we adopted. We found substantial variation in training cost depending on the representation and feature strategy employed. Cosine-distance (CD) approaches using TF-IDF or T5 required approximately 200 seconds per epoch, whereas complete-vector (VT) strategies with T5 increased the cost to approximately 324 seconds per epoch. More expressive configurations imposed significantly higher costs: VT+TF-IDF required approximately 8,025 seconds per epoch, while the VT+Hybrid strategy reached approximately 9,427 seconds per epoch, representing a cost nearly 47 times higher than the CD baseline. In addition, heavily balancing configurations combined with completevector strategies further increased computational overhead. Therefore, reproducing the experiment with a larger number of models, datasets, or hyperparameter configurations may become costly in terms of computational resources and execution time. On the other hand, it is important to emphasize that, once trained, the models presented practical inference times, which supports the feasibility of applying approaches of this nature in real-world settings. Naturally, scenarios involving pairwise comparison among a very large number of bug reports may still require substantial computational resources; however, the prediction stage itself does not constitute a prohibitive bottleneck after training. Also, some optimization opportunities remain open for future work, both in the training/testing pipeline and in the methodology itself. For example, dimensionality reduction techniques, approximate nearest-neighbor search, model distillation, or more efficient sampling and balancing strategies could help reduce computational costs.

  Lexical and semantic representations for similar bug report detection: A TF-IDF and frozen T5 comparative study · 2026 · DOI

• This result has implications for the code domain that have not been tested: if code models from different providers (with different training data, architectures, and optimiza- tion targets) generate structurally different solutions to the same problems, then a development ecosystem that uses multiple models may be more resistant to monoculture than one dominated by a single provider.

  Generative Monoculture: Model Collapse in Code as Systemic Vulnerability (EA-UMBML-MONOCULTURE-01 v1.0) · 2026 · DOI

• Future studies are needed to validate the impact of our recommendations, and community efforts are needed to develop normative guidelines on LLM usage in SRs.

  Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations · 2026

• However, we are still lacking specialized tools for technical debt management (TDM), specifically ones that can be easily integrated into existing development workflows.

  TagDebt: a bot to support technical debt management · 2026 · DOI

• Built on Claude SDK running on AWS AgentCore runtime, Sherlock includes a human-in-the-loop mechanism for graceful degradation when information is insufficient.

  From Bug Report to Pull Request: An Autonomous Agent Pipeline for Production Issue Resolution · 2026 · DOI

• When applied to seven software products, PVDetector discovered 55 vulnerabilities, including 10 silently patched flaws that had not been previously reported.

  PVDetector: Pretrained Vulnerability Detection on Vulnerability-enriched Code Semantic Graph · 2026 · DOI

• We set out to investigate the impact of AI assistants on software maintainability. To this end, we formulated two research questions targeting evolution efficiency and code quality, respectively. We conducted a preregistered two-phase controlled experiment. In Phase 2, a randomized control trial (N=75), participants manually evolved a Phase 1 solution developed by someone else – who had either worked with AI assistants such as GitHub Copilot, ChatGPT, and Cursor (treatment), or not (control). In total, we collected 151 solutions to our realistic development tasks, 95.4% of which were completed by professional software developers. Our results provide no clear evidence that code co-developed with AI assistants is more efficient to evolve manually. Task 2 completion times were highly variable – largely due to variations in learning and setup times – and any speed advantage for evolving AI-assisted code was small and statistically unreliable. For code quality, traditional frequentist analyses found no significant differences between treatment and control. Bayesian analysis, however, suggests a small CodeHealth improvement when the original Task 1 solution was co-developed with AI by a habitual AI user. These findings indicate that, in our setting, prior AI use neither clearly improves nor degrades downstream manual evolution, with at most a small positive signal for CodeHealth in this specific subgroup. We found additional evidence that individual variation in AI proficiency matters: in Phase 1, the posterior mean effect on completion time for habitual AI users was a 55.9% speedup, compared to a 30.7% median decrease across the entire sample. We conclude that learning to work effectively with AI assistants is a valuable developer skill – and that such proficiency may even benefit others working manually downstream. But being a proficient Java developer mattered even more in our study. Some Phase 1 participants assisted by chat-based programming shared enthusiastic comments about their experience. Developers who reported the highest levels of perceived productivity all worked with AI assistants that went beyond code completion, most notably Cursor. This boost to perceived productivity did not carry over to Phase 2, where our results are consistent with a null effect. The Bayesian analysis shows a small and uncertain tendency toward lower perceived productivity when building on work by habitual AI users, but the evidence is too weak for firm conclusions. Future work should explore in what situations overly enthusiastic AI use might lead to downstream maintainability backlashes. We found no treatment effect on test coverage in Phase 2, but we observed that the few outstanding test suites in Phase 1 were all submitted by AI-assisted participants. Our observational findings from Phase 1 adds to the growing body of evidence that AI assistants can effectively accelerate development.

  Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability · 2026 · DOI

• This paper addresses the demand for a comparative study on the robustness of code generation models across multiple popular programming languages, including Java, C++, and JavaScript. Building upon the foundation of Re- Code (Wang et al., 2022), which primarily focuses on Python, we extend the robustness evaluation to a multilingual setting by constructing EvalPlus-X, A Multi-Language Perspective on the Robustness of LLM Code Generation 45 a new benchmark derived from HumanEval-X and EvalPlus with substan- tially more rigorous test cases. We evaluate six LLMs, Incoder-1B, Incoder-6B, CodeGen-2B-Multi, CodeGen-6B-Multi, Magicoder-7B, and QwenCode-2.5- 7B, under 29 distinct semantic-preserving perturbations across four prompt scopes: DocString, function name, syntax, and format. Our investigation yields several key findings. All models consistently de- grade under perturbations across all three languages, but the magnitude of robustness drops varies depending on the language and perturbation type, confirming that robustness is a language-dependent property that cannot be adequately assessed from a single language alone. Semantic and format per- turbations prove at least as disruptive as syntactic ones, underscoring the need for diverse perturbation types in robustness benchmarks. Larger model size does not reliably predict robustness; in several cases, larger models are more brittle than smaller ones under semantics-preserving perturbations. Our feature-level analysis (RQ2) further reveals that the factors driving robustness drops differ across languages, with DocString and function name perturbations most strongly associated with failures in Java and JavaScript, while C++ is more sensitive to syntax-level perturbations. Our LLM-based docstring repair strategy (RQ3) yields only marginal gains for surface-level perturbations and is ineffective for semantic ones, highlighting the limits of lightweight prompt- level mitigation. Several directions remain open for future work. This study evaluates each perturbation type in isolation, following ReCode (Wang et al., 2022), which allows the individual effect of each perturbation to be isolated. However, real- world scenarios may involve multiple simultaneous changes to a single prompt, and prior NLP robustness work has shown that compound perturbations can reveal interaction effects beyond what single perturbations expose (Dong et al., 2023; Gupta et al., 2024). Extending our framework to support combined per- turbations across multiple scopes represents a promising direction. Beyond that, evaluating robustness using white-box strategies, which provide access to model internals, could offer deeper insights beyond the black-box setting adopted here. Extending the evaluation to broader and more complex bench- marks, additional programming languages, and larger closed-source models would further strengthen the generalizability of the findings.

  A multi-language perspective on the robustness of LLM code generation · 2026 · DOI

• optimal metaheuristic algorithm for DNN hyperparameter tuning in CPDP scenarios for mobile applications. the most regarding This study provides several important contributions to SDP. First, it investigates the impact of metaheuristicbased hyperparameter tuning on DNN performance within a CPDP framework for mobile applications. Second, it provides empirical evidence on the effectiveness of PSO, GA, and GWO in optimizing DNN hyperparameters through comprehensive evaluation using ROC-AUC, F1- Score, PR-AUC, and MCC metrics, supported by statistical validation using the Wilcoxon Signed-Rank Test. Third, the application of metaheuristic-based hyperparameter tuning from WPDP scenarios to CPDP scenarios. Fourth, it identifies the most for DNN hyperparameter tuning within CPDP frameworks in the mobile application domain. reliable metaheuristic algorithm this study extends This paper is organized as follows. Section II presents the materials and methods, including the dataset, data preprocessing, data partitioning, DNN architecture, metaheuristic algorithms, and evaluation procedures. Section III reports the experimental results. Section IV discusses the interpretation of the results, comparisons with previous studies, research implications. Finally, Section V concludes the study by summarizing the main findings and outlining directions for future research. limitations, and II. Materials and Method This study presents a structured procedural workflow, as illustrated in Fig. 1. It integrates metaheuristic algorithms to optimize hyperparameters of DNN within a CPDP framework. The process begins with data collection from open-source Android mobile application datasets comprising 14 projects, each of which contains code change metrics and defect labels. This is followed by a preprocessing stage that handles missing values and normalizes the data using MinMaxScaler to scale feature values to 0-1. The processed data are then partitioned into source and target projects using the Leave-One-Out Corresponding author: Rudy Herteno, [email protected], Department of Computer Science, Faculty of Mathematics and Natural Science, Banjarbaru,Indonesia DOI: https://doi.org/10.35882/ijeeemi.v8i2.340 Copyright © 2026 by the authors. Published by Jurusan Teknik Elektromedik, Politeknik Kesehatan Kemenkes Surabaya Indonesia. This work is an open-access article and licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). 249 Indonesian Journal of Electronics, Electromedical Engineering, and Medical Informatics Homepage: https://ijeeemi.org/; Vol. 8, No. 3, pp.

  Metaheuristic-Based Hyperparameter Optimization Analysis of Deep Neural Network for Cross-Project Defect Prediction in Mobile Applications · 2026 · DOI

• In this work, we designed CLCFinder, a tool for identifying cross-language code, and constructed a real-world dataset for CLB detection from GitHub. Based on this dataset, we conducted experiments to evaluate the performance of widely-used CodeLMs in the CLB detection task. The results show significant performance variation across different CodeLMs, with small CodeLMs (the parameter size is 220M or less) tending to achieve better performance than large CodeLMs within our experimental setting. To further validate the differences between our CLB dataset and single-language bug datasets, we fine-tuned CodeLMs on SOTA single-language bug detection datasets and applied these fine-tuned CodeLMs to CLB detection. The results demonstrate significant differences in feature characteristics between single-language bug and CLB datasets, validating the effectiveness ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article. Publication date: April 2026. Fine-Tuning Code Language Models to Detect Cross-Language Bugs 37 and necessity of our CLB dataset. However, although the mainstream CodeLMs have demonstrated preliminary effectiveness in CLB detection, they are still primarily used in research settings and are not yet ready for real-world applications. We then explored factors affecting the performance of CodeLMs in CLB detection. The results show that increasing dataset size improves CodeLM performance, while the impact of token sequence length varies across CodeLMs. Some CodeLMs perform better with a shorter token sequence length, while others perform better with a longer token sequence length. Finally, we investigated the impact of code comments on model performance in CLB detection. Experimental results indicate that code comments affect CodeLMs differently - some CodeLMs perform better with commented code, some CodeLMs show no significant performance change, and others perform worse. For future work, we will advance research on CLB detection along several key directions. First, we intend to conduct a more in-depth analysis of the CLB dataset itself. This includes investigating the characteristics of CLBs, such as common interaction patterns across language boundaries, as well as analyzing CLB fix characteristics to better understand how CLBs are resolved in practice. Such analyses will provide deeper insights into the nature of CLBs and help inform the design of more effective detection and repair approaches. Second, we plan to expand our dataset beyond Python and Java to include more PLs. This will enable more comprehensive CLB studies and improve the generalizability of detection models. Additionally, since our dataset includes bugs related to high-severity security vulnerabilities, we aim to explore cross-language security vulnerability detection in future research. Third, while this study focused on relatively small CodeLMs (no larger than 7B), we plan to investigate the performance of large CodeLMs in CLB detection.

  Fine-Tuning Code Language Models to Detect Cross-Language Bugs · 2026 · DOI

• Tight coupling between generative AI models and formal verification tools to enable AI-generated code to be accompanied by machine-checkable correctness proofs has been proposed but not yet realized at scale. The feasibility of automatically generating formal specifications and proofs for AI-generated code across diverse programming paradigms requires exploration.

  Generative AI in Software Engineering · 2026 · DOI

• Longitudinal empirical studies tracking developer skill trajectories over multi-year career periods under varying levels of generative AI assistance are needed to resolve unresolved questions about skill development, competency erosion, and workforce implications. Current evidence on skill-erosion mechanisms is limited to cross-sectional or short-term observations.

  Generative AI in Software Engineering · 2026 · DOI

• Repository-aware retrieval-augmented generation architectures that ground generative AI model outputs in specific codebases, documentation, and project constraints have been identified as high-potential but remain largely unimplemented and untested. The effectiveness of RAG approaches in reducing the reliability gap for complex judgment-intensive tasks like architecture trade-off analysis and security threat modelling needs empirical investigation.

  Generative AI in Software Engineering · 2026 · DOI

• The governance framework's six principles have been grounded in empirical evidence but have not themselves been empirically evaluated in a prospective deployment study across real software engineering organizations. Validation of the Scope-Limited Use, Mandatory Human Review, Security Scanning, Provenance Tracking, Competency Assurance, and Continuous Review principles in operational settings remains a priority.

  Generative AI in Software Engineering · 2026 · DOI

Most-cited papers in Software Engineering Research

• Large Language Models for Software Engineering: A Systematic Literature Review · ACM Transactions on Software Engineering and Methodology · 2024 · 664 citations
• Program comprehension during software maintenance and evolution · Computer · 1995 · 391 citations
• Using an LLM to Help With Code Understanding · 2024 · 283 citations
• An empirical comparison of seven programming languages · Computer · 2000 · 229 citations
• A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges · 2024 · 140 citations
• ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing · 2024 · 138 citations
• GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning · Journal of Systems and Software · 2024 · 135 citations
• Future of software development with generative AI · Automated Software Engineering · 2024 · 132 citations
• BoneJ2 - refactoring established research software · Wellcome Open Research · 2021 · 127 citations
• Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning · 2024 · 122 citations

Most recent work

• Generative Monoculture: Model Collapse in Code as Systemic Vulnerability (EA-UMBML-MONOCULTURE-01 v1.1) · Zenodo (CERN European Organization for Nuclear Research) · 2026
• A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models · ACM Transactions on Software Engineering and Methodology · 2026
• Abstract Syntax Tree for Programming Language Understanding and Representation · ACM Transactions on Software Engineering and Methodology · 2026
• Generative Monoculture: Model Collapse in Code as Systemic Vulnerability (EA-UMBML-MONOCULTURE-01 v1.0) · Zenodo (CERN European Organization for Nuclear Research) · 2026
• Do code LLMs do static analysis? · Empirical Software Engineering · 2026
• PVDetector: Pretrained Vulnerability Detection on Vulnerability-enriched Code Semantic Graph · ACM Transactions on Software Engineering and Methodology · 2026
• ViBench: A Benchmark on Vibe Coding · 2026
• A Multi-Perspective Investigation into Code Migration for Large Language Models · Applied and Computational Engineering · 2026
• A Comprehensive Study of LLM-Based Code Clone Detection · Applied and Computational Engineering · 2026
• The Metamorphosis of Code: How Artificial Intelligence is Reshaping Software Architecture · INTERNATIONAL JOURNAL OF MULTIDISCIPLINARY RESEARCH AND ANALYSIS · 2026

Related topics in Computer Science