Open research questions in Reinforcement Learning in Robotics

What the literature leaves open

• Similar to SFT in implementation complexity; avoids online rollouts, learned reward-model inference, and critic training, but remains limited by the coverage and quality of the offline preference data [15]. This crisis suggests that static, scalar reward functions are insufficient for both super-human alignment and rigorous problem-solving.

  A Technical Survey of Reinforcement Learning Techniques for Large Language Models · 2026 · DOI

• Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood.

  ISO: An RLVR-Native Optimization Stack · 2026

• Although our study shows that DDQN works well with the DAC framework of (1+(𝐿,𝐿))-GA solving the O&%M#’ problem, a critical open question is how this deep reinforcement learning method and debugging approach will perform on non-linear, multi-modal, or deceptive !tness landscapes. Finally, beyond the model-free RL approaches examined in this study, we plan to explore model-based RL, which remains largely under-investigated in DAC and may o#er promising avenues for improving the learning’s sample e"ciency.

  Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the \((1+(\lambda,\lambda))\) -GA · 2026 · DOI

• Future work could explore model-based reinforcement learning techniques , establish the internal representation of maze dynamics, and 63 Proceedings of CONF-APMM 2025 Symposium: Applied Mathematics and Quantitative Modeling DOI: 10. Whether such analogues can inform the design of better RL algorithms remains an open and fascinating question. Moreover, limited studies have considered scenarios where the maze itself is dynamic, such as walls that can move, doors that can open or close, and new obstacles that can appear.

  An Investigation on Reinforcement Learning Approaches to Maze Solving · 2026 · DOI

• Prior work has documented this coupling and established a diagnostic framework (EPC) to measure it, but has not investigated whether calibration techniques can mitigate the effect.

  Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops? · 2026

• Although joint regression-ranking supervision often improves BIQA, the relation between the two paradigms remains largely empirical and underexplored.

  MR-IQA: A Unified Margin View of Regression and Ranking for Blind Image Quality Assessment · 2026

• Fourth, the dynamic colony-size formula has not been validated against a fixed-size baseline; it is possible that 100 ants with 300 iterations would match or exceed the dynamic formula on most instances at lower computational cost. First, all instances were generated from a single synthetic distribution; the algorithms have not been tested on real warehouse order data where box-size distributions are typically skewed by fast-moving SKUs.

  Proximal Policy Optimisation Versus Ant Colony Optimisation For The Three-Dimensional Bin Packing Problem: A Comparative Study · 2026 · DOI

• Overall, our work contributes an empirical study of active continual learning for autoregressive VLAs, establishing that uncertainty-guided recovery demonstrations can improve adaptation efficiency while also revealing open challenges when targeted new data is incorporated into large robot policies.

  RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models · 2026

• By embedding incentive-compatible mechanisms into the agent interaction protocol, our approach induces condi- tional cooperation, belief propagation, and sequential adap- tation—capabilities rarely addressed in existing multi-agent LLM systems.

  Everyone Contributes! Incentivizing Strategic Cooperation in Multi-LLM Systems via Sequential Public Goods Games · 2026 · DOI

• Algorithm Design Guidelines: Consider value statistics (frequency, string lengths) when deciding how to order values within each row Consider how to group rows that share common values to maximize consecutive prefix matches Handle missing values and mixed data types appropriately Both simple and sophisticated approaches can achieve high scores; the key is maximizing prefix overlap between consecutive rows Constraints: 32 Do not add/remove rows or columns.

  Improving Coherence and Persistence in Agentic AI for System Optimization · 2026 · DOI

• Although online RL has shown promise for text-based agents, its potential for training visual web agents directly on live websites remains largely underexplored.

  OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents · 2026

• Control barrier functions (CBFs) provide a principled mechanism for enforcing forward invariance through minimally invasive safety filters, but their use in model-free RL is limited by the need for accurate dynamics and hand-designed barrier certificates.

  Robust Koopman Control Barrier Filters for Safe Actor-Critic Reinforcement Learning · 2026

• However, conventional DRL methods are fundamentally limited by their reliance on unimodal policy distributions, inefficient exploration in high-dimensional action spaces, and poor adaptability to dynamic and heterogeneous environments.

  From Denoising to Decision Making: A Survey on Diffusion Model-Enabled Deep Reinforcement Learning for Wireless Networks · 2026

• We propose Fluid-Agent Reinforcement Learning as a framework for modeling real-world inter- actions where the assumption of a fixed number of agents is overly restrictive. In this setting, a team of agents can dynamically expand its population through a spawning mechanism integrated into the environment. We evaluate the performance of different MARL algorithms under various reward structures and show that the fluid-agent setting enables teams to dynamically adjust their population size in response to task variability, optimize the characteristics of spawned agents and also discover richer policies that are inaccessible in fixed-agent settings. This work primarily introduces the fluid-agent framework and focuses on the aspect of agent creation. Future directions include studying the more general case where environments support agent creation and death, as well as investigating the theoretical properties of fluid systems—such as conditions for equilibrium convergence and sample complexity under fluid dynamics.

  Fluid-Agent Reinforcement Learning · 2026 · DOI

• In this paper, we put forward a generic adaptive exploration framework that uses a generic uncertainty-based action selection mechanism to decide in a principled manner when to alternate between exploration and exploitation. The generality of that mechanism allows (i) our framework to effectively incorporate any uncertaintymeasuring mechanism of choice; and (ii) existing adaptive exploration techniques to be viewed as special cases of our framework. In problems requiring adaptive exploration, ADEU is a promising and easily deployable solution. Additionally, it allows the user to select an existing uncertainty measurement mechanism or to define a heuristic one tailored to its problem. Our experiments verified the superiority of ADEU in difficult testbeds. Ongoing and future work includes extending ADEU in various directions. To begin, we intend to perform tests to further verify ADEU’s ability to exploit background knowledge; and its ability to recover from being fed with incorrect background information or to detect and escape clear suboptimal trajectories by optimizing the rollout procedure. Moreover, we are actively working to extend ADEU to guarantee safe exploration [42, 43]. In ongoing work, we are calculating the upper bound of allowed exploration in neighboring safe states, so that ADEU can guarantee safe exploration. We also aim to extend ADEU to multiagent environments. By leveraging agents’ independent exploration, ADEU agents can adaptively explore the environment. Furthermore, by allowing a centralized agent to define 𝑓𝑖 (𝑠) for each agent 𝑖, we can reduce the non-stationarity issues that usually arise in multiagent RL. Finally, we intend to design mechanisms to appropriately set the third and fourth moments of 𝐷, to guide exploration to particular directions. ACKNOWLEDGMENTS The research described in this paper was carried out within the framework of the National Recovery and Resilience Plan Greece 2.0, funded by the European Union - NextGenerationEU (Implementation Body: HFRI. Project name: DEEP-REBAYES. HFRI Project Number 15430). Figure 4: Timesteps required for each algorithm to solve the DeepSea domain for 𝑁 × 𝑁 grid. The lower the curve, the better the performance of the algorithm. from the ADEU framework, hereafter referred to as 𝜖z-adeu, in this demanding domain. Figure 4 shows the timesteps required for an algorithm to solve (i.e., find the tressure) a DeepSea domain with a 𝑁 × 𝑁 grid. The 𝑥-axis of that figure shows the different values of 𝑁 while the 𝑦-axis shows the aforementioned timesteps. As seen in Figure 4, 𝜖z-adeu achieves better performance in terms of scalability in the DeepSea domain. Specifically, 𝜖z-adeu solves the DeepSea domain for 𝑁 ≈ 80 when its best opponents struggle even for 𝑁 > 60.

  A Novel Framework for Uncertainty-Driven Adaptive Exploration · 2026 · DOI

• Although AltNet demonstrates strong empirical gains and stability across a range of continuous-control tasks, there are several directions for further study. Our experiments focus on challenging continuous-action problems from the DeepMind Control Suite, a widely used benchmark for modern reinforcement learning. Extending the evaluation to more diverse environments is likely to lead to further insights. In addition, AltNet relies on a choice of predetermined reset frequency. An important direction for future work is to develop adaptive scheduling mechanisms that select reset frequencies automatically based on the environment and replay ratio. REFERENCES Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. 2023. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents. PMLR, 620–636. Alessandro Achille, Matteo Rovere, and Stefano Soatto. 2017. Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08856 (2017). Jordan Ash and Ryan P Adams. 2020. On warm-starting neural network training. Advances in neural information processing systems 33 (2020), 3884–3894. Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. 2021. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042 (2021). Zhiyuan Chen and Bing Liu. 2018. Lifelong machine learning. Morgan & Claypool Publishers. Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. 2024. Loss of plasticity in deep continual learning. Nature 632, 8026 (2024), 768–774. Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. 2021. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325 (2021). Pierluca D’Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. 2022. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022. William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. 2020. Revisiting fundamentals of experience replay. In International conference on machine learning. PMLR, 3061– 3071. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. Pmlr, 1861– 1870. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoÃĢo GM AraÚjo. 2022. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research 23, 274 (2022), 1–18. M Igl, G Farquhar, J Luketina, W Boehmer, and S Whiteson. 2021.

  AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learning · 2026 · DOI

• Yet, our approach is 8 Ocean Engineering 361 (2026) 125579 generally applicable, and future work should focus on more complex traffic situations with more traffic participants and static obstacles. Thus, future research should focus on developing such rules, which are a necessary precondition not only for falsification but also for assured interpretable behavior of autonomous vessels.

  Falsification-driven reinforcement learning for maritime motion planning · 2026 · DOI

• To continuously refine the integration of artificial intelligence in wealth management, future research should focus on multiple distinct vectors. • Expansion of Inverse Reinforcement Learning (IRL): Future iterations of the system should heavily explore Inverse RL, such as the GIRL algorithm, which infers unobserved rewards by observing the behavior of expert human advisors (Dixon & Halperin, 2020). This would allow the robo-advisor to implicitly learn nuanced investment strategies and behavioral coaching techniques simply by cloning the actions of top-performing human managers. • Behavioral and Psychological A/B Testing: Given that formal verification fails to build subjective trust (Tausch et al., 2025), the future work must psychological framing of algorithmic advice. Large-scale behavioral experiments should be conducted to determine how different natural (e.g., emphasizing potential gains versus mitigating losses) generated by the LLM impact user retention and their investigate language framings willingness to collaborate with the robo-advisor during periods of high market volatility (Mahmud et al., 2025).

  Next-Generation Robo-Advisory Systems: Integrating Dynamic Optimization, Reinforcement Learning, and Explainable AI for Robust Wealth Management · 2026 · DOI

• Building on this foundation, several directions remain open for future research. These include: • Multi-agent decomposition: Extending the framework to multi-UAV coordination using decentralized or hierarchical policies. • Online learning: Incorporating reinforcement learning to adapt cost parameters or transition models in real time. • Adversarial resilience: Introducing robust or risk-sensitive formulations to better handle adversarial or evolving threats. • Hardware integration: Validating the approach through hardware-in-the-loop simulation or field deployment with physical UAV platforms. In summary, this study establishes that MDP decomposition, supported by theoretical guarantees and priority-based recombination, is a scalable and practical solution for resilient UAV mission management in uncertain environments. Its demonstrated efficiency and near-linear scalability highlight the potential of decomposition methods for broader AI decision-making challenges, providing a bridge between theory and real-world engineering applications.

  A novel MDP decomposition framework for scalable UAV mission planning in complex and uncertain environments · 2026 · DOI

• The design of action space and state observation for three-dimensional UAV swarm scenarios requires systematic optimization. Current validation uses 2D state representations and discrete action spaces defined for planar interception; extension to 6-DOF agent dynamics with continuous action spaces needs investigation regarding observability and computational scalability.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

• Complex meteorological interference patterns beyond the tested wind speed condition (0.1 m/s) have not been investigated. The robustness evaluation of DAFRL is limited to a single disturbance magnitude, and the performance under combined environmental disturbances (wind shear, turbulence, varying pressure gradients) remains unvalidated.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

• The applicability bounds regarding agent population size have not been empirically characterized. The paper states the method is not applicable to scenarios with 'excessively small' agent numbers but does not specify the minimum agent threshold (e.g., whether 5, 10, or 50 agents) at which the mean field approximation becomes valid for heterogeneous weighted games.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

• The generalization ability of the DAFRL algorithm across different scenario types and task objectives has not been empirically validated. While the paper suggests meta-learning and transfer learning as solutions, there is no systematic evaluation of how well weights and policy parameters learned in one cooperative/confrontational scenario transfer to structurally different multi-agent tasks.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

• The weight network learning efficiency degrades significantly in extremely dynamic environments. The paper identifies this as a limitation but does not specify what threshold of environmental change rate (e.g., reward function variation per timestep) causes performance degradation or propose adaptive learning rate mechanisms for such conditions.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

• The theoretical analysis of equilibrium uniqueness, global stability, and equilibrium selection for heterogeneous weighted mean field games is incomplete. The current work verifies Nash equilibrium existence (Theorem 1) but does not establish conditions for equilibrium uniqueness or characterize the stability properties of equilibria under perturbations in heterogeneous agent populations.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

Most-cited papers in Reinforcement Learning in Robotics

• A survey on large language model based autonomous agents · Frontiers of Computer Science · 2024 · 1,205 citations
• A survey of inverse reinforcement learning: Challenges, methods and progress · Artificial Intelligence · 2021 · 477 citations
• Diffusion policy: Visuomotor policy learning via action diffusion · The International Journal of Robotics Research · 2024 · 421 citations
• Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. · Psychological Review · 2014 · 391 citations
• SSL: A Theory of How People Learn to Select Strategies. · Journal of Experimental Psychology General · 2006 · 372 citations
• Reward is enough · Artificial Intelligence · 2021 · 369 citations
• 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations · 2024 · 170 citations
• A survey of multi-agent deep reinforcement learning with communication · Autonomous Agents and Multi-Agent Systems · 2024 · 122 citations
• Safe multi-agent reinforcement learning for multi-robot control · Artificial Intelligence · 2023 · 122 citations
• Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes · Annual Review of Control Robotics and Autonomous Systems · 2024 · 119 citations

Most recent work

• Data-Driven Health Index Estimation and Multiagent Deep Reinforcement Learning for Optimizing Aeroengine Maintenance Strategies · Journal of Aerospace Engineering · 2026
• Sequential Knockoffs for Variable Selection in Reinforcement Learning · Journal of the American Statistical Association · 2026
• 2D to 3D Placement for Monolithic Systems using Reinforcement Learning with Dynamic Hierarchical Cluster Assignment · ACM Transactions on Design Automation of Electronic Systems · 2026
• Online Weighted Majority Rule with Provably Optimal Weights for Multi-Agent Robotics · 2026
• Attack and defense cooperation mechanism of robot football team based on eligibility trace Q-learning algorithm · Discover Artificial Intelligence · 2026
• Beyond Final Reward: A Deployment-Oriented Evaluation Framework for Online Hyperparameter Adaptation in Reinforcement Learning · International Journal for Research in Applied Science and Engineering Technology · 2026
• Conditional Sequence Modeling for Safe Reinforcement Learning · Mathematics · 2026
• Continuous-Time Risk-Sensitive Reinforcement Learning via Quadratic Variation Penalty · Applied Mathematics & Optimization · 2026
• Isolated Q-learning for offline reinforcement learning · Applied Intelligence · 2026
• Adaptive curriculum reinforcement learning with sim-to-real strategy in balance control of underactuated triple pendulum robots · Robotica · 2026

Related topics in Computer Science