Open research questions in Reinforcement Learning in Robotics

What the literature leaves open

• We introduced RL4RLA, a curriculum-driven reinforcement learning framework for discovering randomized linear al- gebra algorithms as explicit symbolic programs. By de- composing deep algorithmic search into staged refinements around numerical failure modes, the framework reliably rediscovers classical RLA paradigms – including sketch- and-precondition, sketch-and-project, and Newton Sketch – within practical search budgets. Equivalent states are merged via Monte Carlo Graph Search, reducing total search cost by 2–3× across all curricula. We also demonstrate that adapting our framework to a new problem class requires only minimal interface changes. Limitations. The discovery environment is synthetic by design, and adapting to a new problem class requires speci- fying a compact task interface with domain-specific prim- itives and reward definitions. Moving from rediscovery to validated novel algorithm discovery would further require larger search budgets, richer operator libraries, and system- atic post-search formal analysis. Future work. Our work opens several promising direc- tions. First, the operator grammar can be extended to richer iterative solvers and adaptive sketching strategies. Second, learned value or proposal models could be incorporated to further reduce search cost. Finally, building on our eigen- value results, curriculum-guided discovery can be applied to broader numerical domains such as PDE solvers and large-scale convex optimization.

  RL4RLA: Teaching ML to Discover Randomized Linear Algebra Algorithms Through Curriculum Design and Graph-Based Search · 2026

• In this paper, we propose the interpretable programmatic reinforcement learning framework ProRL for scheduling with human-readable programs. We introduce a scheduling DSL, named DSL-S, that leverages understandable concepts and existing heuristics for perception and actions. We propose an iterative bilevel optimization method that performs local search over the program architectures within the programmatic space defined by DSL-S, while using Bayesian optimization to learn the program parameters. We establish that the expressiveness of the programmatic policy is bounded by its depth d, i.e., complexity can be traded off against performance. The inference cost of ProRL programs is low. Experiments on classic benchmarks demonstrate the outstanding performance of ProRL compared to heuristic and DRL baselines. Low-budget performance validation also underlines the effectiveness of ProRL in resource-limited scenarios, still outperforming PDR heuristics and PPO with a 100-episode budget. In the future, we will explore how to automatically discover concepts and heuristics beyond handcrafting. Extending ProRL to other COPs like vehicle routing is also a promising research direction. Generative AI Declaration We used generative AI for proofreading and to generate natural-language explanations of programmatic policies in discussion. All scientific content, claims, and results were produced and verified by the authors.

  Scheduling That Speaks: An Interpretable Programmatic Reinforcement Learning Framework · 2026

• The framework is developed for Gaussian state-space models with random-walk latent dynam- ics, and the closed-form CAUSE index inherits this scope; extension to non-Gaussian distributions or change-point dynamics is an open question.

  Not all uncertainty is alike: volatility, stochasticity, and exploration · 2026

• In this work, we identify mode collapse as a key limitation of on-policy reinforcement learning in Large Reasoning Models, caused by reverse KL minimization’s mode-seeking behavior, leading to premature convergence.

  Beyond Mode Collapse: Distribution Matching for Diverse Reasoning · 2026

• The paper identifies the challenge of reducing the number of samples needed to train control policies. The paper also notes the challenge of adapting to shifting dynamics, such as when the payload condition is randomized. The paper identifies the technical challenge of analyzing the joint dynamics of the online model learning and policy learning problems.

  Efficient model-based reinforcement learning for robot control via online optimization · 2026 · DOI

• Further analysis of the joint dynamics of the two learning problems. Investigation of the algorithm's performance in different robotic tasks. Exploration of the algorithm's potential in other fields.

  Efficient model-based reinforcement learning for robot control via online optimization · 2026 · DOI

• Furthermore, we provide a simple regret bound that leverages a novel strategy for encouraging exploration in CMDPs, namely a constraint on the epistemic uncertainty which encourages exploration only when the greedy policy is insufficiently informative.

  Sampling-Based Safe Reinforcement Learning · 2026

• We argue that this convention is insufficient to reliably characterize an algorithm's true safety: it fails to capture how often and how severely the safety bound is violated, whether this holds consistently across tasks and safety bounds, and whether training-time behavior is representative of behavior of the final converged policy.

  Evaluation Metrics for Safe Reinforcement Learning · 2026

• Despite its promise, the application of meta-RL in real-world tasks is hindered by safety requirements, which have been underexplored in prior work.

  Safe Meta-Reinforcement Learning via Information Space Reachability · 2026

• While DR can achieve robust performance simply using controllers synthesized via policy gradient (PG) methods, the optimization landscape is not well understood, even in the case of linear quadratic regulator (LQR) objectives.

  Policy Gradient over History-Dependent Policy Classes for LQR with Domain Randomization · 2026

• We further examine evaluation methodologies, benchmark ecosystems, and sim-to-real transfer challenges, and synthesize open problems in long-horizon consistency, physical constraint enforcement, data efficiency, and safety.

  A survey of world models for physical AI with uncertainty representation and control · 2026 · DOI

• However, incorporating temporal information of sequential data and utilizing it to guide diffusion models to perform better generation is still an open challenge.

  Instructed Diffuser With Temporal Condition Guidance for Offline Reinforcement Learning · 2026 · DOI

• Existing work mitigating memory overfitting either lacks data diversity or hardness or is hard to train.

  Release the Potential of Memory Buffer in Continual Learning: A Dynamic System Perspective · 2025 · DOI

• The challenges faced in robot soccer can serve as a proxy for real-world applications. The development of collaborative robots that can work together in teams is essential for enhancing the efficiency and effectiveness of complex tasks. The algorithm needs to be evaluated in a simulation environment.

  Attack and defense cooperation mechanism of robot football team based on eligibility trace Q-learning algorithm · 2026 · DOI

• The current methods used for attacking and defending cooperation in robot football teams have low efficiency. There is a need for a novel algorithm to improve the attack and defense cooperation mechanism.

  Attack and defense cooperation mechanism of robot football team based on eligibility trace Q-learning algorithm · 2026 · DOI

• The paper does not report specific limitations of the proposed framework. The paper highlights the challenges of evaluating adaptive RL methods, including the lack of a shared evaluation standard and the difficulty of comparing methods fairly.

  Beyond Final Reward: A Deployment-Oriented Evaluation Framework for Online Hyperparameter Adaptation in Reinforcement Learning · 2026 · DOI

• The paper suggests that future research should explore richer representations of training phase, uncertainty, advantage distributions, and non-stationarity. The paper highlights the importance of considering multiple dimensions when evaluating adaptive RL methods. The paper proposes several open problems for future research, including the development of more efficient and effective adaptive RL methods.

  Beyond Final Reward: A Deployment-Oriented Evaluation Framework for Online Hyperparameter Adaptation in Reinforcement Learning · 2026 · DOI

• The reliance on bootstrapping to propagate value estimates, - The myopic behaviors induced by discounting future rewards, - The need for policies with zero-shot adaptability across multiple cost thresholds

  Conditional Sequence Modeling for Safe Reinforcement Learning · 2026 · DOI

• Further analysis of the theoretical basis for viewing RTG/CTG conditioning as a mechanism for zero-shot control of the return-cost trade-off, - Investigation of the application of RCDT to other domains, - Exploration of the potential of RCDT in safety-critical domains

  Conditional Sequence Modeling for Safe Reinforcement Learning · 2026 · DOI

• Future research can focus on extending the proposed algorithm to more general cases. Future research can explore the application of the proposed algorithm to other domains.

  Continuous-Time Risk-Sensitive Reinforcement Learning via Quadratic Variation Penalty · 2026 · DOI

• The existing literature on RL largely focuses on optimizing the expectation of a flow of time-additive rewards, which may not capture the agent's risk attitude. The conventional policy gradient representation is inadequate for risk-sensitive problems.

  Continuous-Time Risk-Sensitive Reinforcement Learning via Quadratic Variation Penalty · 2026 · DOI

• There is a need for a comprehensive introduction to the fundamentals and contemporary applications of reinforcement learning. The course aims to fill this gap by providing a solid theory of reinforcement learning principles and practical tools to solve real-world engineering problems.

  Reinforcement Learning: A Comprehensive Open-Source Course · 2026 · DOI

• Investigating the application of the proposed framework to more complex environments. Conducting user studies to directly measure human interpretability.

  Surrogate Fitness Metrics for Interpretable Reinforcement Learning · 2026 · DOI

• The lack of a principled and scalable way to approximate which trajectories are likely to be informative for human analysis. The need for a holistic policy evaluation that captures both expected and edge-case behaviors.

  Surrogate Fitness Metrics for Interpretable Reinforcement Learning · 2026 · DOI

• To apply the proposed method to more complex scenarios with multiple types of agents and dynamic environments. To improve the computational efficiency of the DAFRL algorithm for very large-scale systems. To explore the application of the proposed method in other domains such as economics and finance.

  DAFRL: a dynamic adaptive mean field game-based multi-agent cooperative decision-making method · 2026 · DOI

Most-cited papers in Reinforcement Learning in Robotics

• Human-level control through deep reinforcement learning · Nature · 2015 · 24,213 citations
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning · Artificial Intelligence · 1999 · 1,776 citations
• A survey on large language model based autonomous agents · Frontiers of Computer Science · 2024 · 1,205 citations
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning · Nature · 2025 · 1,079 citations
• Frustrative nonreward in partial reinforcement and discrimination learning: Some recent history and a theoretical extension. · Psychological Review · 1962 · 673 citations
• Reinforcement learning algorithms: A brief survey · Expert Systems with Applications · 2023 · 627 citations
• Dense reinforcement learning for safety validation of autonomous vehicles · Nature · 2023 · 582 citations
• Learning to act using real-time dynamic programming · Artificial Intelligence · 1995 · 518 citations
• Exploration in deep reinforcement learning: A survey · Information Fusion · 2022 · 505 citations
• A survey of inverse reinforcement learning: Challenges, methods and progress · Artificial Intelligence · 2021 · 477 citations

Most recent work

• Data-Driven Health Index Estimation and Multiagent Deep Reinforcement Learning for Optimizing Aeroengine Maintenance Strategies · Journal of Aerospace Engineering · 2026
• Reinforced Refinement With Self-Aware Expansion for End-to-End Autonomous Driving · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026
• From Reasoning to Code: GRPO Optimization for Underrepresented Languages · Theory and Practice of Logic Programming · 2026
• A Technical Survey of Reinforcement Learning Techniques for Large Language Models · ACM Transactions on Intelligent Systems and Technology · 2026
• Beyond Discounted Returns: Robust Markov Decision Processes with Average and Blackwell Optimality · Operations Research · 2026
• Hierarchical decision-making for UAVs’ game via LLM enhanced multi-agent reinforcement learning · Aerospace Science and Technology · 2026
• Decentralized Topology Robustness Optimization for IoT via Multi-Agent Graph Reinforcement Learning · IEEE Transactions on Mobile Computing · 2026
• Instructed Diffuser With Temporal Condition Guidance for Offline Reinforcement Learning · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026
• Sequential Knockoffs for Variable Selection in Reinforcement Learning · Journal of the American Statistical Association · 2026
• 2D to 3D Placement for Monolithic Systems using Reinforcement Learning with Dynamic Hierarchical Cluster Assignment · ACM Transactions on Design Automation of Electronic Systems · 2026

Related topics in Computer Science