Evaluating the Foundations and Challenges of Deep Reinforcement Learning for Continuous Control: A Critical Review and Conceptual Synthesis

Abstract

Reinforcement learning (RL) has emerged as a powerful framework for sequential decision-making in uncertain environments. With the advent of deep learning, Deep Reinforcement Learning (DRL) methods have enabled agents to achieve human-level or even superhuman performance in domains ranging from games to continuous control tasks. However, despite impressive empirical successes, fundamental theoretical and methodological challenges remain — especially when applying DRL to continuous-control domains that were historically addressed by classical methods such as Dynamic Programming and Markov Decision Process (MDP) frameworks. This article critically reviews the foundations of RL, particularly MDP-based formulations and value-based dynamic programming, contrasts them with the empirical DRL paradigm as exemplified in continuous control benchmarks, and identifies key limitations, research gaps, and future directions. Building on classical theory (Bellman, 1957; Puterman, 1994; Sutton & Barto, 2018) and modern empirical work (Duan et al., 2016; Mnih et al., 2015), we offer a conceptual synthesis that highlights the tension between theoretical guarantees and practical performance, data efficiency and sample complexity, stability and reproducibility. We argue for a renewed research focus on bridging theory and practice — emphasising the need for reproducible benchmarks, rigorous evaluation, and extensions of classical stochastic dynamic programming to high-dimensional, non-linear environments. Our discussion outlines a research agenda to strengthen the foundations of DRL for continuous control.