论文标题
时间差异学习的有限时间分析：离散时间线性系统视角
Finite-Time Analysis of Temporal Difference Learning: Discrete-Time Linear System Perspective
论文作者
论文摘要
TD学习是强化学习领域（RL）中的基本算法，它通过估算Markov决策过程的相应价值函数来评估给定策略。尽管在TD学习的理论分析中取得了重大进展，但最近的研究发现了通过开发有限的时间误差界限的统计效率的保证。本文旨在通过展示对表格时间差异（TD）学习的新颖有限时间分析来为现有的知识体系做出贡献，该分析可直接有效地利用离散时间随机的线性系统模型并利用Schur矩阵属性。提出的分析可以以统一的方式涵盖上政策和非政策。通过采用这种方法，我们希望提供新的直接模板，不仅可以进一步阐明TD学习和相关RL算法的分析，而且还为该领域的未来研究提供了宝贵的见解。
TD-learning is a fundamental algorithm in the field of reinforcement learning (RL), that is employed to evaluate a given policy by estimating the corresponding value function for a Markov decision process. While significant progress has been made in the theoretical analysis of TD-learning, recent research has uncovered guarantees concerning its statistical efficiency by developing finite-time error bounds. This paper aims to contribute to the existing body of knowledge by presenting a novel finite-time analysis of tabular temporal difference (TD) learning, which makes direct and effective use of discrete-time stochastic linear system models and leverages Schur matrix properties. The proposed analysis can cover both on-policy and off-policy settings in a unified manner. By adopting this approach, we hope to offer new and straightforward templates that not only shed further light on the analysis of TD-learning and related RL algorithms but also provide valuable insights for future research in this domain.
