Researchers propose using low-rank tensors to solve finite-horizon Markov Decision Processes (MDPs).Policies and value functions in finite-horizon MDPs are not stationary, which poses challenges in high-dimensional MDPs.The low-rank tensor representation enables scalable learning of optimal policies in finite-horizon MDPs.Block-coordinate descent and block-coordinate gradient descent algorithms are introduced for solving the Bellman equations.