Algorithms and approaches for continual reinforcement learning have gained increasing attention.Traditional foundations of reinforcement learning may not be well-suited for continual reinforcement learning.Four foundations of traditional RL are identified as antithetical to the goals of continual reinforcement learning.Rethinking and proposing alternative foundations is necessary for the development of better-suited algorithms and approaches.