Reinforcement learning in partially observable environments is challenging, especially in multi-agent settings.The authors propose using learned beliefs on the underlying system state to overcome these challenges.Belief states are pre-trained in a self-supervised fashion and used in a state-based reinforcement learning algorithm.The proposed method simplifies learning tasks, improves convergence speed, and final performance.