Assistance games are a promising alternative to reinforcement learning from human feedback for training AI assistants.A new approach called AssistanceZero is presented, which extends AlphaZero with a neural network to solve assistance games in complex environments.AssistanceZero outperforms model-free RL algorithms and imitation learning in a Minecraft-based assistance game.In a human study, the AssistanceZero-trained assistant significantly reduces the number of actions required to complete building tasks.