Study focuses on the scalability of offline reinforcement learning (RL) algorithms.Current offline RL algorithms show poor scaling behavior despite scaling up data.Long horizons identified as the main cause behind poor scaling of offline RL.Horizon reduction techniques, like SHARSA, enhance scalability on challenging tasks.