Modern neural networks are usually highly over-parameterized.This work studies the rank of convolutional neural networks (CNNs) trained by gradient descent.CNNs trained with gradient descent are found to be robust to image background noises.Theoretical case study and experiments on synthetic and real datasets support the claim.