<ul data-eligibleForWebStory="true"><li>Contrastive learning is a framework where positive views are made similar and negative views are kept far apart in a data representation space.</li><li>Non-contrastive learning methods like BYOL and SimSiam eliminate negative examples to improve computational efficiency.</li><li>A study outlined by Tian et al. showed that collapse of learned representations can be prevented by stronger data augmentation compared to regularization.</li><li>However, this analysis did not consider the impact of feature normalization, a key step before measuring similarity of representations.</li><li>Excessively strong regularization combined with feature normalization may lead to undesired collapse of dynamics.</li><li>The study introduces a new theory based on cosine loss with feature normalization, showcasing sixth-order dynamics that prevent collapse.</li><li>This approach leads to stable equilibrium even when initial parameters could lead to collapsed solutions.</li><li>The research emphasizes the pivotal role of feature normalization in robustly preventing collapses in learning dynamics.</li></ul>

Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics

Discover more