<ul><li>Sparse autoencoders have produced dictionaries of high-dimensional vectors corresponding to the universe of concepts.</li><li>The concept universe shows interesting structure at three levels: 'atomic' small-scale structure, 'brain' intermediate-scale structure, and 'galaxy' scale large-scale structure.</li><li>The atomic structure contains parallelograms or trapezoids, with improved quality when global distractor directions are removed.</li><li>The brain structure exhibits significant spatial modularity, with clusters of co-occurring features spatially clustering together more than expected.</li><li>The large-scale structure follows a power law of eigenvalues with steepest slope in middle layers.</li></ul>

The Geometry of Concepts: Sparse Autoencoder Feature Structure

Discover more