A naukri.com initiative
Arxiv
2d
259
Image Credit: Arxiv
Some Theoretical Results on Layerwise Effective Dimension Oscillations in Finite Width ReLU Networks
- The study analyzes the layerwise effective dimension in fully-connected ReLU networks of finite width.
- Closed-form expressions for the expected rank of the hidden activation matrices for a fixed batch of inputs are derived.
- Main result indicates that the rank deficit decays geometrically with a ratio of 0.3634.
- The oscillatory rank behavior observed is a finite-width phenomenon in random ReLU networks, shedding light on deep network expressivity.
Read Full Article
15 Likes
For uninterrupted reading, download the app