menu
techminis

A naukri.com initiative

google-web-stories
Home

>

ML News

>

The Cost o...
source image

Arxiv

23h

read

86

img
dot

Image Credit: Arxiv

The Cost of Avoiding Backpropagation

  • Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization have been proposed as memory-efficient alternatives to backpropagation (BP) for gradient computation.
  • A new study presents a comparison of BP, FmAD, and ZO methods, highlighting theoretical and empirical findings.
  • Theoretical analysis suggests that FmAD and ZO reduce memory usage but at the cost of accuracy, convergence speed, and computation compared to BP with checkpointing.
  • Empirical experiments on large models demonstrate that BP with checkpointing outperforms FmAD and ZO variants, indicating BP with checkpointing as the most effective strategy for model training in memory-constrained settings.

Read Full Article

like

5 Likes

For uninterrupted reading, download the app