<ul><li>Researchers have used integer arithmetic for both forward and backward propagation in the fine-tuning of language models like BERT to save memory and computation.</li><li>The metric performance of fine-tuning 16-bit integer BERT matches both 16-bit and 32-bit floating-point baselines. Using 8-bit integer data type, integer fine-tuning loses an average of 3.1 points compared to the FP32 baseline.</li><li>A new efficient sparse training method with completely sparse forward and backward passes has been proposed for deep neural networks. This method achieves faster training and saves memory usage.</li><li>The proposed algorithm for sparse training is much more effective in accelerating the training process, up to an order of magnitude faster compared to previous methods.</li></ul>

Backward Propagation is here to stay part3(AI 2024)

Discover more