A naukri.com initiative
Unite
1M
409
Image Credit: Unite
Rethinking Scaling Laws in AI Development
- A recent study has indicated that efficiency in AI models can be more influenced by precision (number of bits used to represent numbers during computations) than increasing model size or increasing training data.
- Researchers tested models with varying precisions, ranging from 3 to 16 bits, containing up to 1.7 billion parameters and trained on as many as 26 billion tokens. They found over-trained models, those involving more training data than the optimal ratio for their size, were especially affected by performance degradation when subjected to quantization (precision reduction after training).
- Scaling laws for precision have now been introduced alongside traditional variables like parameter count and training data. The study identified a precision range of 7–8 bits as being generally optimal for larger models, as this balances computational efficiency and performance, whereas defaulting to 16-bit precision often wastes resources. However, fixed-size models benefit from higher precision levels.
- Hardware compatibility poses a limitation as modern GPUs and TPUs are particularly optimised for 16-bit precision, with limited support for the more compute-efficient 7–8-bit range. Another challenge is marrying the amount of training data with correct precision in order to take advantage of lower computational budgets.
- The findings present an opportunity for AI development to consider precision as a core consideration, allowing for optimal compute budgets and more resourceful resource usage. The future of AI scaling may move towards more human-centered, specialised models directed at real-world needs, pivoting away from the relentless pursuit of larger models and amounts of data.
Read Full Article
24 Likes
For uninterrupted reading, download the app