• Scaling Laws for Precision

  • Nov 18 2024
  • Length: 19 mins
  • Podcast

Scaling Laws for Precision

Listen for free

View show details

  • Summary

  • ⚖️ Scaling Laws for Precision

    This research paper investigates the impact of precision in training and inference on the performance of large language models. The authors explore how precision affects the effective parameter count and propose scaling laws that predict performance degradation due to low-precision training and post-training quantization. They find that overtrained models are more sensitive to post-training quantization, and that training larger models in lower precision might be computationally optimal. Their unified scaling law accounts for both training and post-training effects and predicts loss in varied precision settings, ultimately suggesting that the standard practice of training models in 16-bit might be suboptimal.

    📎 Link to paper
    🌐 Read their Tweet
    Show More Show Less
Show More Show Less

What listeners say about Scaling Laws for Precision

Average Customer Ratings

Reviews - Please select the tabs below to change the source of reviews.

Audible.com.au reviews

Amazon Reviews
No Reviews are Available
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.