Question 12: Consider the scaling law for pretraining test loss: L(N,…

Consider the scaling law for pretraining test loss:

$L(N, D) = \left(\frac{N}{N_c}\right)^{-\alpha_N} + \left(\frac{D}{D_c}\right)^{-\alpha_D}$

$L(N, D) = (A)^{-\alpha_N} + (B)^{-\alpha_D}$

where $A = \frac{N}{N_c}, B = \frac{D}{D_c}$

• $N$ : the number of non-embedding parameters
• $D$ : the size of the dataset (in tokens)
• $N_c = 8.8 \times 10^{13}$
• $D_c = 5.4 \times 10^{13}$
• $\alpha_N = 0.076$
• $\alpha_D = 0.095$

Assume $A = 2B$. What is the approximate ratio of the new loss after doubling $N$ and $D$ to the original loss i.e., $\frac{L(2N, 2D)}{L(N, D)}$? (Answer correct upto 3 digits after the decimal)