Question 21
Which of the following statements correctly explain the importance of deduplication during preprocessing of large-scale datasets used for training Deep Learning or Large Language Models?
Deduplication reduces the risk of overfitting by preventing repeated samples from dominating the gradient updates.
Deduplication guarantees that the trained model will achieve higher accuracy on all downstream tasks.
Deduplication helps avoid data leakage between training and evaluation sets, leading to more reliable performance metrics.
Deduplication eliminates the need for regularization techniques such as dropout and weight decay.