Source: https://www.wandb.courses/courses/take/training-fine-tuning-LLMs/lessons/49181391-conclusions
Advices from Jonathan Frankle:
- Start small and work your way up.
- ensure an end-to-end system for constant evaluation.
- Chinchilla scaling laws
- Be skeptical of what you read in the literature. Test for yourself. _“Not everything you’ve seen in the literature is 100% reliable. And also not everything is going to translate to your setting, your domain and what you are working on.”
- Don’t trust intuition. Test for yourself. “A lot of counterintuitive things happen in deep learning.”
- This is data science in its truest form. “Test things for yourself. This really is science in its truest form and data science in its truest form and you’re gonna have to be a scientist and follow scientific methods.