Hey HN! I've been working on a series capturing the industry best practices in producing fine-tuned LLMs. Our first post covered training data preparation, and for this one we'll be talking about how to select your base model and hyperparameters, covering both closed and open models as well as models of different sizes. This was created in collaboration with Kyle Corbitt at OpenPipe (kcorbitt) who will be in the comments as well!
We cover:
- Proprietary (well mainly OpenAI) vs open source models: OAI has really great performance w/ relatively small training datasets but you hit a ceiling on max performance sooner than you do w/ open source models. We're not sure exactly what drives this, but it very well could be that OAI has made some technical decisions under the hood that leans towards this "Red Mage" approach in order to serve a broader audience (ie users w/ less training data).
- Large vs Small Models: Main thing is larger models typically let you "get away" with less training data all things equal. But where possible you want to deploy the smallest model that achieves acceptable performance in prod. Smaller is less "costly" across a variety of dimensions (not just price).
- Hyperparameter Tuning: We cover this in some detail for those who are curious, but to be frank hparam tuning is generally not a high ROI use of your engineering time. Plow that time into dataset curation instead :)
Hope this is useful for folks! Happy to answer questions here (and hope to learn something new too)
We cover:
- Proprietary (well mainly OpenAI) vs open source models: OAI has really great performance w/ relatively small training datasets but you hit a ceiling on max performance sooner than you do w/ open source models. We're not sure exactly what drives this, but it very well could be that OAI has made some technical decisions under the hood that leans towards this "Red Mage" approach in order to serve a broader audience (ie users w/ less training data).
- Large vs Small Models: Main thing is larger models typically let you "get away" with less training data all things equal. But where possible you want to deploy the smallest model that achieves acceptable performance in prod. Smaller is less "costly" across a variety of dimensions (not just price).
- Hyperparameter Tuning: We cover this in some detail for those who are curious, but to be frank hparam tuning is generally not a high ROI use of your engineering time. Plow that time into dataset curation instead :)
Hope this is useful for folks! Happy to answer questions here (and hope to learn something new too)