LLMs evaluation

Advices from Jonathan Frankle

source

• The most important topic and the things we understand the least. We know so little about evaluation and how to evaluate properly.
• There aren’t really best practice in the field.
• If you have a process you’re already integrating a model into, measure the model in the same way you measure success for that process.
• Go beyond loss
  • Incredibly bespoke, use case by use case
• Focus on success metrics specific to the task at hand.
  • The general advice is try to get as close to the real world as possible, the rest is up to the domain expert.