Cite
Schaeffer, Rylan, et al. Are Emergent Abilities of Large Language Models a Mirage?
Metadata
Title: Are Emergent Abilities of Large Language Models a Mirage? Authors: Rylan Schaeffer, Brando Miranda, Sanmi Koyejo Cite key: schaeffer2023
Links
Abstract
Recent work claims that large language models display emergent abilities: abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher’s choice of metric rather than due to fundamental changes in models with scale. Specifically, nonlinear or discontinuous metrics produce seemingly emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes in model performance. We present our alternative explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities; (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on the Beyond the Imitation Game Benchmark (BIG-Bench); and (3) show how to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep network architectures. Via all three analyses, we provide evidence that emergent abilities disappear with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.
Notes
From Zotero
From Obsidian
Are-Emergent-Abilities-of-Large-Language-Models-a-Mirage
Annotations
Highlighting colour codes
Link to original
- Note: highlights for quicker reading or comments stemmed from reading the paper but might not be too related to the paper
- External Insight: Insights from other works but was mentioned in the paper
- Question/Critic: questions or comments on the content of paper
- Claim: what the paper claims to have found/achieved
- Finding: new knowledge presented by the paper
- Important: anything interesting enough (findings, insights, ideas, etc.) that’s worth remembering
From Zotero
Imported: 2024-11-30
Note | View in local Zotero: page 1
Note
“aring at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher’s choice of metric rather than due to fundamental c”
sfsdfds
Note | View in local Zotero: page 1
Note
sfsdfs
Note | View in local Zotero: page 2
Note
Note | View in local Zotero: page 2
Note
Note | View in local Zotero: page 3
Note
“2 Alternative Explanation for Emergent Abilities”
ajsfklskldfklsda