RoPE - Rotary Position Embeddings

Intuition

$$R_{\Theta ,m}^d=\left[\begin{array}{ccccccc}cos(m{\theta }_1)& -sin(m{\theta }_1)& 0& 0& \cdots & 0& 0\\ sin(m{\theta }_1)& cos(m{\theta }_1)& 0& 0& \cdots & 0& 0\\ 0& 0& cos(m{\theta }_2)& -sin(m{\theta }_2)& \cdots & 0& 0\\ 0& 0& sin(m{\theta }_2)& cos(m{\theta }_2)& \cdots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& 0& \cdots & cos(m{\theta }_{d/2})& -sin(m{\theta }_{d/2})\\ 0& 0& 0& 0& \cdots & sin(m{\theta }_{d/2})& cos(m{\theta }_{d/2})\end{array}\right]$$

• RoPE applies rotation of different frequency on $d/2$ 2D subspaces
  • This treats the standard basis vectors as the bases
  • The generalization rotating on 2D spaces spanned by non-standard basis vectors is described in Rotate from one vector to another vector in high dimensional space
• ” the intuition behind RoPE is that we can represent the token embeddings as complex numbers and their positions as pure rotations that we apply to them. If we shift both the query and key by the same amount, changing absolute position but not relative position, this will lead both representations to be additionally rotated in the same manner---as we will see in the derivation---thus the angle between them will remain unchanged and thus the dot product will also remain unchanged.” ¹
  •  Instead of working in the usual $R^d$, we will work in $C^{d/2}$ by considering consecutive pairs of elements of the query and key vectors to be a single complex number. Specifically, instead of viewing $q=(q_1,q_2,q_3,q_4,\cdots ,q_d)$ as a d-dimensional real vector we view it as $q=(q_1+i{q}_2,q_3+i{q}_4,\cdots q_{d-1}+i{q}_d)\in C^{d/2}$.

Notes

From Training and Fine-tuning LLMs - W&B course

• actually learn positional embeddings
• a popular choice
• tend to slow things down a little bit
• now there are lots of interesting ways of interpolating to get extrapolation with RoPE
• used by LLaMa, Adept.ai (for the 8B model)

Footnotes

1. Rotary Embeddings: A Relative Revolution | EleutherAI Blog ↩