intuition
- uses multiple sinusoidal waves of different frequency at time
- related to the idea that (some) functions are vectors (infinite dimensional objects) and Fourier Transform, where
- functions in the frequency domain is laid out as row vectors
- then the functions in the time domain is column vectors
- these are the positional embeddings
- related to the idea that (some) functions are vectors (infinite dimensional objects) and Fourier Transform, where
- it is like using a number system but with infinite digits, which are real numbers between -1 and 1
- each wave is a number place and its values are the digits
- each number place changes value at a different rate
- thus it has an exponential scale instead of linear (the relative distance grows exponentially)
- so it also behaves like a number system with some linearity properties