RoPE: Rotary Position Embeddings Explained

TL;DR

RoPE encodes position by rotating query and key vectors in 2D subspaces. The beauty is that the dot product between rotated vectors depends only on their relative position, not absolute. This gives you relative position encoding without the memory overhead of learned relative embeddings. I'll derive the math and implement it step by step.