LLM parallelism strategies

training

• Pipeline parallelism

  What

  

  • have each layer of the network on a different GPU
  • constantly sending data forward and backward through the GPUs

  Link to original

• Data parallelism

  What

  

  • multiple GPUs
  • a separate copy of the model on each GPU
  • feed different data on each GPU
  • average the gradients when done

  Link to original

• Tensor parallelism

  What

  

  • Matrix multiplication factorization
    • split matrix to create smaller matrices to do multiplication.
    • $Z=f(XA)B->Z=f(X{A}_1)B_1+f(X{A}_2)B_2$
  • Do $A_1{B}_1$ on one GPU and $A_2{B}_2$ on another GPU, or both on the same GPU but separately

  Link to original