Tensor parallelism

distributed

What

• Matrix multiplication factorization
  • split matrix to create smaller matrices to do multiplication.
  • $Z=f(XA)B->Z=f(X{A}_1)B_1+f(X{A}_2)B_2$
• Do $A_1{B}_1$ on one GPU and $A_2{B}_2$ on another GPU, or both on the same GPU but separately

Hence

• reduce the amount of memory needed for both the weights and activations