Transforming column vectors of model weights to standard bases

theoryhypothesisresearchquestion

• Suppose that the column vectors of model weights $W$ are nearly orthogonal (unit ?) vectors
• That would mean if we apply a transformation $W_{rot}^{d_{feat}\times d_{feat}}$ such that 1 of those vectors become a standard basis, then all other vectors will align with other standard bases $W^{\prime }=W\cdot W_{rot}$
• Then we would have a nearly sparse matrix $W^{\prime }$ with a few nearly 1 but not exactly 1 and a lot of nearly 0 but not exactly 0
• Then take the delta of $W^{\prime }$ with the sparse matrix $$\begin{array}{rl}{W}^{\ast }& =round(W^{\prime })\\ delt{a}_W& =W^{\ast }-W^{\prime }\end{array}$$ $delt{a}_W$ would be a matrix of near 0 values
• What if the values in $delt{a}_W$ follows a Gaussian ? Then we have decomposed a weight matrix $W$ into
  • a lower-rank matrix $W_{rot}$
  • a sparse vector $W^{\ast }$
  • and a Gaussian representing $delt{a}_W$
• Possible applications:
  • A more efficient way of storing model weights
  • We could force one of the column matrix to be a standard basis before training and then we would have a near-sparse weight matrix for free after training
  • If that works then we can instead of learning a weight matrix, we can learn a sparse matrix and a Gaussian
    • then we can look at this from a graphical modelling perspective or like a VAE