ideatraining

  • randomly perform full weight update while doing PEFT
  • heuristically perform full weight update while doing PEFT
    • if the variance of the PEFT gradients are above some threshold, run the step again to perform a full weight update