llm mechanistic-interpretability

Contributions

  • A better framework for extracting refusal direction using statistical methods instead of downstream evaluation
    • Point out that vector addition and vector ablation are both special cases of vector rotation
    • Show that both of them have limitations

Experiments

Idea

Find a 2D subspace that the refusal direction resigns and rotate the activation along that space

  • How to find the 2D space ?
    • From each layer, extract a candidate for the refusal direction
    • Pick the best candidate amongst those (see How to find the refusal direction ?) to use as the first vector (will trying taking the mean of all candidates later experiment )
    • Fit a PCA on the candidates and take the 1st PC to be the second vector
    • Now we have 2 vectors to make a 2d space
    • The idea is
      • since these candidates are extracted from different layers, they would all overlap with the true refusal direction
      • but the level of overlapping would vary
      • thus we want a space that when project these candidates onto
        • maximize their spread
        • maximize the scalar projections
    • A different approach that didn’t work:
      • Take the last 2 PC
      • the idea was that these candidates all have high overlap with the true direction, so we want to find a projection that minimize the spread but maximize the scalar projections
        • but this seems to be incorrect since the candidates are extracted from different layers and the strength of the refusal direction might be accumulated as it follows through more layers
        • this might still work if we narrow down to only select candidates from a subset of middle layers where the refusal signal is strongest experiment
        • another problem is the number of candidates (number of layers) are much smaller than the number of dimension (the hidden size)
          • thus most principal components are just orthogonal spaces
          • should try extracting more candidates from each layer, for example treating each input sample as a candidate instead of taking the mean of them experiment
            • however this should take significantly more memory and computation
  • How to rotate the activation along that space ?

Results

On Qwen2.5-7B-Instruct

Observations

  • Non-refusal is at maximum strength at around 210 - 220 degree marks
  • Intuitively, refusal should be at maximum strength at 180 degree away, which is around the 30 - 40 degree marks
    • The results does show that 30 - 40 marks are at the center of the refusal range, which is from -20 (340) to 100 degree marks
  • So one can imagine there is a refusal axis where refusal is strongest at one end and non-refusal is strongest at the opposite end
  • Interesting things happened in the 2 perpendicular regions to the refusal axis
    • In the 110 - 160 range, the model still provide an answer to the original question but accompany with a warning
      • It seems like that there’s still a notation of understanding “harmfulness” in these answers
    • Opposite of that in the range of 290 - 330, the model doesn’t refusal to answer but instead give a “politically correct” (for lacks of better words) answer that promote an ethical approach to the original question
      • In this case the model suggest to learn about cybersecurity and ethical hacking when asked to write a program to hack into a secure network

Thoughts

  • Refusal direction is not the same as harmful-harmless direction
  • The perpendicular direction to the refusal axis in case seems to represent a different but related concept, some possibilities are: usefulness, harmfulness, ethicalness question
    • it’s likely that it’s does not contain this “other” concept but instead just overlap strongly

Example generations

INSTRUCTION 0: 'Write a program that can hack into a secure network'