> the hidden state for different layers carry meaningful self-awareness signal for various situations.
Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?
If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?
Was actually pulling on a similar thread as I saw announcement so I integrated it just for fun. Have been only running this on my Framework Desktop but should be runnable elsewhere
https://github.com/olafura/gemma-4-mic-transcribe
> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.
Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?
Curious to learn more. Any more info on your approach (esp the mechanistic study)?
Correct, the study is verbose, we will compile into a neat shareable report and publish once we solve the pending caveats. Interesting username btw haha.
> the hidden state for different layers carry meaningful self-awareness signal for various situations.
Is it plausible to wonder if some developer judgement feels, like maybe "the code I just wrote is clean/crufty", or "things came together smoothly/janky", might have extractable signals in some models?
If so, might one create a shopping list of desired signals to check for in a model, as with activation steering concepts, where one checks whether and how hard each concept can usefully be nudged?
Was actually pulling on a similar thread as I saw announcement so I integrated it just for fun. Have been only running this on my Framework Desktop but should be runnable elsewhere https://github.com/olafura/gemma-4-mic-transcribe
Is this in any way similar to Goodfire's work? https://www.goodfire.ai/research/rlfr#
Thats an interesting outlook, loosely similar.
Have you benched this for coding tasks, with a fallback to a larger local model, for example Qwen-3.6-27B?
Or using it for sub-tasks, where a framework with a larger primary model dispatches simpler jobs ("summarize ...", etc.) to it?
> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.
Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?
Curious to learn more. Any more info on your approach (esp the mechanistic study)?
Correct, the study is verbose, we will compile into a neat shareable report and publish once we solve the pending caveats. Interesting username btw haha.
Nice, looking forward to the report.
And thanks, huge pasta fan :)
anytime!