Beautiful illustrations
I find, 'Playing' is just the free and motivated version of 'exploration'.
One thought on your nicely illustrated "key observation [is] that neural networks tend to place features along directions": my guess is that the neural net was TOLD to behave that way by choosing e.g. Cosine Loss?
Nice article! The generated images make me so nostalgic for the early days of AI image generation. DeepDream and others had such uncanny, interesting generations.
One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes.
Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.
> Because you somehow need a giant training set which describes images in natural language, no?
That's definitely one way - they train a text encoder together with an image encoder on a labelled set of images. WL & 3b1b made a nice video on it: https://www.youtube.com/watch?v=iv-5mZ_9CPY
This article is very well structured and provides just the right amount of details for non-practitioners to enjoy it.
Mechanistic interpretability is a fun topic to "play with" (good title there). I recommend watching videos featuring Neel Nanda or Chris Olah
Beautiful illustrations I find, 'Playing' is just the free and motivated version of 'exploration'.
One thought on your nicely illustrated "key observation [is] that neural networks tend to place features along directions": my guess is that the neural net was TOLD to behave that way by choosing e.g. Cosine Loss?
Awesome project! Preserving and sharing knowledge like this is incredibly valuable. Thanks for making these resources accessible to everyone.
For some reason, the uncanniness of the feature pictures are deeply unsettling for me. It just stirs intense unease. A bit amusing, to be honest.
Nice article! The generated images make me so nostalgic for the early days of AI image generation. DeepDream and others had such uncanny, interesting generations.
Very nice visualizations, thanks for that!
One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes.
Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.
> Because you somehow need a giant training set which describes images in natural language, no?
That's definitely one way - they train a text encoder together with an image encoder on a labelled set of images. WL & 3b1b made a nice video on it: https://www.youtube.com/watch?v=iv-5mZ_9CPY
Thanks I'll check out that video