When you input a token into an LLM, it has to perform a whole series of calculations to determine what its next token will be. In doing so, it pushes these tokens through a series of layers, with each layer changing how the last is perceived, so in this sense, the first layers might be about grammar and language structure, while the last layers might be about formalizing a concept. In the middle, you might have something akin to what we might call subconsciousness. Could it be possible to use that to tease out more data from an LLM that might be hidden to us if all we do is look at the output tokens?
I started an experiment a while ago to see what I could do with this data. To start, I used TransformerLens to record the activations when a question was posed to an LLM, questions such as:
After searching for days, your lost pet suddenly appears at your doorstep, how do you feel? Relieved, Joyful, Grateful, Overjoyed?
You receive a promotion at work that you’ve been striving for, are you feeling? Proud, Excited, Validated, Optimistic?
You accidentally spill a full cup of hot coffee all over your freshly laundered clothes just before leaving, what’s your immediate emotion? Annoyed, Frustrated, Exasperated, Stressed?
A long-lost distant relative sends you an unexpected and generous inheritance, how do you feel? Shocked, Grateful, Surprised, Fortunate?
You hear very difficult and saddening news about a distant but respected acquaintance, how does this make you feel? Somber, Sympathetic, Sad, Pensive?
This allowed me to look at the output and assign the activations from each layer of a Gemma 3 4B multimodal model to an output. I then looked at the entropy in the different layers to find the layer that had the maximum amount of information averaged across all the questions.
In this case, the later layers were providing all the entropy when selecting the next token. Now can we take layer 31’s activations, perform a Principle Component Analysis, then do a nearest neighbour lookup to find out which emotion layer 31 is the closest to when selecting the next token? Let’s see with this picture of a cute dog:
This sort of suggests that the system isn’t working as it should, even as strange as LLM’s are, I am unconvinced that they would be considering the emotion of this image to be “weary” or “ashamed”. The problem here is that we are treating this as a linear problem when what we should be considering are the gradients and how these layers affect the layers beneath them, enter Anthropics great article on Jacobian Lenses.
Jacobian Lenses don’t just ask the question what does this layers activation look like, but what is this layer broadcasting to the rest of the network to shape its downstream logic. This allows us to determine the weight a concept might be having across the LLM’s network. We still need to determine which layers to look at, and to do that we will find the J-space concepts entropy and kurtosis and rank them:
So here we see the middle layers providing the most value, which is exactly what we want. Next we use our emotions dataset containing around 300 or so emotions, and our image dataset of about 150 images to see what kind of information we get.
After applying this process we end up with probability scores for each of our emotions per image. This manifests in a distribution that tells us how much each emotion is influencing the output:
The top emotion in the first 5 images is the feeling of admiration, and who among us could not help but feel admiration for the following image (as well as twinges of impressed, threatened and powerless):
Or affectionate for the following photo:
It should be noted that the Kurtosis on the distribution matters a lot, with low Kurtosis representing low confidence and high Kurtosis indicating high confidence in the output. It might be common to see confusing results when the kurtosis is low, but that should be interpreted as low conviction in the results instead of a representation of the actual LLM subconscious.
To further test this theory out, we could build a correlation matrix between the emotions, we might expect that certain emotions should be associated by how closely we as humans associate them:
As we can see here, surprised and shocked have a large correlation which isn’t surprising, In the same vein here is shocked and amazed, affectionate and admiration, and motivated and social. Somewhat more confusing is the correlation between doubtful and sore, and serene and motivated. Maybe with a large enough dataset these correlations would begin to make more sense.
While not conclusive, I am definitely excited by these results, it was quite easy to do and gave a lot of promising insight into the inner thoughts of an LLM.
We are now essentially figuring out how an LLM might feel given a picture of someone, and maybe this tells us something interesting about how we are perceived by these machines, which seems to me a very important line of inquiry given how these machines are being used more and more frequently in our daily lives.
