A lot of research is happening in this space, so I thought it’d be a good idea to compile some of the things that I have read over the last few months and ran a couple of experiments myself. All the plots & tables are based on my experiments unless stated otherwise.
I wish I had the time and ability to make the illustrated Feed-Forward, looking at things like:
have given me immense intuition for how attention is supposed to work in language, but Geometry has always been more methodical for me. It is close to my heart because I really enjoy seeing mathematics, I consider it a language of visualization and geometry is the game of visualization.
These networks, MAST3R, VGGT, DA3, Deja View are one of the most amazing advancements we have seen in a long time. These models have done a lot, things that required full SLAM systems can now be done in a single forward pass. And it is truly interesting what the attention is attending to.
Before we get into their attention and some analysis, here is the VGGT architecture which is what a LOT of these methods are using now:
fig from VGGT. Their decision for keeping it “simple” was, after discussing it with the goats, they ended up deciding that this has the least inductive bias. The idea is to build scalable systems with as little inductive bias and let them learn.
The camera head takes in aggregated token list and inside it there is only token index 0. The dense DPT head on the other hand reads from 4, 11, 17 & 23 and uses feature pyramid network.
We have covered enough ground. The network has to figure out epipolar constraints, multi-view geometry, correspondences, structure from motion, tracking, point maps all on its own.
So one must ask: how? and if you can answer that, where?
Sanity check: if we have images whose correspondences we truly knew, can we figure out does he know? if he does, then perhaps where?
Mathematically:
Inside each layer we would have, for let’s say K frames, N patch tokens and R register & camera tokens.
\(Z^{(\ell)} \in \mathbb{R}^{M \times d}, \qquad M = K\ (R + N ) \)
now pretty standard attention stuff, I am losing out a bit on notation here, but I average out over all the multi-heads, but per head it’d go something like:
\(Q^{(h)} = Z^{(\ell)} W_Q^{(h)} \)
\(K^{(h)} = Z^{(\ell)} W_K^{(h)}\)
\(V^{(h)} = Z^{(\ell)} W_V^{(h)}\)
\(A^{(\ell,h)} \;=\; \mathrm{softmax}\!\left( \frac{Q^{(h)} {K^{(h)}}^{\top}}{\sqrt{d_h}} \right) \;\in\; \mathbb{R}^{M \times M}\)
So each block outputs:
\(\sum_h A^{(\ell,h)} V^{(h)} W_O^{(h)}\)
So, I will write
\(A^{(\ell)} \;=\; \frac{1}{H} \sum_{h=1}^{H} A^{(\ell,h)} \)
Now that we have established this, beauty emerges.
So M runs over all the frames and patches. So if the patch p is a patch in frame i and a column q is a patch in frame j, then A_{p,q} is exactly how much of that patch looked at another patch.
Now we let the probing begin!
Run VGGT with your images
VGGT takes in 518px with patch size 14 so a 37x37 grid. N = 1369 patches. (Attention readers will know it’s 1374 because 1 camera token and 4 register :D) So from here you know pixel u belongs to some p patch in this image and pixel v in some another image to some q patch.
Normalize wrt the another image’s patches because we only care about frame j’s patches. Note: Here I only did 1369 and not camera and register token.
\(a_p^{(\ell)}(q) \;=\; \frac{A^{(\ell)}_{p,q}}{\sum_{q' \in \mathcal{P}_j} A^{(\ell)}_{p,q'}}, \qquad q \in \mathcal{P}_j, \qquad |\mathcal{P}_j| = N = 1369 \)
So think of this as some distribution which tells you some score, now intuitively speaking. If frame j has no similarity whatsoever it’d make sense that the value is 1/N. So let’s define that:
\(c(\ell) = \frac{N}{|\mathcal{S}|} \sum_{(X,i,j) \,\in\, \mathcal{S}} a_p^{(\ell)}(q^{*})\)
Where S is just our ground truth information of which X ( 3D point ) is coming from i and j, and q* denote the “correct” patch. So now ideally attention should be random/chance if it’s not the match.
To put it simply this is just: let’s say attention for q* the correct patch is 0.0209 which times 1369 would give 28.6 and we have to average this over all the correct correspondences. And we can compare by doing it with random or inductively biased patches to see what happens if:
a) If it’s the same position in other image, i.e correspondences are usually at the same place around, so the model might learn that?
so, if you do that, you get c = 1.52 and
b) the correct patch/ the 3D correspondence and c becomes 28.61, almost 30 times.
So this weird finding of how the attention knows where the matches are. So the next natural question is where is this happening?
Run the same above thing for each layer and you get the above plot.
This is very interesting, nothing almost happens in the first 9 layers, suddenly the model is giving attention to correspondences now?
The middle layer, especially the 14th layer is almost 250x chance, which suggests this is where the matching intensifies.
It is tempting to say that this is where the matching is happening, we see some more spike in around layer 21-22, which we happen to know from VGGT-SLAM-2.0’s analysis measures Dopplegangers internally and can be used to figure out places that might look “visually” similar but are not actually the same place.
This works because our attention is measuring the actual 3D point correspondences and not visual similarity!
This has been established in On Geometric Understanding and Learned Priors in Feed-forward 3D Reconstruction Models and they show this across DUSt3R, VGGT & Depth Anything 3 and even intervene on the attention to show it’s causal. AVGGT: Rethinking Global Attention for Accelerating VGGT finds these and uses it to make the model 2-10x faster by making the dead layers cheaper.
Now there are still natural progressions to this analysis, can’t we just take them out of the model if we know this is exactly where the matching is happening?
In an ideal world if layer 14 truly knows, take arg max over the patches, call it a hard match and let a solver do its thing.
I used 3 Indoor-6 scenes, 40 pairs, so total 120. With a frame gap of 30, so they are decently further apart. RANSAC with 0.9999 confidence, 0.01 threshold in normalized coordinates, minimum 12 inliers.
Attached images of 0.4m, 6.5m, 1m apart.
Tried hard vs soft, maybe arg-max is throwing away a lot and the shape of the distribution matters?
Mass-weighted expectation over top 64 destination now, same solver with thresholds.
So clearly, this is random.
I had to investigate why, obviously look into statistics and the answer is very obvious.
\(\Pr\left[\arg\max_q a_p^{(14)}(q) = q^{*}\right] = 0.113\)
So the highest patch is right only 11 percent :p. Ain’t no solver solving almost 9/10 wrong matches.
Here is how the blob attention looked like compared to epipolar lines, I was curious if it’d be along it or follow geometry like we do but alas, it’s more about visual similarity than anything else.
Well, this marks the end of Part 1/3 of Structure of Attention, I have few more analysis and next question on top of them.
This part summarizes how and where the matching is happening which is a crucial step for Structure from Motion, Part 2 will ask questions like was the decision of reading from the end bad if a lot of stuff is happening in the middle? Dense heads read from 4, 11, 17 & 23 due to spacing but what if try to game that with attention? If some layers are working harder than the others, can you kill the non-workers? what happens then?
Part 3 will go for the head! Do all the heads agree with each other? Can you make them agree with each other? Is there any benefit of decoding pose, depth & pointmaps parallely? Deja View’s model acts as in implicit solver ;p it starts unlearning after you run it for more than training, can you prevent that?
I’d love some feedback on how I can improve these writings, is there a way to present some ideas better, some questions perhaps?