Description: VGGT paper note — a single large (~1.2B) feed-forward transformer that ingests any number of views at once and predicts cameras, depth, pointmaps, and point tracks jointly in one forward pass (seconds), no optimization/global-alignment. Core mechanism: an Alternating-Attention backbone that interleaves frame-wise self-attention and global (cross-view) self-attention — no cross-attention. Beats DUSt3R-line and often optimization-based methods. CVPR 2025 best paper; the endpoint of the arc and the model the xattn-spillover attack targets
My Notion Note ID: K2F-C-4
Created: 2026-08-02
Updated: 2026-08-02
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Summary

Title: VGGT: Visual Geometry Grounded Transformer Authors: J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, D. Novotny (Meta AI / Visual Geometry Group, University of Oxford) Paper: arXiv:2503.11651 (CVPR 2025, Best Paper) Github: facebookresearch/vggt

VGGT (Visual Geometry Grounded Transformer) — a single large feed-forward transformer that takes any number of views of a scene and predicts, in one forward pass (seconds), all the core 3D quantities at once: camera parameters, depth maps, pointmaps, and point tracks — with no optimization, no bundle adjustment, no global alignment.

Core problem: the DUSt3R line proved feed-forward pointmap regression can replace SfM, but it is fundamentally two-view: many-view scenes need a graph of pairs plus a post-hoc global-alignment optimization, which is slow and scales badly (DUSt3R runs out of memory beyond ~32 frames). Can a single network just directly handle all views jointly?

Key idea: yes — with a transformer of minimal 3D inductive bias, trained on lots of 3D-annotated data, whose only structural trick is Alternating-Attention (AA): interleave frame-wise self-attention (tokens attend within each image) with global self-attention (tokens attend across all images jointly). Frame-wise attention normalizes per-image activations; global attention integrates information across views. That alternation is the entire multi-view reasoning mechanism — notably only self-attention, no cross-attention anywhere.

Why predict redundantly: cameras, depth, and pointmaps are related in closed form (e.g. cameras can be solved from pointmaps by PnP; depth from pointmap + camera). VGGT nonetheless predicts them all explicitly — the paper shows this over-complete supervision gives large gains, and at inference combining separately-predicted depth + camera yields better 3D points than the pointmap branch directly.

Main results: on camera pose estimation VGGT reaches AUC@30 85.3 (RealEstate10K, unseen) / 88.2 (CO3Dv2) in ~0.2 s, beating feed-forward peers (Fast3R, CUT3R, FLARE) and even optimization-based DUSt3R/MASt3R + global alignment; its tracking head beats the SOTA two-view matcher on ScanNet-1500. ~1.2B parameters.

2. Key Contributions

One feed-forward transformer for all of multi-view 3D — cameras + depth + pointmaps + tracks for N images jointly, in seconds, no optimization/alignment; often beats optimization-based methods.
Alternating-Attention backbone — the frame-wise ↔ global self-attention alternation as the whole multi-view mechanism (no cross-attention, minimal 3D bias).
Over-complete prediction helps — explicitly predicting redundant quantities improves accuracy; and combining independently-predicted depth+camera beats the direct pointmap branch at inference.
A versatile 3D foundation model — its features transfer to downstream tasks (non-rigid tracking, feed-forward novel-view synthesis).

3. Method

3.1 Problem & Outputs

Input: a sequence of $N$ RGB images $(I_i)_{i=1}^N$ of one scene. The transformer maps it to per-frame 3D annotations:

f\big((I_i)_{i=1}^N\big) = (g_i, D_i, P_i, T_i)_{i=1}^N

$g_i \in \mathbb{R}^9$ = camera parameters = $[\,q, t, f\,]$ : rotation quaternion $q\in\mathbb{R}^4$ , translation $t\in\mathbb{R}^3$ , field of view $f\in\mathbb{R}^2$ (principal point assumed centered).
$D_i \in \mathbb{R}^{H\times W}$ = depth map.
$P_i \in \mathbb{R}^{3\times H\times W}$ = pointmap (per-pixel 3D, à la DUSt3R).
$T_i \in \mathbb{R}^{C\times H\times W}$ = dense tracking features (not tracks directly — fed to a separate tracking module).

The first image is the reference frame; the network is permutation-equivariant for all but the first frame (reordering the others reorders outputs identically).

3.2 Feature Backbone: Alternating-Attention

Each image is patchified into $K$ tokens via DINO. All frames' tokens $t^I = \bigcup_i \{t^I_i\}$ go through the backbone, which alternates two self-attention types:

Frame-wise self-attention — attends to tokens within each frame separately.
Global self-attention — attends to tokens across all frames jointly.

Default $L = 24$ layers of each, alternating. This balances cross-view information integration (global) against per-image activation normalization (frame-wise). Crucially, the architecture uses only self-attention — no cross-attention layers at all (unlike DUSt3R's cross-attention decoders). The global self-attention layers are where all cross-view information mixes — the load-bearing part of the model.

3.3 Tokens & Coordinate Frame

Each image's tokens are augmented with 1 camera token $t^g$ and 4 register tokens $t^R$ . The first frame's camera/register tokens use a different set of learnable tokens ( $\bar t^g, \bar t^R$ ) than all other frames ( $t^g, t^R$ ) — this lets the model distinguish the reference frame and express predictions in the first camera's frame. After the AA transformer, register tokens are discarded; image + camera tokens are used for prediction. The first camera's extrinsics are fixed to identity ( $q_1=[0,0,0,1]$ , $t_1=0$ ).

3.4 Prediction Heads

Camera head — from the output camera tokens, 4 extra self-attention layers + a linear layer predict $g_i$ (intrinsics + extrinsics).
Dense head (DPT) — output image tokens $\hat t^I_i$ → a DPT layer → dense features $F_i$ , then a $3\times3$ conv → depth $D_i$ and pointmap $P_i$ . The DPT head also outputs dense tracking features $T_i$ and aleatoric uncertainty maps $\Sigma^D_i, \Sigma^P_i$ (used in the loss; proportional to model confidence after training).
Tracking head — a CoTracker2-style module: bilinearly sample the query feature, correlate against all frames' feature maps, refine with self-attention → 2D correspondences $\hat y_{j,i}$ for query point $y_j$ across all images. No temporal ordering assumed → works on any image set.

3.5 Training

End-to-end multi-task loss:

\mathcal{L} = \mathcal{L}_\text{camera} + \mathcal{L}_\text{depth} + \mathcal{L}_\text{pmap} + \lambda\,\mathcal{L}_\text{track}

Camera / depth / pointmap losses have similar ranges (no weighting needed); tracking is down-weighted with $\lambda = 0.05$ . Depth and pointmap losses use the predicted aleatoric uncertainty (uncertainty-weighted regression). Ground-truth is normalized by the pointmap's scale (average distance to origin), and — unlike DUSt3R — this normalization is not applied to the network outputs; the model is forced to learn the chosen normalization from data.

Implementation: ~1.2B parameters; AdamW, 160K iterations, peak lr $2\times10^{-4}$ , 8K warmup, cosine schedule; each batch samples 2–24 frames from one scene; images resized to max 518 px, aspect ratio randomized 0.33–1.0; trained on 64 A100 GPUs over ~9 days.

4. Experiments & Results

Camera pose estimation (AUC@30, higher better; runtime for the full estimate):

Method	RealEstate10K (unseen)	CO3Dv2	Runtime
CUT3R	75.3	82.8	~0.6 s
FLARE	78.8	83.3	~0.5 s
Fast3R	72.7	82.5	~0.2 s
VGGT (feed-forward)	85.3	88.2	~0.2 s

VGGT leads by a wide margin and without any optimization — DUSt3R/MASt3R need global alignment / bundle adjustment (up to ~10 s) to compete. On two-view matching (ScanNet-1500) its tracking head beats the specialized SOTA matcher RoMa, despite not being designed for two views. It also delivers strong multi-view depth and pointmap accuracy, and its features transfer to downstream 3D tasks.

Key ablation: the Alternating-Attention design and the over-complete prediction both give substantial gains; combining independently-estimated depth + camera at inference yields more accurate 3D points than reading the pointmap branch directly.

5. Strengths / Limitations / Legacy

Strengths

Closes the arc: removes DUSt3R's two-view + global-alignment seam — one forward pass over all views, seconds, often beating optimization-based pipelines.
Minimal 3D inductive bias + scale: the model learns geometry, and the only architectural commitment (Alternating-Attention) is simple and effective.
One model, many outputs (cameras/depth/points/tracks) and transferable features — a genuine 3D foundation model.

Limitations

Huge: ~1.2B params, 64×A100·9 days to train; global attention over all frames' tokens is $O((NK)^2)$ — memory/compute grow fast with #views × resolution.
Feed-forward accuracy still leans on massive 3D-annotated training data; behavior on out-of-distribution scenes is a learned prior, not a guarantee.
Predictions are entangled through the shared global-attention trunk — a property that is exactly what makes it interesting to probe/attack (see below).

Legacy / relevance to my work

VGGT is the endpoint of this reading arc (Transformer → ViT → DPT → CroCo → CroCo v2 → DUSt3R → VGGT): every prior note is a component here — Transformer attention, ViT patch-tokens, DPT dense heads, CroCo/DUSt3R pointmaps.
The global (cross-view) self-attention layers of the Alternating-Attention backbone are the surface the xattn-spillover work targets — the mechanism by which information leaks/mixes across frames lives entirely in those layers. Understanding VGGT here is the prerequisite for reasoning about that attack.

References

Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., & Novotny, D. (2025). VGGT: Visual Geometry Grounded Transformer. CVPR (Best Paper). arXiv:2503.11651, code. — source paper (Fig. 2 architecture above)
DUSt3R: the two-view pointmap-regression precursor VGGT generalizes to N views — read first
Oquab, M., et al. (2023). DINOv2. — the patch-tokenizer backbone VGGT uses
Karaev, N., et al. (2023). CoTracker. — the tracking architecture VGGT's tracking head is based on (CoTracker2)
Wang, J., et al. (2024). VGGSfM: Visual Geometry Grounded Deep SfM. CVPR. — the camera parametrization and differentiable-SfM lineage VGGT descends from
Transformer · ViT · DPT: the backbone components reused inside VGGT

Table of Contents