K2F-C-2
CroCo v2: Improved Cross-View Completion Pre-training
- Description: CroCo v2 paper note — scales up cross-view completion pre-training on three axes: (a) millions of real-world image pairs (not just synthetic Habitat) via a scalable overlap-controlled collection method, (b) RoPE relative position embeddings for robustness to resolution/cropping, (c) a bigger ViT-Large encoder + decoder. Reaches SOTA on stereo matching and optical flow with a plain DPT head and no task-specific tricks; provides the weights DUSt3R initializes from
- My Notion Note ID: K2F-C-2
- Created: 2026-08-02
- Updated: 2026-08-02
- License: Free to share: please credit Yu Zhang and link back to yuzhang.io
Table of Contents
- 1. Summary
- 2. Key Contributions
- 3. Method
- 4. Experiments & Results
- 5. Strengths / Limitations / Legacy
- References
1. Summary
Title: CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow Authors: P. Weinzaepfel, T. Lucas, V. Leroy, Y. Cabon, V. Arora, R. Brégier, G. Csurka, L. Antsfeld, B. Chidlovskii, J. Revaud (NAVER LABS Europe) Paper: arXiv:2211.10408 (ICCV 2023) Github: naver/croco
CroCo v2 — a scaled-up, improved version of CroCo's cross-view completion pre-training, pushed until it reaches state of the art on stereo matching and optical flow — with a plain transformer + DPT head and no task-specific machinery (cost volumes, iterative refinement, feature pyramids).
Core problem: CroCo showed cross-view completion is a good geometric pretext task, but underdelivered because of two limits: (a) it trained on synthetic (Habitat) data only — real pairs are hard to collect at scale, yet scale is what makes pre-training work; (b) it used absolute position embeddings, which don't generalize well to the varying resolutions and crops that dense geometric tasks demand.
Three fixes (the whole paper):
- Scale the data to the real world — a method to gather millions of suitable real image pairs from diverse real datasets (ARKitScenes, MegaDepth, 3DStreetView, IndoorVL) plus Habitat, with controlled visual overlap.
- Relative position embeddings via RoPE — inject relative token positions in attention, robust to resolution changes and cropping (which absolute embeddings are not).
- Scale up the network — a bigger ViT-Large encoder and a larger decoder.
Main result: fine-tuned on stereo / flow with a DPT head that directly regresses disparity (stereo) or flow, CroCo v2 matches or beats task-specific SOTA on KITTI 2015, ETH3D, Spring, MPI-Sintel — showing a generic pre-trained cross-view model can replace hand-built matching architectures.
2. Key Contributions
- Scalable real-world pair collection — a practical recipe to mine millions of cross-view pairs with the right overlap from real datasets, breaking CroCo's synthetic-only limitation.
- RoPE in cross-view completion — showing relative position embeddings substantially improve dense geometric transfer (resolution/crop robustness).
- Scaled ViT + SOTA without task tricks — a larger encoder/decoder that reaches state-of-the-art stereo and optical flow using only a generic architecture + DPT head.
3. Method
3.1 Recap: Cross-View Completion
Same pretext task as CroCo: split two views into patches, mask 90% of the first image, feed the rest to a shared Siamese ViT encoder, and let a ViT decoder with cross-attention reconstruct the masked patches by attending to the second (reference) view. To leverage the reference and fill the ~90% hole, the model must reason about scene geometry and the spatial relationship between views — which primes it for geometric downstream tasks. (See CroCo for the base task, MSE loss, and MAE lineage.)
3.2 Scalable Real-World Pair Collection
The bottleneck was data. A good training pair needs the right overlap: too much overlap makes completion trivial; too little collapses it to single-image MIM. CroCo v2 controls visual overlap explicitly when mining pairs, and gathers them from a mix of real datasets — ARKitScenes, MegaDepth, 3DStreetView, IndoorVL — plus the original synthetic Habitat. This yields diverse, large-scale real pairs (indoor, outdoor, street-level) instead of synthetic-only.
3.3 Relative Position Embeddings (RoPE)
A ViT treats patches as an unordered set, so it needs position information. CroCo used absolute (added cosine) position embeddings — but dense pixel-level tasks must handle varying resolutions and be robust to cropping, where absolute positions break. RoPE (Rotary Position Embedding) applies a rotation to each key/query by its absolute position such that the attention similarity between two tokens depends only on their relative positions (and feature similarity). So RoPE models relative positions at any resolution — the property dense geometric transfer needs. The paper finds RoPE substantially better than absolute embeddings, and prior work notes relative attention is especially crucial in the cross-attention for flow.
3.4 Scaling Up + Downstream Heads
The larger real-data corpus lets the model scale: a ViT-Large encoder and a larger decoder (both encoder and decoder grow). For downstream stereo matching and optical flow — both dense correspondence problems — a plain DPT head on the pre-trained encoder+decoder directly predicts the output (disparity for stereo, 2D flow for optical flow), with no cost volumes, warping, iterative refinement, or feature pyramids. Variants: CroCo-Stereo and CroCo-Flow.
4. Experiments & Results
- Stereo matching / optical flow benchmarks: the paper reaches state-of-the-art on KITTI 2015, ETH3D, Spring, MPI-Sintel without task-specific techniques (its headline claim). On Middlebury it is best on 6/15 sequences but has a worse average error (large-disparity outliers on a few sequences), so it is not an overall Middlebury SOTA.
- Ablations confirm each of the three axes pulls its weight: real-world pairs > synthetic-only; RoPE > absolute position embeddings; larger encoder/decoder helps. The 90% masking ratio from CroCo is retained.
The takeaway is a proof of scaling: the same cross-view-completion idea, given real data + relative positions + more capacity, beats hand-engineered matching pipelines.
5. Strengths / Limitations / Legacy
Strengths
- Turns CroCo from "promising pretext task" into "SOTA on real geometric benchmarks" by fixing the two things that held it back (data source, position encoding) — a clean scaling study.
- The "generic architecture + DPT head beats task-specific pipelines" result is the strongest evidence that cross-view pre-training genuinely learns matching/geometry.
- RoPE-for-dense-geometry is a reusable lesson (carried into later multi-view transformers).
Limitations
- Still a two-view / pixel-reconstruction pretext task; not yet a full 3D-reconstruction model (that is DUSt3R's step).
- Requires posed/overlapping real pairs to mine — heavier data pipeline than single-image MIM.
- Fine-tuning per downstream task (stereo vs flow) rather than one unified geometry model.
Legacy (why it's in this reading arc)
- CroCo v2 provides the improved pre-trained weights and architecture (ViT-Large encoder + cross-attention decoder + RoPE) that DUSt3R builds on — DUSt3R initializes from CroCo/CroCo v2 and repurposes the two-view encoder/decoder to regress pointmaps instead of reconstructing pixels. The scaling recipe here (real data, relative positions, bigger ViT) is the substrate DUSt3R and then VGGT stand on.
References
- Weinzaepfel, P., Lucas, T., Leroy, V., Cabon, Y., Arora, V., Brégier, R., Csurka, G., Antsfeld, L., Chidlovskii, B., & Revaud, J. (2023). CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow. ICCV. arXiv:2211.10408, code. — source paper (Fig. 1 above)
- CroCo: the base cross-view completion pretext task — read first
- Su, J., et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE). — the relative position embedding CroCo v2 adopts
- DPT: the dense-prediction head used to regress disparity / flow
- DUSt3R: next in the arc — reuses CroCo v2's encoder/decoder to regress pointmaps