Description: Paper note on "Image Generators are Generalist Vision Learners" (Vision Banana, arXiv:2604.20329, Google DeepMind). Instruction-tunes the Nano Banana Pro image generator, at a very low data-mix ratio, to solve visual-understanding tasks by generating the answer as an RGB image (segmentation masks as colored images, depth/normals as false-color). One shared set of weights, no task-specific heads or losses; reaches SOTA-level results on 2D segmentation, monocular metric depth, and surface normals while keeping its generation ability. Thesis: image-generation pretraining plays the role LLM pretraining plays for language.
My Notion Note ID: K2G-D-3
Created: 2026-09-06
Updated: 2026-09-06
License: Free to share: please credit Yu Zhang and link back to yuzhang.io

1. Paper Information

Title: Image Generators are Generalist Vision Learners
Authors: Valentin Gabeur, Shangbang Long, Songyou Peng (equal contribution), Paul Voigtlaender, Shuyang Sun, Jonathan T. Barron, Kyle Genova, Saining Xie, Kaiming He, Thomas Funkhouser, Jean-Baptiste Alayrac, Radu Soricut, et al. (25 authors, Google DeepMind)
Paper: arXiv:2604.20329
Project: vision-banana.github.io
Code: no public release found at time of writing

2. Summary

Vision Banana turns an image generator into a generalist vision model. The base model is Nano Banana Pro (Google's Gemini 3 Pro Image, a natively multimodal image generator). The authors instruction-tune it by mixing a small amount of vision-task data into the generator's own training mixture, at a very low ratio, so the model gains understanding tasks without losing its generative priors.

The trick that unifies everything: reframe each vision task as generating an RGB image. A segmentation result is emitted as a colored image (one color per class or instance); depth and surface normals are emitted as false-color images. So there is one set of weights and no task-specific architecture or loss: only the text prompt changes to select the task.

Under a zero-shot-transfer protocol (no training on the evaluation benchmark splits), Vision Banana reaches SOTA-level results across 2D segmentation, monocular metric depth, and surface normals, beating strong specialists (SAM 3 on segmentation, Depth Anything V3 on metric depth, Lotus-2 on normals), while retaining its image generation and editing quality. The paper's thesis: generative image pretraining is to vision what LLM pretraining is to language - a route to foundational vision models that both generate and understand.

3. Key Contributions

Evidence that image generators are generalist vision learners: a pretrained generator (Nano Banana Pro) already carries rich visual understanding, unlockable by light instruction-tuning.
Vision Banana: a single instruction-tuned model that is SOTA-competitive across 2D and 3D understanding tasks with no specialized architecture and no custom loss.
A universal interface: express every task's output as an RGB image (masks as colors, depth/normals as false-color), so one generative model serves segmentation, depth, normals, and generation at once.
Capability preservation: mixing vision data at a low ratio adds understanding while keeping generation/editing quality nearly intact.

4. Background

Image generators (diffusion / autoregressive image models) are trained to synthesize pixels. Historically, understanding tasks (segmentation, depth) use discriminative models with task-specific heads. This paper collapses that generative-vs-discriminative divide.
Nano Banana Pro = Google's Gemini 3 Pro Image: a natively multimodal image generator (image generation is a modality of a Gemini model). Vision Banana is a lightweight tuned variant of it.
Baselines it is measured against: SAM 3 (Segment Anything, promptable segmentation), Depth Anything V3 (monocular depth foundation model), Lotus-2 (diffusion-based dense-prediction model for normals), plus DINO-X and others.
Zero-shot transfer protocol: Vision Banana is not trained on the benchmark train splits, so its numbers are compared against other zero-shot / foundation approaches rather than benchmark-specialized models.

5. Method

5.1 Instruction-tuning the image generator

Start from Nano Banana Pro (frozen recipe, same weights family).
Mix vision-task data into the model's own training mixture at a very low ratio - a light touch that adds understanding tasks without eroding the generative prior.
Training data: in-house model annotations on web-crawled 2D images for the 2D tasks, and synthetic data from rendering engines for the 3D tasks. No data from the evaluation benchmarks' splits.
No specialized architecture and no custom training loss: all tasks share the base generator's weights; the task is selected by the text prompt.

5.2 Tasks as RGB image generation

Every output is an image:

Segmentation (semantic / instance / referring): generate a colored visualization where each pixel's color encodes its class or instance, driven by a prompt that supplies the color mapping (e.g. "generate a semantic segmentation image using {cat: red, ...}").
Surface normals: emit camera-space normals as RGB (x -> pink, y -> light green, z -> light blue).
Metric depth: emit a false-color image (see 5.3).

5.3 Depth encoding

Unbounded metric depth d in [0, inf) must fit into a bounded color image. The paper uses an invertible power transform that maps depth into [0,1), then lays that scalar along a color path (a Hilbert-curve traversal of the RGB cube) for a smooth false-color code:

$f(d,\lambda,c) = 1 - \left(1 - \frac{d}{\lambda c}\right)^{\lambda+1} \tag{1}$

Symbols: d = metric depth (meters); f in [0,1) = normalized code turned into color; lambda = -3 is the fixed shape parameter and c = 10/3 the fixed scale parameter (paper constrains lambda < -1). The map is invertible, so a predicted color image is decoded back to metric depth.

6. Results

All under the zero-shot-transfer protocol (no benchmark-split training). Directional headline: beats SAM 3 on segmentation, Depth Anything V3 on metric depth, Lotus-2 on surface normals.

6.1 2D semantic understanding

Semantic segmentation (Cityscapes val, mIoU): Vision Banana 0.699 vs SAM 3 0.652.
Referring-expression segmentation (vs SAM 3 Agent): RefCOCOg cIoU 0.738 vs 0.734; ReasonSeg gIoU 0.793 vs 0.770 - on par / slightly ahead.
Instance segmentation (SA-Co/Gold, pmF1): 0.540 vs DINO-X 0.552 - still behind the strongest specialist, one of the paper's acknowledged gaps (evaluated on a 500-query subset).

6.2 3D understanding from a single image

Monocular metric depth (delta_1 accuracy, higher better): average 0.882 across the reported benchmark suite; on the 4-benchmark subset (NYU / ETH3D / DIODE / KITTI) about 0.929 vs Depth Anything V3 0.918.
Surface normals (mean angular error, lower better): average 18.928 deg vs Lotus-2 19.642 deg (avg of 4 datasets); on NYU indoor 15.549 deg vs Lotus-2 16.558 deg.
Predicted depth can be lifted to a 3D point cloud.

6.3 Generation retained

The low-ratio mixing keeps generation quality close to the Nano Banana Pro baseline (head-to-head win rates): text-to-image GenAI-Bench 53.5% vs 46.5% (Vision Banana preferred); image editing ImgEdit 47.8% vs 52.2% (marginally below parity, base model preferred). So understanding is added with text-to-image improved and editing near-parity - no large generation regression.

7. Discussion and Limitations

Thesis: image generation as a universal interface for vision, and generative pretraining as the vision analogue of LLM pretraining - one foundational model for both generation and understanding.
Compute cost: running a full image generator per prediction is far heavier than a lightweight specialist model.
Instance segmentation gap: still trails the best domain-trained specialists on SA-Co/Gold.
Scope: monocular-image inputs only; multi-view and video are future work. The model does not use camera intrinsics in training or inference.

Personal comment: the elegant part is that "output the answer as an image" removes the need for any task head, so a single generator covers segmentation, depth, and normals by prompt alone; that is a genuinely different bet from the DUSt3R/VGGT feed-forward-regression line. The caveat is cost and the zero-shot-transfer framing: the SOTA claims are against zero-shot/foundation baselines, not benchmark-specialized models, so "beats SAM 3 / Depth Anything V3" should be read within that protocol. This is my own reading, not a claim in the paper.

Vision Banana: Image Generators are Generalist Vision Learners

Table of Contents