Rethinking Visual Intelligence: Insights from Video Pretraining

Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.

Paper: https://arxiv.org/pdf/2510.24448

Website: https://pabloacuaviva.github.io/rethinking-visual-intelligence

Composition of Memory Experts for Diffusion World Models

Paper: https://arxiv.org/abs/2605.1881

A paper from our group got accepted to NeurIPS 2025!

KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products

Zixuan Xia, Aram Davtyan, Paolo Favaro, in Neural Information Processing Systems (NeurIPS), 2025.

We propose KOALA++, a scalable Kalman-based optimization algorithm that explicitly models structured gradient uncertainty in neural network training. Unlike second-order methods, which rely on expensive second order gradient calculation, our method directly estimates the parameter covariance matrix by recursively updating compact gradient covariance products. This design improves upon the original KOALA framework that assumed diagonal covariance by implicitly capturing richer uncertainty structure without storing the full covariance matrix and avoiding large matrix inversions. Across diverse tasks, including image classification and language modeling, KOALA++ achieves accuracy on par or better than state-of-the-art first- and second-order optimizers while maintaining the efficiency of first-order methods.

Paper: https://arxiv.org/abs/2506.04432

Code: https://github.com/Sumxiaa/KOALA_Plus_Plus

A paper from our group got accepted to ICCV 2025 as an ORAL!

[ORAL] Diffusion Image Prior

Hamadi Chihaoui, Paolo Favaro, in the Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

Zero-shot image restoration (IR) methods based on pretrained diffusion models have recently achieved significant success. These methods typically require at least a parametric form of the degradation model. However, in real-world scenarios, the degradation may be too complex to define explicitly without relying on crude approximations. To handle this general case, we introduce the DIffusion Image Prior (DIIP). We take inspiration from the Deep Image Prior (DIP), since it can be used to remove artifacts without the need for an explicit degradation model. However, in contrast to DIP, we find that pretrained diffusion models offer a much stronger prior, despite being trained without knowledge from corrupted data. We show that, the optimization process in DIIP first reconstructs a clean version of the image before eventually overfitting to the degraded input, but it does so for a broader range of degradations than DIP. In light of this result, we propose a blind image restoration (IR) method based on early stopping, which does not require prior knowledge of the degradation model. We validate DIIP on various degradation-blind IR tasks, including JPEG artifact removal, waterdrop removal, denoising and super-resolution with state-of-the-art results.

Paper: https://openaccess.thecvf.com/content/ICCV2025/html/Chihaoui_Diffusion_Image_Prior_ICCV_2025_paper.html

MIRAGE: Unsupervised Single Image to Novel View Generation with Cross Attention Guidance

Llukman Cerkezi, Aram Davtyan, Sepehr Sameni, Paolo Favaro, in the Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2025.

This paper introduces a novel pipeline to generate novel views of an object from a single image. Our method, MIRAGE, trains a pose-conditioned diffusion model on a dataset of real images of multiple unknown categories, all completely unsupervised. The conditioning is obtained via clustering pre-trained self-supervised features to identify approximate object categories and poses. At inference time, we introduce hard-attention guidance and apply cross-view attention to align the appearance of the objects in the generated views with that in the input image. Through our experiments, we show that MIRAGE generates novel views that are on par or better than supervised methods in terms of image realism and 3D consistency. Furthermore, MIRAGE is robust to diverse textures and geometries, not restricted to simple rigid rotations, and is capable of generating plausible deformations of nonrigid objects, such as animals.

Paper: https://openaccess.thecvf.com/content/ICCV2025W/3D-VAST/html/Cerkezi_MIRAGE_Unsupervised_Single_Image_to_Novel_View_Generation_with_Cross_ICCVW_2025_paper.html

Code: https://github.com/llukmancerkezi/mirage

A paper from our group got accepted to CVPR 2025!

GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control

Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Pedro M B Rezende, Yasaman Haghighi, David Brüggemann, Isinsu Katircioglu, Lin Zhang, Xiaoran Chen, Suman Saha, Marco Cannici, Elie Aljalbout, Botao Ye, Xi Wang, Aram Davtyan, Mathieu Salzmann, Davide Scaramuzza, Marc Pollefeys, Paolo Favaro, Alexandre Alahi, in the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

We present GEM, a Generalizable Ego-vision Multimodal world model that predicts future frames using a reference frame, sparse features, human poses, and ego-trajectories. Hence, our model has precise control over object dynamics, ego-agent motion and human poses. GEM generates paired RGB and depth outputs for richer spatial understanding. We introduce autoregressive noise schedules to enable stable long-horizon generations. Our dataset is comprised of 4000+ hours of multimodal data across domains like autonomous driving, egocentric human activities, and drone flights. Pseudo-labels are used to get depth maps, egotrajectories, and human poses. We use a comprehensive evaluation framework, including a new Control of Object Manipulation (COM) metric, to assess controllability. Experiments show GEM excels at generating diverse, controllable scenarios and temporal consistency over long generations. Code, models, and datasets are fully open-sourced.

Paper: https://arxiv.org/abs/2412.11198