Institution: Agroscope
Starting date: As soon as possible
Duration: 6–12 months
Working place: Agroscope Posieux
Language: French/English
Contacts: Hassan-Roland Nasser, Helge Aasen
Email: roland.nasser@agroscope.admin.ch, helge.aasen@agroscope.admin.ch

Background

Training a detection model for a new plant species or a new field site still depends on hours of manual annotation by experts. Vision foundation models such as SAM 3 (Meta, 2025) and Grounding DINO can now produce a first set of labels from a text prompt or a few example images, and tools like Autodistill or ARAMSAM (Oehme et al., 2026) already use them to speed up labelling. The open question is where these automatic labels can be trusted and where an expert is still needed.

Published results show that foundation models work well for large, common objects but struggle with visually similar species and small objects, and that detectors trained on one site or one month of the flowering season can lose most of their accuracy at another (Gallmann et al., 2022; Frontiers in Plant Science, 2025).

Project

The thesis is embedded in an Agroscope project on flowering field margins, in which 25 margins in three cantons are flown by drone roughly monthly from April to August to estimate the flower resources available to pollinators under different mowing regimes. A second Master student with botanical training, based at Reckenholz, will provide expert labels on a subset of the images; this thesis provides the computer vision side. Together the two theses build an annotated dataset of drone images of Swiss flowering plants.

The research questions, to be refined with the student, are:

• How much expert annotation does an iterative, semi-automatic labelling loop need to reach expert-level labels on a multi-species flower dataset, and how does this vary with species and flower size?
• Can the quality of automatically generated labels be estimated without ground truth (for example through agreement between models or prompts), and does that estimate predict real accuracy per species?
• How much does detection accuracy drop between months and between sites, and how much of it can be recovered by re-prompting with a few current examples?

The work covers the whole chain: drone image preparation, running and adapting foundation models, designing the labelling loop, evaluation against the botanist's labels, and writing up. The same methods will be tested on other public datasets to check that the findings transfer.

Since this is an active research field, the exact scope and methods may be adjusted during the project as the literature evolves.

What we offer

• Real field data with a clear ecological question behind it
• Close supervision and regular exchange with both the vision and the ecology teams
• Access to GPU computing, annotation tools and model tracking infrastructure
• A dataset and results intended for publication

Technical stack

Python, Git, Jupyter / VS Code, PyTorch, annotation tools, CSCS Cluster stack (SLURM, PyFireCrest, etc.), MLFlow, NVIDIA GPUs, Linux.

Prior knowledge

Good proficiency in Python and basic knowledge of deep learning. Experience with computer vision, drones or GIS is a plus; the rest can be learned during the project. Interest in plants is welcome but botanical expertise is not required.

References

• Gallmann et al., 2022. Flower mapping in grasslands with drones and deep learning. Frontiers in Plant Science 12:774965.
• Oehme et al., 2026. Orchestrating segment anything models to accelerate segmentation annotation on agricultural image datasets. Frontiers in Artificial Intelligence 8:1748468.
• Wang et al., 2026. Accurate weed detection in UAV grassland images via intelligent annotation. Frontiers in Plant Science 17.
• Meta AI, 2025. SAM 3: Segment Anything with Concepts.

Institution: Agroscope
Starting date: As soon as possible
Duration: 6–12 months
Working place: Agroscope Posieux
Language: French/English
Contacts: Hassan-Roland Nasser, Helge Aasen
Email: roland.nasser@agroscope.admin.ch, helge.aasen@agroscope.admin.ch

Background

Building a detection model for a new agricultural dataset still requires an engineer to make a long series of decisions: which foundation model to use, how to prompt it, whether to tile the images, which thresholds to apply, how to filter the results, and when the labels are good enough to train on. Large language and vision-language models can now call tools, inspect intermediate results and plan multi-step workflows, and several recent papers use them to orchestrate annotation (Xia et al., 2025; Zhang et al., 2026). Almost all of this work is on text data or general-purpose images. Whether such an agent can make sound pipeline decisions on fine-grained agricultural imagery, where visually similar species and small objects are the norm, has not been studied.

Project

The thesis develops an agent that receives a raw drone image dataset and a task description, and must configure and run a detection pipeline built from existing components (SAM 3, Grounding DINO, embedding models, a lightweight detector). The agent is not the goal; it is the object of study. The research questions, to be refined with the student, are:

• Given only the data, does the agent reach the same pipeline configuration and accuracy as an experienced engineer, and on which datasets does it diverge?
• Which information does the agent need in order to decide well: raw images, model outputs, agreement between models, embedding clusters, sparse human feedback?
• When the agent declares a class finished or unlabelable, how often is it right against hidden expert labels? Where is it confidently wrong?
• Does a configuration learned on one dataset (pasture weeds) transfer to another (flowering field margins), and does the agent recognise when it should not?
• What does autonomy cost in compute and time, compared with the expert time it saves?

Development starts on RumexDrone, an existing Agroscope drone dataset of pasture weeds with full ground truth. The final experiments use drone images of flowering field margins collected in an Agroscope project on pollinator resources, for which expert labels are produced by a parallel Master thesis. The two students work in the same team and share tools, but the theses are independent.

Since this is an active research field, the exact scope and methods may be adjusted during the project as the literature evolves.

What we offer

• A well-defined experimental question in a field that is currently moving fast
• Existing datasets with ground truth, so experiments can start early
• Close supervision and a second student working on a complementary topic
• Access to GPU computing, model tracking and a budget for API calls
• Results intended for publication

Technical stack

Python, Git, PyTorch, an LLM orchestration library (LangGraph or similar), Weights & Biases, NVIDIA GPUs, Linux.

Prior knowledge

Good proficiency in Python and basic knowledge of deep learning. Prior experience with LLM tool use or computer vision is a plus but can be learned during the project. The student should be comfortable designing experiments and reading recent papers, since much of the relevant work is less than a year old.

References

• Xia et al., 2025. From Selection to Generation: A Survey of LLM-based Active Learning. ACL 2025.
• Zhang et al., 2026. From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models. arXiv:2606.18846.
• Oehme et al., 2026. Orchestrating segment anything models to accelerate segmentation annotation on agricultural image datasets. Frontiers in Artificial Intelligence 8:1748468.
• Meta AI, 2025. SAM 3: Segment Anything with Concepts.

Context and Motivation:

Medical imaging plays a critical role in diagnostic workflows, but the diagnostic utility of MRI and CT scans can be significantly degraded by various types of quality issues. These include patient or organ motion artifacts (see, for example, the blurry edges caused by the heartbeat in the figure below), poor contrast, scanner miscalibration, and incorrect scan volume positioning or coverage. Currently, image quality checks are often performed manually by radiologists or technicians, which is time-consuming and subjective or sometimes too late. There is a growing need for robust, automated and instant tools that can perform image quality assessment (IQA) reliably and in real-time to aid clinical decision-making and reduce the need for repeated scans.

Thesis Objective:

The goal of this thesis is to develop a machine learning-based system that automatically assesses the quality of MRI and/or CT scans. The model should detect and categorize common quality issues such as:

• Patient motion artifacts (blurring, ghosting, ringing)
• Scan misalignments (head/body not centered in scan volume)
• Field-of-view clipping or partial coverage of anatomy
• Noise and low signal-to-noise ratio (SNR)
• Beam hardening artifacts (in CT)
• Metal or implant-induced artifacts
• Contrast agent timing issues (e.g., poor vascular contrast)
• Slice thickness inconsistencies or spacing artifacts
• and others

The model should output a quality score and, where possible, localize regions with detected artifacts.

Data:

The Department of Diagnostic, Interventional and Pediatric Radiology (DIPR) will provide a dataset of anonymized MRI and CT scans, along with partial annotations indicating known quality issues. These may include binary or categorical labels per scan, or image-level artifact annotations. Additional manual annotations may be curated as needed for evaluation and training.

Tasks:

1. Literature review on existing approaches to medical image quality assessment and artifact classification.
2. Data preparation: preprocessing and augmentation, designing train/test splits.
3. Model development: investigate and implement CNNs, transformers, or hybrid architectures for volumetric data; evaluate existing pre-trained models.
4. Multi-label classification and localization: develop techniques for identifying and attributing multiple artifacts in a single scan.
5. Evaluation: using standard metrics (AUC, F1, mean Average Precision), possibly stratified by artifact type.
6. User-facing output: produce interpretable visualizations (e.g., saliency maps) to explain detected quality issues.

Requirements:

• Strong background in machine learning and deep learning
• Experience with medical imaging data and 3D image processing is a plus
• Interest in healthcare applications and interdisciplinary collaboration

Deliverables:

• A trained and evaluated AI model for scan quality assessment
• An annotated dataset and data curation tools
• A comprehensive thesis document describing methodology, experiments, and results
• Source code and user documentation

Expected Impact:

This work aims to reduce the burden on radiologists, improve patient throughput, and decrease repeat scan rates by flagging unusable scans early. The project also has potential for real-world deployment in clinical imaging pipelines.

Supervisors

Department of Computer Science (INF)

Prof. Dr. Paolo Favaro, email: paolo.favaro@unibe.ch

Department of Diagnostic, Interventional and Pediatric Radiology (DIPR)

Prof. Dr. med. Hendrik von Tengg-Kobligk, email: Hendrik.vonTengg@insel.ch

Background
Precision agriculture has emerged as a critical approach to enhancing agricultural productivity and sustainability. In this context, semantic segmentation, a computer vision technique that classifies each pixel in an image into a predefined category, holds immense potential for monitoring crop health and development. However, models are usually bound to specific conditions during experiment acquisition and often suffer from generalization. This thesis aims at improving the generalization by exploring the design of unconventional augmentation techniques. One of the proposed techniques is to use AI-generated backgrounds from multiple scenarios and overlay masks of plants and weeds and compare the performance with other conventional techniques.

Aim

• Create unconventional augmentation techniques such as mixing randomly overlapping cropped masks on backgrounds generated using generative models simulating multiple scenarios (invitro, field, drone).
• Develop semantic segmentation models to automate the detection of faba beens and colza.
• Evaluate the performance of the proposed model using a dataset of field images acquired over time and with different augmentation scenario.

Methodology

• Data Collection: we already have a big dataset of images annotated with segmentation masks.
• Data Preprocessing: Preprocess the images to ensure consistency in size, format, and color calibration.
• Model Development: Employ a deep learning architecture, such as U-Net or DeepLabV3+, for semantic segmentation.
• Model Training: Train the model using the preprocessed image dataset, optimizing hyperparameters for improved performance.
• Model Evaluation: Evaluate the trained model's performance on a separate validation dataset, assessing its accuracy and robustness.
• Analysis in context: the results of the developed models will be used by other researchers’ experts in the agricultural field to draw conclusions about their research.

Requirements

• Interest/Experience with image processing.
• Python programming knowledge (Pytorch bonus).

Language
English.

Starting date
As soon as possible.

Duration
6 months minimum (can be increased to 12 month if student is interested).

Supervisors
Paolo Favaro, Hassan-Roland Nasser.

Institutes
Computer Vison Group, Posieux (home office possible).

Contact
Hassan-Roland Nasser, hassan-roland.nasser@agroscope.admin.ch .

Background
The welfare of pigs in commercial barns is a cornerstone for what it bears from economic and ethical point of view. On the one hand, commercial barns are increasingly aware of the pig’s welfare by sustaining less aggressive environment for their pigs. On the other hand, the higher the welfare in a barn, the higher the economic return (better meat quality, less incident, …). For this, commercial barns usually monitor aggressive pigs and exclude them from their next breeding programs. Manual assessment is often resources hungry and not consistent as it is operator depends. The aim of this thesis is to exploit part of the 15TB video dataset that we have in order to find clues about aggressive behavior in pigs.

Aim

• Review an object detection model we created and improve it.
• Create a projection map from the acquired videos to the barn map.
• From the Object detection model, obtain a proxy dataset of heads and tails position in the actual barn.
• Analyze the relative motions of head and tails and explore the relationship between the speed of head-to-tail motion in collaboration with a expert animal scientist.

Methodology

• Data discovery: we already have a big dataset of 15TB of videos over 4 cameras.
• Model Development: Employ a deep learning architecture, such as FasterRCNN or Yolov8.
• Deep Learning Model Training and evaluation: Train the model using the preprocessed image dataset, optimizing hyperparameters for improved performance.
• Comparing supervised and unsupervised ML models to assess the predictability of aggressive behavior.
• Analysis in context: the results of the developed models will be used by other researchers’ experts in the animal field to draw conclusions about their research.

Requirements

• Interest/Experience with image processing.
• Python programming knowledge (Pytorch bonus).

Language
English.

Starting date
As soon as possible.

Duration
6 months minimum (can be increased to 12 month if student is interested).

Supervisors
Paolo Favaro, Hassan-Roland Nasser.

Institutes
Computer Vison Group, Posieux (home office possible).

Contact
Hassan-Roland Nasser, hassan-roland.nasser@agroscope.admin.ch .

Background
Light microscopy became an indispensable tool in life sciences research. Deconvolution is an important image processing step in improving the quality of microscopy images for removing out-of-focus light, higher resolution, and beter signal to noise ratio. Currently classical deconvolution methods, such as regularisation or blind deconvolution, are implemented in numerous commercial software packages and widely used in research. Recently AI deconvolution algorithms have been introduced and being currently actively developed, as they showed a high application potential.

Aim
Adaptation of available AI algorithms for deconvolution of microscopy images. Validation of these methods against state-of-the -art commercially available deconvolution software.

Material and Methods
Student will implement and further develop available AI deconvolution methods and acquire test microscopy images of different modalities. Performance of developed AI algorithms will be validated against available commercial deconvolution software.

Nature of thesis

• Al algorithm development and implementation: 50%.
• Data acquisition: 10%.
• Comparison of performance: 40 %.

Requirements

• Interest in imaging.
• Solid knowledge of AI.
• Good programming skills.

Supervisors
Paolo Favaro, Guillaume Witz, Yury Belyaev.

Institutes
Computer Vison Group, Digital Science Lab, Microscopy imaging Center.

Contact
Yury Belyaev, Microscopy imaging Center, yury.belyaev@unibern.ch, + 41 78 899 0110.

In the 1600s, a pioneering Dutch scientist named Antonie van Leeuwenhoek embarked on a remarkable journey that would forever transform our understanding of the natural world. Armed with a simple yet ingenious invention, the light microscope, he delved into uncharted territory, peering through its lens to reveal the hidden wonders of microscopic structures. Fast forward to today, where cryo-electron tomography (cryo-ET) has emerged as a groundbreaking technique, allowing researchers to study proteins within their natural cellular environments. Proteins, functioning as vital nano-machines, play crucial roles in life and understanding their localization and interactions is key to both basic research and disease comprehension. However, cryo-ET images pose challenges due to inherent noise and a scarcity of annotated data for training deep learning models.

Credit: S. Albert et al./PNAS (CC BY 4.0)

To address these challenges, this project aims to develop a self-supervised pipeline utilizing diffusion models for instance segmentation in cryo-ET images. By leveraging the power of diffusion models, which iteratively diffuse information to capture underlying patterns, the pipeline aims to refine and accurately segment cryo-ET images. Self-supervised learning, which relies on unlabeled data, reduces the dependence on extensive manual annotations. Successful implementation of this pipeline could revolutionize the field of structural biology, facilitating the analysis of protein distribution and organization within cellular contexts. Moreover, it has the potential to alleviate the limitations posed by limited annotated data, enabling more efficient extraction of valuable information from cryo-ET images and advancing biomedical applications by enhancing our understanding of protein behavior.

Methods
The segmentation pipeline for cryo-electron tomography (cryo-ET) images consists of two stages: training a diffusion model for image generation and training an instance segmentation U-Net using synthetic and real segmentation masks.

    1. Diffusion Model Training:
        a. Data Collection: Collect and curate cryo-ET image datasets from the EMPIAR
            database (https://www.ebi.ac.uk/empiar/).
        b. Architecture Design: Select an appropriate architecture for the diffusion model.
        c. Model Evaluation: Cryo-ET experts will help assess image quality and fidelity
            through visual inspection and quantitative measures
    2. Building the Segmentation dataset:
        a. Synthetic and real mask generation: Use the trained diffusion model to generate
            synthetic cryo-ET images. The diffusion process will be seeded from either a real
            or a synthetic segmentation mask. This will yield to pairs of cryo-ET images and
            segmentation masks.
    3. Instance Segmentation U-Net Training:
        a. Architecture Design: Choose an appropriate instance segmentation U-Net
            architecture.
        b. Model Evaluation: Evaluate the trained U-Net using precision, recall, and F1
            score metrics.

By combining the diffusion model for cryo-ET image generation and the instance segmentation U-Net, this pipeline provides an efficient and accurate approach to segment structures in cryo-ET images, facilitating further analysis and interpretation.

References
    1. Kwon, Diana. "The secret lives of cells-as never seen before." Nature 598.7882 (2021):
        558-560.
    2. Moebel, Emmanuel, et al. "Deep learning improves macromolecule identification in 3D
        cellular cryo-electron tomograms." Nature methods 18.11 (2021): 1386-1394.
    3. Rice, Gavin, et al. "TomoTwin: generalized 3D localization of macromolecules in
        cryo-electron tomograms with structural data mining." Nature Methods (2023): 1-10.

Contacts
Prof. Thomas Lemmin
Institute of Biochemistry and Molecular Medicine
Bühlstrasse 28, 3012 Bern
(thomas.lemmin@unibe.ch)

Prof. Paolo Favaro
Institute of Computer Science
Neubrückstrasse 10 3012 Bern
(paolo.favaro@unibe.ch)

Nature of the Thesis:

Literature review: 10%

Replication of Blob Loss paper: 10%

Implementation of the sliding window metrics:10%

Training on MS lesion segmentation task: 30%

Extension to other datasets: 20%

Results analysis: 20%

Fig. Results of an existing lesion filling algorithm, showing inadequate performance

Requirements:

Interest/Experience with image processing

Python programming knowledge (Pytorch bonus)

Interest in neuroimaging

Supervisor(s):

PD. Dr. Richard McKinley

Institutes: Diagnostic and Interventional Neuroradiology

Center for Artificial Intelligence in Medicine (CAIM), University of Bern

References: [1] Brain Imaging Generation with Latent Diffusion Models, Pinaya et al, Accepted in the Deep Generative Models workshop @ MICCAI 2022, https://arxiv.org/abs/2209.07162

Contact : PD Dr Richard McKinley, Support Centre for Advanced Neuroimaging (richard.mckinley@insel.ch)

References: [1] blob loss: instance imbalance aware loss functions for semantic segmentation, Kofler et al, https://arxiv.org/abs/2205.08209

Contact : PD Dr Richard McKinley, Support Centre for Advanced Neuroimaging (richard.mckinley@insel.ch)

References: [1] Isensee, F, Schell, M, Pflueger, I, et al. Automated brain extraction of multisequence MRI using artificial neural networks. Hum Brain Mapp. 2019; 40: 4952– 4964. https://doi.org/10.1002/hbm.24750

Contact : PD Dr Richard McKinley, Support Centre for Advanced Neuroimaging (richard.mckinley@insel.ch)

Correlating plant phenotypes such as leaf area or number of leaves to the genotype (i.e. changes in DNA) is a common goal for plant breeders and molecular biologists. Such data can not only help to understand fundamental processes in nature, but also can help to improve ecotypes, e.g., to perform better under climate change, or reduce fertiliser input. However, collecting data for many plants is very time consuming and automated data acquisition is necessary.

The project aims at building a machine learning model to automatically detect plants in top-view images (see examples below), segment their leaves (see Fig C) and to estimate the leaf area. This information will then be used to determine the leaf area of different Arabidopsis ecotypes. The project will be carried out in collaboration with researchers of the Institute of Plant Sciences at the University of Bern. It will also involve the design and creation of a dataset of plant top-views with the corresponding annotation (provided by experts at the Institute of Plant Sciences).

Contact: Prof. Dr. Paolo Favaro (paolo.favaro@unibe.ch)

The Gerontechnology and Rehabilitation group at the ARTORG Center for Biomedical Engineering is offering multiple MSc thesis projects to students, which are interested in working with real patient data, artificial intelligence and machine learning algorithms. The goal of these projects is to transfer the findings to the clinic in order to solve today’s healthcare problems and thus to improve the quality of life of patients.

• Assessment of Digital Biomarkers at Home by Radar. [PDF]
• Comparison of Radar, Seismograph and Ballistocardiography and to Monitor Sleep at Home. [PDF]
• Sentimental Analysis in Speech. [PDF]

Contact: Dr. Stephan Gerber (stephan.gerber@artorg.unibe.ch)

A 6 month intership at Prophesee, Grenoble is offered to a talented Master Student.

The topic of the internship is working on burst imaging following the work of Sam Hasinoff, and exploring ways to improve it using event-based vision.

A compensation to cover the expenses of living in Grenoble is offered. Only students that have legal rights to work in France can apply.

Anyone interested can send an email with the CV to Daniele Perrone (dperrone@prophesee.ai).

This Master’s project lies at the intersection of psychiatry and computer science and aims to use machine learning techniques to improve health. Using sensors to detect sleep and waking behavior has as of yet unexplored potential to reveal insights into health.  In this study, we make use of a watch-like device, called an actigraph, which tracks motion to quantify sleep behavior and waking activity. Participants in the study consist of healthy and depressed adolescents and wear actigraphs for a year during which time we query their mental health status monthly using online questionnaires.  For this masters thesis we aim to make use of machine learning methods to predict mental health based on the data from the actigraph. The ability to predict mental health crises based on sleep and wake behavior would provide an opportunity for intervention, significantly impacting the lives of patients and their families. This Masters thesis is a collaboration between Professor Paolo Favaro at the Institute of Computer Science (paolo.favaro@inf.unibe.ch) and Dr Leila Tarokh at the Universitäre Psychiatrische Dienste (UPD) (leila.tarokh@upd.unibe.ch).  We are looking for a highly motivated individual interested in bridging disciplines. 

The Gerontechnology and Rehabilitation group at the ARTORG Center for Biomedical Engineering is offering multiple BSc- and MSc thesis projects to students, which are interested in working with real patient data, artificial intelligence and machine learning algorithms. The goal of these projects is to transfer the findings to the clinic in order to solve today’s healthcare problems and thus to improve the quality of life of patients.

• Machine Learning Based Gait-Parameter Extraction by Using Simple Rangefinder Technology. [PDF]
• Detection of Motion in Video Recordings [PDF]
• Home-Monitoring of Elderly by Radar [PDF]
• Gait feature detection in Parkinson's Disease [PDF]
• Development of an arthroscopic training device using virtual reality [PDF]

Contact: Dr. Stephan Gerber (stephan.gerber@artorg.unibe.ch), Michael Single (michael.single@artorg.unibe.ch)

Visual Transformers have obtained state of the art classification accuracies [ViT, DeiT, T2T, BoTNet]. Mixture of experts could be used to increase the capacity of a neural network by learning instance dependent execution pathways in a network [MoE]. In this research project we aim to push the transformers to their limit and combine their dynamic attention with MoEs, compared to Switch Transformer [Switch], we will use a much more efficient formulation of mixing [CondConv, DynamicConv] and we will use this idea in the attention part of the transformer, not the fully connected layer.

Visual Transformers have obtained state of the art classification accuracies for 2d images[ViT, DeiT, T2T, BoTNet]. In this project, we aim to extend the same ideas to 3d data (videos), which requires a more efficient attention mechanism [Performer, Axial, Linformer]. In order to accelerate the training process, we could use [Multigrid] technique.

Goals:

• Better video understanding by attention blocks.

Publication Opportunity: LOVEU (a CVPR workshop), Holistic Video Understanding (a CVPR workshop), ActivityNet (a CVPR workshop)

Related Papers:

• Visual Transformers: Token-based Image Representation and Processing for Computer Vision [ViT]
• DeiT: Data-efficient Image Transformers [DeiT]
• Bottleneck Transformers for Visual Recognition [BoTNet]
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet [T2TViT]
• Rethinking Attention with Performers [Performer]
• Axial Attention in Multidimensional Transformers [Axial]
• Linformer: Self-Attention with Linear Complexity [Linformer]
• A Multigrid Method for Efficiently Training Video Models [Multigrid]

Contact: Sepehr Sameni

GIRAFFE is a newly introduced GAN that can generate scenes via composition with minimal supervision [GIRAFFE]. Generative methods can implicitly learn interpretable representation as can be seen in GAN image interpretations [GANSpace, GanLatentDiscovery]. Decoding GIRAFFE could give us per-object interpretable representations that could be used for scene manipulation, data augmentation, scene understanding, semantic segmentation, pose estimation [iNeRF], and more. 

In order to invert a GIRAFFE model, we will first train the generative model on Clevr and CompCars datasets, then we add a decoder to the pipeline and train this autoencoder. We can make the task easier by knowing the number of objects in the scene and/or knowing their positions. 

Goals:  

• Scene Manipulation and Decomposition by Inverting the GIRAFFE 

Publication Opportunity: DynaVis 2021 (a CVPR workshop on Dynamic Scene Reconstruction) 

Related Papers: 

• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields [GIRAFFE] 
• Neural Scene Graphs for Dynamic Scenes 
• pixelNeRF: Neural Radiance Fields from One or Few Images [pixelNeRF] 
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis [NeRF] 
• Neural Volume Rendering: NeRF And Beyond 
• GANSpace: Discovering Interpretable GAN Controls [GANSpace] 
• Unsupervised Discovery of Interpretable Directions in the GAN Latent Space [GanLatentDiscovery] 
• Inverting Neural Radiance Fields for Pose Estimation [iNeRF] 

Contact: Sepehr Sameni

Visual Transformers have obtained state of the art classification accuracies [ViT, CLIP, DeiT], but the best ViT models are extremely compute heavy and running them even only for inference (not doing backpropagation) is expensive. Running transformers cheaply by quantization is not a new problem and it has been tackled before for BERT [BERT] in NLP [Q-BERT, Q8BERT, TernaryBERT, BinaryBERT]. In this project we will be trying to quantize pretrained ViT models. 

Goals:  

• Quantizing ViT models for faster inference and smaller models without losing accuracy 

Publication Opportunity: Binary Networks for Computer Vision 2021 (a CVPR workshop) 

Extensions:  

• Having a fast pipeline for image inference with ViT will allow us to dig deep into the attention of ViT and analyze it, we might be able to prune some attention heads or replace them with static patterns (like local convolution or dilated patterns), We might be even able to replace the transformer with performer and increase the throughput even more [Performer]. 
• The same idea could be extended to other ViT based models [DETR, SETR, LSTR, TrackFormer, CPTR, BoTNet, T2TViT] 

Related Papers: 

• Learning Transferable Visual Models From Natural Language Supervision [CLIP] 
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision [ViT] 
• DeiT: Data-efficient Image Transformers [DeiT] 
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [BERT] 
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT [Q-BERT] 
• Q8BERT: Quantized 8Bit BERT [Q8BERT] 
• TernaryBERT: Distillation-aware Ultra-low Bit BERT [TernaryBERT] 
• BinaryBERT: Pushing the Limit of BERT Quantization [BinaryBERT] 
• Rethinking Attention with Performers [Performer] 
• End-to-End Object Detection with Transformers [DETR] 
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers [SETR] 
• End-to-end Lane Shape Prediction with Transformers [LSTR] 
• TrackFormer: Multi-Object Tracking with Transformers [TrackFormer] 
• CPTR: Full Transformer Network for Image Captioning [CPTR] 
• Bottleneck Transformers for Visual Recognition [BoTNet] 
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet [T2TViT] 

Contact: Sepehr Sameni

Recently contrastive learning has gained a lot of attention for self-supervised image representation learning [SimCLR, MoCo]. Contrastive learning could be extended to multimodal data, like videos (images and audio) [CMC, CoCLR]. Most contrastive methods require large batch sizes (or large memory pools) which makes them expensive for training. In this project we are going to use non batch size dependent contrastive methods [SwAV, BYOL, SimSiam] to train multimodal representation extractors. 

Our main goal is to compare the proposed method with the CMC baseline, so we will be working with STL10, ImageNet, UCF101, HMDB51, and NYU Depth-V2 datasets. 

Inspired by the recent works on smaller datasets [ConVIRT, CPD], to accelerate the training speed, we could start with two pretrained single-modal models and finetune them with the proposed method.  

Goals:  

• Extending SwAV to multimodal datasets 
• Grasping a better understanding of the BYOL 

Publication Opportunity: MULA 2021 (a CVPR workshop on Multimodal Learning and Applications) 

Extensions:  

• Most knowledge distillation methods for contrastive learners also use large batch sizes (or memory pools) [CRD, SEED], the proposed method could be extended for knowledge distillation. 
• One could easily extend this idea to multiview learning, for example one could have two different networks working on the same input and train them with contrastive learning, this may lead to better models [DeiT] by cross-model inductive biases communications. 

Related Papers: 

• Self-supervised Co-training for Video Representation Learning [CoCLR] 
• Learning Spatiotemporal Features via Video and Text Pair Discrimination [CPD] 
• Audio-Visual Instance Discrimination with Cross-Modal Agreement [AVID-CMA] 
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering [XDC] 
• Contrastive Multiview Coding [CPC] 
• Learning Transferable Visual Models From Natural Language Supervision [CLIP] 
• Contrastive Learning of Medical Visual Representations from Paired Images and Text [ConVIRT] 
• A Simple Framework for Contrastive Learning of Visual Representations [SimCLR] 
• Momentum Contrast for Unsupervised Visual Representation Learning [MoCo] 
• Bootstrap your own latent: A new approach to self-supervised Learning [BYOL] 
• Exploring Simple Siamese Representation Learning [SimSiam] 
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments [SwAV] 
• DeiT: Data-efficient Image Transformers [DeiT] 
• Contrastive Representation Distillation [CRD] 
• SEED: Self-supervised Distillation For Visual Representation [SEED] 

Contact: Sepehr Sameni

Neural Networks have been found to achieve surprising performance in several tasks such as classification, detection and segmentation. However, they are also very sensitive to small (controlled) changes to the input. It has been shown that some changes to an image that are not visible to the naked eye may lead the network to output an incorrect label. This thesis will focus on studying recent progress in this area and aim to build a procedure for a trained network to self-assess its reliability in classification or one of the popular computer vision tasks.

The Personalised Medicine Research Group at the sitem Center for Translational Medicine and Biomedical Entrepreneurship is offering multiple MSc thesis projects to the biomed eng MSc students that may also be of interest to the computer science students.

• Automated quantification of cartilage quality for hip treatment decision support. PDF
• Automated quantification of massive rotator cuff tears from MRI. PDF
• Deep learning-based segmentation and fat fraction analysis of the shoulder muscles using quantitative MRI. PDF
• Unsupervised Domain Adaption for Cross-Modality Hip Joint Segmentation. PDF

Contact: Dr. Kate Gerber

3-6 months internships on event-based computer vision. Chronocam is a rapidly growing startup developing event-based technology, with more than 15 PhDs working on problems like tracking, detection, classification, SLAM, etc. Event-based computer vision has the potential to solve many long-standing problems in traditional computer vision, and this is a super exciting time as this potential is becoming more and more tangible in many real-world applications. For next year we are looking for motivated Master and PhD students with good software engineering skills (C++ and/or python), and preferable good computer vision and deep learning background. PhD internships will be more research focused and possibly lead to a publication. 
For each intern we offer a compensation to cover the expenses of living in Paris. 
List of some of the topics we want to explore:

• Photo-realistic image synthesis and super-resolution from event-based data (PhD)
• Self-supervised representation learning (PhD)
• End-to-end Feature Learning for Event-based Data
• Bio-inspired Filtering using Spiking Networks
• On-the fly Compression of Event-based Streams for Low-Power IoT Cameras
• Tracking of Multiple Objects with a Dual-Frequency Tracker
• Event-based Autofocus
• Stabilizing an Event-based Stream using an IMU
• Crowd Monitoring for Low-power IoT Cameras
• Road Extraction from an Event-based Camera Mounted in a Car for Autonomous Driving
• Sign detection from an Event-based Camera Mounted in a Car for Autonomous Driving
• High-frequency Eye Tracking

Email with attached CV to Daniele Perrone at dperrone@chronocam.com.

Today we have many 3D scanning techniques that allow us to capture the shape and appearance of objects. It is easier than ever to scan real 3D objects and transform them into a digital model for further processing, such as modeling, rendering or animation. However, the output of a 3D scanner is often a raw point cloud with little to no annotations. The unstructured nature of the point cloud representation makes it difficult for processing, e.g. surface reconstruction. One application is the detection and segmentation of an object of interest. 
In this project, the student is challenged to design a system that takes a point cloud (a 3D scan) as input and outputs the names of objects contained in the scan. This output can then be used to eliminate outliers or points that belong to the background. The approach involves collecting a large dataset of 3D scans and training a neural network on it.

A photograph accurately captures the world in a moment of time and from a specific perspective. Since it is a projection of the 3D space to a 2D image plane, the depth information is lost. Is it possible to restore it, given only a single photograph? In general, the answer is no. This problem is ill-posed, meaning that many different plausible depth maps exist, and there is no way of telling which one is the correct one. 
However, if we cover one of our eyes, we are still able to recognize objects and estimate how far away they are. This motivates the exploration of an approach where prior knowledge can be leveraged to reduce the ill-posedness of the problem. Such a prior could be learned by a deep neural network, trained with many images and depth maps.

Deblurring finds many applications in our everyday life. It is particularly useful when taking pictures on handheld devices (e.g. smartphones) where camera shake can degrade important details. Therefore, it is desired to have a good deblurring algorithm implemented directly in the device. 
In this project, the student will implement and optimize a state-of-the-art deblurring method based on a deep neural network for deployment on mobile phones (Android). 
The goal is to reduce the number of network weights in order to reduce the memory footprint while preserving the quality of the deblurred images. The result will be a camera app that automatically deblurs the pictures, giving the user a choice of keeping the original or the deblurred image.

If an object in front of the camera or the camera itself moves while the aperture is open, the region of motion becomes blurred because the incoming light is accumulated in different positions across the sensor. If there is camera motion, there is also parallax. Thus, a motion blurred image contains depth information. 
In this project, the student will tackle the problem of recovering a depth-map from a motion-blurred image. This includes the collection of a large dataset of blurred- and sharp images or videos using a pair or triplet of GoPro action cameras. Two cameras will be used in stereo to estimate the depth map, and the third captures the blurred frames. This data is then used to train a convolutional neural network that will predict the depth map from the blurry image.

The idea of this project is that we have two types of neural networks that work together: There is one network A that assigns images to k clusters and k (simple) networks of type B perform a self-supervised task on those clusters. The goal of all the networks is to make the k networks of type B perform well on the task. The assumption is that clustering in semantically similar groups will help the networks of type B to perform well. This could be done on the MNIST dataset with B being linear classifiers and the task being rotation prediction.

The student designs a data augmentation network that transforms training images in such a way that image realism is preserved (e.g. with a constrained spatial transformer network) and the transformed images are more difficult to classify (trained via adversarial loss against an image classifier). The model will be evaluated for different data settings (especially in the low data regime), for example on the MNIST and CIFAR datasets.

People with sensory impairment (hearing, speech, vision) depend heavily on assistive technologies to communicate and navigate in everyday life. The mass production of media content today makes it impossible to manually translate everything into a common language for assistive technologies, e.g. captions or sign language. 
In this project, the student employs a neural network to learn a representation for lip-movement in videos in an unsupervised fashion, possibly with an encoder-decoder structure where the decoder reconstructs the audio signal. This requires collecting a large dataset of videos (e.g. from YouTube) of speakers or conversations where lip movement is visible. The outcome will be a neural network that learns an audio-visual representation of lip movement in videos, which can then be leveraged to generate captions for hearing impaired persons.

Satellite images have many applications, e.g. in meteorology, geography, education, cartography and warfare. They are an accurate and detailed depiction of the surface of the earth from above. Although it is relatively simple to collect many satellite images in an automated way, challenges arise when processing them for use in navigation and cartography.
The idea of this project is to automatically convert an arbitrary satellite image, of e.g. a city, to a map of simple 2D shapes (streets, houses, forests) and label them with colors (semantic segmentation). The student will collect a dataset of satellite image and topological maps and train a deep neural network that learns to map from one domain to the other. The data could be obtained from a Google Maps database or similar.