Visual Prompt Tuning survey for Remote Sensing Domain

Basic Information

Project Title / Domain: Visual Prompt Tuning survey for Remote Sensing Domain
Name: Savaliya Abhishek
Guide: Subhasis Chaudhuri
Project Type: implementation-driven

Short Description of Project

It is a Deep Learning project where we are exploring various Visual Prompt Tuning (VPT) approaches and figuring out what are the best VPT variants for the remote sensing domain classification problem. We are experimenting with various approaches (like using fourier transforms in prompts, dynamic prunings of negative prompts and federated learning) to figure out best accuracy models for the remote sensing benchmark.

Whom did you work with?

PG students, The prof directly, Peers (Other UG students)

Tools / Simulation / Software / Hardware

We worked primarily on IITB’s GPUs. All implementations are done on insti’s remote servers which are connected with GPUs.

Expectations from Guide

My expectations were mainly that I wanted a flexible work schedule and research in an interesting topic in the field of Deep learning. My both expectations were met.

Expectations for 8th Sem & Summer

Prof allowed me for my internship. Expectation for 8th semester was to finish all literature review and to create the initial pipeline for training vision transformer models. I did not work in summers and continued work after 9th sem started.

Load: 8th Sem vs 9th (Placement) Sem

Less load during 9th sem. I had already finished literature survey and created pipeline for training scripts in 8th sem itself. So in 9th sem, I was only putting models for training on GPU. So my 9th sem load was almost half compared to 8th sem.

Summer on Campus

not much productive, since I was working on remote servers, I could also work from home.

Is DDP Guide Same as SRE/RnD Guide?

Yes

How did your SRE help with your DDP?

My SRE was also in Deep Learning so it gave me a very good experience about working with GPUs and Vision Transformers. Both these skills helped me greatly to set up pipeline quickly for training models in 8th sem.

End Deliverables

A survey paper on VPT performance over remote sensing benchmarks.

Advice

Prof is flexible to work with but strict in grading. Moreover, IITB GPUs are usually very much busy so you might have to struggle with securing good compute power for your project. I myself started with very less compute power and kept getting better GPU accesses as I showcased better progress. You will have to coordinate with other students who will share the same GPU. Vision transformers are very interesting topic to explore in DL but very resource constrained as well. Most of your struggles in the project will be regarding optimizing GPU usage.