Self Supervised[Vit] vs. Supervised[CNN]Discussion

[u/Just-m_d]

Hi everyone,

I am working in the Multi-Task ML projects in that i have done many experiments to train the model
Key finding:

CNN based architecture outperform the Vision transformer based architecture

Question:
is the Dataset only be the problem for Vit architecture to underperform the CNN bases architecture?

[u/OneOrZeroGG]

Short answer: no, the dataset is only one of several possible reasons. Some of the usual suspects:

1. Inductive bias. CNNs have locality and translation equivariance built in, so they learn good features from less data. A ViT has to learn that from scratch, which is why it usually needs much more data (or a pretrained checkpoint) to catch up. On small or medium datasets a CNN winning is the normal result, not a bug.

2. Training recipe. ViTs are much more sensitive than CNNs. AdamW, learning rate warmup, a cosine schedule, weight decay, and strong augmentation (RandAugment, mixup, CutMix) matter a lot. If you used the same recipe for both, the CNN probably benefited from it more.

3. Your comparison changes two things at once. You said Self-Supervised ViT vs Supervised CNN, so it's not just architecture, it's also the training paradigm. A few things to check there:

   - Are you evaluating with a linear probe or full fine-tuning? Some SSL methods (MAE for example) look weak under a linear probe and much better after fine-tuning.

   - How big was the pretraining set, and how long did you pretrain? SSL ViTs need a lot of both.

   - If possible, also run a supervised ViT and an SSL CNN, so you can tell whether the gap comes from the architecture or from the SSL.

4. Task type. In multi-task setups, dense or localization-heavy tasks often favor CNNs, and ViT patch size/resolution can hurt fine details. Also check how the task losses are weighted, since one task can dominate the shared backbone.

So I'd say the dataset is probably part of it, but I wouldn't conclude that before checking the recipe and the fair-comparison points above. What dataset size, model sizes and tasks are you using? That would help narrow it down.

[u/Just-m_d → u/OneOrZeroGG]

First of all thank u for ur time u gave me the deeper reason and thanks for the question I used only 15k images to train the Vit based approach specific a JEPA based architecture

The domain problem which I am solving is not the general nature problem the images differ from the natural image net dataset also

[u/OneOrZeroGG → u/Just-m_d]

Thanks for the details, that helps a lot. 15k images is the main thing here.

I-JEPA-style models are normally pretrained on datasets like ImageNet-1k (1.28M images) or bigger. With 15k images a ViT has too little data to learn good general features from scratch with a self-supervised objective, and a supervised CNN gets to use your labels directly and has the locality bias built in. So the CNN winning at this size is expected, not a sign that you did something wrong.

Some things I'd try:

1. Don't pretrain the ViT from scratch. Start from a public pretrained checkpoint (I-JEPA, DINOv2 or similar), then continue the self-supervised pretraining on your 15k images for a while (domain adaptation), then fine-tune on your tasks. A different domain from ImageNet is actually a good reason to do the continued pretraining step.

2. Make the baseline fair: also fine-tune an ImageNet-pretrained CNN (ResNet or ConvNeXt). If your CNN is trained from scratch and still wins, the gap is even more about data size.

3. Use layer-wise learning rate decay and a low LR for the backbone when fine-tuning the ViT, with a higher LR for the heads. ViTs are sensitive to this.

4. Check for collapse. JEPA-type models can collapse to near-constant embeddings without it being obvious from the loss. Look at the variance or effective rank of the embeddings across the dataset.

5. Think about the masking. The default block masking assumes objects of a certain size and layout. If your images are very different (e.g. medical, satellite, microscopy), the mask scale and patch size may not fit.

What is the domain, and are the tasks classification or something more dense like segmentation? That changes the advice quite a bit.

[u/Just-m_d → u/OneOrZeroGG]

Thanks for the detailed explanation and sorry for the late reply Objective is the multi task classification with the medical images

[u/OneOrZeroGG → u/Just-m_d]

Thanks, that helps. Medical images change the advice a bit:

1. Look for a checkpoint from your modality instead of plain ImageNet. For example RAD-DINO for chest X-rays, BiomedCLIP for general biomedical images, UNI or similar for histopathology. Fine-tuning one of these usually beats training a JEPA from scratch on 15k images. Which modality is it (X-ray, CT, MRI, histology...)?
   
2. Check for patient-level leakage. If the same patient has several images, they must all be in the same split (train, val or test). Otherwise your numbers will look better than they really are, for both models.
   
3. Be careful with augmentations. Flips, rotations and color jitter can change the meaning of medical images (left vs right, intensity values). Use ones that make sense for your modality.
   
4. Resolution and patch size. Small lesions can disappear with 16x16 patches at low resolution. A CNN handles that more naturally, which is another reason it may win here.
   
5. For multi-task, check the loss weighting (uncertainty weighting or GradNorm are common) and class imbalance. Report per-task metrics like AUC or F1 instead of only accuracy.

I'd still run the same ImageNet-pretrained CNN baseline next to the ViT, so you know whether the gap comes from the architecture or from the pretraining.

[u/Just-m_d → u/OneOrZeroGG]

The data are the endo scope images Do u aware of the any of these domain specific pretrained model or the dataset? My moto is to train the well represented encoder that can be used for any medical domain downstream tasks.

[u/OneOrZeroGG → u/Just-m_d]

Endoscopy has some domain-specific options, though I'd check the licenses and availability yourself since this field moves fast:

- Pretrained models: EndoViT (a ViT pretrained with MAE on a collection of public endoscopy images, mostly surgical/laparoscopic) and Endo-FM (a video foundation model for endoscopy). It depends whether your data is GI endoscopy or surgical, so check which one matches.

• Public datasets for pretraining or evaluation: HyperKvasir and Kvasir (GI tract, ~100k images in HyperKvasir), plus the datasets collected in the EndoViT paper (Endo700k). Pooling public data like this with your own 15k images gives you far more than 15k to pretrain on.

About your goal of one encoder for any medical downstream task: that's a much bigger target than 15k images from one modality can reach. A more realistic path is:
1. Start from a strong checkpoint (DINOv2 or one of the endoscopy models above).
2. Continue the self-supervised pretraining on a pooled set of public endoscopy data plus yours.
3. Evaluate the frozen encoder on several different downstream tasks with kNN or linear probe, and compare against an ImageNet-pretrained CNN. That tells you whether the encoder is really more general.

With only one modality in the training data, I'd describe the result as an endoscopy encoder rather than a general medical one.

[u/Just-m_d → u/OneOrZeroGG]

And I have another thing to say Semantic learning in Vit is better than the just learning local structure in CNN

[u/OneOrZeroGG → u/Just-m_d]

True in some settings, ViTs can learn more global, semantic features, but usually only with a lot of data or a strong pretrained checkpoint. With 15k images from a non-ImageNet domain, that advantage often doesn't show up, so a CNN can still win.

Easy way to check: compare kNN or linear-probe accuracy on frozen features from both models. And what kind of images are they, and is the task classification?