I’m seeking help in reproducing the zero-shot ImageNet-1k results for SigLIP-2 models.
While my evaluation setup reproduces SigLIP v1 results almost perfectly, it consistently fails for SigLIP-2 models with a significant accuracy drop.
Summary of Results
Here is a summary of the performance I’m observing compared to the reported numbers in the paper:
| Model Version | Model Size | Paper Accuracy | My Accuracy | Difference |
|---|---|---|---|---|
| google/siglip-base-patch16-256 | Base | 76.7% | 76.6% | -0.1% |
| google/siglip-large-patch16-256 | Large | 80.5% | 80.4% | -0.1% |
| google/siglip2-base-patch16-256 | Base | 79.1% | 70.6% | -8.5% |
| google/siglip2-large-patch16-256 | Large | 82.5% | 73.2% | -9.3% |
As the table shows, the v1 models are working as expected. However, both v2 models suffer from a consistent and large drop of ~9%. This suggests the issue is not with my general framework but with a v2-specific detail I might be missing.
This problem has also been reported by other users: *(Link: <[ google/siglip2-base-patch16-224 · The accuracy on the ImageNet dataset is low ]>)*
My Evaluation Logic
My code follows a standard zero-shot approach with 80-template ensembling.
- Load model : I use
AutoModel.from_pretrained(model_path) - Image Processing : For all image transformations, I use
processor.image_processorfrom the loadedAutoProcessor. - Text classifier generation: Here is the function I use to generate the text classifier weights (slightly modified from open_clip’s zero-shot evaluation code):
python
import torch
from tqdm.auto import tqdm
from functools import partial
from itertools import batched
# Assuming OPENAI_IMAGENET_TEMPLATES and IMAGENET_CLASSNAMES are predefined
def build_text_classifier(model, tokenizer, device, num_classes_per_batch=10):
num_templates = len(OPENAI_IMAGENET_TEMPLATES)
num_classes = len(IMAGENET_CLASSNAMES)
num_iter = (num_classes - 1) // num_classes_per_batch + 1
iter_wrap = partial(tqdm, total=num_iter, unit_scale=num_classes_per_batch, desc=“Building text classifier”)
def _process_batch(batch_classnames):
num_batch_classes = len(batch_classnames)
texts = [template.format(c) for c in batch_classnames for template in OPENAI_IMAGENET_TEMPLATES]
inputs = tokenizer(texts, padding=“max_length”, max_length=64, return_tensors=“pt”).to(device)
class_embeddings = model.get_text_features(**inputs)
class_embeddings = class_embeddings.reshape(num_batch_classes, num_templates, -1).mean(dim=1)
class_embeddings = class_embeddings / class_embeddings.norm(dim=1, keepdim=True)
return class_embeddings.T # Returns (embed_dim, num_batch_classes)
with torch.no_grad():
batched_embeds = [_process_batch(batch) for batch in iter_wrap(batched(IMAGENET_CLASSNAMES, num_classes_per_batch))]
zeroshot_weights = torch.cat(batched_embeds, dim=1) # Final shape: (embed_dim, num_classes)
return zeroshot_weights
Image features are also L2-normalized. I computed the image features by using
vision_output = model.vision_model(images)
image_features = vision_output.pooler_output
image_features = image_features / image_features.norm(p=2, dim=-1, keepdim=True)
The final logits are computed via logits = image_features @ zeroshot_weights.
My Question
Given the consistent ~9% performance drop only for v2 models, I suspect I’m handling something incorrectly. Could you please help me identify if I’m missing a step required for SigLIP-2?
Specifically, I’m wondering about:
logit_scale/logit_bias: Is there a specific way these must be applied for v2 models during inference, which is different from v1?- Model Architecture: Does the v2 architecture, despite inheriting from v1, require passing through an extra projection layer or a different method to extract the final embeddings?
- My Code: Is there a subtle bug in my
build_text_classifierfunction that would affect only v2 models?
Any guidance on the correct procedure to evaluate SigLIP-2, or any pointers on what I might be doing wrong, would be greatly appreciated.
Thank you!
Environment:
- transformers: 4.54.1
- torch: 2.7.0+cu128