Peter Robicheaux 1† Matvei Popov1† Anish Madan 2 Isaac Robinson 1
Joseph Nelson 1 Deva Ramanan 2 Neehar Peri 2
Roboflow Carnegie Mellon University
† Equal Contribution
Introduced in the paper "Roboflow 100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models", RF100-VL is a large-scale collection of 100 multi-modal datasets with diverse concepts not commonly found in VLM pre-training.
The benchmark includes images, with corresponding annotations, from seven domains: flora and fauna, sport, industry, document processing, laboratory imaging, aerial imagery, and miscellaneous datasets related to various use cases for which detection models are commonly used.
You can use RF100-VL to benchmark fully supervised, semi-supervised and few-shot object detection models, and Vision Language Models (VLMs) with localization capabilities.
To download RF100-VL, first install the rf100vl pip package:
RF100-VL is hosted on Roboflow Universe, the world's largest repository of annotated computer vision dataset. You will need a free Roboflow Universe API key to download the dataset. Learn how to find your API key
Export your API key into an environment variable called ROBOFLOW_API_KEY:
export ROBOFLOW_API_KEY=YOUR_KEY
Several helper functions are available to download RF100-VL and its subsets. These are split up into two categories: functions that retrieve Dataset objects with the name of each project and its category. (that start with get_), and data downloaders (that start with download_).
get_rf100vl_fsod_projects RF100-VL-FSOD get_rf100vl_projects RF100-VL get_rf20vl_fsod_projects RF20-VL-FSOD get_rf20vl_full_projects RF20-VL download_rf100vl_fsod RF100-VL-FSOD download_rf100vl RF100-VL download_rf20vl_fsod RF20-VL-FSOD download_rf20vl_full RF20-VL
Each dataset object has its own download method.
Here is an example showing how to download the full dataset:
from rf100vl import download_rf100vl download_rf100vl(path="./rf100-vl/")
The datasets will be downloaded in COCO JSON format to a directory called rf100-vl. Every dataset will be in its own sub-folder.
Organized by: Anish Madan, Neehar Peri, Deva Ramanan
This challenge focuses on few-shot object detection (FSOD) with 10 examples of each class provided by a human annotator. Existing FSOD benchmarks repurpose well-established datasets like COCO by partitioning categories into base and novel classes for pre-training and fine-tuning respectively. However, these benchmarks do not reflect how FSOD is deployed in practice.
Rather than pre-training on only a small number of base categories, we argue that it is more practical to download a foundational model (e.g., a vision-language model (VLM) pretrained on web-scale data) and fine-tune it for specific applications. We propose a new FSOD benchmark protocol that evaluates detectors pre-trained on any external dataset (not including the target dataset), and fine-tuned on K-shot annotations per C target classes.
We evaluate a subset of 20 datasets from Roboflow-VL. Each dataset is independently evaluated using AP. Roboflow-VL includes datasets that are out-of-distribution from typical internet-scale pre-training data, making it a particularly challenging (even for VLMs) for Foundational FSOD.
🚨 Top performing teams can win cash prizes! 🚨
🥇 1st Place: $750
🥈 2nd Place: $500
🥉 3rd Place: $250
To be eligible for prizes, teams must submit a technical report, open source their code, and provide instructions on how to reproduce their results. Teams must also beat our best performing official baseline to be eligible for prizes. Many thanks to Roboflow for sponsoring prizes!
Goal: Developing robust object detectors using few annotations provided by annotator instructions. The detector should detect object instances of interest in real-world testing images.
Environment for model development:
Evaluation metrics:
Submit a zip file with pickle files for each dataset. The name of each pickle file should match the name of each dataset. Each pickle file should use the following COCO format.
[
"image_id": int,
"instances":
[{ "image_id": int,
"category_id": int,
"bbox": [x,y,width,height],
"score": float
},
{"image_id": int,
"category_id": int,
"bbox": [x,y,width,height],
"score": float }, ... ,],
...,
]
We've provided a sample submission for your reference. Submissions should be uploaded to our EvalAI server.
We pre-train Detic on ImageNet21-K, COCO Captions, and LVIS. We evaluate this pre-trained model zero-shot on the datasets in RF20-VL.
Our baseline code is available here.
This work was supported in part by compute provided by NVIDIA, and the NSF GRFP (Grant No. DGE2140739).
The datasets that comprise RF100-VL are licensed under an Apache 2.0 license.
If you find our paper and code repository useful, please cite us:
@article{robicheaux2025roboflow100vl,
title={Roboflow100-vl: A multi-domain object detection benchmark for vision-language models},
author={Robicheaux, Peter and Popov, Matvei and Madan, Anish and Robinson, Isaac and Nelson, Joseph and Ramanan, Deva and Peri, Neehar},
journal={arXiv preprint arXiv:2505.20612},
year={2025}
}
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4