ImageCLEF 2007 - Object Retrieval Task

The ImageCLEF 2007 Object retrieval Task is part of the Cross Language Evaluation Forum (CLEF), a benchmarking event for multilingual information retrieval held annually since 2000. CLEF first began as a track in the Text Retrieval Conference (TREC,

Retrieval task

In ImageCLEF 2007 there are two tasks dealing with the IAPR TC12 database. Both tasks will likely require the use of image retrieval techniques for best results. The object retrieval task will not contain any text as input for the task and is aimed at image analysis research groups. On request, we will try to make results of GIFT and FIRE available to participants without access to an own CBIR system. This page is concerned with the Object Retrieval Task.

You might also be interested in

Object Retrieval Task

The aim of this task is to test how well object recognition techniques perform in an image retrieval scenario. The retrieval results should be based only on visual object recognition algorithms, text search should not be used.

Database & Download

For the training data, we will use the train+val images from the PASCAL Visual Object classes challenge 2006 (part of the PASCAL Network of Excellence). The data can be downloaded by following this download link. Please read the Database Rights before downloading this data.

We will use the IAPR-TC12 data for test data. This data is available for download. The login information is available from Carol Peters after registration for CLEF/ImageCLEF 2007.



2007-07-23: Results are available now!

Submission of Results

For the submission of the results we will provide a web interface and you will have to specify the following information:
To clarify:

training data: PASCAL 2006 VOC training and val data

test data: IAPR TC-12 2007 data

Submission file according to trec_eval format described e.g. here:

The queries are:
Query 1: bicycle
Query 2: bus
Query 3: car
Query 4: motorbike
Query 5: cat
Query 6: cow
Query 7: dog
Query 8: horse
Query 9: sheep
Query 10: person    

So, you end up with a file of up to 10000 lines!

Submissions can be uploaded at


Use the PASCAL VOC 2006 train+val data to train classifiers to recognise the 10 objects (DO NOT use the PASCAL VOC test dataset for training purposes). Use these classifiers to retrieve images corresponding to the ten classes (queries) from the IAPR-TC12 dataset.

There are 10 queries, corresponding to the 10 classes in the PASCAL VOC 2006 training data. They should be numbered in the "topic number" field of the result submission as follows:


Submission format

Participants are required to submit ranked lists of (up to) the top 1000 images ranked in descending order of similarity (i.e. the highest nearer the top of the list). The format of submissions for this task can be found here and the filenames should distinguish different types of submission. Participants can submit as many system runs as they require.

Please note that there should be at least 1 document entry in your results for each topic (i.e. if your system returns no results for a query then insert a dummy entry, e.g. 25 1 annotations/16/16019 0 4238 xyzT10af5 ). The reason for this is to make sure that all systems are compared with the same number of topics and relevant documents. Submissions not following the required format will not be evaluated.

Optional but useful: We also request that you calculate the ROC curve and the AUC measure of your classifier on the PASCAL test dataset. This will allow us to make a useful correlation between object recognition and object retrieval performance. Code for calculating these measures is provided on the PASCAL VOC 2006 webpage and documented here.

Questions & Comments

If you have any questions or comments on these information, feel free to contact us. Questions or comments can be addressed to Allan Hanbury and Thomas Deselaers.
Thomas Deselaers
Last modified: Mon Jul 23 17:45:46 CEST 2007

Valid HTML 4.01! xemacs logo debian logo