The aim of this task is to test how well object recognition techniques perform in an image retrieval scenario. The retrieval results should be based only on visual object recognition algorithms, text search should not be used.
For the training data, we will use the train+val images from the PASCAL Visual Object classes challenge 2006 (part of the PASCAL Network of Excellence). The data can be downloaded by following this download link. Please read the Database Rights before downloading this data.
We will use the IAPR-TC12 data for test data. This data is available for download. The login information is available from Carol Peters after registration for CLEF/ImageCLEF 2007.
To clarify: training data: PASCAL 2006 VOC training and val data http://www.pascal-network.org/challenges/VOC/voc2006/index.html -> http://www.pascal-network.org/challenges/VOC/download/voc2006_trainval.tar test data: IAPR TC-12 2007 data -> http://eureka.vu.edu.au/~grubinger/ImageCLEFphoto2007/adhoc.htm Submission file according to trec_eval format described e.g. here: http://eureka.vu.edu.au/~grubinger/ImageCLEFphoto2007/adhoc_submission.txt The queries are: Query 1: bicycle Query 2: bus Query 3: car Query 4: motorbike Query 5: cat Query 6: cow Query 7: dog Query 8: horse Query 9: sheep Query 10: person So, you end up with a file of up to 10000 lines! Submissions can be uploaded at http://www-i6.informatik.rwth-aachen.de/~deselaers/imageclef07/objretsub/
Use the PASCAL VOC 2006 train+val data to train classifiers to recognise the 10 objects (DO NOT use the PASCAL VOC test dataset for training purposes). Use these classifiers to retrieve images corresponding to the ten classes (queries) from the IAPR-TC12 dataset.
There are 10 queries, corresponding to the 10 classes in the PASCAL VOC 2006 training data. They should be numbered in the "topic number" field of the result submission as follows:
| 1 | bicycle |
| 2 | bus |
| 3 | car |
| 4 | motorbike |
| 5 | cat |
| 6 | cow |
| 7 | dog |
| 8 | horse |
| 9 | sheep |
| 10 | person |
Participants are required to submit ranked lists of (up to) the top 1000 images ranked in descending order of similarity (i.e. the highest nearer the top of the list). The format of submissions for this task can be found here and the filenames should distinguish different types of submission. Participants can submit as many system runs as they require.
Please note that there should be at least 1 document entry in your results for each topic (i.e. if your system returns no results for a query then insert a dummy entry, e.g. 25 1 annotations/16/16019 0 4238 xyzT10af5 ). The reason for this is to make sure that all systems are compared with the same number of topics and relevant documents. Submissions not following the required format will not be evaluated.
Optional but useful: We also request that you calculate the ROC curve and the AUC measure of your classifier on the PASCAL test dataset. This will allow us to make a useful correlation between object recognition and object retrieval performance. Code for calculating these measures is provided on the PASCAL VOC 2006 webpage and documented here.