ImageCLEF 2006 - Object Annotation Task

formerly known as non-medical automatic annotation task
The ImageCLEF 2006 Object Annotation Task is part of the Cross Language Evaluation Forum (CLEF), a benchmarking event for multilingual information retrieval held annually since 2000. CLEF first began as a track in the Text Retrieval Conference (TREC, This task was added to the ImageCLEF campaing in 2006.

Retrieval tasks

In ImageCLEF 2006, there are two automatic image annotation tasks. The medical and the object annotation task. These tasks will not contain any text as input for the task and is aimed at image analysis research groups. On request, we will try to make results of GIFT and FIRE available to participants without access to an own CBIR system. This page is concerned with the Object Annotation Task

You might also be interested in

Automatic image annotation

Automatic image annotation or image classification can be an important step when searching for images from a database. We will provide an image database containing images of 21 different objects. These images are provided with given classes. That is, it is given which object is shown on the image. In addition, a set of 1000 unlabelled images will be given as test data that have to be labelled or classified.

Database & Download

Training data

To download the data you have to sign this form and fax it to LTUTech. Please follow the following procedure to obtain authorization to use the LTUtech Object Dataset:
  1. Complete and sign the agreement.
  2. Fax it to Dr. Chahab Nastar, LTU Technologies SA at fax N: +33 (0)1 53 43 01 69. Drop the (0) if sending from outside of France. Include a cover page with your clear name, organization, e-mail address and fax number. To avoid the fax going missing, make sure that the fax is clearly addressed to Dr. Chahab Nastar.
  3. The agreement will be signed and faxed back to you.
  4. Fax this signed agreement to Dr. Allan Hanbury at fax N: +43 1 58801 18392. Include a cover page with your clear name, organization, e-mail address and fax number. You will then receive by e-mail the instructions to download the dataset.
The form is available as word-file and PDF-file.

Relevant images

The training data consists of very many classes. But only 21 of them will be used for the evaluation. A list of relevant images is now available here

Relevant classes are:

Development data

To give an impression of the coming test data, we supply a set of 100 images from our test images that can be considered as development test data. That is, use these data to optimize your system.

Download: 100 images development set (9.8 MB)

Test data

Test data is available here. The archive contains 1000 images, each showing exactly one of the 21 object categories.

Download: 1000 images test data set (369 MB)

Submission of Results

The submission website is now online:
In case you experience problems with submission over this site, please contact me.
You have to specify the following information

Results have to be submitted by June 16, 2006.

Submission format

The submission format is the same that is used for the medical automatic annotation task. With classes being ordered alphabetically. That is: (class. classname in the following)
  1. ashtrays
  2. backpacks
  3. balls
  4. banknotes
  5. benches
  6. books
  7. bottles
  8. calculators
  9. cans
  10. chairs
  11. clocks
  12. coins
  13. computer_equipment
  14. cups_mugs
  15. hifi_equipment
  16. knives_forks_spoons
  17. mobilephones
  18. plates
  19. sofas
  20. tables
  21. wallets
A script to check a submission file is now available and the required list of filenames.

Together, these can be used to check whether you have a valid submission. If you have a valid submission a run of this program will look like this:

# ./check_submission.python -c filenames [submissionfile]
conffile=../MYSUBMISSIONS/idmsubmission classfile=filenames
2034 is classified as X
2229 is classified as Y
2630 is classified as Z
373446 is classified as XX
373447 is classified as YY
373450 is classified as ZZ
classified: 1000 wrong: 0 correct: 0 illegal: 0 missing: 0
If it looks differently, e.g. there is something as illegal or missing you should check your submission file.


To classify image 123.png as class balls you need a line of this type:
123.png 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 

Questions & Comments

If you have any questions or comments on these information, feel free to contact us: Allan Hanbury and or Thomas Deselaers.


The results of the object annotation task are now available here together with the groundtruth of the test data
Thomas Deselaers
Last modified: Thu Aug 3 16:21:25 CEST 2006

Valid HTML 4.01! xemacs logo debian logo ;