ImageCLEF 2006 - Object Annotation Task
formerly known as non-medical automatic annotation task
The ImageCLEF 2006 Object Annotation Task is part of
the Cross
Language Evaluation Forum (CLEF), a benchmarking event for
multilingual information retrieval held annually since 2000. CLEF
first began as a track in the Text Retrieval Conference (TREC,
trec.nist.gov). This task was added to the ImageCLEF campaing
in 2006.
Retrieval tasks
In ImageCLEF 2006, there are two automatic image annotation
tasks. The medical and
the object annotation task. These tasks will not contain any text
as input for the task and is aimed at image analysis research
groups.
On request, we will try to make results of GIFT
and FIRE available to
participants without access to an own CBIR system.
This page is concerned with the Object Annotation Task
You might also be interested in
Automatic image annotation
Automatic image annotation or image classification can be an
important step when searching for images from a database. We
will provide an image database containing images of 21 different
objects. These images are provided with given classes. That is,
it is given which object is shown on the image. In addition, a
set of 1000 unlabelled images will be given as test data that
have to be labelled or classified.
Database & Download
Training data
To download the data you have to sign this form and fax it to LTUTech.
Please follow the following procedure to obtain authorization to use the LTUtech Object Dataset:
- Complete and sign the agreement.
- Fax it to Dr. Chahab Nastar, LTU Technologies SA at fax N°: +33 (0)1 53 43 01 69.
Drop the (0) if sending from outside of France. Include a cover page with your clear
name, organization, e-mail address and fax number. To avoid the fax going missing,
make sure that the fax is clearly addressed to Dr. Chahab Nastar.
- The agreement will be signed and faxed back to you.
- Fax this signed agreement to Dr. Allan Hanbury at fax N°: +43 1 58801 18392.
Include a cover page with your clear name, organization, e-mail address and fax
number. You will then receive by e-mail the instructions to download the dataset.
The form is available as word-file and PDF-file.
Relevant images
The training data consists of very many classes. But only 21 of
them will be used for the evaluation. A list of
relevant images is now available here
Relevant classes are:
- ashtrays
- backpacks
- balls
- banknotes
- benches
- books
- bottles
- calculators
- cans
- chairs
- clocks
- coins
- computer_equipment
- cups_mugs
- hifi_equipment
- knives_forks_spoons
- mobilephones
- plates
- sofas
- tables
- wallets
Development data
To give an impression of the coming test data, we supply a set of
100 images from our test images that can be considered as
development test data. That is, use these data to optimize your
system.
Download: 100 images development set (9.8 MB)
Test data
Test data is available here. The archive contains 1000 images,
each showing exactly one of the 21 object categories.
Download: 1000 images test data set (369 MB)
Submission of Results
The submission
website is now online: http://www-i6.informatik.rwth-aachen.de/~deselaers/imageclef06/medaat-submission/ic06submit.py.
In case you experience problems with submission over this site,
please contact me.
You have to specify the following information
- Contact person
- Name, Email, Contact address
- Group
- Group identifier (for presentation of results)
- Complete name of the group
- Address
- Run
- Identifier (for presentation of results)
- description (200-500 words)
- file with classification results following the format explained below
- do you consider this run your primary run?
Results have to be submitted by June 16, 2006.
Submission format
The submission format is the same that is used for the medical automatic
annotation task. With classes being ordered
alphabetically. That is: (class. classname in the following)
- ashtrays
- backpacks
- balls
- banknotes
- benches
- books
- bottles
- calculators
- cans
- chairs
- clocks
- coins
- computer_equipment
- cups_mugs
- hifi_equipment
- knives_forks_spoons
- mobilephones
- plates
- sofas
- tables
- wallets
A script to check a
submission file is now available and the required list of filenames.
Together, these can be used to check whether you have a
valid submission. If you have a valid submission a run of this
program will look like this:
# ./check_submission.python -c filenames [submissionfile]
conffile=../MYSUBMISSIONS/idmsubmission classfile=filenames
2034 is classified as X
2229 is classified as Y
2630 is classified as Z
....
373446 is classified as XX
373447 is classified as YY
373450 is classified as ZZ
classified: 1000 wrong: 0 correct: 0 illegal: 0 missing: 0
If it looks differently, e.g. there is something as illegal or
missing you should check your submission file.
Example
To classify image 123.png as class balls you need a line of this type:
123.png 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Questions & Comments
If you have any questions or comments on these information, feel
free to contact us:
Allan Hanbury and
or Thomas
Deselaers.
Results
The results of
the object annotation task are now available here together with the groundtruth of the test data
Thomas Deselaers
Last modified: Thu Aug 3 16:21:25 CEST 2006
;