ImageCLEF 2007 - Object Retrieval Task: results

The ImageCLEF 2007 Object retrieval Task is part of the Cross Language Evaluation Forum (CLEF), a benchmarking event for multilingual information retrieval held annually since 2000. CLEF first began as a track in the Text Retrieval Conference (TREC, trec.nist.gov).

Annotation

For the annotation, we created pools of hypothesized relevant images. All these images were shown to judges asking to tell whether the hypothesized object is contained or not. Additionally to answering these questions, the judges had the possibility to give information on which other types of objects are contained in the images.

Statistics of the annotation

query	query name	images judged	images relevant	additional relevan images
1	bicycle	1437	66	254
2	bus	1449	23	69
3	car	1578	200	522
4	motorbike	1478	7	28
5	cat	1502	5	18
6	cow	1504	7	23
7	dog	1488	9	22
8	horse	1513	13	94
9	sheep	1693	5	42
10	person	1435	554	3939

Results

In the following we give the results as mean average precision over all queries:

2007-07-25: The result tables were updated because HUT submitted there initial runs with a wrong order of queries. The original results are available from the source code of this webpage in comments.

2007-08-20: All result tables and the qrels on this page were updated because we figured out that we had a severe error in the qrel-creation script.

Using the normal qrels

rank	run id	MAP
1	clefrun.HUTCIS_SVM_FULLIMG_ALL.relabeled	0.0742
2	clefrun.HUTCIS_SVM_FULLIMG_IP+SC.relabeled	0.0610
3	clefrun.HUTCIS_SVM_FULLIMG+BB.relabeled	0.0505
4	clefrun.HUTCIS_SVM_FULLIMG_IP.relabeled	0.0413
5	MSRA-MSRA_RuiSp	0.0336
6	clefrun.HUTCIS_SVM_BB_BAL_IP+SC.relabeled	0.0209
7	clefrun.HUTCIS_PICSOM1.relabeled	0.0193
8	clefrun.HUTCIS_PICSOM2.relabeled	0.0178
9	clefrun.HUTCIS_SVM_BB_ALL.relabeled	0.0161
10	clefrun.HUTCIS_SVM_BB_BB_IP+SC.relabeled	0.0151
11	clefrun.HUTCIS_SVM_BB_FULL_IP+SC.relabeled	0.0150
12	RWTHi6-HISTO-PASCAL	0.0150
13	clefrun.HUTCIS_SVM_BB_BB_IP.relabeled	0.0087
14	MSRA-MSRA-VLM_8_8_640_ful	0.0080
15	clefrun.HUTCIS_SVM_BB_BAL_IP.relabeled	0.0078
16	MSRA-MSRA-VLM-8-8-800-HT	0.0065
17	PRIP-PRIP_HSI_ScIvHarris	0.0061
18	budapest-acad-budapest-acad314	0.0043
19	clefrun.HUTCIS_SVM_BB_FULL_IP.relabeled	0.0041
20	NTU_SCE_HOI-NTU_SCE_HOI_1	0.0041
21	PRIP-PRIP_cbOCS_ScIvHarr2	0.0035
22	budapest-acad-budapest-acad315	0.0028
23	INAOE-TIA-INAOE_SSAssemble	0.0017
24	INAOE-TIA-INAOE-RB-KNN+MRFI_ok	0.0006
25	INAOE-TIA-INAOE-RB-KNN+MRFI	0.0006
26	INAOE-TIA-INAOE-RB-KNN	0.0006

Using the qrels with additional information

rank	run id	MAP
1	clefrun.HUTCIS_SVM_FULLIMG_ALL.relabeled	0.0369
2	clefrun.HUTCIS_SVM_FULLIMG_IP+SC.relabeled	0.0331
3	clefrun.HUTCIS_SVM_FULLIMG_IP.relabeled	0.0291
4	clefrun.HUTCIS_SVM_FULLIMG+BB.relabeled	0.0235
5	clefrun.HUTCIS_PICSOM1.relabeled	0.0167
6	clefrun.HUTCIS_PICSOM2.relabeled	0.0156
7	MSRA-MSRA_RuiSp	0.0147
8	clefrun.HUTCIS_SVM_BB_BAL_IP+SC.relabeled	0.0118
9	clefrun.HUTCIS_SVM_BB_BB_IP+SC.relabeled	0.0113
10	clefrun.HUTCIS_SVM_BB_BB_IP.relabeled	0.0111
11	clefrun.HUTCIS_SVM_BB_ALL.relabeled	0.0107
12	budapest-acad-budapest-acad314	0.0106
13	clefrun.HUTCIS_SVM_BB_FULL_IP+SC.relabeled	0.0105
14	RWTHi6-HISTO-PASCAL	0.0083
15	clefrun.HUTCIS_SVM_BB_BAL_IP.relabeled	0.0078
16	budapest-acad-budapest-acad315	0.0069
17	clefrun.HUTCIS_SVM_BB_FULL_IP.relabeled	0.0062
18	MSRA-MSRA-VLM_8_8_640_ful	0.0057
19	MSRA-MSRA-VLM-8-8-800-HT	0.0046
20	PRIP-PRIP_HSI_ScIvHarris	0.0045
21	NTU_SCE_HOI-NTU_SCE_HOI_1	0.0041
22	PRIP-PRIP_cbOCS_ScIvHarr2	0.0028
23	INAOE-TIA-INAOE_SSAssemble	0.0021
24	INAOE-TIA-INAOE-RB-KNN	0.0016
25	INAOE-TIA-INAOE-RB-KNN+MRFI_ok	0.0014
26	INAOE-TIA-INAOE-RB-KNN+MRFI	0.0014

QRELs

The QREL files are available here and can be used for further analysis of the results with trec_eval

Questions & Comments

If you have any questions or comments on these information, feel free to contact us. Questions or comments can be addressed to Allan Hanbury and Thomas Deselaers.

More detailed results

Here are the results with query-wise MAP values:

Using the normal qrels

rank	run id	query 1	query 2	query 3	query 4	query 5	query 6	query 7	query 8	query 9	query 10	all
1	clefrun.HUTCIS_SVM_FULLIMG_ALL.relabeled	0.2128	0.0151	0.2808	0.0034	0.0000	0.0020	0.0016	0.0081	0.2095	0.0082	0.0742
2	clefrun.HUTCIS_SVM_FULLIMG_IP+SC.relabeled	0.1018	0.0119	0.2579	0.0021	0.0000	0.0024	0.0010	0.0136	0.2034	0.0160	0.0610
3	clefrun.HUTCIS_SVM_FULLIMG+BB.relabeled	0.1299	0.0146	0.1141	0.0009	0.0000	0.0044	0.0010	0.0058	0.2235	0.0108	0.0505
4	clefrun.HUTCIS_SVM_FULLIMG_IP.relabeled	0.0925	0.0126	0.2362	0.0009	0.0000	0.0014	0.0009	0.0264	0.0302	0.0122	0.0413
5	MSRA-MSRA_RuiSp	0.0253	0.0194	0.0792	0.0346	0.0091	0.0002	0.0031	0.0067	0.0207	0.1374	0.0336
6	clefrun.HUTCIS_SVM_BB_BAL_IP+SC.relabeled	0.0488	0.0129	0.0235	0.0002	0.0000	0.0149	0.0000	0.0008	0.1041	0.0040	0.0209
7	clefrun.HUTCIS_PICSOM1.relabeled	0.0319	0.0131	0.1352	0.0014	0.0000	0.0006	0.0019	0.0000	0.0038	0.0050	0.0193
8	clefrun.HUTCIS_PICSOM2.relabeled	0.0169	0.0136	0.1319	0.0008	0.0000	0.0002	0.0038	0.0000	0.0058	0.0048	0.0178
9	clefrun.HUTCIS_SVM_BB_ALL.relabeled	0.0583	0.0157	0.0236	0.0002	0.0000	0.0138	0.0002	0.0013	0.0443	0.0032	0.0161
10	clefrun.HUTCIS_SVM_BB_BB_IP+SC.relabeled	0.0522	0.0129	0.0278	0.0005	0.0000	0.0082	0.0001	0.0013	0.0443	0.0038	0.0151
11	clefrun.HUTCIS_SVM_BB_FULL_IP+SC.relabeled	0.0805	0.0165	0.0153	0.0005	0.0000	0.0095	0.0000	0.0014	0.0238	0.0028	0.0150
12	RWTHi6-HISTO-PASCAL	0.0034	0.0265	0.0165	0.0013	0.0000	0.0006	0.0024	0.0039	0.0014	0.0938	0.0150
13	clefrun.HUTCIS_SVM_BB_BB_IP.relabeled	0.0398	0.0070	0.0121	0.0007	0.0000	0.0013	0.0003	0.0019	0.0209	0.0028	0.0087
14	MSRA-MSRA-VLM_8_8_640_ful	0.0057	0.0021	0.0116	0.0028	0.0000	0.0000	0.0001	0.0201	0.0014	0.0359	0.0080
15	clefrun.HUTCIS_SVM_BB_BAL_IP.relabeled	0.0392	0.0162	0.0116	0.0006	0.0000	0.0020	0.0002	0.0006	0.0050	0.0028	0.0078
16	MSRA-MSRA-VLM-8-8-800-HT	0.0088	0.0006	0.0114	0.0036	0.0000	0.0000	0.0000	0.0057	0.0028	0.0323	0.0065
17	PRIP-PRIP_HSI_ScIvHarris	0.0022	0.0000	0.0062	0.0020	0.0258	0.0016	0.0014	0.0003	0.0000	0.0218	0.0061
18	budapest-acad-budapest-acad314	0.0231	0.0014	0.0002	0.0053	0.0113	0.0004	0.0000	0.0000	0.0000	0.0009	0.0043
19	clefrun.HUTCIS_SVM_BB_FULL_IP.relabeled	0.0023	0.0121	0.0095	0.0005	0.0000	0.0006	0.0003	0.0012	0.0121	0.0021	0.0041
20	NTU_SCE_HOI-NTU_SCE_HOI_1	0.0203	0.0011	0.0181	0.0000	0.0000	0.0003	0.0002	0.0000	0.0000	0.0013	0.0041
21	PRIP-PRIP_cbOCS_ScIvHarr2	0.0008	0.0000	0.0025	0.0003	0.0075	0.0000	0.0003	0.0003	0.0133	0.0100	0.0035
22	budapest-acad-budapest-acad315	0.0034	0.0002	0.0000	0.0151	0.0001	0.0007	0.0025	0.0000	0.0008	0.0057	0.0028
23	INAOE-TIA-INAOE_SSAssemble	0.0050	0.0007	0.0010	0.0011	0.0002	0.0025	0.0000	0.0001	0.0000	0.0064	0.0017
24	INAOE-TIA-INAOE-RB-KNN+MRFI_ok	0.0011	0.0007	0.0003	0.0002	0.0000	0.0000	0.0003	0.0001	0.0009	0.0021	0.0006
25	INAOE-TIA-INAOE-RB-KNN+MRFI	0.0011	0.0007	0.0003	0.0002	0.0000	0.0000	0.0003	0.0001	0.0009	0.0021	0.0006
26	INAOE-TIA-INAOE-RB-KNN	0.0001	0.0000	0.0022	0.0000	0.0000	0.0000	0.0007	0.0000	0.0000	0.0031	0.0006

Using the qrels with additional information

rank	run id	query 1	query 2	query 3	query 4	query 5	query 6	query 7	query 8	query 9	query 10	all
1	clefrun.HUTCIS_SVM_FULLIMG_ALL.relabeled	0.0715	0.0125	0.1492	0.0044	0.0001	0.0043	0.0007	0.0413	0.0272	0.0581	0.0369
2	clefrun.HUTCIS_SVM_FULLIMG_IP+SC.relabeled	0.0368	0.0072	0.1409	0.0049	0.0003	0.0114	0.0004	0.0352	0.0302	0.0634	0.0331
3	clefrun.HUTCIS_SVM_FULLIMG_IP.relabeled	0.0334	0.0074	0.1299	0.0078	0.0004	0.0138	0.0004	0.0395	0.0084	0.0502	0.0291
4	clefrun.HUTCIS_SVM_FULLIMG+BB.relabeled	0.0472	0.0164	0.0649	0.0074	0.0002	0.0053	0.0004	0.0199	0.0313	0.0416	0.0235
5	clefrun.HUTCIS_PICSOM1.relabeled	0.0149	0.0119	0.0746	0.0018	0.0001	0.0030	0.0009	0.0041	0.0055	0.0507	0.0167
6	clefrun.HUTCIS_PICSOM2.relabeled	0.0103	0.0107	0.0744	0.0026	0.0002	0.0022	0.0017	0.0026	0.0056	0.0455	0.0156
7	MSRA-MSRA_RuiSp	0.0135	0.0074	0.0540	0.0104	0.0025	0.0005	0.0014	0.0025	0.0034	0.0512	0.0147
8	clefrun.HUTCIS_SVM_BB_BAL_IP+SC.relabeled	0.0187	0.0233	0.0109	0.0032	0.0004	0.0065	0.0001	0.0046	0.0177	0.0321	0.0118
9	clefrun.HUTCIS_SVM_BB_BB_IP+SC.relabeled	0.0191	0.0224	0.0130	0.0048	0.0005	0.0053	0.0002	0.0057	0.0096	0.0323	0.0113
10	clefrun.HUTCIS_SVM_BB_BB_IP.relabeled	0.0147	0.0049	0.0092	0.0377	0.0020	0.0023	0.0001	0.0068	0.0043	0.0288	0.0111
11	clefrun.HUTCIS_SVM_BB_ALL.relabeled	0.0221	0.0128	0.0114	0.0050	0.0001	0.0068	0.0002	0.0066	0.0111	0.0309	0.0107
12	budapest-acad-budapest-acad314	0.0911	0.0019	0.0003	0.0066	0.0034	0.0009	0.0000	0.0011	0.0000	0.0002	0.0106
13	clefrun.HUTCIS_SVM_BB_FULL_IP+SC.relabeled	0.0298	0.0135	0.0076	0.0023	0.0005	0.0058	0.0001	0.0060	0.0095	0.0303	0.0105
14	RWTHi6-HISTO-PASCAL	0.0020	0.0097	0.0146	0.0033	0.0003	0.0002	0.0012	0.0019	0.0005	0.0497	0.0083
15	clefrun.HUTCIS_SVM_BB_BAL_IP.relabeled	0.0154	0.0080	0.0073	0.0056	0.0027	0.0024	0.0001	0.0060	0.0021	0.0286	0.0078
16	budapest-acad-budapest-acad315	0.0024	0.0000	0.0004	0.0616	0.0001	0.0004	0.0010	0.0000	0.0018	0.0014	0.0069
17	clefrun.HUTCIS_SVM_BB_FULL_IP.relabeled	0.0027	0.0076	0.0064	0.0044	0.0010	0.0010	0.0001	0.0084	0.0035	0.0267	0.0062
18	MSRA-MSRA-VLM_8_8_640_ful	0.0028	0.0032	0.0129	0.0009	0.0000	0.0001	0.0004	0.0034	0.0006	0.0324	0.0057
19	MSRA-MSRA-VLM-8-8-800-HT	0.0036	0.0007	0.0111	0.0009	0.0000	0.0000	0.0001	0.0030	0.0010	0.0254	0.0046
20	PRIP-PRIP_HSI_ScIvHarris	0.0008	0.0000	0.0043	0.0014	0.0154	0.0013	0.0026	0.0005	0.0008	0.0181	0.0045
21	NTU_SCE_HOI-NTU_SCE_HOI_1	0.0121	0.0016	0.0230	0.0000	0.0000	0.0002	0.0003	0.0003	0.0001	0.0036	0.0041
22	PRIP-PRIP_cbOCS_ScIvHarr2	0.0005	0.0002	0.0019	0.0079	0.0037	0.0000	0.0003	0.0008	0.0023	0.0106	0.0028
23	INAOE-TIA-INAOE_SSAssemble	0.0014	0.0010	0.0015	0.0003	0.0003	0.0030	0.0001	0.0012	0.0014	0.0108	0.0021
24	INAOE-TIA-INAOE-RB-KNN	0.0011	0.0000	0.0033	0.0010	0.0000	0.0000	0.0003	0.0001	0.0000	0.0100	0.0016
25	INAOE-TIA-INAOE-RB-KNN+MRFI_ok	0.0025	0.0007	0.0011	0.0001	0.0001	0.0002	0.0005	0.0002	0.0003	0.0078	0.0014
26	INAOE-TIA-INAOE-RB-KNN+MRFI	0.0025	0.0007	0.0011	0.0001	0.0001	0.0002	0.0005	0.0002	0.0003	0.0078	0.0014

Thomas Deselaers

Last modified: Tue Oct 23 12:10:41 CEST 2007

;