Results of the PASCAL Visual Object Classes Challenge 2005 |
The goal of the The PASCAL Visual Object Classes Challenge is to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). It is fundamentally a supervised learning problem in that a training set of labelled images was provided. The task was to decide whether an object of a certain class is present in an image or not.
Using the approach described in our CVPR paper and some extensions, we (Thomas Deselaers and Daniel Keysers) participated in this evaluation and obtained quite good results.
T. Deselaers, D. Keysers, and H. Ney. Discriminative Training for Object Recognition using Image Patches. In CVPR 2005, International Conference on Computer Vision and Pattern Recognition, San Diego, CA, June 2005. In press.
Here are the slides of the talk I gave at the PASCAL workshop in April 2005
The detailed results are given in the following tables. More information on the tasks and the submissions of other groups can be obtained from
ECR=Equal Classification Rate = 1-EER
AUC=Area Under ROC Curve
for both performance measures higher values are better
Note that in the tables below the results of INRIA are not
exactly comparable to the other results because INRIA performed
several runs on the test data and only submitted the best runs.
Task 2.1: Classification of motorbikes |
ECR | AUC | Group |
---|---|---|
0.798 | 0.865 | INRIA |
0.769 | 0.829 | Aachen |
0.767 | 0.825 | Aachen |
0.698 | 0.765 | MPI Tuebingen |
0.698 | 0.710 | Edinburgh |
0.683 | 0.716 | Darmstadt |
0.663 | 0.706 | Darmstadt |
0.635 | 0.675 | Helsinki |
0.624 | 0.693 | Helsinki |
0.614 | 0.666 | Helsinki |
0.594 | 0.637 | Helsinki |
Task 2.2: Classification of bicycles |
ECR | AUC | Group |
---|---|---|
0.728 | 0.813 | INRIA |
0.667 | 0.724 | Aachen |
0.665 | 0.729 | Aachen |
0.616 | 0.654 | MPI Tuebingen |
0.616 | 0.645 | Helsinki |
0.604 | 0.647 | Helsinki |
0.575 | 0.606 | Edinburgh |
0.527 | 0.567 | Helsinki |
0.524 | 0.546 | Helsinki |
Task 2.3: Classification of people |
ECR | AUC | Group |
---|---|---|
0.719 | 0.798 | INRIA |
0.669 | 0.739 | Aachen |
0.663 | 0.721 | Aachen |
0.614 | 0.661 | Helsinki |
0.601 | 0.650 | Helsinki |
0.591 | 0.655 | MPI Tuebingen |
0.587 | 0.630 | Helsinki |
0.574 | 0.618 | Helsinki |
0.519 | 0.552 | Edinburgh |
Task 2.4: Classification of cars |
ECR | AUC | Group |
---|---|---|
0.720 | 0.802 | INRIA |
0.716 | 0.780 | Aachen |
0.703 | 0.767 | Aachen |
0.692 | 0.744 | Helsinki |
0.677 | 0.717 | MPI Tuebingen |
0.676 | 0.740 | Helsinki |
0.655 | 0.709 | Helsinki |
0.644 | 0.694 | Helsinki |
0.633 | 0.655 | Edinburgh |
0.551 | 0.572 | Darmstadt |
Task 1.1: Classification of motorbikes |
ECR | AUC | Group |
---|---|---|
0.977 | 0.998 | INRIA |
0.972 | 0.994 | Southampton |
0.968 | 0.997 | INRIA |
0.964 | 0.996 | INRIA |
0.949 | 0.989 | Southampton |
0.940 | 0.987 | Aachen |
0.940 | 0.985 | Southampton |
0.926 | 0.979 | Aachen |
0.921 | 0.974 | Helsinki |
0.917 | 0.970 | Helsinki |
0.912 | 0.952 | Helsinki |
0.903 | 0.966 | Ankara |
0.898 | 0.960 | Helsinki |
0.875 | 0.945 | MPI Tuebingen |
0.856 | 0.882 | Darmstadt |
0.829 | 0.919 | Darmstadt |
0.722 | 0.765 | Edinburgh |
Task 1.2: Classification of bicycles |
ECR | AUC | Group |
---|---|---|
0.930 | 0.982 | INRIA |
0.930 | 0.981 | INRIA |
0.918 | 0.974 | INRIA |
0.895 | 0.961 | Southampton |
0.868 | 0.954 | Aachen |
0.868 | 0.943 | Southampton |
0.851 | 0.930 | Southampton |
0.842 | 0.931 | Aachen |
0.816 | 0.895 | Helsinki |
0.795 | 0.891 | Helsinki |
0.781 | 0.864 | Helsinki |
0.781 | 0.822 | Ankara |
0.767 | 0.880 | Helsinki |
0.754 | 0.838 | MPI Tuebingen |
0.689 | 0.724 | Edinburgh |
Task 1.3: Classification of people |
ECR | AUC | Group |
---|---|---|
0.917 | 0.979 | INRIA |
0.917 | 0.972 | INRIA |
0.901 | 0.965 | INRIA |
0.881 | 0.943 | Southampton |
0.861 | 0.936 | Aachen |
0.861 | 0.928 | Aachen |
0.857 | 0.921 | Helsinki |
0.850 | 0.927 | Helsinki |
0.845 | 0.919 | Helsinki |
0.841 | 0.925 | Southampton |
0.833 | 0.931 | Helsinki |
0.833 | 0.918 | Southampton |
0.803 | 0.816 | Ankara |
0.731 | 0.834 | MPI Tuebingen |
0.571 | 0.597 | Edinburgh |
Task 1.4: Classification of cars |
ECR | AUC | Group |
---|---|---|
0.961 | 0.992 | INRIA |
0.938 | 0.987 | INRIA |
0.937 | 0.983 | INRIA |
0.925 | 0.978 | Aachen |
0.920 | 0.979 | Aachen |
0.913 | 0.972 | Southampton |
0.909 | 0.971 | Helsinki |
0.908 | 0.968 | Helsinki |
0.901 | 0.961 | Southampton |
0.898 | 0.959 | Southampton |
0.869 | 0.956 | Helsinki |
0.847 | 0.934 | Helsinki |
0.840 | 0.920 | Ankara |
0.831 | 0.918 | MPI Tuebingen |
0.793 | 0.798 | Edinburgh |
0.644 | 0.717 | Darmstadt |
0.548 | 0.578 | Darmstadt |
;