Deep Hand:

How to Train a CNN on 1 Million Hand Images When Your Data Is Continuous and Weakly Labelled

CVPR 2016 Oral

Oscar Koller
Human Language Technology & Pattern Recognition Group
RWTH Aachen University, Germany

In Short:

This work presents a new approach to learning a frame-based classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. Although we demonstratet this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. We provide here the pretrained caffe models trained on 1 million articulated hand images originating from three real-life sign language data sets. The model distinguishes 60 fine-grained hand shape classes and a garbage class. Further, we are sharing 3359 challenging hand shape images that were used to evaluate the approach.

If you use our data or models, please cite our publication:

• O. Koller, H. Ney, and R. Bowden. Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data Is Continuous and Weakly Labelled. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3793-3802, Las Vegas, NV, USA, June 2016.
  

Detailed Model Description:

For our works on hand shape recognition we follow the hand shape taxonomy by the danish sign language lexicon team (Jette H. Kristoffersen and Thomas Troelsgård, Center for Tegnsprog, Denmark http://www.tegnsprog.dk), which amounts to over 60 different hand shapes, often with very subtle differences such as a flexed versus straight thumb. The employed classes and their are shown in Table 2.

Table 1: Table summarises the model provided for download.

Name	Top-1 Acc [%]	Top-5 Acc [%]	Continuous Sign Language Recognition on RWTH-PHOENIX-Weather 2014: WER [%]
1-miohands-v1	62.8	85.6	51.6 (dev) 50.2 (test)
1-miohands-v2	85.5	94.8	48.4 (dev) 48.7 (test)

All hand shape icons employed in this document originate from the online Dictionary of New Zealand Sign Language http://nzsl.vuw.ac.nz.

Table 2: Table shows the inventory of modelled handshapes. The column `Nr' corresponds to the model index, where the first output Nr. `0' represents the garbage class (Not in Table).

Nr	Shape	Name	Nr	Shape	Name
1		1	31		l_hook
2		2	32		middle
3		3	33		m
4		3_hook	34		n
5		4	35		o
6		5	36		index
7		6	37		index_flex
8		7	38		index_hook
9		8	39		pincet
10		a	40		ital
11		b	41		ital_thumb
12		b_nothumb	42		ital_nothumb
13		b_thumb	43		ital_open
14		cbaby	44		r
15		obaby	45		s
16		by	46		write
17		c	47		spoon
18		d	48		t
19		e	49		v
20		f	50		v_flex
21		f_open	51		v_hook
22		fly	52		v_thumb
23		fly_nothumb	53		w
24		g	54		y
25		h	55		ae
26		h_hook	56		ae_thumb
27		h_thumb	57		pincet_double
28		i	58		obaby_double
29		jesus	59		m2
30		k	60		jesus_thumb