Research - Sign Language Recognition

Automatic Sign Language Recognition (ASLR)

Developing sign language applications for deaf people can be very important, as many of them, being not able to speak a language, are also not able to read or write a spoken language. Ideally, a translation systems would make it possible to communicate with deaf people. Compared to speech commands, hand gestures are advantageous in noisy environments, in situations where speech commands would be disturbing, as well as for communicating quantitative information and spatial relationships.

A gesture is a form of non-verbal communication made with a part of the body and used instead of verbal communication (or in combination with it). Most people use gestures and body language in addition to words when they speak. A sign language is a language which uses gestures instead of sound to convey meaning combining hand-shapes, orientation and movement of the hands, arms or body, facial expressions and lip-patterns. Contrary to popular belief, sign language is not international. As with spoken languages, these vary from region to region. They are not completely based on the spoken language in the country of origin.

Sign language is a visual language and consists of 3 major components:

finger-spelling: used to spell words letter by letter
word level sign vocabulary: used for the majority of communication
non-manual features: facial expressions and tongue, mouth and body position

Similar to automatic speech recognition (ASR), we focus in automatic sign language recognition (ASLR) on automatically recognizing sign language videos as gloses, which can be later translated by a statistical machine transaltion system into written text (see a demo video).

Benchmark Databases for Sign Language Recognition and Translation

In the course of the diploma thesis work Appearance-Based Gesture Recognition of Philippe Dreuw, a new database of fingerspelling letters of German Sign Language (Deutsche Gebärdensprache, DGS) was created. The RWTH Fingerspelling Database contains 35 gestures with video sequences for the signs A to Z and SCH, the German umlauts Ä, Ö, Ü, and for the numbers 1 to 5. Five of the gestures contain inherent motion (J, Z, Ä, Ö and Ü). The recording was done under non-uniform daylight lighting conditions, the back- ground and the camera viewpoints are not constant, and the persons had no restrictions on the clothing while gesturing.

Other ASLR sign language recognition databases used at our institute:

RWTH German Fingerspelling Database: German sign language, fingerspelling, 1400 utterances, 35 dynamic gestures, 20 speakers
RWTH-Phoenix Tagesschau: German sign language database, 95 German weather forecast records, 1353 sentences, 1225 signs, fully annotated, 11 speakers
RWTH-BOSTON-50: American sign language database, 483 utterances, 50 isolated signs, 83 pronunciations, 3 speakers
RWTH-BOSTON-104: American sign language database, 201 sentences, 104 signs, continuous sign language, 3 speakers
RWTH-BOSTON-400: American sign language database, 843 sentences, about 400 signs, continuous sign language, 5 speakers
RWTH-BOSTON-Hands: hand tracking database, 1000 frames with annotated hand positions to evaluate hand tracking algorithms
ATIS Corpus: Irish sign language database, 680 sentences, about 400 signs, continuous sign language, several speakers, with annotated hand and head positions to evaluate hand tracking algorithms
Corpus NGT: An online corpus of video data from Sign Language of the Netherlands with annotations
BSL Corpus Project

If you are interested in using these sign language recognition databases too, please contact Philippe Dreuw.

Automatic Sign Language Recognition Related Publications

P. Dreuw, and H. Ney. SignSpeak - Bridging the Gap Between Signers and Speakers. In Elektronische Sprachsignalverarbeitung (ESSV), pages 278-286, Dresden, Germany, September 2009.
P. Dreuw, and H. and Ney. SignSpeak - Bridging the Gap Between Signers and Speakers. Elektronische Sprachsignalverarbeitung (ESSV), Dresden, Germany, September 2009. Invited Talk for the special session "Strukturierte Sitzung: Sprachtechnologie zur Unterstützung von Menschen mit Sinnesbehinderungen".
P. Dreuw, P. Steingrube, T. Deselaers, and H. Ney. Smoothed Disparity Maps for Continuous American Sign Language Recognition. In Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), LNCS, pages 24-31, Póvoa de Varzim, Portugal, June 2009.
P. Dreuw, D. Stein, and H. Ney. Enhancing a Sign Language Translation System with Vision-Based Features. Gesture-Based Human-Computer Interaction and Simulation, volume 5085, number 1, pages 108-113, Lisbon, Portugal, January 2009.
P. Dreuw, and H. Ney. Visual Modeling and Feature Adaptation in Sign Language Recognition. In ITG Conference on Speech Communication (ITG), Aachen, Germany, October 2008.
P. Dreuw, and H. Ney. Visual Modeling and Feature Adaptation in Sign Language Recognition. Invited Talk at ITG Conference on Speech Communication, Aachen, Germany, Aachen, Germany, October 2008.
P. Dreuw, J. Forster, T. Deselaers, and H. Ney. Efficient Approximations to Model-based Joint Tracking and Recognition of Continuous Sign Language. In IEEE International Conference Automatic Face and Gesture Recognition (FG), Amsterdam, The Netherlands, September 2008.
©
P. Dreuw. Visual Modeling and Tracking Adaptation for Automatic Sign Language Recognition. In International Computer Vision Summer School (ICVSS), Sicily, Italy, July 2008.
P. Dreuw, and H. Ney. Towards Automatic Sign Language Annotation for the ELAN Tool. In LREC Workshop on the Representation and Processing of Sign Languages: Construction and Exploitation of Sign Language Corpora, Marrakech, Morocco, June 2008.
P. Dreuw, D. Stein, T. Deselaers, D. Rybach, M. Zahedi, J. Bungeroth, and H. Ney. Spoken Language Processing Techniques for Sign Language Recognition and Translation. Technology and Dissability, volume 20, number 2, pages 121-133, Amsterdam, The Netherlands, June 2008.
J. Forster. An Integrated Tracking And Recognition Approach For Video. Master Thesis, Aachen, Germany, May 2008.
P. Dreuw, C. Neidle, V. Athitsos, S. Sclaroff, and H. Ney. Benchmark Databases for Video-Based Automatic Sign Language Recognition. In International Conference on Language Resources and Evaluation (LREC), Marrakech, Morocco, May 2008.
J. Bungeroth, D. Stein, P. Dreuw, H. Ney, S. Morrissey, A. Way, and L. van Zijl. The ATIS Sign Language Corpus. In International Conference on Language Resources and Evaluation (LREC), Marrakech, Morocco, May 2008.
P. Dreuw. Visual Modeling and Feature Adaptation in Sign Language Recognition. Invited Talk at FBK-IRST, FBK-IRST, Trento, Italy, May 2008.
D. Stein, P. Dreuw, H. Ney, S. Morrissey, and A. Way. Hand in Hand: Automatic Sign Language to Speech Translation. In Conference on Theoretical and Methodological Issues in Machine Translation (TMI), pages 214-220, Skövde, Sweden, September 2007.
P. Dreuw, D. Rybach, T. Deselaers, M. Zahedi, and H. Ney. Speech Recognition Techniques for a Sign Language Recognition System. In Interspeech, pages 2513-2516, Antwerp, Belgium, August 2007. ISCA best student paper award Interspeech 2007.
P. Dreuw, D. Stein, and H. Ney. Enhancing a Sign Language Translation System with Vision-Based Features. In International Workshop on Gesture in Human-Computer Interaction and Simulation, pages 18-20, Lisbon, Portugal, May 2007.
M. Zahedi, P. Dreuw, D. Rybach, T. Deselaers, and H. Ney. Using Geometric Features to Improve Continuous Appearance-based Sign Language Recognition. In British Machine Vision Conference (BMVC), volume 3, pages 1019-1028, Edinburgh, UK, September 2006.
M. Zahedi, P. Dreuw, D. Rybach, T. Deselaers, J. Bungeroth, and H. Ney. Continuous Sign Language Recognition - Approaches from Speech Recognition and Available Data Resources. In LREC Workshop on the Representation and Processing of Sign Languages: Lexicographic Matters and Didactic Scenarios, pages 21-24, Genoa, Italy, May 2006.
P. Dreuw, T. Deselaers, D. Rybach, D. Keysers, and H. Ney. Tracking Using Dynamic Programming for Appearance-Based Sign Language Recognition. In IEEE International Conference Automatic Face and Gesture Recognition (FG), IEEE, pages 293-298, Southampton, UK, April 2006.
©

Home > Research > Sign Language Recognition

Philippe Dreuw

Last modified: Tue Sep 15 19:51:08 CET 2009 Disclaimer. Created Wed Dec 22 18:04:32 CET 2004