EvalTrans - Fast Evaluation for MT Research



Introduction  

In machine translation, evaluation is necessary to compare system performance and rate progress. Unfortunately, automatically computable criteria like word error rate (WER) etc. depend fundamentally on the choice of sample translations.
Subjective, manually given criterions like subjective sentence error rate are very useful for this task, but require labourous evalation by human experts.
To faciliate evaluation as much as possible, and to give easy acess to collected evaluation data, we developed a tool: EvalTrans.
It has been successfully applied in various national and international projects. Recently, it has been used for the final comparative assessment of translation quality in the European project EuTrans.

Features  

Evaluation

Database GUI Structure

Screenshots  

Main Menu [17k] Main menu and the most important submenus. The most important pieces of database statistics are shown in the main window.
List of target sentences assigned to a source sentence in database. These sentences are sorted by their score here; some sentences have been selected for further operations. Target sentences list [23k]
Manual evaluation dialog [37k] Dialog for the manual evaluation of a source/target hypothesis sentence pair. At the top you will find the sentence pair; below there is a list of the most similar target sentences in database. Similarity is indicated by the left-hand exclamation marks; the exact differences can be shown in the white box in the lower half. At the bottom there is the information item error indicator and some control buttons.
During an extrapolation session: Out of the 147 hypothesis sentences, 42 have not been found in database and have to be evaluated. As a quick check, scores for these sentences have been extrapolated; in addition, sentence 4 has been re-evaluated manually. Score, WER, information item error etc. are listed for each hypothesis sentence and for the whole test/hypothesis corpus. Manual evaluation dialog [23k]
Online help [27k] Most dialogs have a "help" button which links directly to the corresponding section of the hypertext online help.

System/Software requirements  

To use EvalTrans, you need a system supported by the following software:

Obtaining EvalTrans / Contact  

Click here to download EvalTrans.

EvalTrans is registered in the Natural Language Software Registry (an initiative of the ACL). Click here to visit the EvalTrans page there.

Contact us if you have any questions or wish to use EvalTrans for your MT or other NLP projects.

Publications / Ressources  

Publications

Ressources