Assessing the quality of automatic subtitles

The demand for subtitling has grown rapidly since the European Parliament passed legislation on making audiovisual content available to all citizens in the member states of Europe. As a result, broadcasters and the audiovisual industry have been seeking more productive subtitling alternatives than the traditional manual processes. Automatic Speech Recognition (ASR) is proving useful in this context. The recently completed EU-funded SAVAS project focused on developing ASR technology and applications tailored to automatic intralingual subtitling in live and pre-recorded settings and assessing the quality of their output. Within the project, transcription and dictation systems were developed and tested for Basque, Spanish, Portuguese, Italian, French, German and the Swiss variants of the latter three. We will present an overview of the methodology, quality features and metrics employed to evaluate the SAVAS technology, including parameters linked to subtitle layout, duration and text editing such as speaker colours, delay, persistence on screen, capitalization, punctuation, splitting and the use of acronyms, apostrophes and numerals. An extended version of the NER model that also considers errors related to the automatic allocation of speaker colours, time stamps and segmentations will be introduced. The quality performance achieved by the SAVAS applications will be presented together with the results of a preliminary experiment testing the productivity gain achievable through post-editing automatic subtitles. Finally, the main challenges that still remain will be highlighted.