When details count
Input representations and evaluation for neural text-to-speech synthesis of complex text
Time: Fri 2026-11-13 14.00
Location: Kollegiesalen, Brinellvägen 8, Stockholm
Video link: https://kth-se.zoom.us/j/66813631609
Language: English
Subject area: Speech and Music Communication
Doctoral student: Doktorand Christina Tånnander , Tal, musik och hörsel, Myndigheten för tillgängliga medier, MTM, Malmö Sweden
Opponent: Associate Professor Catherine Lai, University of Edinburgh, Edinburgh, United Kingdom
Supervisor: Professor Joakim Gustafsson, Tal, musik och hörsel; Jonas Beskow, Tal, musik och hörsel
QC 20261021
Abstract
For some people, synthetic speech is primarily a convenient way to access written text. For people with print disabilities, it may instead be the primary means of accessing written language, including long and complex texts that place particular demands on text-to-speech synthesis (TTS). Developing TTS systems for different users and applications involves choices about how linguistic information is represented and how the systems are evaluated.
The development of a TTS system necessarily involves abstractions. One such abstraction concerns how the linguistic input is represented: a system cannot explicitly represent every distinction that could potentially influence speech. Similarly, evaluation of speech reduces a complex listening experience to selected aspects. Which details need to be represented explicitly, and which aspects need to be captured in evaluation, depend on the phenomenon being studied, the purpose of the system, the text, the listener, the task, and the listening situation.
This thesis investigates some of these choices and their consequences for neural TTS. One group of studies examines explicit representations of phonetic, phonological and prosodic information, primarily at the word and sub-word levels, and their effects on synthetic speech. The results show that the choice of representation affects both the resulting speech and the possibilities for controlling specific properties of speech. Two studies address the synthesis of English insertions in Swedish text, investigating how different representations can control English-accentedness and how this variation is perceived by listeners.
Synthetic speech and TTS models are also explored as experimental tools for studying speech through analysis-by-synthesis. Some of the studies use systematic manipulation of linguistic input representations, together with analyses of the resulting speech and listener judgements, to investigate phonetic and phonological phenomena. The results show how controlled manipulation of synthetic speech can provide insights into how variation in the training data is represented by the model, and reveal how explicitly represented information and variation learned from the training data relate to the resulting speech and its perception.
Two further studies examine how listening test results may be influenced by choices made in experimental design. One study investigates how the same synthetic speech is evaluated under different contextual framings and listening tasks.The other uses an Audience Response System (ARS) to collect judgements continuously during listening, allowing responses to be studied over longer stretches of speech and providing diagnostic information about how listener judgements relate to specific parts of the speech.The studies confirm that what becomes observable in an evaluation depends both on the listening context and on the evaluation method.
Altogether, the eight studies show that details count in different ways throughout TTS research and development: in what information is represented, in what properties of speech can be controlled and investigated, and in what aspects of the listening experience are captured by evaluation.There is therefore no single appropriate level of detail: which details count depends on what the system or study is intended to achieve.