Joakim Gustafsson

I am a professor in speech technology and head of departement at TMH. My research activity covers design and development of multimodal systems, conversational speech synthesis and social robots, as well as interactional analysis of spontaneous spoken dialogue phenomena.

In my current position I am doing long-term research in design and development of multimodal systems and social robots, as well as interactional analysis of spontaneous spoken dialogue phenomena.

I recently headed the VR-funded project CONNECTED (context-aware speech synthesis for conversational AI) that develops a spontaneous speech synthesizer that uses breath and disfluencies as an implicit control of the manner of speaking. I was co-PI in the VR funded project STANCE (Perception of speaker stance - using spontaneous speech synthesis to explore the contribution of prosody, context and speaker) and the RJ-funded project CAPTivating (Comparative Analysis of Public speaking with Text-to-speech). For examples of our Spontaneous speech go to the KTH TTS demo page).

In 2010 I started working on human-robot interaction. In the SRA ICT-TNG funded project SAVIR we collaborated with RPL to develop a robot that talked with humans in order to interpret visual scenes, see video. In the EU project IURO, we developed a mobile robot that could ask pedestrians in Munich for directions, see video. In this project the social robot FurHat was developed. I was recently the technical coordinator and work package leader in the EU project BabyRobot, where developed social robot applications for children. Our goal was to build a robot that is able to achieve joint-attention with the user, see video. I recently collaborated with RPL in two SSF-funded projects: FACT, a project on collaborative robots for manufacturing and EACare where we developed methods for detecting early signs of dementia. Finally, I have a Digital Futures funded collaborative project AAIS (Advanced Adaptive Intelligent Systems), where the aim is to develop social robots that can assist elderly in everyday tasks, e.g. giving sitation-dependent cooking directions,and a WASP funded project PerCorSo where the goal is to design the most appropriate way for a robot to behave in different human-crowded environments.

My current research theme is AI for good. I recently had a PTS-funded project called EmpowerME, where the focus was to make use of large language models to help people with cognitive disabilites to understand and answer to mail from authorities. I also had a Vinnova funded project, Food Talk, that aimed to investigate how intelligent assistants could be used to promote sustainable cooking and behavior changes. In the project will will do multimodal data collections in our Intelligence Augmentation lab. I currently have a WASP-funded project called RAPPORT (Real-time context-aware speech prosthesis for conversational interaction), where we will make use of our spontaneous TTS and large language models to improve AAC devices. I also have two founded research project that start in 2026. The first is a VR-funded 4-year project, that will support my RAPPORT project, called Personalized Voices (Adaptable Speech Synthesis) for Speech-Impaired Users. The second is a large 5-year project funded by Promobilia, called SAInt (Situated Agentic Intelligence). Together with my co-PI (Danica Kragic and Noemie Jaquier) we will aquire 5 humanoid robots (starting with Unitree G1 and Rainbow RB-Y1). The overarching goal is to evaluate how humanoid robots equipped with multimodal conversational AI can support domestic activities such as cooking, and how they may function as social companions for mitigating loneliness among elderly users.

I defended my PhD thesis at KTH in 2002. My supervisor was professor Björn Granström and my faculty opponent was professor Julia Hirschbergfrom Columbia university, NY, US.

I am examiner and lecturer in the speech technology course. I am member of the Editorial Boards of the journal Speech Communication and the International Journal of Human-Computer Studies, and Review Editor on the Editorial Board of Human-Robot Interaction, specialty section of Frontiers in Robotics and AI. I was Technical Program co-Chair of Interspeech 2017 and spoken dialogue Area Chair of Interspeech 2015,2016,2018,2019.  I am member of the Board of ISCA (International Speech Communication Association).