Skip to main content

Master Thesis Proposal

Evaluating modern gesture generation models in an interactive conversational agent

 

Gesticulation is a crucial part of non-verbal communication and conveys a large share of non-verbal content [1]. Gesture generation is hence an important part of animation as well as of human-agent interaction research.

This project is about evaluating the recent gesture generation models such as Gesticulator [2] and StyleGestures [3] in an interactive scenario.

For that purpose, we provide recently developed a proof-of-concept framework for integrating a gesture generation model into an interactive conversational agent in Unity, which enables virtual agents to gesticulate while speaking using the state-of-the-art end-to-end models.

The project has three parts:

  1. Further development of the framework so that it is easy to use as a stand-alone program;

  2. Designing an interaction scenario appropriate for evaluating gesturing behavior;

  3. Evaluating several recent end-to-end gesture generation models such as [2,3] in an interaction with a user;

 

The project requirement:

  • very good programming skills

  • basic understand of Unity is a big plus

  • some machine learning skills is a plus

 

If performed in a successful manner, the results of the project will be publishable in an international peer-reviewed conference.

 

Main supervisor: Taras Kucherenko

Co-supervisor: Andre Pereira

Examiner: Hedvig Kjelström

 

[1] David McNeill. Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press. 1992

[2] Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexanderson, Iolanda Leite, and Hedvig Kjellström. “Gesticulator: A framework for semantically-aware speech-driven gesture generation”. (to appear in) Proceedings of the  22nd International Conference on Multimodal Interaction. 2020

[3] Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows. Computer Graphics Forum. 2020.