Area 2: Human Communication and Behavior

In this project, which is a collaboration with the Department of Women’s and Children’s health at Karolinska Institutet, we develop representation learning methods to detect various kinds of bio-markers, typically connected to underlying motor or cognitive conditions, from non-verbal behavior. The currently primary application is detection of motor conditions in neonates, but we will also work with datasets from other applications, containing 3D body, face, hand pose, gaze behavior, RGB video, IMU data, and other kinds of measurements. The underlying mechanisms are modeled using deep generative approaches such as VAE and Diffusion Models.

Publications

This project, which is part of the WARA Media and Language and a collaboration with the Perceptual Neuroscience group at KI, entails two contributions.

Firstly, we develop a 3D embodied, integrated representation of head pose, gaze and facial micro expression, that can be extracted from a regular 60 Hz video camera and a desk-mounted gaze sensor. The embodied, integrated 3D representation of head pose, gaze and facial micro expression provides a preprocessing step to the second contribution, a deep generative model for inferring the latent emotional state of the human from the non-verbal communicative behavior. The model is employed in three different contexts: 1) estimating user affect for a digital avatar, 2) analyzing human non-verbal behavior connected to sensor stimuli, e.g., quantify approach/avoidance motor response to smell, 3) estimating frustration in a driving scenario.

Publications

Human communication is multimodal in nature, and occurs through combinations of speech,
language, gesture, facial expression, and similar signals. To enable natural interactions with human beings, artificial agents must be capable of both analysing and producing these rich and
interdependent signals, and connect them to their semantic implications. Unfortunately, even the strongest machine learning methods currently fall short of this goal: automated semantic understanding of human behaviour remains superficial, and generated agent behaviours are empty gestures lacking the ability to convey meaning and communicative intent.

The STING NEST, part of the WARA Media and Language, intends to change this state of affairs by uniting synthesis and analysis with transducers and invertible neural models. This involves connecting concrete, continuous­ valued sensory data such as images, sound, and motion, with high­ level, predominantly discrete, representations of meaning, which has the potential to endow synthesis output with human­ understandable high­level explanations, while simultaneously improving the ability to attach probabilities to semantic representations. The bi­directionality also allows us to create efficient mechanisms for explainability, and to inspect and enforce fairness in the models.
Recent advances in generative models suggest that our ambitious research agenda is likely to be met with success. Normalising flows and variational autoencoders permit both extract­ing disentangled representations of observations, and (re­-)generating observations from these abstract representations, all within a single model. Their recent extensions to graph­ structured data are of particular interest because graphs are commonly­ used semantic representations.
This opens the door not only to generating structured information, but also to capturing the com­position of the generation itself (which is a graph in its own right) by exploiting and transferring techniques from finite ­state transducers and graph grammars.

Publications

• Wenjie Yin, Ruibo Tu, Hang Yin, Danica Kragic, Hedvig Kjellström, and Mårten Björkman. Controllable motion synthesis and reconstruction with autoregressive diffusion models. In IEEE International Conference on Robot and Human Interactive Communication, 2023.

Project home page

The main goal of the multidisciplinary project EACare is to develop an embodied agent – a robot head with communicative skills – capable of interacting with especially elderly people at a clinic or in their home, analyzing their mental and psychological status via powerful audiovisual sensing and assessing their mental abilities to identify subjects in high risk or possibly at the first stages of cognitive decline, with a special focus on Alzheimer’s disease. The interaction is performed according to the procedures developed for memory evaluation sessions, the key part of the diagnostic process for detecting cognitive decline.
This new diagnostic system will be one of the means by which medical doctors evaluate people for cognitive decline, in parallel to the existing methods such as memory evaluation sessions with a (human) clinician, MRI scans, blood tests, etc. Different parts of the framework can also be used for other purposes, such as to develop tools for dementia preventive training and for decision support during clinical memory evaluation sessions.

Publications

• Olga Mikheeva, Ieva Kazlauskaite, Adam Hartshorne, Hedvig Kjellström, Carl-Henrik Ek, and Neill DF Campbell. Aligned multi-task Gaussian process. In International Conference on Artificial Intelligence and Statistics, 2022.
• Taras Kucherenko, Rajmund Nagy, Michael Neff, Hedvig Kjellström, and Gustav Eje Henter. Multimodal analysis of the predictability of hand-gesture properties. In International Conference on Autonomous Agents and Multi-Agent Systems, 2022.
  • Videos and code
• Taras Kucherenko, Rajmund Nagy, Patrik Jonell, Michael Neff, Hedvig Kjellström, and Gustav Eje Henter. Speech2Properties2Gestures: Gesture-property prediction as a tool for generating representational gestures from speech. In ACM International Conference on Intelligent Virtual Agents, 2021.
  • Videos and code
  • IVA 2021 Honorable Mention extended abstracts
• Rajmund Nagy*, Taras Kucherenko*, Birger Moëll, André Pereira, Hedvig Kjellström, and Ulysses Bernardet. A framework for integrating gesture generation models into interactive conversational agents. In International Conference on Autonomous Agents and Multiagent Systems, demo track, 2021. (*Joint first authors)
  • Videos and code
• Patrik Jonell*, Birger Moëll*, Krister Håkansson*, Gustav Eje Henter, Taras Kucherenko, Olga Mikheeva, Göran Hagman, Jasper Holleman, Miia Kivipelto, Hedvig Kjellström, Joakim Gustafson and Jonas Beskow. Multimodal capture of patient behaviour for improved detection of early dementia: Clinical feasibility and preliminary results. Frontiers in Computer Science, doi: 10.3389/fcomp.2021.642633, 2021. (*Joint first authors)
• Taras Kucherenko, Dai Hasegawa, Naoshi Kaneko, Gustav Eje Henter, and Hedvig Kjellström. Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation. International Journal of Human-Computer Interaction, doi: 10.1080/10447318.2021.1883883, 2021.
• Krister Håkansson, Jonas Beskow, Hedvig Kjellström, Joakim Gustafsson, Alexandre Bonnard, Marie Rydén, Sara Stormoen, Göran Hagman, Ulrika Akenine, Kristal Morales Peres, Gustav Henter, Maria Sundström, and Miia Kivipelto. Robot-assisted detection of subclinical dementia: progress report and preliminary findings. Alzheimer's & Dementia 16(S6):e043311, 2020.
• Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexanderson, Iolanda Leite, and Hedvig Kjellström. Gesticulator: A framework for semantically-aware speech-driven gesture generation. In ACM International Conference on Multimodal Interaction, 2020.
  • Videos and code
  • ICMI 2020 Best Paper award
• Taras Kucherenko, Dai Hasegawa, Gustav Eje Henter, Naoshi Kaneko, and Hedvig Kjellström. Analyzing input and output representations for speech-driven gesture generation. In ACM Intelligent Virtual Agents Conference, 2019.
  • Videos and code
• Olga Mikheeva, Carl Henrik Ek, and Hedvig Kjellström. Perceptual facial expression representation. In IEEE International Conference on Automatic Face and Gesture Recognition, 2018.

Project home page

This project aims to investigate fundamentals of situated and collaborative multi-party interaction and collect the data and knowledge required to build social robots that are able to handle collaborative attention and co-present interaction. In the project we will employ state-of-the art motion- and gaze tracking on a large scale as the basis for modelling and implementing critical non-verbal behaviours such as joint attention, mutual gaze and backchannels in situated human-robot collaborative interaction, in a fluent, adaptive and context sensitive way.

Publications

• Kalin Stefanov, Giampiero Salvi, Dimosthenis Kontogiorgos, Hedvig Kjellström, and Jonas Beskow. Modeling of human visual attention in multiparty open-world dialogues. ACM Transactions on Human-Robot Interaction 8(2):1-21, 2019.
• Yanxia Zhang, Jonas Beskow, and Hedvig Kjellström. Look but don't stare: Mutual gaze interaction in social robots. In International Conference on Social Robotics, 2017.

The overwhelming majority of human activities are interactive in the sense that they relate to the world around the human (in Computer Vision called the "scene"). Despite this, visual analyses of human activity very rarely take scene context into account. The objective in this project is modeling of human activity with object and scene context.

The methods developed within the project will be applied to the task of Learning from Demonstration, where a (household) robot learns how to perform a task (e.g. preparing a dish) by watching a human perform the same task.

Publications

• Cheng Zhang, Carl Henrik Ek, Andreas Damianou, and Hedvig Kjellström. Factorized topic models. In International Conference on Learning Representations, 2013.
• Cheng Zhang, Dan Song, and Hedvig Kjellström. Contextual modeling with labeled multi-LDA. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013.
• Alessandro Pieropan, Carl Henrik Ek, and Hedvig Kjellström. Functional object descriptors for human activity modeling. In IEEE International Conference on Robotics and Automation, 2013.
  • Video
• Hedvig Kjellström, Javier Romero, and Danica Kragic. Visual object-action recognition: Inferring object affordances from human demonstration. Computer Vision and Image Understanding, 115:81-90, 2011.
• Hedvig Kjellström, Danica Kragic, and Michael J. Black. Tracking people interacting with objects. In IEEE Conference on Computer Vision and Pattern Recognition, 2010.