Research
In my research I explore the concept of embodiment, and how our embodiment is shaped by and shapes our perception, cognition and behavior. This includes understanding about emotional drivers behind behavior, and how a human's or animal's affective states, attitudes, and experiences can be inferred from observations of their behavior in video. Another aspect is modeling of communicative processes to create immersive multimodal interfaces that react in purposeful ways to the user's behavior. A third strand of research is to understand more about the coupling between animal genetics, body shape and biomechanics, primarily to learn about the animal body to increase welfare, but also potentially for bio-inspired robotics.
These questions are investigated via a broad range of interdisciplinary collaborations with neuroscientists, artists, veterinarians, clinicians, interaction designers, and behavioral scientists. The general research approach is to identify relevant scenarios and capture multimodal data from these together with domain experts, and then to design computer vision models and train them with the collected data. The models can then be studied in order to discover the underlying mechanisms in the modeled scenario, or be used in computer applications such as medical diagnostics tools or immersive user interfaces.
The work in my group is organized into three partly overlapping themes:
AI for Performing Arts
In this theme we explore the aesthetic aspects of human communicative behavior. For example, what are the mechanisms of communication between a musical conductor and an orchestra, how do the musicians interpret the conductor motion? Why do humans get stimulation from watching a dancer? And, how can a completely different embodiment, e.g., a swarm of drones, express feelings and attitudes while performing on stage? We investigate these questions in collaboration with a range of performing arts professionals; musicians, conductors, and dancers, as well as interaction designers.
Current Projects
- OrchestrAI: Deep generative models of the communication between conductor and orchestra (WASP, SeRC 2023-present)
Selected Papers
- Mert Mermerci, Emile Pascoe, Fredrik Edström, and Hedvig Kjellström. Real-time control of a virtual orchestra by recognition of conducting gestures. arXiv preprint arXiv:2604.27957, 2026.
- Sara Eriksson, Åsa Unander-Scharin, Vincent Trichon, Carl Unander-Scharin, Hedvig Kjellström, and Kristina Höök. Dancing with drones: Crafting novel artistic expressions through intercorporeality. In ACM CHI Conference on Human Factors in Computing Systems, 2019.
- Kelly Karipidou*, Josefin Ahnlund*, Anders Friberg, Simon Alexanderson, and Hedvig Kjellström. Computer analysis of sentiment interpretation in musical conducting. In IEEE Conference on Automatic Face and Gesture Recognition, 2017. (*Joint first authors)
AI for Healthcare
In this theme, the main focus is on how human cognitive processes and health status can be inferred from observable behavior. Together with healthcare professionals we are developing precision diagnostics methods, e.g. to detect early symptoms of dementia and motor disorder from behavior and motion in video. We also develop methods to estimate physiological parameters, breathing pattern, and underlying emotional state from facial video; such methods are valuable for uninvasive sensors for both clinical practice and cognitive-scientific studies.
Current Projects
- EACare Revisited (KTH 2026-present)
- VITAL: Video inference of physiological parameters (SeRC, KTH 2026-present)
- The relation between motion and cognition in infants (SeRC 2023-present)
- UNCOCO: UNCOnscious COmmunication (WASP 2023-present)
Selected Papers
- Chen Ling, Henglin Shi, and Hedvig Kjellström. FIELDS: Face reconstruction with accurate inference of expression using learning with direct supervision. arXiv preprint arXiv:2511.21245v3, 2026.
- Patrik Jonell*, Birger Moëll*, Krister Håkansson*, Gustav Eje Henter, Taras Kucherenko, Olga Mikheeva, Göran Hagman, Jasper Holleman, Miia Kivipelto, Hedvig Kjellström, Joakim Gustafson and Jonas Beskow. Multimodal capture of patient behaviour for improved detection of early dementia: Clinical feasibility and preliminary results. Frontiers in Computer Science 3, 2021. (*Joint first authors)
- Xueru Zhang*, Ruibo Tu*, Yang Liu, Mingyan Liu, Hedvig Kjellström, Kun Zhang, and Cheng Zhang. How do fair decisions fare in long-term qualification? In Neural Information Processing Systems, 2020. (*Joint first authors)
- Ruibo Tu, Kun Zhang, Bo Christer Bertilson, Hedvig Kjellström, and Cheng Zhang. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. In Neural Information Processing Systems, 2019.
- Cheng Zhang, Hedvig Kjellström, Carl Henrik Ek, and Bo Christer Bertilson. Diagnostic prediction using discomfort drawings with IBTM. In Machine Learning for Healthcare, 2016.
AI for Animal Science
In this theme we develop methods to do inference from animal motion, behavior, and communication, with the aim to enhance animal welfare. Together with experts in animal biomechanics and 3D computer modeling we develop highly accurate 3D pose and shape models of horses and dogs. Moreover, we develop diagnostics tools for pain and stress from video together with horse veterinarians, and collaborate with ethologists to model the communicative patterns in dogs.
Current Projects
- Modeling of social relationships among wild animals (KTH 2026-present)
- The code that shapes the body: AI and digital phenotyping to decode evolutionary morphology in domesticated animals (DDLS 2026-present)
- ANITA: ANImal TrAnslator (VR 2024-present)
Selected Papers
- Ci Li, Yi Yang, Zehang Weng, Elin Hernlund, Silvia Zuffi, and Hedvig Kjellström. Dessie: Disentanglement for articulated 3D horse shape and pose estimation from images. In Asian Conference on Computer Vision, 2024.
- Ci Li, Ylva Mellbin, Johanna Krogager, Senya Polikovsky, Martin Holmberg, Nima Ghorbani, Michael J. Black, Hedvig Kjellström, Silvia Zuffi, and Elin Hernlund. The poses for equine research dataset (PFERD). Nature Scientific Data 11, 497, 2024.
- Silvia Zuffi, Ylva Mellbin, Ci Li, Markus Hoeschle, Hedvig Kjellström, Senya Polikovsky, Elin Hernlund, and Michael J. Black. VAREN: Very accurate and realistic equine network. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
- Sofia Broomé, Marcelo Feighelstein, Anna Zamansky, Gabriel Carreira Lencioni, Pia Haubro Andersen, Francisca Pessanha, Marwa Mahmoud, Hedvig Kjellström, and Albert Ali Salah. Going deeper than tracking: A survey of computer-vision based recognition of animal pain and affective state. International Journal of Computer Vision 131, 2023.
- Sofia Broomé, Karina Bech Gleerup, Pia Haubro Andersen, and Hedvig Kjellström. Dynamics are important for the recognition of equine pain in video. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
