Hoppa till huvudinnehållet
Till KTH:s startsida Till KTH:s startsida

Area 3: Animal Communication and Behavior

Icelandic horsesIn this area we collaborate with veterinarians at the Swedish University of Agricultural Sciences (SLU) in developing computer methods to analyze the behavior and motion of four-legged animals in video. The research so far has focused on the detection of pain, lameness and other signs of illness in horses. Recognition of pain in horses and other animals is important, because pain is a manifestation of disease and decreases animal welfare. Horses cannot verbalize their pain as humans can, and being prey animals, they also hide signs of illness in order lower the interest from predators. We investigate different AI approaches to this problem; a data-driven self-supervision approach where horse behavior is analysed from RGB video, and a model-based approach where we collaborate with researchers at the Max Planck Institute for Intelligent Systems in Germany in developing highly detailed 3D models of horse pose, shape and motion.

People

Research Engineers

  • Ernest Pokropek (2021-2022)

MSc Students

  • Theo Wieland
  • Michaela Söderström (MSc 2021)
  • Zhenghong Li (MSc 2020)
  • Ci Li (MSc 2020)

PhD Students

  • Ci Li
  • Sofia Broomé (PhD 2022, now at TheraPanacea, France)

Affiliated PhD Students

  • Jeanne Parmentier (Utrecht University and University of Twente, Netherlands)
  • Maheen Rashid (University of California at Davis, USA, PhD 2021, now at Univrses, Sweden)

Collaborators

Current Projects

ANITA: ANImal TrAnslator (VR 2024-present)

Beagle
In recent years there has been an explosive growth in neural network based algorithms for interpretation and generation of natural language. One task that has been addressed successfully using neural approaches is machine translation from one language to another.
The goal of the project proposed here is to make an automated interpreter of animal communicative behavior to human language, in order to allow humans to get an insight into the mind of animals in their care. In this project we focus on the dog species, and will potentially explore common denominators with other species such as horse and cattle.
Despite the high potential impact, automated animal behavior recognition is still an undeveloped field. The reason is not the signal itself; the dog species has developed for thousands of years together with humans, and have rich communication and interaction both with humans and with other animal individuals. The main difference from the human language field is instead the lack of data; large data volume is a key success factor in training large neural language models. Thus, data collection is an important venture in this project. Using this data, we propose to develop a deep generative approach to animal behavior recognition from video.
Publications

MARTHA: MARkerless 3D capTure for Horse motion Analysis (KTH, FORMAS 2020-present)

Lameness in horses is a sign of disease or injury, and associated with experiences of pain for the animal. Changed behavior, body pose and motion pattern accompany lameness and many other diseases. Caretakers have difficulties in recognizing early signs of disease, since these signs are subtle and since efficient observation of animals takes resources and time. At the same time, it is crucial to detect these early signs, as the injury that cause them is easy to treat at this early stage, but will potentially be lethal if untreated.

A system for automatic detection of lameness would therefore have a huge positive impact on horse welfare.
This project aims at developing such a system. We propose a Computer Vision and Machine Learning method, MARTHA, that reconstructs the 3D horse shape, pose and motion from video, and recognizes deviations lameness and pain-related behavior from this model.

The developed method will require data. Firstly, we will record 3D motion capture data and time-correlated video in controlled indoor settings. This data will be used to train and evaluate the developed 3D shape, pose and motion model. Secondly, we will record, as well as leverage previous recordings of, horses with induced lameness or with clinically obtained ground truth lameness diagnoses. This data will be used to train methods for recognition and localization of orthopaedic injuries, and for detection of pain-related behavior.

Publications

  • Ci Li, Ylva Mellbin, Johanna Krogager, Senya Polikovsky, Martin Holmberg, Nima Ghorbani, Michael J. Black, Hedvig Kjellström, Silvia Zuffi, and Elin Hernlund. The Poses for Equine Research Dataset (PFERD). Scientific Data (to appear), 2024.
  • Silvia Zuffi, Ylva Mellbin, Ci Li, Markus Hoeschle, Hedvig Kjellström, Senya Polikovsky, Elin Hernlund, and Michael J. Black. VAREN: Very accurate and realistic equine network. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
  • Felix Järemo Lawin, Anna Byström, Christoffer Roepstorff, Marie Rhodin, Mattias Almlöf, Mudith Silva, Pia Haubro Andersen, Hedvig Kjellström, and Elin Hernlund. Is Markerless More or Less? Comparing a smartphone computer vision method for equine lameness assessment to multi-camera motion capture. Animals, doi: 10.3390/ani13030390, 2023.
  • Ci Li, Elin Hernlund, Hedvig Kjellström, and Silvia Zuffi. The Sound of Motion: Multimodal horse motion estimation from video and audio. In CVPR Workshop on Sight and Sound, 2022.
  • Hedvig Kjellström. AI and Machine Learning methods in equine research. Invited short paper in International Conference on Equine Exercise Physiology, 2022.
  • Ci Li, Nima Ghorbani, Sofia Broomé, Maheen Rashid, Michael J. Black, Elin Hernlund, Hedvig Kjellström, and Silvia Zuffi. hSMAL: Detailed horse shape and pose reconstruction for motion pattern recognition. In CVPR Workshop on Computer Vision for Animal Behavior Tracking and Modeling, 2021.

Past Projects

EquineML: Machine Learning methods for recognition of the pain expressions of horses (VR, FORMAS 2017-2022)

Recognition of pain in horses and other animals is important, because pain is a manifestation of disease and decreases animal welfare. Pain diagnostics for humans typically includes self-evaluation and location of the pain with the help of standardized forms, and labeling of the pain by an clinical expert using pain scales. However, animals cannot verbalize their pain as humans can, and the use of standardized pain scales is challenged by the fact that animals as horses and cattle, being prey animals, display subtle and less obvious pain behavior - it is simply beneficial for a prey animal to appear healthy, in order lower the interest from predators. The aim of this project is to develop methods for automatic recognition of pain in horses, with the help of Computer Vision.

Publications

Project home page
Profilbild av Hedvig Kjellström