Conversational social robotics
Robots that can interact with humans in a social way using spoken natural language, facial expression and eye gaze have applications in a wide range of areas - education, service, retail, health, elderly care, simulation, training and entertainment. Research from KTH and ICTTNG is now enabling this transformation worldwide: Deutsche Bahn is currently deploying social robots as information guides in airports and train stations in Germany and Japan to guide passengers in.
Swedish recruitment agency TNG is pioneering an unbiased recruitment process where candidates are being interviewed by a robot that is inherently agnostic to gender and ethnicity. MERCK is developing health screening robots to detect underdiagnosed diseases. Bandai Namco is exploring roboticized versions of their computer game characters to serve as greeters at theme parks. All of the above based on technologies originating from KTH Department of Speech, Music and Hearing by Prof. Beskow, Prof. Skantze and colleagues, since 2010 supported by the ICT TNG environment.
Research area background
During recent years, we have witnessed the start of a revolution in personal robotics. But as robots are entering human domains, there is a need for robots that are able to interact with humans in a socially intelligent way, using spoken language as well as non-verbal cues that humans intuitively understand.
This research inherently interdisciplinary line of research encompasses robotics, natural language processing, dialogue modelling, non-verbal behavior, speech technology and more. Speech-only devices have already taken center stage in human machine interaction with massive proliferation of smart speakers and virtual assistants. Conversational social robots take the interaction paradigm one step further and aim to include all the cues and channels that humans use in face-to-face communication, which opens up for a large array of new application domains. The European Strategic Research Agenda on Robotics5 lists among the targets in robotics for 2020 a number of aspects central to social communication such as “To extend basic interaction capabilities to exploit gestural, emotional, and intentional cues” and “To develop interfaces that can assess the emotional and cognitive state of the user and respond appropriately.''
Underpinning research
The speech group at TMH (dept. of speech music and hearing) has a long record of world leading research on human-robot conversational face-to-face interaction[1],[2],[3],[4],[5]. Traditionally, conversational systems have been designed to handle the exchange of speech in task-oriented dialog, such as ticket booking or weather information. In such voice-only applications, the physical space where the interaction takes place is neglected, and the visual channel is not used at all. Think Alexa or Siri – it doesn’t matter where exactly the user is standing, what direction the user is facing, or whether the user is smiling or grimacing; the system is audio only.
Moreover, the system is assumed to interact with a single user and does not recognise the difference between one user speaking or several. With the Furhat system, in contrast, several users may interact with the robot in one interaction. The visual channel (such as facial expressions) and the physical situation are taken into account, and many different types of interactions can be modelled.
Robotics 2020 Strategic Research Agenda for Robotics in Europe Produced (PDF) (5)
Details of impact
Furhat Robotics is a KTH spin-off company that develops conversational robots and social robotics applications. Furhat’s core product is a social robotics platform, that enables creation of engaging spoken social human robot applications. These applications can be in any conceivable domain, but most use cases target public settings (as opposed to domestic applications).
When the first version of the Furhat platform was released in 2014, many of the initial customers were from academia, interested in using the robot as a research tool in human-robot interaction, psychology and other areas. Since then, there has been a shift in the clientele towards commercial companies and organizations interested in deploying social robotics technologies in their operations. Today, Furhat has customers around the world, including: ICA-gruppen, Unicef, Ericsson, Arbetsförmedlingen, TNG Rekrytering, Uppsala Universitet, Örebro Universitet, KTH, Deutsche Bahn, BMW, MERCK, KPMG, University of Glasgow, University of Bielefeld, Disney Research, Northeastern University, University of Texas El Paso, Honda Robotics Institute, Bandai-Namco Studios and more.
In 2017, Balderton Capital & Local Globe invested 20MSEK in the company and in 2019 Furhat received a 10MSEK grant through the European Union H2020 SME instrument. These funds allowed Furhat to scale up it’s robot production (currently ongoing), mature its software platform and focus on developing several strong partnerships, aiming to establish the utility of social robots in real-world applications in wildly different segments. Below are some of the most prominent projects outlined.
Deutsche Bahn – FRAnny: FRAnny is a robot concierge, built by Deutsche Bahn on the Furhat Platform, deployed in the Frankfurt Airport. FRAnny is able to answer a wide range of questions ranging from identifying the correct gate for airline departures, directing the way to a specific restaurant, and how to access the free Wi-Fi. In April 2018, Deutsche Bahn launched a first version of the system Furhat platform. The pilot ran for a month under which the robot had 4400 passenger interactions (75% of which were rated positively by users). The trial was deemed a success by DB management, and a more advanced version of the system was developed; this system is being piloted at Berlin central station, and trials has been conducted also at Fraport and Tokyo central station.
TNG – Unbiased Recruitment Robot: TNG Rekrytering, a Swedish recruitment agency specializing in unbiased recruitment, is partnering with Furhat Robotics on the development of the world's first unbiased recruiter robot. The vision behind the robot is to better analyze, understand and perform competency-based interviews and assessments eliminating unconscious bias[6]. The first version of the robot recruiter is being launched by TNG in May 2019.
Merck - Medical Screening: Together with MERCK Pharmaceuticals, Furhat has developed Petra, a medical screening robot. Petra will interview the user in order to discover signs of three of common, yet under-diagnosed diseases: diabetes, alcoholism and hypothyroidism. The MERCK robot is designed to be placed in public areas e.g. shopping malls or train stations and was showcased at Epicenter in Stockholm in 2019.
Stockholms Stad – Robot Teaching Assistant: Furhat Robotics and Stockholms Stad are collaborating on the use of social robots in schools. The aim of the project is to reduce the workload on teachers and find new ways for students to collaborate with each other. Furhat will be used as an interactive teaching assistant which can be customized by students with content such as presentations, lectures, Q&A’s and quiz games. Stockholms Stad’s long-term vision is to have a social robot in every school, further solidifying Stockholm as one of the most innovative and forward-thinking cities in the world. A first pilot study was carried out in two Stockholm schools during spring 2019.
Bandai Namco – Greeter for theme parks: Japanese game developer Bandi Namco (PAC-MAN, Tekken...) has done a pilot project with Furhat on building a greeter robot that can be deployed in theme parks or arcade halls. The robot was designed as a clone one of the company’s own characters, Mirai Komachi.
The medical pre-screening robot Petra developed by Merck and Furhat Robotics.
References
For research
[1] Al Moubayed, S., Skantze, G., & Beskow, J. (2013). The Furhat Back-Projected Humanoid Head - Lip reading, Gaze and Multiparty Interaction. International Journal of Humanoid Robotics. 10(1).
[2] Skantze, G., Johansson, M., & Beskow, J. (2015). A Collaborative Human-robot Game as a Test-bed for Modelling Multi-party, Situated Interaction. In Proceedings of IVA. Delft, Netherlands.
[3] Al Moubayed, S., Edlund, J., & Beskow, J. (2012). Taming Mona Lisa: communicating gaze faithfully in 2D and 3D facial projections. ACM Transactions on Interactive Intelligent Systems, 1(2), 25.
[4] Skantze, G., & Al Moubayed, S. (2012). IrisTK: a statechart-based toolkit for multi-party face-to-face interaction. In Proceedings of ICMI. Santa Monica, CA.
[5] G. Skantze och M. Johansson, "Modelling situated human-robot interaction using IrisTK," i Proceedings of the SIGDIAL 2015 Conference, 2015, s. 165-167.
To corroborate impact
- Selected global media coverage on Deutsche Bahn robot concierge:
- Selected global media coverage on TENGAI Unbiased recruiter:
- Selected global media coverage on Bandai Namco Mirai Komachi anime greeter robot