Researcher helps characters get their lines right
Innovation
A new movie-making app uses technology developed at KTH to literally put the words in the characters’ mouths.
Inside Swedish start-up Plotagon’s offices in Stockholm’s fashionable Stureplan neighbourhood, designers, artists and musicians sit side-by-side with programmers, doing all the set design, character drawing and incidental music that every animated movie needs.
The only difference here is they have no script to work with.
Plotagon makes a desktop app that lets the user create their own animated movies. The interface offers a selection of scenarios and characters – you can also buy extra locations and characters with thematic packets like Jane Austen’s “Pride & Prejudice” and “Stan Lee’s Superhero Pack”.
All the user needs to do is write a script.
“You write the script and the software takes care of everything else - the craftsmanship of creating a movie,” says Plotagon founder Christopher Kingdon.
And making sure the actors get their lines right is ensured by lip synchronisation technology developed at KTH. Plotagon’s chief scientist is Jonas Beskow, Associate Professor in Speech Communication and Multimodal Embodied Systems at the Department of Speech, Music and Hearing.
“I’ve been working on speech synthesis and facial animation in my research, and this felt like a great thing to be involved in,” Beskow says. “At KTH we are looking at how we can simulate human interaction in computer terms both as a tool to understand communication better and also for practical applications of building human-to-machine interfaces that are based on spoken dialog.”
Plotagon can also be a tool for further research work, Beskow says. “This fits in really well with the way we work, because we try to understand things via modelling.
“We develop models of human verbal and non-verbal behaviours during spoken interaction and we can test these behaviours to see how they work with different animated characters,” he says. “And the real user group can also be involved in the testing. We see this as a strategic cooperation in which both parties can benefit.”
Beskow is also supervising a master's student at Plotagon, Markus Hahn, who is doing his thesis on something speech experts call “back channeling”, or the physical and audible reactions that every listener provides during a conversation. “Like when you nod your head and say, ‘OK’ or ‘uh-huh’,” explains Hahn.
“Markus is working on making these short references in speech synthesis that can be automatically generated with the listening character and paired with animated gestures like nods,” Beskow says.
“This is an example of what we’ve been studying at KTH – how this happens in human dialog. We have been analysing many hours of videotaped recordings of dialog and using motion capture to record the movements.
“Markus will generate a set of example dialogs with this behaviour and then evaluate them using real test subjects to judge the dialogs with and without these back channels,” he says.
Research like this ensures that Plotagon will keep evolving and retaining its uniqueness,” Kingdon says. “The speech technology gives us a real edge,” he says. “There’s nothing compared to this.
“And by working together with research, it ensures we will continue to be at the cutting edge and hopefully we can accelerate it.”
David Callahan
Jonas Beskow's homepage