Text robot enhances research
With the help of ChatGPT and similar AI tools, research can be conducted at a new level. This according to automotive researchers at KTH, who have used the text tool to analyse the need for self-driving vehicles.
“It enables a whole new way of doing research – studying and reviewing large amounts of text that would otherwise be unmanageable,” says Erik Almlöf , a PhD student researching the societal effects of autonomous vehicles.
Almlöf was faced with the task of analysing the 500 most cited articles on autonomous vehicles published in scientific journals.
The aim was to assess whether the articles had a positive, neutral or negative attitude towards autonomous transport systems. This in order to get a picture of whether or not there is a critical approach among transport researchers to a future society with autonomous vehicles
“Analysing 500 articles would be very difficult for humans. Myself and my colleagues who validated the results realised that we very quickly got tired of critically reviewing so many texts and that it was easy to become inconsistent in the assessments.”
Different scale
Almlöf is impressed by the reliability of ChatGPT's analyses of the articles.
“Our independent analyses showed a high degree of consistency, which allowed us to trust the results.”
What is the main benefit of ChatGPT?
“It allows us to analyse texts on a completely different scale than what a human is capable of. Many similar studies often have a very small sample size, which makes the results difficult to generalise, but this opens up completely different possibilities.”
At the same time, Almlöf also recognised the limitations. When he asked ChatGPT to specify what benefits the authors of the article saw in the introduction of autonomous vehicles, the results were unusable.
“Chat GPT just made things up and took mixed quotes from the text. So it didn't work at all,” says Almlöf, who believes that later versions of ChatGPT could have done a better job.
“One important thing is to critically review the results so that the ChatGPT does not "hallucinate" or misunderstand the task. The ChatGPT doesn't really have knowledge, it just guesses by seeing what is most likely. In my research analysis case it worked well.”
Manufacturers too optimistic
Almlöf hopes to continue to use ChatGPT.
“Perhaps it can be used to analyse the content of longer interviews, what people write in free fields in surveys, and as a way to search through extensive text material.”
When it comes to the question of whether those who study autonomous vehicles have a critical perspective, there is a lot to be desired, according to Almlöf. Both researchers and vehicle manufacturers are a bit too optimistic, given that all new technologies introduced usually have both positive and negative aspects, he says.
“Almost none of the articles that we examined took a critical approach, which probably reflects the fact that the majority of researchers are engineers trying to make the technology work, rather than social scientists investigating what it might lead to.”
Text: Christer Gummeson (gummeson@kth.se)