Large language models are not fit for use in the real world, scientists warn – even small changes will cause their world models to collapse

Generative artificial intelligence (AI) systems may produce some eye-opening results, but new research shows they lack a coherent understanding of the world and real rules.

In a new study published in the arXiv preprint database, scientists from MIT, Harvard, and Cornell found that the large language models (LLMs), such as GPT-4 or anthropic Claude 3 Opusfail to produce underlying models that accurately represent the real world.

For example, when LLMs were tasked with providing turn-by-turn directions in New York City, they delivered them with nearly 100% accuracy. But the underlying maps used were full of non-existent streets and routes when the scientists extracted them.

The researchers found that when unexpected changes were added to a guideline (such as detours and closed streets), the accuracy of the directions the LLMs provided plummeted. In some cases this led to total failure. As such, it raises concerns that AI systems deployed in a real-world situation, for example in a driverless car, may malfunction when presented in dynamic environments or tasks.

“One hope is that because LLMs can achieve all these amazing things in language, maybe we can use the same tools in other parts of science. But the question of whether LLMs learn coherent world models is very important if we want to use these techniques to make new discoveries,” said senior author Ashesh Rambachanassistant professor of economics and principal investigator at the MIT Laboratory for Information and Decision Systems (LIDS), in a statement.

Difficult transformers

The core of generative AIs is based on the ability of LLMs to learn in parallel from massive amounts of data and parameters. To do this they depend on transformer modelsthe underlying set of neural networks that process data and enable the machine learning aspect of LLMs. This process creates a so-called ‘world model’ that a trained LLM can then use to derive answers and produce output for questions and tasks.

An example of such a theoretical use of world models would be using data from taxi rides through a city to generate a map without having to painstakingly plot each route as required by current navigation tools. But if that map isn’t accurate, deviations from a route can cause AI-based navigation to underperform or fail.

To assess the accuracy and consistency of transformer LLMs when it comes to understanding real-world rules and environments, the researchers tested them using a class of problems called deterministic finite automations (DFAs). These are problems with a sequence of states, such as rules of the game or intersections in a route on the way to a destination. In this case, the researchers used DFAs from the board game Othello and navigation through the streets of New York.

To test the transformers with DFAs, the researchers looked at two metrics. The first was ‘sequencing’, which assesses whether a transformer LLM has formed a coherent world model if it has seen two different states of the same thing: two Othello signs or one map of a city with road closures and another without. The second metric was ‘sequence compression’ – a sequence (in this case an ordered list of data points used to generate outputs) that should show that an LLM with a coherent world model can understand that two identical states (say two Othello boards which are exactly the same) have the same order of possible steps to follow.

Relying on LLMs is risky

Two general classes of LLMs were tested on these measures. One was trained on data generated from randomly produced sequences, while the other on data generated by tracking strategic processes.

Transformers trained on random data formed a more accurate world model, the scientists found. This may have been because the LLM saw a greater variety of possible steps. Lead author Keyon Vafaa researcher at Harvard, explained in a statement: “In Othello, if you saw any two computers playing instead of champion players, you would theoretically see the full range of possible moves, even the bad moves that champion players would not make.” By seeing more possible moves, even if they are bad, the LLMs were theoretically better prepared to adapt to random changes.

However, despite generating valid Othello movements and accurate directions, only one transformer generated a coherent world model for Othello, and neither type produced an accurate map of New York. When the researchers introduced things like detours, all the navigation models used by the LLMs failed.

“I was surprised at how quickly performance deteriorated once we added a detour. If we close just 1 percent of the possible streets, the accuracy immediately plummets from almost 100 percent to just 67 percent,” Vafa said.

This shows that different approaches to using LLMs are needed to produce accurate world models, the researchers said. What these approaches might be is not clear, but it does highlight the vulnerability of transformer LLMs when confronted with dynamic environments.

“We often see these models doing impressive things and thinking they have understood something about the world,” Rambachan concluded. “I hope we can convince people that this is a question that we need to think about very carefully, and that we don’t have to rely on our own intuition to answer this question.”