Scientists have shown that AI will always hallucinate and guess „like a student on an exam;” artificial intelligence assessment systems are to blame

AI hallucinations pose a major issue since we can never predict when it will invent the information it delivers. The problem is that a lot of ChatGPT users use writing assistance tools, and if they don't verify the generated content, they might accidentally pass on errors. Scientists from OpenAI stated that this cannot be avoided (see Computer World).

AI will always hallucinate

In the published document "Why Language Models Hallucinate," a team of four researchers presented their conclusions, and one of the main culprits is the AI benchmark system, which rates any answer, even a wrong one, higher than admitting ignorance. That's why artificial intelligence will try to guess any solution.

The behavior was compared to students who would rather write something, anything, on an exam question than leave the page blank:

Models competing with ChatGPT were subjected to an experiment, which revealed that AI systems tend to provide incorrect answers. So the question was asked how many letters "d" are in the word "deepseek." DeepSeek-V3 in ten independent tests gave values such as "2" or "3." Claude 3.7 Sonnet responded even with "6" and "7."

ChatGPT-5 is also prone to hallucinations, although according to scientists, to a lesser extent. The model already showed it in August when it responded "I don't know" to a question from an internet user, which impressed many, including Elon Musk, because it was seen as a very human reaction. Interestingly, in the experiment, less errors were made by the more primitive models than the more advanced ones (o1 with 16% hallucinations, o3 with 33% hallucinations, and o4-mini with 48% hallucinations).

Researchers have found that it's impossible to avoid hallucinations, so we need to learn to control them. They also suggest that we need to make changes to benchmark systems so that they stop rewarding guessing and start penalizing for admitting ignorance. However, this cannot be achieved without appropriate regulations and industry requirements.

• tech
• AI / Artificial Intelligence
• OpenAI
• chatbots / ChatGPT