Limits, hallucinations and the real cost

When a user interacts with an advanced language model and receives an answer written with impeccable fluency, an empathetic tone and perfect grammar, it is almost inevitable that an ancient cognitive bias kicks in: anthropomorphism. We intuitively assume that behind those words there is a mind that understands, a subject that reflects and a consciousness that weighs up the truth. The phenomenon is as old as conversational computing: as early as 1966, Joseph Weizenbaum was startled to see his users confide their private lives to ELIZA, a program that did no more than hand their own sentences back to them, rephrased. Yet this conversational mirror is an elegant mathematical illusion.

Mistaking verbal fluency for real reasoning is the greatest misunderstanding of today’s technology ecosystem. While human cognition rests on experience, intentionality, lived meaning and an understanding of physical causality, language models process massive statistical correlations between symbols (tokens) in very high-dimensional vector spaces: they calculate probabilities over sequences of text. How far that can be called reasoning is an open question among researchers, but their failures make it clear that they do not reason as we do.

Unravelling the limits, structural flaws and material bill of AI means leaving the realm of magic behind and going into the workings of its own architecture. A rigorous analysis shows that synthetic intelligence not only suffers from memory lapses and invention errors, but also carries a heavy physical, environmental and legal mortgage that the commercial gloss of the interfaces tends to hide.

When AI “lies” and gets distracted

In journalistic and commercial jargon it has become common to say that language models “hallucinate” when they give false information. From an engineering point of view, however, the term is inaccurate and misleading: it suggests a pathological departure from a normal state of consciousness. In reality, what we call hallucination is not a system fault but its normal way of working: the same mechanism that produces the correct answers also produces the false ones.

Three factors explain the erratic behaviour of artificial intelligence.

Stochastic completion

Large language models (LLMs) do not consult an internal encyclopaedia or weigh up abstract concepts before answering; what they do is stochastic completion. Given a sequence of tokens (words or fragments of words), the model calculates the probability of every possible next token and picks one, as we saw in part one, based on the statistical patterns learned during its training.

This nature led the linguist Emily Bender and her co-authors to popularise the idea of the “stochastic parrot”: the system repeats the formal patterns of language with astonishing perfection, but with no real anchor in the meaning of the physical world. When an AI “hallucinates” —that is, invents a historical quote, a non-existent scientific paper or a false financial figure— it is not lying maliciously; it is simply completing the sentence with what is statistically likely, stitching together words that sound coherent together, even if they match no real fact.

Training itself makes the problem worse. The tests used to evaluate models usually score an “I don’t know” the same as a wrong answer, so it pays a model to take a gamble, like a student in a multiple-choice exam with no penalty for wrong answers. That is why a model would rather make something up confidently than admit it does not know.

The giant context window

The technical specifications of modern models boast of enormous “context windows”, able to process anywhere from 128,000 to more than a million tokens in a single request, the equivalent of several whole books. However, their ability to retain and accurately use the information in those texts is not uniform.

• The lost in the middle effect: several studies have shown that the models’ performance traces a U-shaped curve depending on where the information they need is located. They make very good use of information at the beginning and at the end of the text, but their accuracy drops markedly for what lies in the middle: in one of the first experiments, by more than 20 points. The most recent models have softened the effect, but not eliminated it.
• The needle in the haystack: finding a specific sentence hidden in a huge document is no longer a challenge for today’s models. The difficulty comes when they have to connect several scattered pieces of information or reason about them: there, performance falls as the text grows, even though the whole file formally “fits” in their working memory. In a 2024 evaluation, only half of the models that advertised windows of 32,000 tokens or more kept acceptable performance at that length.

The lack of common sense and physical causality

Humans have a common sense grounded in physics, causality and bodily experience. We know that if we drop a glass on the floor it will break, or that we cannot fit a large object inside a smaller one.

AI lacks this empirical frame of reference. It knows the relationship between the words that describe the world, not the world itself. As a result, it is prone to blatant logical errors as soon as it is taken out of the patterns that appear explicitly in its training data. In a 2024 experiment, simply changing the names and numbers in school maths problems was enough to lower the models’ accuracy, and adding one irrelevant sentence to the problem made it plummet by up to 65% in some of them. They operate in a purely relational universe of symbols, free of material friction.

The data wall

To train today’s frontier models, technology labs have had to devour colossal volumes of information: practically all digitised books, scientific articles, historical newspapers, Wikipedia, code repositories and public forums produced by humanity over decades. This development model is now approaching a very concrete limit: the exhaustion of the human internet.

The scarcity of original raw material

According to the most widely cited estimates, at the current pace the industry will have used up all the good-quality public human text between 2026 and 2032. Faced with the prospect of that desert of fresh data, the big corporations are increasingly turning to synthetic data: using powerful AI models to generate millions of texts and images on which to train the next generations of algorithms.

Model collapse

This practice has brought to light a mathematical problem known as model collapse, or “digital inbreeding”. What happens when an artificial intelligence feeds on content generated by other artificial intelligences?

1. First generation: the model learns from diverse human data, full of nuances, exceptions and stylistic richness.
2. Second generation: it is trained on the texts produced by the previous one. Since that one tends to repeat what is most frequent, the model starts to lose the uncommon cases, the tails of the statistical distribution. Diversity is impoverished and the rarest, most minority or most creative nuances of human culture disappear.
3. Later generations: the network feeds almost exclusively on predictable synthetic data. Errors, residual hallucinations and lack of variety are amplified in a loop. The model loses the ability to represent reality, suffers irreversible degradation and ends up as a repertoire of hyper-stereotyped, empty answers.

Collapse appears above all when synthetic data replaces human data. If it accumulates alongside it and is carefully filtered, the risk is greatly reduced, and labs already train on synthetic data with good results. The problem is that the internet is filling up with unlabelled AI-generated text, and telling the two apart is getting harder and harder.

Physical and legal costs

The popular perception of artificial intelligence as an ethereal technology, housed in a weightless, immaterial “cloud”, clashes head-on with the harsh reality of its industrial infrastructure, deeply rooted in physics, geography and the consumption of natural resources. Silicon has weight, electricity pollutes and data has owners.

The energy and water bill

Training and running large language models requires supercomputing infrastructure on a colossal scale, equipped with tens of thousands of graphics processing units (GPUs).

• Electricity consumption: the largest data centres draw hundreds of megawatts around the clock, and training a single frontier model can take as much electricity as thousands of homes use in a year. Taken together, the world’s data centres consumed about 415 terawatt hours in 2024, around 1.5% of global electricity, and the International Energy Agency expects that figure to double by 2030, driven above all by AI.
• Quadratic computational cost: in today’s architectures, the computation of the attention mechanism does not grow in proportion to the length of the text but with its square. Doubling the text we give an AI quadruples that computation on the server, which sends the energy cost of long requests soaring.
• Water footprint: server farms generate extreme heat. To keep the circuits from overheating, many use evaporative cooling towers that consume millions of litres of fresh water, often drinking water, putting pressure on the water reserves of the regions where they are located. Training GPT-3 alone may have evaporated some 700,000 litres.

The intellectual property crisis

The mass collection of data (web scraping) to feed AI was carried out for years in a legal grey area, absorbing copyrighted works without the consent, attribution or compensation of their creators.

Today, courts around the world are hearing lawsuits from writers, visual artists, photographers and media organisations. The legal debate pits the doctrine of “fair use” defended by the tech companies, which argue that the model only “learns” abstract patterns, against accusations of data extractivism and unfair competition, since the commercial models compete in the market with the very creators whose works they were fed on.

The first rulings paint a nuanced picture. In 2025, a US judge held that training a model on purchased books was fair use, but that downloading pirated copies to build a library was not, and Anthropic paid 1.5 billion dollars to settle the case. That same year, a British court sided largely with Stability AI against Getty Images, while The New York Times’ lawsuit against OpenAI is still ongoing. In Europe, the AI Act already requires providers of general-purpose models to publish a summary of their training data and to respect authors who object to their works being used.

Algorithmic bias and the social mirror

Since models feed on the historical archive of human society, they inevitably inherit its inertia, prejudices and systemic discrimination. If the texts contain gender, racial or socio-economic biases, the model will encode them as valid statistical correlations. A 2024 study showed this starkly: models assigned less prestigious jobs and harsher sentences to people who wrote in African American English, even though, when asked directly, they expressed favourable views about them.

Without rigorous human oversight to adjust its ethics, AI acts as an automated amplifier of the prejudices of the past, disguising them as supposed mathematical objectivity. This also raises a question with no easy answer: who has the authority to fine-tune the ethics of an AI? Who owns morality?

References and scientific support

• ELIZA, the first conversational program, and its author’s surprise at how attached his users became to it. Weizenbaum, J. (1966). ELIZA—A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36-45. View paper (DOI)
• The “stochastic parrots” paper: models that string together linguistic forms with no access to their meaning, and the risks of making them ever larger. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. View study (DOI)
• Why models hallucinate: evaluations reward guessing an answer over admitting not knowing. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why language models hallucinate. arXiv, 2509.04664. View study on arXiv
• The lost in the middle effect: models make worse use of information placed in the middle of a long text. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. View paper (DOI)
• A model’s real context window is often much smaller than the advertised one. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What’s the real context size of your long-context language models? arXiv, 2404.06654. View study on arXiv
• Changing the numbers in a maths problem, or adding an irrelevant sentence to it, is enough to make models fail far more often. Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2025). GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models. International Conference on Learning Representations (ICLR). View study on arXiv
• When the public human text available for training models will run out: between 2026 and 2032, at the current pace. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2024). Position: Will we run out of data? Limits of LLM scaling based on human-generated data. Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 49523-49544. View study on arXiv
• Model collapse: training on data generated by other models erases rare cases and degrades the result generation after generation. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755-759. View paper (DOI)
• The nuance: if synthetic data accumulates alongside real data instead of replacing it, collapse is avoided. Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., … & Koyejo, S. (2024). Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data. arXiv, 2404.01413. View study on arXiv
• The Transformer architecture, whose attention mechanism has a cost that grows with the square of the length of the text. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008. View study on arXiv
• How much electricity data centres consume and how much that consumption will grow with AI. International Energy Agency. (2025). Energy and AI. IEA. View the IEA report
• AI’s water footprint: how much water evaporates to cool the servers that train and run the models. Li, P., Yang, J., Islam, M. A., & Ren, S. (2025). Making AI less ‘thirsty’. Communications of the ACM, 68(7), 54-61. View paper (DOI)
• The first major ruling on training and copyright: fair use with purchased books, not with pirated copies. United States District Court for the Northern District of California. (2025). Bartz v. Anthropic PBC, 787 F. Supp. 3d 1007. View the ruling
• The European AI regulation, which requires providers of general-purpose models to summarise their training data and respect copyright. European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union, L 2024/1689. View the text on EUR-Lex
• Human biases become embedded in the representations that models learn from language. Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186. View paper (DOI)
• Covert racism: models judge people who write in African American English more harshly, even while claiming to have no prejudice. Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024). AI generates covertly racist decisions about people based on their dialect. Nature, 633(8028), 147-154. View paper (DOI)