AI has an incest problem: Why our future will be stuck in the past

It's now common knowledge about large-scale language models like Open AI, Chat-GPT, Google's Gemini, or Elon Musk's Grok that they learn to "think" using past data. Less widespread is the resulting concern: that aligning language models with yesterday's data will hinder social progress. When discussing the ethical and social risks of language models, one often hears the phrase "value lock-in."

This complaint is only partially honest. After training on the dataset, language models undergo fine-tuning. The perspectives learned from the data, which are as full of prejudice as society itself, are overwritten in this process by morally desirable perspectives. This, as various studies show, causes the language models to shift to the left. Since these more politically correct perspectives then determine the outputs of the language models and thus also influence the thinking of their users, language models cause less a fixation of values ​​than a change in values.

To what extent this shift in values, which is taking place without public discussion in the labs of AI companies, corresponds to the demands of a democratic society remains open here. Rather, we will examine what is known in AI research as the incest problem or model collapse – and what speaks for the lock-in effect.

Suppose an AI is trained with 100 cat images, of which 10 show cats with blue fur and 90 show cats with yellow fur. The AI ​​learns that yellow cats are more common, so when it generates an image of a cat, it will mostly show a yellow cat and only occasionally a blue one, to which it will then add a bit of yellow. This color shift becomes more pronounced with each new training cycle, which then includes the cat images generated by the AI: the yellow cats and the blue cats with a yellow tinge. Until the blue cats make up less than 1 percent and are ignored by the AI.

More than two years ago, computer scientist Ilia Shumailov warned of this "curse of recursion." While blue cats, if they actually exist, will continue to cross paths with people in real life, other phenomena that only appear mediated by AI are becoming statistically irrelevant: our image of Napoleon or Gorbachev, our concepts of freedom and happiness. In all of these cases, we will soon be completely dependent on what our language model tells us.

If we ask our language model about Napoleon, for example, we learn what most people think about him: most of the people present in the language model's dataset. Of course, these aren't just historians, because many people have an opinion about Napoleon—especially when a new blockbuster about him has just hit the cinemas.

Historians can't compete with this majority, even if their professional association were to bring itself to issue a joint communiqué on Napoleon and produce a slew of academic articles, blogs, and TikTok videos. Not to mention the young scholar who offers a completely new perspective on this belligerent modernizer. How is she ever going to get a majority in the AI ​​dataset for that?

But it's not just that the completely new perspective of a young researcher or the established perspectives of the Society for History no longer hold up against the texts of amateur historians and moviegoers. Human texts also no longer hold up against those of AI when the latter's outputs constantly repeat what the majority in its dataset believes, and when these outputs then become the AI's new inputs.

Thus, yesterday's perspective on the world is exponentially amplified in the AI's data set, immunizing itself against any re-evaluation. Whereas the present previously allowed the past to speak only as far as it saw fit, the past now determines what the present thinks about it and itself.

What computer scientists call model collapse and is often referred to as the "textual incest" of AI can also be described as the "Ouroboros" effect. "Ouroboros" means "self-consuming" in ancient Greek and refers to the ancient Egyptian motif of the snake biting its tail: a self-contained cycle, completely self-sufficient from its environment, as it feeds on its own excretions.

Friedrich Schiller's "Letters on the Aesthetic Education of Man" speaks of a tension between the material drive and the formal drive. The material drive represents the human being's sensual openness to the world: a "state of selflessness" toward what is. The formal drive represents the human being's urge to give the world (the material) a rational order: just as the historian gives meaning to the material in an archive.

According to Schiller, the material and formal drives must be in balance. The predominance of the material drive leads to the abolition of man, who, in absolute devotion to the world, no longer opposes himself to it; the predominance of the formal drive leads to the exclusion of the world, when man no longer incorporates any new aspects into his views of it: In the first case, man "will never be himself; in the second, he will never be anything else."

Humans' drive for form—their views on Napoleon, for example—leads to new data for the AI. This data, however, is ignored by the incestuous AI, which is caught in its own tail, because its quantity is below its threshold of perception. The human drive for form is no match for the AI's drive for form.

The world—as humans see and shape it—thus remains outside of AI, which can never be anything other than what it was yesterday. And to the extent that humans hardly ever look at the world without AI, they too ultimately remain stuck in yesterday. The end of innovation, the end of history, the eternal recurrence of the same.

As a possible remedy, Ilia Shumailov recommended a "prestige copy" of the original, human-created dataset, with the language model regularly updated. While such a "reboot" would give the blue cats a chance of survival, it would not improve the new perspective on Napoleon.

In addition, "new, clean, human-generated data sets" would have to be regularly introduced into AI training. However, this "clean" data would not only have to be clearly identifiable as such. The question also arises as to what extent it is itself already contaminated by AI data if humans no longer read or write texts without AI.

Another suggestion to prevent a collapse would be to subject AI-generated texts to human evaluation to exclude nonsensical texts from the next AI generation's data pool. But how do you define nonsense? According to yesterday's criteria? Does the Society for History then decide which statements about Napoleon are permissible? The spokesmen of the public? Or the general public?

The experts' almost desperate suggestions demonstrate the hopeless situation we are heading toward. Identifying synthetic texts requires a comprehensive and ultimately bureaucratic investigation of the tools used: Did the AI ​​already help with the structuring of the text or merely with the fine-tuning? Or did it even help with reading the texts to which the written text refers? Human review of synthetic texts, in turn, points to a central censorship authority whose democratic status remains completely unclear.

The collapse of AI models simultaneously means the collapse of the modern measurement paradigm. With language models, however, measurement no longer remains confined to reading existing texts, but also determines the writing of new texts—with the prospect that the formative processing of the world will tip into a kind of self-cannibalism of AI. The quantification of knowledge leads to society no longer knowing more about itself, but rather increasingly less: about blue cats, new perspectives on Napoleon, and everything that leaves no sufficient data trail.

Cultural and media scholar Roberto Simanowski is a Distinguished Fellow of Global Literary Studies at the Excellence Cluster "Temporal Communities" at Freie Universität Berlin. This text is an excerpt from his new book "Language Machines: A Philosophy of Artificial Intelligence," published by CHBeck.