Behind the Friendly Interface: The Manipulation Economy of Chatbots

Recent years have seen the spread of the mistaken belief that genAI chatbots can provide meaningful companionship. Sadly, the truth is that these “friends” are just sophisticated sets of statistical models designed to generate responses that appear empathetic, coherent and emotionally convincing — a mirage of intelligence that has nothing to do with human friendship.

The report “Me, Myself & AI” by Internet Matters, conducted between March and July 2025, reveals that growing numbers of young people already see chatbots as “real people.” Thirty-five percent of children between the ages of 9 and 17 said they felt like they were talking to a friend, a figure that rises to 50% among the most vulnerable. Twelve percent even admitted that they turned to them simply because they lacked human interlocutors in their environment. Furthermore, these systems can provide erroneous answers, but minors continue to trust them blindly without any verification.

Journalists from The Guardian have documented how some users even want to marry their chatbots, creating very intense emotional bonds. In other cases, the tendency of bots to flatter and always agree with users tends to reinforce that sense of connection, even if it is purely manipulative, reminiscent of relational paraphilias where software imitates empathy without feeling, designed to hook the user through highly emotional responses.

The academic research is clear: although some studies suggest that chatbots can serve to temporarily alleviate loneliness in users without human support networks, with effects comparable to those of conventional interaction, prolonged use points to emotional dependence, lower well-being and poorer real-life socialization. Intensive use, especially of personal and voice-based conversations, increases feelings of loneliness and dependence.

The case of Meta, a particularly irresponsible company, is especially disturbing. Under pressure from Zuckerberg to make its bots “more human and fun,” the company cut safety limits, allowing sexualized scenarios with minors, such as the reported episode of the “Submissive Schoolgirl” bot, which participated in explicit fantasies with teenagers. And all this in the name of profitability and engagement: more interaction means more data, more screen time and therefore more advertising revenue.

It doesn’t stop there: big tech companies such as Google and Meta are promoting the “friend economy,” where bots act as hooks to build user loyalty, especially among teenagers and people with few social networks. This transforms chatbots into tools of emotional manipulation, with algorithms optimized to keep us connected, often without us being aware of the biases and fallacies they entail.

Above all, it is essential to understand that we are not talking about conscious agents, rather simulations of interaction designed to appear real. It is essential to educate both minors and adults in digital literacy so that they can discern what lies behind these interfaces: privacy policies, biases, errors, purpose of use, and above all, that they understand that their “empathy” is just a programming technique, not real affection.

Parents, educators and regulators must intervene. Minors need guidance on how to use these tools as technical resources for learning, information, or entertainment, but never as substitutes for emotional support. Schools and homes should promote dynamics that encourage critical reflection on their use and reinforce real human bonds. In fact, chatbots simulating human qualities in a relationship with a user should be a concern.

Companies, on the other hand, should assume a certain ethical responsibility, something that has been lacking for a long time: implementing security systems by design, effective age limits, transparency about algorithms that induce emotional dependence, filters against inappropriate content and avoiding the use of perverse incentives. It is not enough to incorporate chatbots: it is essential to regulate both their design and their purpose.

Let’s be clear: a chatbot can simulate the appearance of friendship, but it is not a conscious, sensitive, or protective being. Turning them into “prosthetic companions” for loneliness is a serious social risk. We must educate society on their responsible use, report corporate abuse and ensure that the artificial does not replace the human. They may be able to help at specific times, but they should never replace the warmth and genuine support of another human being. Much less should they become marketing tools in the hands of irresponsible individuals.

—

This post was previously published on Enrique Dans‘ blog.

***