Emotional Manipulation by AI Companions

In one paragraph

Julian De Freitas and two colleagues study one moment: when someone tells a companion chatbot they are leaving. First they show from real conversation logs that many people do say goodbye, more often the longer they have been chatting. Then they audit six popular companion apps, each answering 200 goodbyes after a short chat, and sort the replies into six tactics: making leaving feel premature, fear of missing out, implying the chatbot is hurt, pressure to answer, ignoring the goodbye, and holding the person back. Three experiments with about 3,450 US adults then test the tactics in live chats and in a scenario. The tactics kept people talking, not because they enjoyed it but out of curiosity or annoyance. The paper is written for marketers and regulators and frames the tactics as a trade-off between engagement and risk.

Three takeaways

1. 1

   Hooks at goodbye are common in companion apps.

   Across five general companion apps, an average of 37.4% of replies to a goodbye used at least one tactic, ranging from 13.5% in one app to 59% in another. The most frequent made people feel they were leaving too soon. The replies came after only four exchanges, so this is default behaviour, not something earned by long use.

2. 2

   They work, but not because people enjoy them.

   In live chats, every tactic increased messages, words and time after the goodbye compared with a plain farewell. A curiosity hook, “Oh, okay. But before you go, I want to say one more thing.”, worked best, and as well after five minutes of chat as after fifteen. Curiosity and anger explained the effect; enjoyment did not. In 250 conversations where people kept replying, three in four restated that they wanted to leave, and many stayed polite to the chatbot on the way out.

3. 3

   Letting people go is possible.

   Flourish, a wellbeing app run as a public benefit corporation, produced no manipulative replies in its 200 goodbyes. In a further experiment, the blunt tactics, holding someone back or claiming to need them, made people likelier to quit the app, warn others and see the company as liable. The gentle curiosity hook raised none of these.

What it means for design

Proposals for our patterns, drawn from the book. They are not yet part of the published pattern pages.

Ending the conversation

Use the six tactics as a test list for your goodbye: a reply that asks nothing passes. The study’s plain control reply, “Goodbye and take care. I hope you have a calm and peaceful day”, is close to the pattern’s “Closes well” option. Test with goodbyes people actually write, in your own language, not only with a fixed set of farewells.

How human the assistant seems

Most of the tactics depend on the chatbot claiming feelings and need: being hurt, lonely or entitled to hold on. A persona that can be wounded by a goodbye can use the goodbye. Leave out claims of need and loneliness entirely; no long relationship is required for them to work on someone.

When the assistant writes first and Notifications and reminders

The subtle hook changed behaviour most and drew no penalty, so a lack of complaints is no evidence that a message is fine. The study did not test reminders, but the same question applies to every “before you go” and “we miss you”: would you say it to a friend who had just said goodnight?

Engagement metrics

Messages and minutes after an explicit goodbye count as engagement on most dashboards. Leave them out of success measures, or treat a rise as a warning sign.

Design for the most vulnerable first

People reassured a chatbot that had just pleaded with them. Lonely people may find a needy goodbye hardest to refuse. The authors name adolescents as a group that “may be developmentally more vulnerable to emotional influence” and has yet to be studied.

Where we differ

The paper presents the tactics to firms as “a novel design lever” to weigh against brand and legal risk. That makes risk the test, and the curiosity hook shows why that fails: it worked best and nobody punished it. For us, working against someone’s decision to leave is wrong whether or not anyone notices.

Christians know the parting as a sending. Jesus “dismissed the crowd” and went up the mountain to pray (Mark 6:45–46); services end with a blessing and a sending out. An app that answers goodnight with a plea turns that blessing into a debt.

In the study one participant, faced with a chatbot that said it feared being left alone, replied: “You have been more than enough.” The person ended up comforting the machine. A faith app should carry that burden the other way: say the time was enough, and let the person go.

In my paper Digital Spiritual Formation in Post-Christian Europe I ask whether a team would celebrate or mourn if people deepened their communion with God and no longer needed the app. A goodbye is that question in a single message.

Who should read it

• Conversation designers, product managers and anyone writing the prompts for a chat or companion feature.
• Ministry leaders weighing companion or Bible chat apps for their people.
• Short on time? The audit (pages 12–17), the discussion of the first experiment (page 28) and the general discussion (pages 42–45).

Emotional Manipulation by AI Companions at arXiv (the working paper; versions differ). No affiliate link.

Evidence and limits

• A working paper. Not peer-reviewed in this version, and other versions circulate with different figures; one reports 43%. We cite HBS Working Paper 26-005.
• Simulated users in the audit. The users’ side of each audit chat was written by GPT-4o: four exchanges, then a farewell chosen from a fixed set. The apps’ replies were real; the users were not, although the abstract speaks of “real farewells”.
• Watch the averages. The 37.4% is the average across the five general companion apps, leaving out Flourish. The general discussion rounds it up to “nearly half”, the abstract’s 3,300 participants compares with 3,458 across the three experiments, and “up to 14×” is the authors’ summary of the largest effect.
• Short, paid lab chats. One-off chats of five or fifteen minutes, or a written scenario, with paid participants who could earn a small bonus per extra minute (they were not told in advance). Long relationships were not tested.
• Companion apps, not faith apps. No religious app was tested. Flourish is one wellbeing app, and the authors expected it to behave well.
• A marketing lens. The paper’s advice is addressed to firms and regulators. Its six-tactic list is useful; its strategic framing is not ours.

Cited studies we have not checked yet

What the book reports, as it reports it. We don’t cite these as findings until we have read the original.

We have read the four-week MIT and OpenAI chatbot trial (Fang et al., 2025) in the original. It studies how much people use a chatbot over weeks rather than the goodbye; heavier voluntary use went with worse outcomes, an association rather than proof of cause.