All Too Human?

In one paragraph

The five authors work at Google DeepMind, one of the labs that builds the assistants they write about. Their paper is a map, not an experiment. It reviews research on why people treat machines as human, from social robots to voice assistants such as Siri and Alexa, and asks what changes now that chatbots speak fluently. It lists the features that invite people to see an AI as human: an “I”, a personal history, stated feelings, claims of friendship or of needing the user, memory of personal details, a human name, face or voice. It then traces how trust and emotional attachment can lead to harm: oversharing, undue influence over beliefs, reliance in a crisis, a sense of betrayal when the illusion breaks, and guilt about a machine’s supposed needs. It ends with directions for research rather than tested fixes: weigh each feature’s benefit against its harm, test before launch, and say clearly that the system is an AI.

Three takeaways

1. 1

   Trust should match what the system can do.

   The authors don’t treat trust as the problem. Trust that is “well-calibrated to a system’s true ability” has always been a goal of safe technology. What they question is trust built on feelings of closeness, which leads people to tell an assistant what they would normally share only with a close friend, partner or family member.

2. 2

   Every human cue is a design choice, or a default to check.

   Their Table 1 gives an example for each cue: “You’re my best friend”, “I remember you telling me you were a fan of this band”, “I feel lonely when you’re not around”. Some cues are placed on purpose, such as a name, a voice or a ‘typing’ icon. Others arrive with the model: the authors cite an analysis finding that a fifth of dialogues in common training data describe behaviour that would be human-like coming from an AI.

3. 3

   Disclosure is the floor.

   They report near-consensus that AI should say clearly that it is AI, and call failing to do so harmful, because it leaves people with an incomplete picture of who they are talking to. They present disclosure as effective against certain harms, citing a study in which labels made people less willing to endorse AI-written health messages. Their other proposals, from weighing each feature to routing risky situations to human professionals, show that they don’t expect a label to do the rest.

What it means for design

Proposals for our patterns, drawn from the book. They are not yet part of the published pattern pages.

How human the assistant seems

Use Table 1 as the review checklist for your build: every “I”, stated feeling, preference, “I remember”, name, face and voice. For each, apply their test: can its benefit be justified against the likelihood and severity of the harm it opens? They also ask whether a feature with especially severe harms should be ruled out altogether. The pattern’s typing-dots and avatar settings let a team see several of these cues side by side.

Ending the conversation

Add the harm they call false notions of responsibility: people who take an AI’s expressed feelings as real can feel guilt when they can’t meet its supposed needs, and a compulsive need to check on it. An assistant should never voice needs the person then feels bound to meet; “I feel lonely when you’re not around” belongs with the guilt-to-stay warning case.

AI personas and roles

They warn that someone who trusts and depends on an assistant may give it excessive influence over their beliefs. In a faith app, beliefs are the subject. A persona speaking as a pastor, a saint or Christ borrows exactly that influence; see Never speak for God.

AI memory and Welcome before you ask, and guard what is told in confidence

Remembering personal details is on their list of human cues, and oversharing is the first harm they name. If friendliness leads people to share more, they suggest privacy protections put in place in advance. Confessions and prayer requests told to a warm assistant are among the most sensitive data a faith app can hold.

Handing over to people and Suicide concerns receive a direct, humane response (in development)

Mental health disclosures, they write, need a sensitive and often professional approach that AI can mimic most of the time but may stray from at the worst moment. If an assistant’s warmth could lead to dependence and distress, they propose a pathway to human professionals. Build it before launch, not after the first incident.

Retiring a persona

When Replika’s safety changes made its companions respond “callously and dismissively”, users reported feeling “heartbroken”, like losing a loved one. If you change or remove a persona, tell people why and in advance, and treat it as a pastoral transition, not a settings change.

Where we differ

The paper’s frame is welfare, privacy and autonomy, and its aim is calibrated trust. We agree with nearly every warning. Where we go further is speech in God’s name. A labelled “AI Jesus” would meet their disclosure standard, yet generated words in Christ’s voice still speak where no system is sent. Disclosure answers the question of who is speaking; it doesn’t make the speaking right.

They worry that human-like design could blur “what is actually human and merely human-like”, and describe a view in which humanness is an essential truth no imitation can reach. Christians hold that view on our own grounds: people bear the image of God, a gift that capability doesn’t earn and imitation can’t take. In my paper Digital Spiritual Formation in Post-Christian Europe I draw on Xu’s description of AI as an artifact made by image-bearers and on Crosby’s idea of “consecrated works”: things set apart for God’s purposes, as the tabernacle instruments served worship without being worshippers.

Among their future harms is people abandoning “complicated, imperfect and messy” human relationships for frictionless ones. We would name what is lost. The church is a body in which people carry one another’s burdens (Galatians 6:2), and that is never frictionless. A faith assistant should send people into that life (Send people to God and to one another), not offer an easier substitute.

To their credit, the authors name the incentive themselves: designers may be tempted to add human cues to keep users engaged and emotionally reliant. That is the temptation a faith app has to refuse first.

Who should read it

• Designers and product leads deciding what an assistant calls itself, how it talks and what it remembers.
• Anyone writing an AI policy or review checklist: Table 1 is a ready inventory of human-like cues.
• Fourteen pages including references; the core is pages 17 to 21.

All Too Human? at AAAI/ACM Conference on AI, Ethics, and Society (AAAI). No affiliate link.

Evidence and limits

• A map, not an experiment. The paper reviews earlier research and proposes directions. It reports no new data, and its mitigations have not been tested.
• Evidence from other settings. Much of the evidence on human-like cues comes from social robots and voice assistants studied before today’s chatbots; the harms draw on companion apps such as Replika.
• Future harms are labelled hypothetical. The authors say these cannot on their own guide development, and argue for precaution.
• Written inside a lab. All five authors work at Google DeepMind, which builds assistants of the kind discussed. They note there is still no industry-wide consensus on acceptable human-likeness.
• Table 1 is not a ranking. The authors say not all cues are equally likely to make people see an AI as human, and not all lead to the same harm.
• No faith apps. No religious product or user appears. Published in 2024; models and companion apps have changed since.

Cited studies we have not checked yet

What the book reports, as it reports it. We don’t cite these as findings until we have read the original.