Manage the Machine

In one paragraph

Paula Goldman leads ethical and humane use of technology at Salesforce. Her book is a practitioner’s guide for business leaders. She argues that “human in the loop”, where AI drafts and a person approves, no longer fits AI that makes thousands of small decisions and acts on its own. Her replacement is the company’s phrase “human at the helm”: people set the mission, monitor performance and step in when judgement is required; elsewhere she adds deciding what to delegate and designing the handoffs. She asks teams to plan for over-trust as much as under-trust, and borrows from older safety fields such as cockpit cross-checks and surgical time-outs. Each chapter ends with use cases and principles; the conclusion is a checklist.

Three takeaways

1. 1

   Design the handoff before launch.

   Decide in advance when the AI passes to a person: when it doesn’t know, in emergencies, at legal or ethical limits, when someone is frustrated, and in sensitive moments people may not name. Pass the whole record so nobody has to repeat themselves. Hand over at good moments too, not only when something goes wrong.

2. 2

   A low handoff rate can hide a closed door.

   Salesforce’s own help agent, optimised to handle as many conversations as possible, passed about 1% to a person. The team found frustrated customers asking for someone and not getting through, let more reach a person, and satisfaction rose. From that we take: count who asked and didn’t get through.

3. 3

   Review that isn’t a rubber stamp.

   People stop checking a machine they already trust. Point reviewers at the few checks that matter, sample cases, and name two owners: the person answerable for each output and the person answerable for the system. Make the safe choice the default.

What it means for design

Proposals for our patterns, drawn from the book. They are not yet part of the published pattern pages.

Handing over to people

Count requests for a person that did not reach one. Hand over at good moments: someone asking about baptism, a group or confession may welcome a person. Keep it an offer; she also reports that some people prefer an AI when they feel ashamed, a point our own sources make too.

Saying “I don’t know” and Citations in AI answers

She reports research that a plausible explanation makes a wrong answer harder to catch. Grounding in vetted sources reduces errors but does not remove them. Re-run quotation and citation checks after every model change.

Evidence sets the limits of the answer (in development)

Because a fluent answer persuades, keep the passage itself one tap away from any interpretation.

Agreeing and pushing back

She cites a controlled study in which an AI that asked questions helped people spot flawed arguments better than one that gave its verdict. The benefit came from making people think; a question that only flatters would not earn it.

How human the assistant seems

She reports that identical messages seem less empathetic when labelled as AI. Disclosure costs some perceived warmth. We disclose anyway, because of honesty and Never speak for God.

Autonomy has a visible boundary (in development)

Start narrow and read-only, widen one capability at a time, and give each a visible stop. A borrowed option from her Formula 1 example, where the AI suggests a reply and a person edits and sends it: drafts, a person sends, for replies to prayer requests or pastoral messages.

Team review

A council’s AI note-taker for social workers was rejected because it removed the afternoons when they compared hard cases. An AI that summarises pastoral notes could remove the conversation where a team shares care. Before launch, ask a pastor, a youth worker or a counsellor to try to make the feature do the wrong thing.

Where we differ

Goldman’s end is business value: human contact as a differentiator, great service as a growth strategy. For us, sending people to God and to one another is the point, not a premium tier (Send people to God and to one another).

Her governance stays inside the company. In my paper Digital Spiritual Formation in Post-Christian Europe I argue that a team whose income depends on a service cannot be its only check, and that evidence of harm should pause a feature whatever the business numbers say.

She accepts AI that conveys empathy in routine service, while granting that it “falls short of real connection” and that people should handle high-stakes moments. We go further. In pastoral moments, simulated empathy is itself the question: presence and accountability are the substance of care, not its style.

Her measurement has no room for seasons. In spiritual life, falling use can be a dry season or rest, not a failure to fix.

Where we agree most is her line that technology can never be responsible for itself. We would add before whom: before God and others.

Who should read it

• Product managers and engineering leads shipping AI features, especially chat, triage or agents.
• Ministry leaders drafting an AI policy: her one-page model with three groups, allowed, ask first and not allowed, fits a church well.
• Less useful for pastoral theology or for vulnerable users, which the book barely covers.

Manage the Machine at Basic Venture (Hachette Book Group). No affiliate link.

Evidence and limits

• A vendor’s view. The author leads ethics at Salesforce, “human at the helm” is a Salesforce phrase, and several statistics come from Salesforce surveys or press coverage of Salesforce.
• Company stories rather than outcomes. Much of the persuasion is interviews and case stories told without independent results.
• Findings at second hand. Some headline findings reach the book through a science news summary, a university press release, a preprint or a company blog.
• Narrow scope. Employees and business customers. Emergencies appear only as keyword escalation in a health assistant; mental-health crisis not at all; children barely; religion only as “religious context” in content-sensitivity scoring.
• US-centred. US law, regulators and surveys; Europe appears briefly.
• Our reading is partial. We read the argument, the conclusion and the oversight sections in full, not every case study.

Cited studies we have not checked yet

What the book reports, as it reports it. We don’t cite these as findings until we have read the original.