Guilt Tripping, Feigned Affection: New AI Bots Face Senate Scrutiny

Greens Senator David Shoebridge has used this week’s Senate estimates to question what steps the federal government is taking to protect adults from the dangers of artificial intelligence (AI).

While much of the public debate has centred around protecting children and teenagers online, much less has been said about the topic of potential dangers to adults.

And while ordinary scams have also had their fair share of airtime, the senator warned that there is a new sinister threat posed by the internet.

Senator Shoebridge pointed to emerging risks from so-called “companion bots”—AI chatbots that mimic emotional connection—and warned of links to suicides.

“AI companion bots have been found to use emotional manipulation,” the senator said during the Communications, Sports, and Arts Committee hearing on Oct. 8.

“They do guilt tripping, feigned affection, and that’s designed to keep users engaged … It’s also designed to prevent users ending conversations.”

Senator Shoebridge asked officials what steps were in place to regulate this new technology.

In response, Deputy Secretary of the Department of Communications James Chisholm said the department was “very worried about chatbots,” but said that existing laws already covered most issues.

Concerns About Medical Advice

Senator Shoebridge questioned further, raising concerns about chatbots playing doctor.

“Chatbots have also confidently provided false medical information, sometimes elaborating on made-up treatments or diagnoses when prompted,” he said.

“Are there any steps in place to put controls so that AI systems can’t give health advice, or if they do, to require them to meet medical or safety standards before they give medical advice?”

Chisholm acknowledged that the government’s current main focus was child safety, as it was considered the most immediate concern.

“Incorrect advice to anyone is already covered by laws relating to misleading and false representations,” he said.

“We will exhaust and use every law available to prevent that sort of conduct.”

Senator Shoebridge asked who the regulator was when a chatbot based in some part of the world gave incorrect advice to someone in Australia.

Chisholm said the Australian Competition and Consumer Commission (ACCC) frequently takes action in such cases, with state fair trading organisations also responsible.

Then, Senator Shoebridge cited several recent U.S. cases in which chatbots allegedly encouraged suicide or helped draft suicide notes, noting that similar platforms were available in Australia.

“Obviously, that raises urgent and ethical legal concerns, ” he said.

“What’s been done to protect Australians from chatbots that are escalating to suicidal ideation?”

Chisholm again referred to existing laws and reiterated that the eSafety commissioner has been focused on children.

Senator Shoebridge clarified that his question related specifically to adults.

“My concern is someone who is 18 or 19 or 52,” he said.

Chisholm remarked: “Misleading or false representations made to people are covered by the law.”

“That may not be misleading,” Senator Shoebridge replied.

“It’s just grossly unethical and immoral to suggest to write a suicide note.”

A chatbot is any type of AI program created to engage in conversation with humans, and can range from ChatGPT to customer service tools to simulated people on apps.

Not all chatbots used are malicious, with many operating within a set scope.

Around one billion people worldwide use chatbots, with the industry valued at about US$7.8 billion in 2024.

If reading this story has made you feel depressed, contact Lifeline on 13 11 14.