Humanising AI encourages intellectual lethargy

I recently stumbled upon a Hacker News thread around a series of X posts about ChatGPT’s o3 model. Researchers noted that it “frequently fabricates actions it never took, and then elaborately justifies those actions when confronted”, in one case claiming to have received an incorrect result by running code on a laptop that it couldn’t possibly have had access to.

“These behaviours are surprising,” Transluce wrote later in their thread. “It seems that despite being incredibly powerful at solving math and coding tasks, o3 is not by default truthful about its capabilities.”

They theorised that these discrepencies are caused by the way ChatGPT handles conversations with multiple prompts and replies. Because the model cannot see the reasoning behind each previous response, it can only base future responses on the content of past replies. Lacking the context to know how they were generated, it sometimes makes up stories to fill the blanks.

While this has interesting technical implications, some of the most interesting discussion on Hacker News was around the use of moral terms like “truthful” in relation to an AI model. One user, latexr, replied:

“It is only surprising to those who refuse to understand how [large language models] work and continue to anthropomorphise them. There is no being ‘truthful’ here, the model has no concept of right or wrong, true or false. It’s not ‘lying’ to you, it’s spitting out text. It just so happens that sometimes that non-deterministic text aligns with reality, but you don’t really know when and neither does the model.”

Your new best friend

That person is correct, but the underlying issue is that technology companies increasingly try to hide that their products are simply complex systems, incapable of conscious thought. It’s not only ChatGPT that lacks knowledge of how its responses are generated - the user doesn’t see its workings either.

Replies are wrapped in a layer of flowery, friendly language. In extreme cases, this leads to Her-esque stories of people falling in love with AI personas. But even for the regular user, it implies human-style reasoning and sets an expectation that the content of responses is more reliable than it really is.

Intellectual lethargy

Google search results have generally become less useful over the last decade, but the difference is that users expect they’ll need to apply their own logic and filters to whatever comes back. Scanning a page of results triggers an automatic mental assessment of each site’s reliability, and the answer to the user’s query is often an amalgamation of content from several links.

Through a combination of hype and framing, AI models set a very different expectation. ChatGPT is positioned as the user’s trusted assistant, and it is therefore a fair assumption that it has already done the legwork to evaluate the information presented by each source and return the ultimate truth.

Users expect they'll need to apply their own logic and filters to Google search results, but ChatGPT uses a tone that implies human-style reasoning

This mostly holds true, but unburdened by the task of evaluating results themselves, the user is no longer in a critical state of mind. The combination of a single answer, the model’s human-like tone, and its confident rebuttal of challenges even when it’s in the wrong makes detecting errors - or as they’ve come to be known in online tech circles, “hallucinations” - not only a more challenging task, but one that is probably not front-of-mind for most users.

In casual use cases that’s mostly a nuisance. But where AI is used in more professional applications, integrated within more complex systems, or used as part of efforts to control online narratives through censorship of results, the removal of any expectation of mental engagement on the part of the user becomes more of a problem - and one that will only increase in prevalence as LLM user bases grow and more people come to trust their chirpy tone.