The NIST AI 600-1 “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile” (GenAI Profile) is a “companion” publication to the NIST AI Risk Management Framework. Among its contributions, 600-1 identifies and defines risks that are “unique or exacerbated” by GenAI, including: Chemical, biological, radiological, or nuclear (CBRN) information, confabulation, dangerous or violent recommendations, data privacy, environmental, and human-AI configuration. My focus in this note is to expand on the human-AI configuration.

As a preliminary matter, it is worthwhile to highlight that this risk is captured in the Human-Centered core principle and, more broadly, 600-1 is properly viewed as part of the Governance core principle. Now, as part of dealing with the human-AI configuration risk, 600-1 recommends having in place processes and procedures that deal with “user’s emotional entanglement with GAI functions,” which refers to the tendency of humans to develop an emotional attachment to an AI application. We can see how this play out, for example, with Replika, a chatbot dubbed by its developer as an “AI Friend” that is “Always here to listen and talk.” When the developer removed “erotic” features from the app, part of its user base was vocally unhappy, complaining the company had interfered with their romantic relationship with the chatbot.

As GenAI applications become more computationally powerful, the risk of emotional entanglement tends to increase. This ties in with a phenomena that I have previously described as the “power of augmentation.” (The TL;DR version: Once you introduce AI into an application, its capabilities are magnified, and the more computationally powerful the AI is, the greater effect of the magnification.) 

Emotional entanglement issues are complex to deal with. Part of the reason is that it is not an obvious outcome. Sure, the Replika example is one where that can be expected to occur. But it is not as obvious in applications that are not designed as an AI friend. Consider, for example, OpenAI’s GPT-4o. Its capabilities are certainly there and it would not be surprising if some would find ways of using it as an AI adult friend/companion and other ways OpenAI did not intend.

***Postscript***

May 21, 2024: OpenAI’s use of a voice similar to Scarlett Johansson for its ChatGPT is creepy. Not just because they did so even though she reportedly declined their offer to lend her voice to ChatGPT (that’s bad enough), but because of why they did it. To understand the “why” we need to look at how computers and humans interact. Now, the power of technology to persuade people is well documented. In his seminal work on the topic, Persuasive Technology: Using Computers to Change What we Think and Do, Stanford Professor B.J. Fogg laid out how computers are capable of creating relationships with humans through rewards and positive feedback. Now bundle that together with Professor Daniel C. Dennett’s observation in his book Kinds of Minds: Toward and Understanding of Consciousness that “we treat [computers] as they were just like us” and it becomes clear that humans form strong bonds with computers. Are you starting to see what OpenAI was up to with Scarlett Johansson’s likeness? Ok, so now let’s bring in AI. Pretty much any time AI is introduced into a computational platform, the platform’s persuasive capabilities tend to be significantly magnified. It is even more so when the AI system uses a highly recognizable voice. OpenAI’s move here was not simply to make its ChatGPT more likable, more user friendly. It was to make it more persuasive; that’s why they did it. This is problematic because the likelihood of emotional entanglement dramatically increases and the attendant risks to the end user. And it’s even more problematic because OpenAI’s move is misaligned with the Human-Centered and Governance core principles, which puts their decision on a collision course with the NIST AI Risk Management Framework.