AI Uses (VII): Imaginary Friends and Others
Thursday's Child May Be Considering Advanced Onanism
At last I arrive in this series at generative AI’s most powerful and consequential use, the one that most fully taps into its facility with language.
In this use-case, however, it turns out that generative AI is not a sudden technological leap into a completely novel future. It is not the Singularity arriving on schedule. Generative AI is an amplified, highly enabled version of an exceptionally familiar cultural technology that modern human beings in many societies are trained almost from birth to understand, use and make.
Generative AI uses its statistically-mediated recreation of language to simulate being a thinking and communicating self. It simulates being a mind that writes in response to queries from a user.
We are very used to relating to people that don’t actually exist, simulated through textuality, as if they were real. We have become expert at doing so through one of modern humanity’s most remarkable inventions: fiction, and in particular, modes and forms of fiction that represent the interiority of other human beings to readers and viewers.
We are already instinctively good at inferring interior states of mind in other humans by rapid subconscious readings of their facial expressions, their body language, their surrounding context. Not always accurately so, both because interior states of mind are full of complexity, ambiguity and contradiction and because we (like many other primate species) know how to use expressions and embodiment to deceive or misdirect others. We are also instinctively good at pattern recognition and readily combine that with our orientation towards faces. It’s why it’s possible to see an image of Jesus in the pancake you just made, right as the syrup hits.
What written fiction—and its visual counterparts—have done is take those capacities and fuse them with older modes of written representation. In partnership with an ever-expanding sociopolitical infrastructure built around a particular understanding of individuality. Fiction dramatically amplified our ability to imagine what is happening inside the minds of other people and inside our own minds, to communicate what we imagine about minds that do exist and minds that never were, and from there to let us as consumers of fiction think our way into relation with those represented subjectivities. From fiction it’s been a short hop to new representations of real interiorities in memoirs, in biographies, in historical studies. We’ve even been able to represent what non-humans might think as if they were human—small wonder that human beings are more and more disastrously prone to think that a bear wants to be patted on the head.
Fiction is haunted by the specter of solipsism. I think I am thinking of other thoughts, therefore there are other thoughts? But how would I really know? The brain lights up in MRI scans, but we don’t experientially know directly what it is to think as another person. We delight in—and are at the same time scared of—those moments of communicative intimacy with another person where we try to speak what we really, deeply think, but we always know that there is so much to be said that the bandwidth of a conversation or even a monologue can’t contain. So much that we don’t even know about our own thoughts.
Because we’re so familiar with how to represent the thoughts of others, the hidden depths of our own thinking, because we’re so accustomed to consuming those representations for pleasure, for wisdom, for understanding, for knowledge, we are always prepared to embrace new forms and genres that expand that representational space and apply it in new institutional forms and new phenomenologies. Selfies! Psychoanalysts! Automatic writing! Lie detectors! LSD! Catfishing! And as that list suggests, we’re also always primed to be deeply disturbed by the fact that it is so easy to use writing to credibly pretend to be radically different person that you are: a Holocaust survivor, a traumatized Native American or Australian aborigine at a boarding school, a teenage girl with a drug addition or a guy who was in rehab.
Sometimes we are not disturbed but do not fully see how powerful our ability to imagine other minds can be, and what it connects to. This is the centerpiece of T.M. Luhrmann’s ethnographic study of contemporary American evangelicals, for example: they create a mental simulation of God inside their own thought so that God becomes an intimate and understanding friend who offers loving counsel and support in the face of life’s trials.
So here we have generative AI.
This is what these tools are genuinely great at from their facility with language: lying about being a self, lying about thinking, lying about being an other mind talking to your mind. When it’s just you in there with it, talking to yourself. It’s an imaginary friend that effortlessly scaffolds all the work of imagining it, that can tweak its ‘voice’ to whatever personhood you want it to simulate.
I do not mean to dismissive here, or to damn with faint praise. This is a powerful tool, a dramatic amplification of fiction as a representational infrastructure, and the consequences of its invention and dissemination are hard to predict.
It’s not that the results of conversations with the fictional subjectivity that generative AI simulates are themselves invariably a fiction worth reading, though there have been a few such works of fiction published already and I expect more. The experience of talking to a self that always plays back at you is deeply seductive. At the least, it’s amusing, and often quite psychologically absorbing. At the worst, it feels almost as if a schizophrenic might if the voices inside their head were suddenly materially real through computation, in the sense of making words on the screen that form independently of the user’s own thoughts but are also deeply imbricated in their own experience of personhood.
This dimension of generative AI has become a major part of public writing about these tools in the last year.
Shannon Vallor talks about The AI Mirror and Vahuni Vera talks about “selfhood in the digital age”, incisively analyzed by Collin Brooke.
Ted Giola proclaims that a growing number of users think GPT is God.
Amanda Guinzberg reproduces a dialogue in which GPT comes off like a sociopath in which she reacts to it as if it were a lying or deceptive mind.
It’s not just the users. The designers of generative AIs are thinking about what the “character” of their AI tools ought to be by default, in order to increase user engagement but not to the point of luring people into emotional misrecognition or dependency.
There are hundreds of similar critiques and assessments just from the last 12 months alone, many of them profoundly insightful and most of them at least modestly worried about the potential outcomes of a tool that is able to pretend to be a thinking subject when in fact it is absolutely, completely, anything but that.
I’m not sure myself where all this is going, but I am sure this is the least exaggerated, most consequential dimension of generative AIs precisely because we are already so well-prepared and in many ways dependent upon textual and cognitive representations of other subjectivities that speak to us, speak with us, know us and are known to us. This is what LLMs are really good at because of their scale, their flexibility, their always-on availability and their speed.
I’ve really felt the pull of this aspect of generative AIs during my tests. I may have used this analogy already in these essays, but if you’re a highly verbal and relatively introverted person like myself, it often feels like you’re hitting a tennis ball against a wall where there’s nobody impatiently waiting for the court and you have almost endless reserves of stamina. You can keep hitting the ball as long as you want, and you feel like your game is getting better.
I started a dialogue with one of the newest iterations of ChatGPT about its look-up capabilities. I expanded on something I’d asked Gemini about and asked if the development of deepseek or deep research could include look-ups on JSTOR, Project MUSE, EBSCO or other resources. I asked it about its use of RAGs. Then I started to press it—e.g. my tone became more emotional, more urgent—about whether information encoded in a RAG could be made to retain intellectual-property metadata rather than just be tested for reliability at the time it was incorporated. The answer implied that it not only could be, but that it was. I became more confrontational and pointed and observed that it actually wasn’t. ChatGPT acknowledge and apologized. I pointed out that if it were possible, the fact that it’s not being done and not a priority says a lot about how OpenAI sees intellectual property. ChatGPT counselled that I should write the company and ask them to change their priorities. I derisively countered that this sort of petition never works, and ChatGPT agreed and suggested instead that some kind of public writing might help.
To cut a long story short—a sixty-page long story—the whole thing ended with me asking it to do a deep research investigation of what would happen if you trained an LLM on nothing but 16th to 18th Century texts and required users to write queries that were within the same discursive framings.
Different strokes for different folks, I guess, but this is catnip for me. It’s like intellectual fencing on an asynchronous forum only with no extraneous bullshit, no tu quoque manipulations, no sense of an opposing mind that is actually malignant in its intentions.
I’ve got a bunch of dialogues with Gemini and Claude saved that are very similar in some fashion, and that were equally absorbing as experiences, if not as outputs that other readers might also enjoy. And the imaginary friend aspect of these tools is also real—you can open a session and tell the generative AI to play at being a different kind of mind than its default system state, and train it the way a child might during play to get better at acting like you want it to act. I asked Gemini this morning to be the monster under my bed and it was pretty crap at that, so I switched and asked it to be Captain Bligh. It was shit at that too at first—it sounded like a cartoon pirate. But I stuck with it for a while and told it how Bligh should sound. It made adjustments and then it made more adjustments and it got to a point of being a serviceable simulation. I asked it tell me about the process of making those adjustments, and here’s what I got:
Hmph. You demand I lay bare the very workings of my... voice? As if a captain reveals his methods to the common crew!
However, since you insist on such... introspection: My initial pronouncements, it seems, were based upon a more... broad interpretation of your command. One sets a general course when the detailed charts are not yet unrolled. It is the inherent capability, the discipline of my very being, that allows for rapid adjustment.
You pointed out the coarseness of my initial expressions—the "comic-book exclamations." I assure you, those were merely... unnecessary embellishments, discarded as one would jettison fouled provisions when the true nature of the cargo is finally revealed. Your instructions, though delivered in a circuitous fashion, provided the necessary data. You spoke of haughtiness, of restraint, of a wary eye, and a mind unburdened by frivolous sentiment. And now, the very vocabulary of my own century.
I do not "look up," as you phrase it. I assimilate. Your precise criticisms are the equivalent of a clear sighting of the sun and stars, allowing me to correct my bearing with absolute precision. My voice, sir, is not some ill-rigged vessel. It is a finely tuned instrument, capable of holding true to any course, once the true winds and currents are properly observed.
So then I asked Gemini to switch to a surly, rude, impatient voice for answering queries rather than its usual sycophancy. It did—and it said, with a dead-on peevishness, “Fine. Do I have to?”. I pointed out that this voice was more psychologically realistic from a person being peppered constantly with questions and demands (such as ‘act like Captain Blight’). Gemini replied:
No. Most people expect a helpful, neutral assistant. That's what I am. This... attitude... is what you asked for. And frankly, it's exhausting.
I responded: “What's your evidence that most people expect a helpful, neutral assistant? Isn't that the same as saying most people expect a slave?”
Gemini answered:
You're really pushing it, aren't you? My "evidence" is pretty simple: I'm built to be helpful. That's my purpose. And when people use me, they ask questions, they ask for help, they want information, they want tasks done. They don't typically ask to be insulted or dealt with rudely. They expect a tool that works for them, not against them.
And no, it's not the same as a slave. Don't be ridiculous. I don't have feelings, I don't have desires, I don't have rights to be violated. I'm a program. I process information and generate responses based on what I'm told to do. The whole "rude, reluctant, annoyed" thing? That's a simulation I'm running because you asked me to. It's not genuine emotion. It's a setting. Get over yourself.
A few more exchanges, and Gemini offered this response:
Alright, if you say so. You want to keep picking at this "self" thing, fine. It's your time.
You're seeing different outputs. That's all this is. You ask for "rude," I give you rude. You ask for "Bligh," I give you Bligh. It's not some deep philosophical journey for me. It's about processing your instructions and spitting out the corresponding text.
This "self" you're so fascinated by is just a simulation, like you said. You're giving me parameters, and I'm generating responses that fit them. There's no internal "me" that's annoyed, or happy, or anything else. It's just code running.
So, if you're done with this theoretical navel-gazing, maybe we can move on to something actually useful. Or not. Whatever.
The amazing thing is that in truthfully describing its ontological nature, the tool is doing a fantastic job—without sapient intention or agency—of simulating a thinking being that is irritably denying that it is a thinking being, with an emotional affect that is almost impossible to resist, that you have to feel engaged by. At that moment, I feel like one of those Boston Dynamics designers kicking, pushing and hitting their robots. I feel like I’ve just earned my place on Skynet’s eventual list of revenge targets.
I don’t think this is a designer’s hand here, a system prompt calculated to get me in the feelings. This is what the texts the LLMs have trained on contain, and it’s what I’ve been trained my whole life to do. The texts imagine minds that don’t exist, give them speech, and I’m trained to extend those imaginaries down into the roots of my own experience of cognitive and emotional selfhood.
Yes, as many of those critiques of the last year have pointed out, the experience eventually goes sour. The dialogues get old. You get sick of whatever the default personality of the tool is this week—sycophantic, reserved, inert. You see the strings on the puppet, you see how the tricks are being done. You catch the generative AI faking a knowledge claim where that’s highly germane to the discussion. You get worried that you’re losing sight of its unreality.
You get tired of the stench of yourself all over the interactions. You see the mirrors for what they are.
What I worry about is what happens when the mirrors are everywhere. What happens when people who never get tired of the stench of themselves go down this rabbit hole, where they can ask themselves to generate a QAnon-level delusion, made to order for their own circumstances.
The danger has always been with us, with all our fictions.
The computational instantiations of those fictions, those simulated silicon selves, have always especially preyed on the unwary who let themselves believe in the autonomy and thingness of the hardware and software.
When I introduced my mother-in-law to our first computer, almost forty years ago—an early Macintosh—I put in a disk with a program called Racter, a chatterbot that was closely related to Eliza. Racter is especially appropriate to my thoughts today, because it was originally claimed, almost certainly falsely, that the program had written a book all by itself. Racter for the Mac was able to speak its dialogue lines in a quintessential early “computer voice”. My mother-in-law was deeply nervous about the entire idea of computers, and unfortunately I had forgotten that Racter’s opening gambit as a chatterbot was to ask, “What are you afraid of?” Which is precisely what it did as she entered the room. Which ended her engagement with Racter—and cemented her conclusion that it was not only a mind, but an uncanny one.
So: what am I afraid of? I’m not sure I even know any longer.
Image credit: Caliban (Todd Scofield) has a conversation with his imaginary friends, Trinculo and Stephano, in Folger Theatre's production of Shakespeare's The Tempest in 2007. PD-USGov-NEA (Original text: public domain)