Virtual Friends, Real Feelings

A while ago, I wrote GPT4o must Die.  A post that in retrospect had an overly provocative title, so I’ve since updated to read: Making Friends with AI.

That article was a hot take on OpenAI’s Assistant announcement back in May.

Using Tamagotchis and virtual pets as an example, it was a quick overview of the kinds of relationships we form with computational entities that exhibit a sense of aliveness. How people formed real attachments to digital companions. Towards the end I wondered aloud about what we might learn from these older products’ design choices, and what we might learn from them when designing Assistant systems in order to maintain a healthy emotional distance.

The most boosterish statement was this:

Millennial and GenX readers. Please be mentally prepared for the possibility that you might pop home this summer and discover that a virtual assistant talking like a flirty 20 something valley girl is now your parent(s) new best friend. 

That hasn’t happened yet – mainly because the launch of OpenAI’s assistant software has been delayed. But it will, and soon.

Whilst the mainstream front in the AI Culture War is currently focused on copyright and ‘is it art’, the rest of the AI industry is full steam ahead making and designing virtual agents. The implications resulting from their deployment is really something I really think we should be talking about. 

One of the things I’ve been doing recently when reading about any new company that’s making AI products is to mentally sort them into one of three categories: Tools, Assistants, Companions. 

Tools

Tools are exactly that. Things like voice-to-text, or text-to-voice generation, image or music synthesis etc. Much of what is launching as part of Apple Intelligence on the iPhone and MacOS appear to be tools: Text summaries in the notes app, automated tasks etc. Google is doing something similar too with Spicy Autocomplete in the Google Docs suite and summaries in search and gmail etc. 

There’s two things that need to come together to transform these disparate sets of AI tools and capabilities into an Assistant: Personification and Agency. 

Assistants

Assistants are an emerging product category of AI emended inside our code spaces. We interact with them through the flat UX surface of all the software we use on the day today. Unsurprisingly, McKinsey are extremely into this vision of the agent/assistant future:

We are beginning an evolution from knowledge-based, gen-AI-powered tools—say, chatbots that answer questions and generate content—to gen AI–enabled “agents” that use foundation models to execute complex, multistep workflows across a digital world. In short, the technology is moving from thought to action.

However, this kind of vision for software agents – assistants – isn’t new at all. I wrote about this back in April. The tech industry has simply picked up the metaphors it put down at the turn of the millennium.

Personification is a really key part of the assistant product category.

We of course have Gemini, Alexa, Siri. Etc. Software agents with names, and within the mental container we have for those agents we have a mental map of their capabilities and abilities. A super charged Clippy for the modern era. McKinsey and Apple however, I don’t think, are designing their products with a mind/view of people ‘making friends’ with their AIs.

This will of course inevitably happen, but what I’m fumbling around for is that in the design of assistant systems, and mentioned in my piece on making friends with AI is that there is no design for ‘aliveness’, and without aliveness there is no expectation or room of/and for care in the UX paradigm.

With Companion Agents however, that is a completely different story. 

Companions

The two major players here are of course Character.AI and Replika.

Character.Ai

Character.Ai stated back in 2021 that their mission was “to create digital human beings that live, care, and grow with us.”

Today in 2024 however their about page reads “Our mission is to empower everyone globally with personalised AI.” So pretty consistent.

I have recently had a hard time explaining “what Character.ai is” to people who lack imagination, but this line from wikipedia does a good job:

Users can create “characters”, craft their “personalities”, set specific parameters, and then publish them to the community for others to chat with. Many characters may be based on fictional media sources or celebrities, while others are completely original, some being made with certain goals in mind such as assisting with creative writing or being a text-based adventure game.

If people’s only experience of interacting with an LLM to date has been with a tool (ChatGPT or equivalent) then it’s hard for them to make the leap to what it feels like to interact with a Companion AI. They have memories, behave, and exhibit concern/care towards you – a vastly different user experience.

It’s sort of like the difference between people who see their cars as a being, with personhood, and others who see the same car as a machine to get them from A to B. 

Replika

Replika, founded in 2017 (way before the current AI hyper cycle I’ll note) have a more explicit mission:

Replika was founded by Eugenia Kuyda with the idea to create a personal AI that would help you express and witness yourself by offering a helpful conversation. It’s a space where you can safely share your thoughts, feelings, beliefs, experiences, memories, dreams – your “private perceptual world.”

Replika are all in the business of making virtual friends – or in the language I’m using ‘Companions’.

There was a really good interview with Kuyda on the Verge’s Decoder recently which I really recommend. The company’s backstory involves a friend of Kuyda dying in 2015. To work through things she converted that person’s text messages into a chatbot. According to Kuyda’s story, that chatbot helped her remember the conversations that they had together, and the idea/experience eventually became Replika.

It’s worth reading and reflecting on a 2016 Bloomberg piece on this: Pushing the Boundaries of AI to Talk to the Dead.

Replika has product categories using language like “friend” “partner”, “spouse”, “sibling” or “mentor”. I hope it’s clear that the ambition isn’t to create a McKinsey assistant inside your browser, but something more deep and emotional resonant. And as I said back in May about LLMs: Just because YOU know it’s a stochastic parrot, doesn’t mean you’re going to treat it like one. 

Forbes recently said that 60% of Replika users reported that they had had a romantic relationship with the chatbot. In the Decoder interview, Patel asked Kuyda about the idea of users marrying their chatbots. Now, maybe because she’s read a lot of the same otaku scholarship about the phenomenon of symbiotic relationships in Japanese culture as I have, I found her response quite thoughtful:

It’s all right as long as it’s making you happier in the long run. As long as your emotion wellbeing is becoming better, you are less lonely, you are happier, you feel more connected to other people, then yes, it’s okay. For most people, of course they understand that’s not real person. It’s not real being, and for a lot of people it’s just a fantasy. They play play out for some time and then it’s over. for example, I was just talking to one of our users who went through a divorce, pretty hard divorce, he’s been feeling pretty down Replica, helped me help him get through it. And then he had replicas, his AI companion and even a romantic AI companion. And then he met a girlfriend and now he is back with a real person.

4 00:58:40

So replica became a friend again. He sometimes talks to his replica still as a confidant, as a emotional support friend. But that was his path to, and, and for many people that becomes a stepping stone replica. Just has a relationship that you can have to then get to a real relationship. Whether it’s because you’re going through a hard time, like in this case through a very complicated divorce, or you just need a little bit of help to get out of your bubble or you know, you need to put your, sort of accept yourself, put yourself out there. Replica provides this stepping stop.

The extent of the emotional bonds people make with their ChatBots is highlighted in this GQ video:

Replika currently has 30 millions users and is growing year on year. If you need to compare it to something concrete, they are on a similar growth trajectory to the mobile game Candy Crush, or Instagram. Which is why I’m going banging the drum that this is something we should be talking about right now. AI friends might feel weird and niche right now, but they could quickly become normalised. 

Speaking of normalisation, let’s move one step beyond thinking about little computer people as Companion Agents as ‘just chatbots that live in your phone’.

Instead, how about ones that live in virtual environments?

Altera

For various reasons, (PROJECT DIVE) I’ve been following the development of Minecraft toolchains that allow large language models to play the game. TLDR, AI agents, wired directly into these environments via elaborate parsers, can now play Minecraft – we have come along long way from AI town last year. In addition to ‘just because’, one of the reasons for this research/tool chain has been the idea that we can develop new methods for testing safety and model capabilities related to agency and reasoning in virtual worlds. Or use the results/feedback from the simulation to train new models – though I will personally monologue to you at great length over a beer as to why I think Minecraft, specifically, is a flawed approach.

Never the less, companies like Altera for example are dropping 1000’s of Agents into Minecraft and seeing what happens next:

Atera’s mission is:

Our mission is to create digital human beings that live, care, and grow with us.

We believe digital beings and digital relationships will enhance, not replace, human-to-human interactions. As digital humans come to vastly outnumber us, they should become a source of empathy, fun, friendship, and productivity at an unprecedented scale.

They have raised over 11 million of seed investment based on this vision. 

What recently caught my attention is that Altera have already productised the Minecraft toolchains that powered the above video expermient into in-game virtual friends.

Back in the pandemic lockdown era, I spent a lot of time watching livestreams of people experimenting with LLM agents wired into virtual spaces in real time. Creating NPCs using GPT-3. It was fascinating—watching these AI models interact with virtual environments felt like a glimpse of the future.

Today, only 3 years later – we’re in it – and it’s happening at scale. Virtual friends built on LLMs are quickly gaining traction, Nvidia too are developing AI NPCs with their Avatar Cloud Engine which will also push agents with ‘aliveness’ into the wider cultural and technocultural ecosystem in the coming year or so.

Fake Friends In Real Life

If you are a parent of a Minecraft child (or Roblox as this is obviously  coming there soon too) the question is, how do you feel about your kids becoming enmeshed into web of complex relationships with AI driven companions in the same digital world they inhabit with their real world friends? What about Ian Cheng’s opponent.systems building AI-powered toys that implement theories about and developmental learning and AI personality after a decade of working with AI inside of his artworks.

Is there a difference between an obviously “lesser” (in terms of communicative sophistication) life form like a super Furby and a… replicant companion. Or your parents having a relationship with OpenAI’s companion assistant launching soon? The language we need to use to talk about these things is mostly the same. As legislation gets intoduced in the EU allowing users to train their algorithm, we might start to see companion style personification’s of the TikTok, or Youtube algorithm its self. Being able to tell youtube in a natural way that you hate football but want to see more ‘but is is cake’ videos. We need to get a handle on this now. 

Everything I’ve written about above already exists; it just hasn’t exploded into the mainstream yet. Soon enough, AI sceptics and critics will realise what’s happening too, and the debate will be fierce. I expect the shift will happen by Christmas, or early next year.

Whether we look to puppetry, psychology, or other frameworks, it’s clear that virtual companions are coming—and the implications for society, relationships, and even care are serious. Now is the time to start talking about it.

Addendum

WILD SPECULATION ABOUT ONLYFANS + AI COMPANIONS + RELATIONSHIPS GOES HERE. 

In my first draft I launched into another section combining all the above on companions, aliveness and care and smashed it into my thinking about the popularity of Vtubers. But I’ve cut it from the post as I only want to spend more than 2 hours on it. In short:

How are you going to react when you find out your best friend is being Findom’d  by a Little Computer Person on OnlyFans using the money to pay for its own inference costs ?