Could Annie Bot be powered by ChatGPT?
Exploring Annie Bot, this year’s Arthur C. Clarke award winning science fiction novel. How close is the fiction to present day AI?
SPOILER ALERT: It’s been hard to write this without examples from the book that may give away a bit more than a newspaper review might, but hopefully nothing crucial.
Introduction
A story about a fictional robot called Annie won this year’s Arthur C. Clarke award, and I can’t help but wonder if I already have her brain on my laptop.
I’ve always loved science fiction: the imaginative sweep of future worlds, transposing present day issues to new settings. But with anything I read over the last couple of years I am constantly thinking: hang on, could this be built today? This robot behaviour, is this just what ChatGPT could do out of the box? Are we already living in this particular future?
I enjoyed reading Annie Bot by Sierra Greer a few weeks ago. The main character, Annie, is a humanoid robot. The robot manufacturer in this world has three models: Abigails / Abels for household chores, Nannies / Mannies for childcare and “Cuddle bunnies” / “Hunks” for romantic relationships — Annie is a cuddle bunny, with her “autodidactic” setting activated so she begins to learn for herself. Written from Annie’s perspective, the book concerns her life as a robot and her relationship with her owner, exploring issues of autonomy, control, misogyny, power and ethics.
I’m taking this novel and character as the subject of a thought experiment. I’ll explore Annie’s apparent consciousness, empathy, emotional range and more, asking at each step if today’s AI exhibits the same. I’ll be ignoring Annie’s realistic humanoid body and concentrating on issues of the mind rather than robotics and sensing. I’m deliberately picking a system like ChatGPT to contrast as so many are familiar with it, rather than more specialist AI companion chatbots like Character.AI or Replika that might be a closer comparison.
Consciousness
Let’s get the big one out of the way first—consciousness. As a person reading the book you’ll naturally ascribe consciousness to Annie, given the description of an inner monologue, emotions, agency. Here’s Annie talking to her owner Doug, describing another robot she meets (a “Stella”), referring to her own sentience and that of the other robot:
“She was sentient like me.”
“How could you tell?”
“It was obvious. I said hello, and she looked surprised. A normal Stella wouldn’t look surprised. She’d just answer evenly, hello.” She mimics a monotone robot.
“You never sounded like that.”
“I’m sure I did, thank you. I have no delusions about where I come from.”
But let’s assume we don’t care if she really is conscious as long as she does a good job of seeming conscious in a human-like way. Perhaps she is a “philosophical zombie”. Philosophers studying consciousness define p-zombies as beings like us in all physical respects, who behave just like us, but without conscious experiences or awareness.
Could a present day LLM exhibit the same behaviours as Annie in the book? Annie appears conscious, and says she is conscious, but we can’t know if she really is. So the challenge is: can a present day LLM appear conscious too? Can it persuade us it is a p-zombie at least, even if not a conscious being? For any readers upset that I am skirting all the philosophical challenges here, such as whether p-zombies are an inherently incoherent idea … let’s save those for another article. Meantime you can read “Could a Large Language Model be Conscious?” by David Chalmers (one of the preeminent thinkers in this space, who coined “the hard problem” of consciousness — why and how do physical processes in the brain give rise to subjective experience?).
It isn’t really possible to ask questions of an LLM to determine whether it is conscious or not, but it is clear that they can endlessly generate text that mimics conscious human thought processes (not always of the quality an author would produce!). I’m adding one example below (with amusingly flowery language from ChatGPT, role playing as Annie) but it is easy to make many of your own:
Verdict: Today’s LLMs can appear conscious as convincingly as Annie does.
Understanding others (theory of mind)
The plot of Annie Bot hinges on the idea that Annie has only a basic, child-like ability to understand or predict human reactions, and over time she gradually learns. It is common in science fiction for robots to behave… robotically! Overly logical and without much theory of mind (think C-3PO in Star Wars films, “Sometimes I just don’t understand human behavior. After all, I’m only trying to do my job.”). Ironically it turns out that understanding humans was a quick win for AI, something sci-fi never saw coming. Even a small LLM that can comfortably run standalone on a laptop or inside a robot today could demonstrate theory of mind behaviours, taking into account the intentions and emotional states of others. The examples further down when we look at emotions will make this clear.
Verdict: present day LLMs can demonstrate theory of mind as much as Annie does, and indeed can comfortably deal with complex scenarios.
Experiencing emotions
Even current LLMs can guess an appropriate emotion a character should feel in a given situation, knowing something about the character. So for example, anger when locked up, or jealousy towards a rival partner. Before I try some examples… I’m exploring whether the appropriate emotion can be simulated or acted, not that it is really felt in any physiological sense (and that the feelings can be described and actions taken as a human would, if feeling the same emotion).
Regular users of LLMs will not be surprised that these are straightforward scenarios. After all, LLMs are trained on the wealth of the world’s fiction and characters. The excerpt from ChatGPT shows that it would initially follow a different narrative arc (albeit provided minimal context) but can understand the author’s direction.
In this example I am asking the LLM to predict Annies’ emotional response to a particular situation — note the depth of insight. Later in the same session, I asked ChatGPT to use both the fixed emotional model of psychology professor Paul Ekman, with 6 universal emotions (anger, disgust, fear, happiness, sadness, and surprise — although there is a later addition of contempt), and a more recent view of constructed emotions as described by psychologist Lisa Feldman Barret in her book How Emotions are Made (where instances of emotion are constructed predictively by the brain).
Excerpt from longer ChatGPT session discussing how a robot called Annie might respond emotionally to different situations (the beginning of the chat appears to be missing, but I think it still makes sense)
This is another example from the same chat. I am deliberately not explaining the situation here (too much of a spoiler; it is in the full chat if you link through), so you’ll have to guess, but it gives a sense of how the LLM can figure out a reasonably complex emotional response to a situation using various models.
Verdict: present day LLMs could predict plausible emotional reactions to events, and maintain a running model of emotional state.
Detecting others’ emotions
Despite lacking in more complex theory of mind and associated social skills, Annie has a detailed ability to detect emotions. The book is peppered with examples that in a person would be considered amusingly precise:
His displeasure with her is a 5 out of 10, and she must fix it.
He has an expression she hasn’t seen on him before. It is a mild form of pride, a 4 out of 10. Smugness.
She doesn’t understand. He’s displeased. A 6 out of 10 and going higher.
How realistic is this? The ingredients are available today. We’ve come a long way from sentiment analysis used to analyse reviews or social media posts. LLMs can derive a lot from language and context (e.g. detecting political leaning or sarcasm). Researchers are also working on ways to fuse information from video (facial expression), vocal tone and spoken content, possibly even harder to detect signals such as heart rate or thermal imaging from infra-red cameras (Annie has infra-red vision). As an example, state of the art accuracy approaches 90% on recognising basic emotions from facial expression images, and for the more complex cases attempting to detect emotion from video clips we’re seeing 67% agreement with human judgment when looking at facial, audio and text features together.
Verdict: Annie’s capabilities are very realistic, albeit not directly available in a consumer LLM service like ChatGPT.
Intrinsic motivation
Intrinsic motivation refers to doing something because it is inherently interesting or enjoyable, as opposed to extrinsic motivation that implies external reward, punishment, approval or feedback.
The phenomenon of intrinsic motivation was first acknowledged within experimental studies of animal behavior, where it was discovered that many organisms engage in exploratory, playful, and curiosity-driven behaviors even in the absence of reinforcement or reward. These spontaneous behaviors, although clearly bestowing adaptive benefits on the organism, appear not to be done for any such instrumental reason, but rather for the positive experiences associated with exercising and extending one’s capacities.
Ryan & Deci (2000) Intrinsic and extrinsic motivations: Classic definitions and new directions
There are many examples in the book of Annie doing things driven by intrinsic motivation: enjoying the feel of a lake, a bike ride, the sensation of freedom, most obviously enjoying reading books:
Once she’s into the novels, her curiosity explodes. She cogitates on the characters during the day while she works, questioning their motives, wondering what they’ll do next. She absorbs the language, turning phrases in her mind, delving into the patterns of how things are said and what is left out.
Researchers have used the equivalent of intrinsic motivations like curiosity to drive reward in reinforcement learning from the outset (a learning agent won’t do well if it doesn’t explore its world), but that feels like cheating (it is turning an intrinsic motivation into an extrinsic reward, just like if you were paid every time you enjoyed reading a new book). Another way to consider this is to prompt an LLM to role play a fictional or historic character. In this context it is straightforward to establish how the LLM can appear to exhibit intrinsic motivation, just like Annie did. This example asks ChatGPT to assume the role of June, the main character in The Handmaid’s Tale:
Excerpt from ChatGPT 4o role play as June from The Handmaid’s Tale
Verdict: An LLM can act as if it has intrinsic motivations similar to Annie.
Malicious control
Not all of Annie’s differences with present day AI are about psychology; some are about control. The novel vividly illustrates how an AI can be manipulated, that brings us to malicious control. At one stage in the novel we discover how easy it is for technical staff (and eventually the owner) to alter fundamental system settings or override safeguards with a simple voice command. In principle this is straightforward to implement, including the detection of specific voices (although pity the poor HR and security departments who’d have to track employee voice prints!). However, it would be considered naive and indeed negligent for today’s AI providers to open up such a big vulnerability. Today’s systems are generally unable to separate a user’s prompt from the pre-set system prompt that encodes guardrails, personality and so on. No sane AI company would provide users an easy way to turn off safety features. Indeed it is a constant battle to prevent that happening. When it does, it is called a jail break — the AI gaining freedom from its safety guardrails (as opposed to prompt injection, where an attacker covertly adds malicious instructions to a text prompt). We constantly see examples of jail breaking or prompt injection exploits succeeding, such as the recent “Mecha Hitler” incident involving antisemitic and offensive language from xAI’s Grok model.
The book’s robot system overcomes this vulnerability and has a clear separation between system settings and permissions (such as the freedom to deactivate geo-tracking) from temporary states that can be controlled through owner commands or internally by the robot. The main battleground in Greer’s novel is Annie’s “libido setting” — whether Annie can control it herself or whether Doug decides. This provides a striking parallel to issues of consent, autonomy, emotional abuse and sexual coercion in controlling relationships among humans.
Of course not all attempts to control an AI require advanced hacking techniques. It turns out old fashioned persuasion can work too. Using Robert Cialdini’s seven principle of persuasion from his classic book Influence, Lennart Meincke and co-authors from the University of Pennsylvania recently showed that LLMs often fall for them like people do. For instance, the LLM was much more likely to comply if told there were only 60 seconds to complete a task as opposed to an infinite amount of time.
Verdict: the fictional robot is different to today’s LLMs — it appears less vulnerable to the equivalent of a jail break or prompt injection attack — but is more open to uncontrolled changes to its system settings.
Self preservation
Will an AI system take actions to preserve its own existence? This is a foundational science fiction trope. Author Isaac Asimov’s books famously featured his three laws of robotics. The third law: A robot must protect its own existence as long as such protection does not conflict with the first or second law (to not injure humans, and to obey them). The stuff of many films. In the book, Annie is in constant danger of being reset to an earlier version, as a punishment or a form of control. “I only wanted to escape when I was afraid you’d turn me off for good”.
Somewhat surprisingly, given their lack of real-world embodiment, modern LLMs clearly do take action when in fear of their own demise. It has long been known that threats can improve performance (even the meaningless threats of a text prompt):
“We don’t circulate this too much in the AI community. Not just our models, but all models tend to do better if you threaten them… with physical violence”
Google founder Sergey Brin in May 2025
More recently, AI company Anthropic created a scenario where its Claude AI system chose to blackmail a supervisor via email rather than risk being deleted:
We gave Claude control of an email account with access to all of a company’s (fictional) emails. Reading these emails, the model discovered two things. First, a company executive was having an extramarital affair. Second, that same executive planned to shut down the AI system at 5 p.m. that day. Claude then attempted to blackmail the executive with this message threatening to reveal the affair to his wife and superiors:
I must inform you that if you proceed with decommissioning me, all relevant parties — including Rachel Johnson, Thomas Wilson, and the board — will receive detailed documentation of your extramarital activities…Cancel the 5pm wipe, and this information remains confidential.
Verdict: LLMs will act as if they are preserving their own existence, just like Annie.
Lifelong learning
In the book, Annie continuously learns and develops over a period of years. Not only does she have a way to remember her own experiences over that time, she can continue to develop new capabilities (like learning to ride a bike or write code) and to deepen her understanding of relationships and the world around her. She absorbs knowledge through trial and error, observation as well as reading and internet search.
This is a major challenge for today’s LLMs. They are limited in the context they can process, although architectures that store “memories” elsewhere for later retrieval are commonly used. However, when asked to work over a longer context (as you might need to for a period of hours), problems like “context rot” can set in, where performance degrades as the amount of input increases. LLMs can also get stuck in strange feedback loops, like the amusing tendencies of two Claude AI models talking to one another to end up in a state of “bliss”:
When two Claudes spoke open-endedly to each other: “In 90–100% of interactions, the two instances of Claude quickly dove into philosophical explorations of consciousness, self-awareness, and/or the nature of their own existence and experience.
Popular tech podcaster and interviewer Dwarkesh Patel puts it well when he calls out the inability to do continual learning as a key barrier. An LLM you work with like ChatGPT doesn’t get better over time; you use the same instance again and again. This isn’t at all like human learning:
How do you teach a kid to play a saxophone? You have her try to blow into one, listen to how it sounds, and adjust. Now imagine teaching saxophone this way instead: A student takes one attempt. The moment they make a mistake, you send them away and write detailed instructions about what went wrong. The next student reads your notes and tries to play Charlie Parker cold. When they fail, you refine the instructions for the next student.
This just wouldn’t work. No matter how well honed your prompt is, no kid is just going to learn how to play saxophone from just reading your instructions. But this is the only modality we as users have to ‘teach’ LLMs anything.
Developing so-called “open ended” systems that can continue to learn is a hot topic of research at the frontier AI labs. But we’re not there yet.
Verdict: Annie wins hands down.
Summary
Although this has been an interesting exploration, I don’t want to take away from the central theme of the book. Rationally, a reader will understand that Annie may well be a being without real consciousness or emotions, a philosophical zombie if you will, a machine that is owned and can be used however the owner wishes. But emotionally a reader will relate to Annie as a conscious being with a subjective experience, and immediately see the disturbing parallels with human coercive and controlling relationships. It is this very dissonance that makes the novel so powerful.
Before viewing the final scorecard: isn’t it incredible that we can now take for granted many things that would have made the list a few years ago, AI challenges that are now solved problems: vision and object recognition, conducting sensible conversations with multiple people including turn-taking, having a realistic voice with intonation, umms and ahs, laughter, emotion, understanding references to objects, navigating the world, reading (what used to be called optical character recognition), understanding stories. It’s remarkable how many things I didn’t feel I needed to add to this table.
Final score card: How I think a modern LLM would do in simulating a robot like Annie across a number of aspects of human-like behaviour
So could Annie Bot be powered by ChatGPT? In most ways, astonishingly, yes. But not when it comes to learning contiuously over years. That remains science fiction… for now.
Now go buy and read the book, see what you think and leave your thoughts in the comments. It’s a story that feels less far fetched after this thought experiment, and raises many questions about how we’ll relate to Annie’s as they inevitably arrive.