Showing posts with label theory of mind. Show all posts
Showing posts with label theory of mind. Show all posts

Saturday, October 28, 2023

No, LLMs do not have a so-called Theory of Mind

Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, Maarten Sap, FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions, EMNLP 2023.  

Abstract: Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM 👻, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering. Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that demand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANToM is challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.

Wednesday, March 20, 2019

Primate mind-reading


Thursday, February 9, 2017

Laughter in Infants

Gina Mireault has an article about infant laugher in Aeon. Infants show laughter about about four months. And then they use it:
For example, infants can employ fake laughter (and fake crying!) beginning at about six months of age, and do so when being excluded or ignored, or when trying to engage a social partner. These little fake-outs show that infants are capable of simple acts of deception much earlier than scholars previously thought, but which parents knew revealed infants’ cleverness. Similarly, the psychologist Vasu Reddy of the University of Portsmouth has found that, by eight months, infants can use a specific type of humour: teasing. For example, the baby might willingly hand over the car keys she’s been allowed to play with, but whip her hand back quickly, just before allowing her dad to take possession, all the while looking at him with a cheeky grin. Reddy calls this type of teasing ‘provocative non-compliance’. She has found that eight- to 12-month-olds use other types of teasing as well, including provocative disruption, as in toppling over a tower someone else has carefully built.

Teasing is the infant’s attempt to playfully provoke another person into interacting. It shows that infants understand something about others’ minds and intentions. In this example, the infant understands that she can make her father think that she will relinquish the car keys. The ability to trick others in this way suggests that infants are maturing toward a Theory of Mind, the understanding that others have minds that are separate from one’s own and that can be fooled. Psychologists have generally thought children don’t reach this milestone until about four and a half years of age. Infants’ ability to humorously tease reveals they are progressing toward a Theory of Mind much earlier than previously thought.

Saturday, September 3, 2016

Rant: Theory of Mind, NOT!

I posted this back in 2010, but I'm thinking about these things these days, so I thought I'd bump it to the top of the queue. Here's a companion piece from last year.
Theory of mind (aka TOM) is all the rage in (some quarters of) cognitive science and evolutionary psychology. It’s driving me batsh¡t crazy. And its use by literary critics makes me super-mega batsh¡t crazy.

Why? By the time literary critics use all that's left is the term itself, undisciplined by the observations that gave rise to the idea. For literary critics it's a "get out of (conceptual) jail free" card.

Let me explain

Well, first, in case you don’t know what TOM is, it refers to the capacity humans have for attending to and wondering about what’s on someone else’s mind. It’s a capacity that’s possibly unique to humans, though perhaps not, and it begins appearing at around four-plus years of age. My problem is not with the research itself or the notion that some-such capacity comes “online” at that point in development. What bothers me is the term and its implications.

It bothers Melvin Konner too. Here’s a passage from an opinion piece* he published in Nature a few years ago:
Meanwhile, social-cognition theorists have come up with a phrase inferential enough to make one almost long for the black-boxers: theory of mind. Freud sought one, Skinner assiduously didn’t, and most people don’t bother to ask themselves whether they have or need one. Yet there is serious debate as to whether chimpanzees or four-year-olds have a theory of mind. Closely inspected, the phrase seems to mean something like perspective-taking or, when mutual, intersubjectivity. True, a four-year-old can see and act on another person’s perspective whereas most three-year-olds can’t.

This is fascinating stuff and something we need to understand. But a term such as ‘theory of mind’ simply stands in the way. It makes for catchy article titles but conveys no meaning. Is the maturing orbitofrontal cortex newly able to calm an impulsive and self-centred limbic circuit? Is there a down-regulation of some neurotransmitter receptor, allowing a younger form of social mirror-imaging to grow into identification and parallel perspectives? As long as we are playing with pretty word-coins that substitute for brain functions, we will never know.
This TOM-talk is rather like Richard Dawkins talking about “selfish” genes. He knows perfectly well that genes aren’t the kind of agents that can be motivated by selfish considerations, but it’s a useful way of talking. And, in a pinch, he’s quite capable of explaining what’s going on without recourse to the personification; that is to say, Dawkins and others can give technical accounts that do not require genes to have mental states.

Similarly, the psychologists don’t believe that four year-old children are reasoning about the mind in the manner of philosophers or cognitive scientists. It’s not that kind of theory they’re imputing to the child. But this odd usage of “theory” is, in the end, grounded in experimental evidence and psychologists can talk and reason about those experiments. Konner’s point, of course, is that too ready explanatory recourse to TOM-talk is likely to get in the way of deeper theoretical investigation.

Wednesday, October 7, 2015

Posturing Robots and Infants: Contra Theory of Mind

Theory of Mind (aka TOM) has been a big deal in psychology for well over a decade, and it’s crept into literary criticism too. The psychological research behind TOM is important and fascinating, but the term itself is unfortunate. As I said some time ago in an email to some colleagues:
I find the way literary critics use the notion of theory of mind to be somewhat problematic in that it doesn't have much to do with the psychology they cite, especially when it gets reified into a theory of mind module, which it sometimes does.

Here's the problem. First, I think the phrase itself is unfortunate, because it promises a lot; but I understand that "theory of X" is a common developmental psych way of thinking about this or that cognitive behavior in children. What's important is that the usage is grounded in, given meaning by, a wealth of observations. That's certainly the case with TOM. And those observations, as far as I know, typically involve either actual face-to-face interaction, or situations where the (human) subject can examine either dolls in a play environment, or some picture. So, TOM behavior is something one observes in a selected class of physical situations.

What's the physical situation of the reader of a book? They're reading written words. They're not interacting with someone who is physically present, nor are they playing will dolls or looking at pictures. They might well imagine character interactions in their mental theatre, but they're doing that, it's not something before them. Beyond that, what's the physical situation of characters in stories? Are they gaze following, for example?

As far as I can tell, all the literary critic takes over from the TOM literature is the term itself. Nothing else. In particular they are not taking explicit mechanisms and putting those mechanisms through their paces in a literary context. The reason I say that is because the TOM literature I'm familiar with doesn't have explicit mechanisms, though some of the literature does break TOM into modules (I'm thinking of Simon Baron-Cohen on mind blindness).
Concerning the name, Melvin Konner made that point in an opinion piece, “Bad Words”, that he published in Nature (Vol. 411, 14 June 2001, p. 743) over a decade ago:
Meanwhile, social-cognition theorists have come up with a phrase inferential enough to make one almost long for the black-boxers: theory of mind. Freud sought one, Skinner assiduously didn’t, and most people don’t bother to ask themselves whether they have or need one. Yet there is serious debate as to whether chimpanzees or four-year-olds have a theory of mind. Closely inspected, the phrase seems to mean something like perspective-taking or, when mutual, intersubjectivity. True, a four-year-old can see and act on another person’s perspective whereas most three-year-olds can’t.

This is fascinating stuff and something we need to understand. But a term such as ‘theory of mind’ simply stands in the way. It makes for catchy article titles but conveys no meaning. Is the maturing orbitofrontal cortex newly able to calm an impulsive and self-centred limbic circuit? Is there a down-regulation of some neurotransmitter receptor, allowing a younger form of social mirror-imaging to grow into identification and parallel perspectives? As long as we are playing with pretty word-coins that substitute for brain functions, we will never know.
I am thus pleased to be able to point to some recent research the indicates another way of thinking about such matters. Christian Kliesch has an interesting post at Replicated Typo, Posture Helps Robots Learn Words, and Infants, Too.
The word learning task used in this study was the Baldwin task: Two objects are presented to the infant multiple times. However, they are not named in the presence of the object. Instead the experimenter hides both objects in two buckets, then looks at one bucket and names the object (e.g. “Modi”). Then the two objects are taken out of their containers, put on a pile, and the child is asked to pick up the Modi. Children as young as 18-20 months do fairly well in this task, and their high performance has generally been interpreted as evidence that they have used a form of mind reading or mental attributions to infer which object is the Modi, as the object and the word do not appear simultaneously.

However, proponents of non-mentalistic approaches to language acquisition have come up with alternative explanations. For example, a previous study by Samuelson et al., (2011) has found that spatial location can greatly contribute to word learning in infants and in computer simulations using Hebbian learning. Hebbian learning is a form of associative learning which is loosely based on the way the neurons in our brain are assumed to learn as well. This very simple way of associative learning does not take into account the intention of the speaker at all, instead the model used by Samuelson et al. only takes into account the spatial location over time.
By “form of mind reading or mental attributions” read TOM. I urge you to read this post and then to check out the study it reports:
Morse A.F., Benitez V.L., Belpaeme T., Cangelosi A., Smith L.B. (2015) Posture Affects How Robots and Infants Map Words to Objects. PLoS ONE 10(3): doi: 10.1371/journal.pone.0116012
The interesting thing, of course, is that a simple robot can match an infant’s performance of this task. No one is about to attribute to TOM to this robot;

The results of both, robot and infant data, suggest that infants may not need to accurately represent the speaker’s intent when learning words, but are able to make the correct word-meaning associations based on visual and spatial information. Furthermore, they do not seem to use complex mental representations in the interference condition and the posture change tasks, in which the posture change has a detrimental effect on children’s word learning.