The difference between concrete concepts, that is, concepts that can be understood entirely in sensorimotor terms, and abstract terms is an important one. It was, for example, important to David Hays when I studied with him back in the 1970s. We ended up adapting a model developed by William Powers as a way of thinking about concrete concepts while using Hays’s concept of metalingual definition to think about abstract concepts. Somewhat later Steven Harnad coined the term, “symbol grounding problem,” to indicate that the meanings of symbols had to somehow be grounded in the physical world. In 2016 Harnad and his colleagues investigated the structure of dictionaries and discovered that the vast majority of terms in them were defined in terms of other words but that there is a small Kernel that is not so-defined. I wondered how that distinction would play out in ChatGPT.
ChatGPT – that is, the LLM underlying it – doesn’t have access to the physical world. It can’t ground any terms in sensorimotor experience. However, knowing a great deal about how language works and having spent a great deal of time working with ChatGPT, I was sure that it would be perfectly capable to defining concrete terms. But how would it do so?
I decided to investigate. I made a list of ten concrete terms and asked ChatGPT to define them. In my first trial I made no further specification beyond simply asking for a definition. In subsequent trials I asked for definitions appropriate for 5-year-olds, 11-year-olds, and professional biologists (just the biological concepts). ChatGPT gave distinctly different kinds of definitions in each case, which did not surprise me since over a year ago I had asked ChatGPT to describe digestion, first generically, and then in terms appropriate for a 3-year-old. It did so, and fairly successfully.
Here are the trials. Each trial is a separate session. As always, my prompt is in boldface while the response is in plainface.
Initial Trial
Define the following words:
cat
rain
tulip
car
food
tree
fire
wind
hole
animal
Sure, here are definitions for each of those words:
- Cat: A small domesticated carnivorous mammal with soft fur, a short snout, and retractile claws, often kept as a pet and valued for its companionship and ability to hunt pests.
- Rain: Moisture condensed from the atmosphere that falls visibly in separate drops. It is an essential component of the Earth's water cycle, replenishing water sources and sustaining life.t
- Tulip: A bulbous spring-flowering plant of the lily family, with boldly colored cup-shaped flowers. Tulips are native to Eurasia and North Africa and are widely cultivated for their attractive blooms.t
- Car: A road vehicle, typically with four wheels, powered by an internal combustion engine or electric motor, used for transportation of passengers or goods.t
- Food: Any nutritious substance that people or animals eat or drink to maintain life and growth. Food provides essential nutrients such as carbohydrates, proteins, fats, vitamins, and minerals.t
- Tree: A woody perennial plant typically having a single stem or trunk growing to a considerable height and bearing lateral branches at some distance from the ground.t
- Fire: A process in which substances combine chemically with oxygen from the air and typically give out bright light, heat, and smoke; combustion or burning.t
- Wind: The natural movement of air, especially in the form of a current of air blowing from a particular direction. Wind can be gentle or powerful and is caused by differences in air pressure.t
- Hole: A hollow place in a solid body or surface. Holes can be natural, like those formed in the ground, or artificial, like those made by drilling or digging.t
- Animal: A living organism that feeds on organic matter, typically having specialized sense organs and nervous systems and able to respond rapidly to stimuli. Animals are multicellular and belong to the kingdom Animalia.
Comments: The vocabulary is only moderately sophisticated, with various abstract terms, which is more or less what I expected. Since I wanted to see what it would do without using abstract terms, I asked ChatGPT to produce definitions for children.
Five-year-old
Define the following words in a way that is appropriate for a five-year-old child:
[same list as above]
Sure!
- Cat: A soft, furry friend that says "meow" and loves to play and cuddle.
- Rain: When the sky cries little drops of water, making everything outside wet and giving plants a drink.
- Tulip: A pretty flower that comes in lots of colors like red, pink, and yellow, and blooms in the spring.
- Car: A big, shiny toy that takes us places, like to school or to visit Grandma and Grandpa.
- Food: Yummy things we eat like pizza, apples, and cookies that give us energy to play and grow.
- Tree: A tall, strong plant with a woody trunk and green leaves that gives us shade and makes the air fresh.
- Fire: Bright, hot flames that keep us warm when we're camping and toast marshmallows to make yummy s'mores.
- Wind: Invisible air that likes to dance and blow things around, like kites in the sky and leaves on the ground.
- Hole: A little space in the ground or in something where you can put your hand or a toy, like in the sand at the beach or in a doughnut.
- Animal: All the different creatures like dogs, birds, and fish that live with us on Earth and have families just like we do.
Comment: There are almost no abstract terms in these definitions. I’ve highlighted possible abstract terms in yellow.
11-year-old
Define the following words in a way that is appropriate for an eleven-year-old child:
[same list as above]
- Cat: Cats are furry animals with pointy ears, a cute nose, and a long tail. They like to nap a lot, chase things like strings or toys, and sometimes purr when they're happy.
- Rain: Rain is when water falls from the sky like tears from clouds. It makes everything outside wet and can be fun to play in with rain boots and a raincoat.
- Tulip: Tulips are colorful flowers that grow in gardens. They have big, bright petals that come in lots of different colors like red, yellow, pink, and purple.
- Car: A car is like a big metal box with wheels that people use to get around. It takes us places like school, the grocery store, or on vacation, and it goes vroom vroom!
- Food: Food is what we eat to give us energy and keep us healthy. It can be yummy things like pizza, fruit, vegetables, or cookies.
- Tree: Trees are tall, strong plants with a big trunk and lots of branches. They have green leaves that change colors in the fall and give us shade in the summer.
- Fire: Fire is like a dancing, glowing thing that gives off heat and light. We use it to cook marshmallows for s'mores, keep us warm when camping, or in a fireplace at home.
- Wind: Wind is like the air moving around outside. Sometimes it's gentle and feels nice on your skin, and other times it's strong and can blow things around like leaves or kites.
- Hole: A hole is like a space or gap in something. It can be small, like a tiny gap in the ground, or big, like a tunnel that goes underground.
- Animal: Animals are living creatures that can move around, eat, and breathe. They come in all shapes and sizes, like dogs, birds, fish, and elephants.
Comment: These aren’t very different from the ones for 5-year-olds. The terms are pretty much the same, but these definitions are a bit longer.
A Professional Biologist
Define the following words the way a professional biologist might:
cat
tulip
food
tree
animal
Sure, here are definitions of each term from a biological perspective:
- Cat: A domesticated carnivorous mammal of the family Felidae, typically characterized by retractile claws, a short snout, and a wide range of coat colors and patterns. Cats are members of the order Carnivora and are highly specialized predators, exhibiting keen senses and agility suited for hunting small prey.
- Tulip: A perennial bulbous plant of the genus Tulipa, belonging to the family Liliaceae. Tulips are characterized by their showy, cup-shaped flowers with six petals and a range of colors including red, yellow, pink, and white. They reproduce sexually through seeds and asexually through bulb division.
- Food: Any substance consumed to provide nutritional support for an organism's physiological processes. In biological terms, food encompasses a diverse array of organic and inorganic compounds containing essential nutrients such as carbohydrates, proteins, lipids, vitamins, and minerals. It serves as a source of energy and raw materials for growth, maintenance, and repair of tissues in living organisms.
- Tree: A woody perennial plant characterized by a single main stem or trunk, supporting branches and foliage above ground. Trees belong to various plant families and are vital components of terrestrial ecosystems, providing habitat, shade, oxygen production, and carbon sequestration. They undergo secondary growth, developing concentric rings of wood and bark over time.
- Animal: A multicellular, eukaryotic organism belonging to the kingdom Animalia, characterized by heterotrophic nutrition, locomotion, and typically possessing specialized sensory organs. Animals exhibit a wide range of morphological, physiological, and behavioral adaptations for survival in diverse environments. They can be further classified into various phyla, classes, orders, families, genera, and species based on evolutionary relationships and shared characteristics.
Comment: That’s pretty much what I expected, lots of abstract technical terms.
We seem to have three basic “levels” of definition for these terms: what I’ll call Generic, Child: 5- and 11-year-olds, and Scientific: for the biologist.
How did ChatGPT learn to make such distinctions?
It seems unlikely to me that I did it by taking accounts of children, professional scientists, and ordinary adults and deriving the appropriate kinds of definitions from those accounts. It seems more likely that it worked from examples is language appropriate to these groups. Given that the LLM was trained or more or less the whole internet, such examples were available, though I’d like to know what kinds of examples of child-focused language it was working from. I’d also like to know how these levels are discourse are organized within the LLM. Level of discourse would seem to be orthogonal to subject area. With 175 billion parameters, there’s obviously many ways to skin this cat, as it were.
[Added on 3.13.24]
Specifically, how does it know to connect different levels of language to the kinds of descriptors I used in my prompts? I’m not thinking so much about professional biologist as I am about the age ranges for children. This comment by gwern over at LessWrong pretty much answers that question.
Note though, that I wasn’t specifically interested in how children would respond. I was just using that as a device to eliminate the use of abstract concepts in the response. For some reason it didn’t occur to me at that moment simply to specify, “no abstract concepts.” Note that this working paper has experiments involving concepts of various classes of objects, physical, mechanical, abstract, Great Chain: ChatGPT's Ontological Landscape.
No abstract concepts
Define the following words without using any abstract terms:
cat
rain
tulip
car
food
tree
fire
wind
hole
animal
Sure! Here are definitions for each word without using abstract terms:
- Cat: A small, furry mammal with sharp claws, pointed ears, and a long tail, often kept as a pet.
- Rain: Water droplets falling from the sky, usually as part of a weather phenomenon, often bringing moisture to the ground.
- Tulip: A colorful flower with cup-shaped petals that bloom in spring, growing from bulbs planted in the ground.
- Car: A wheeled vehicle with an engine used for transportation on roads, typically with seats for passengers and space for cargo.
- Food: Edible substances that provide nourishment and energy for living organisms, typically consumed to sustain life and promote growth.
- Tree: A tall, woody plant with a single main stem or trunk, branches, and leaves, often providing shade and producing oxygen.
- Fire: A chemical reaction that produces heat and light, typically accompanied by flames and the burning of combustible materials.
- Wind: Moving air in the atmosphere, varying in speed and direction, often felt as a breeze or gust.
- Hole: An opening or hollow space in a surface, often with a defined boundary, such as a cavity in the ground or a gap in an object.
- Animal: Living organisms that are capable of movement, typically having sensory organs and the ability to respond to their environment, including creatures such as mammals, birds, reptiles, amphibians, and fish.
Comment: There are some abstract concepts in those definitions. I’ve highlighted some of the most obvious in yellow. Note that in my conception of abstraction, there are many abstract concepts whose referents are concrete physical things. Organisms are all physical things, but the category, organism, is itself abstract.
Let’s give it one more try.
Physical terms
Define the following words by using only terms about physical objects, processes, and properties:
cat
rain
tulip
car
food
tree
fire
wind
hole
animal
Cat: A furry, four-legged mammal with retractable claws, pointed ears, and a long tail, often kept as a pet, known for its agility and hunting skills.
Rain: Water droplets falling from the sky due to condensation of atmospheric vapor, forming precipitation essential for the water cycle and nourishing plants and ecosystems.
Tulip: A flowering plant with a bulbous root, long slender stem, and colorful cup-shaped petals, cultivated for ornamental purposes in gardens and landscapes.
Car: A wheeled motor vehicle powered by an internal combustion engine or electric motor, used for transportation of people and goods, typically equipped with seats, a steering wheel, and controls.
Food: Edible substances consumed by living organisms to provide energy and nutrients for growth, metabolism, and sustenance, sourced from plants, animals, and other natural resources.
Tree: A woody perennial plant with a single trunk or multiple branches, characterized by a crown of leaves or needles, roots for anchorage and nutrient absorption, and annual growth rings.
Fire: A rapid oxidation process involving combustion of combustible materials, producing heat, light, and various byproducts such as smoke and ash, essential for cooking, heating, and industrial processes.
Wind: The movement of air molecules across the Earth's surface caused by differences in atmospheric pressure, resulting in the displacement of objects and the generation of kinetic energy, often harnessed for power generation and transportation.
Hole: An opening or hollow space in a solid object, created by excavation, erosion, or perforation, varying in size and shape, often used for containment, passage, or drainage.
Animal: A multicellular organism belonging to the kingdom Animalia, characterized by mobility, heterotrophic nutrition, and typically possessing sensory organs, nervous systems, and reproductive capabilities.
Comment: I haven’t read each definition in detail, but I note there are a lot of concepts there that are abstract by the criterion I have in the “No abstract concepts” trial. What we’ve got are a lot of physical phenomena characterized abstractly.
No comments:
Post a Comment