Our society needs to foster visual thinking if it is to thrive in a world where computers are ever more important

Temple Grandin, Temple Grandin: Society Is Failing Visual Thinkers, and That Hurts Us All, NYTImes, Jan. 9, 2022.

She opens:

When I was younger, I believed that everybody thought in photo-realistic pictures the same way I did, with images clicking through my mind a little bit like PowerPoint slides or TikTok videos.

I had no idea that most people are more word-centric than I am. For many, words, not pictures, shape thought. That’s probably how our culture got to be so talky: Teachers lecture, religious leaders preach, politicians make speeches and we watch “talking heads” on TV. We call most of these people neurotypical — they develop along predictable lines and communicate, for the most part, verbally.

I was born in the late 1940s just as the diagnosis of autism was being applied to kids like me. I had no language until age 4 and was first diagnosed as brain damaged. Today, many people would say that I’m neurodivergent — a term that encompasses not only autism but also dyslexia, A.D.H.D. and other learning problems. The popularization of the term neurodivergence and society’s growing understanding about the different ways that brains work are unquestionably positive developments for many individuals like me.

Neurodivergence in the workplace:

For over 25 years, I designed equipment to handle livestock and worked with the highly skilled people who built the equipment. When I look back at all the projects I designed for large companies, I estimate that 20 percent of the skilled welders and drafting technicians were either autistic, dyslexic or had A.D.H.D. I remember two people who had autism and held numerous patents for mechanical devices they invented and sold equipment to many companies. Our visual thinking skills were key to our success. Schools:

I often get asked what I would do to improve both elementary and high school. The first step would be to put more of an emphasis on hands-on classes such as art, music, sewing, woodworking, cooking, theater, auto mechanics and welding.

Yes!  

The world of computation is very visual, something I argue in Visual Thinking. Here’s the abstract:

Its ability to deal with visual information is one of the mind's most powerful capacities. Visual thinking, high-level manipulation of visual information, is important to computer science because, with the flowering of computer graphics and image processing, it provides the basis for a rich and intuitively satisfying channel of man-machine interaction. Just as writing evolved to help the verbal mind, so various media have evolved to help the visual mind. I propose that visual thinking involves the internalization of visuo-manipulative activity and of movement through the environment. We move through the physical environment, sometimes in a familiar place, sometimes in a strange place; we handle objects, sometimes to accomplish a specific task, sometimes simply to inspect the object. Visual thinking involves imagined locomotion in imagined settings, imagined manipulation of imagined objects. The settings and objects may be real, but not present, or they may exist only in imagination.

Then there’s my 1985 article explaining why the advent of the Macintosh computer was so important, The Visual Mind and the Macintosh; I illustrated it with images I made with my 1984 “classic” Mac. Then I teamed up with Richard Friedhoff to write Visualization: The Second Computer Revolution (Abrams 1989). A more recent venture into the visual mind: Jamie’s Investigations: The Art of a Young Man with Down Syndrome (2016). And then there’s my work on graffiti, such as: #GVM004: The Demolition Chronicles (2015). Whoops! I almost forgot, Description 3: The Primacy of Visualization (2015), which is about the importance of using diagrams to describe the structure of literary texts.

As I said in a note to Tyler Cowen:

I note that I am a very visual thinker, not like Grandin, I don’t think in “photo-realistic pictures ... with images clicking through my mind a little bit like PowerPoint slides or TikTok videos.” But I do think in images.

In particular, that’s how I think about the mind. Early in my career I became an expert in knowledge representation via. cognitive or semantic networks. I covered large sheets of paper with diagrams. I wrote some of my best stuff by first drawing diagrams and then writing the prose needed to explicate them. And I still do.

And there’s a very important epistemological & methodological point here. If your thinking about the mind, your theory, exists in diagrams, then there is a clear distinction between those diagrams, theory, and the prose commentary you write about them. You can also maintain the distinction by constructing your theory in mathematics or computer programs. A few use all three, diagrams, math, computer simulation.

Back to Temple Grandin, who concludes:

Today, Taiwan produces the majority of the world’s highest tech silicon chips. Much of the specialized mechanical equipment used for processing meat is made in Holland and Germany. When I visited the Steve Jobs Theater in California, pre-Covid, I discovered that the glass walls were created by an Italian company. The massive carbon fiber roof that looks like a spaceship was imported from Dubai. The reason this equipment is coming from outside the United States can be traced in part to differences in educational systems. In Italy and the Netherlands, for instance, a student at about age 14 decides whether to go the university route or the vocational route. The vocational route is not looked down on or regarded as a lesser form of intelligence. And that’s how it should be everywhere, because the skill sets of visual thinkers are essential to finding real-world solutions to society’s many problems.

Addendum: Neel Nanda has worked on language model interpretability – don’t worry about it, it’s about popping the hood on those pesky AIs that are so damn opaque – at Anthropic, a major AI research company. This is from a recent post, Concrete Steps to Get Started in Transformer Mechanistic Interpretability:

A core skill in MI is being good at visualizing data. Neural networks are high dimensional objects, and you need to be able to understand what's going on! My research workflow looks like running an experiment, visualizing the data, staring at the data, being confused, forming more hypotheses, and iterating. Plot data often, and in a diversity of ways.