Sometime in the last two or three years I began thinking about the fact that the same transformer architecture that is used to create large language models (LLMs) is also used to create images. In the case of language models, the learning procedure involves predicting the next token. In image models it’s about predicting the next pixel. Thus, on some level and in some way a written text and a visual image can be treated as the same kind of thing.
So, how can we think about a written text as some kind of visual object. That’s really quite simple. Imagine that each word or each character is a colored bead. If you think at the word level, you’ll need lots of different colors, 10s of thousands of them. If you think at the character level, a much smaller number of colors will do, 48 for the letters (upper and lower case), one for a space, and on the order of 10s for punctuation marks. Whatever level you choose, a text becomes a collection of colored beads. Once you’ve done that it’s easy to arrange the text into one long string and feed all those strings to the transformer. That is to saw, the transformer architecture treats verbal texts as one-dimensional visual objects.
We don’t have to do anything fancy to treat images as strings of pixels of various colors. Just rasterize the images and code the colors with some appropriate convention and you’re done.
Now, all we have to do is come up with a way to think about how to translate back and forth between these two forms of representation so that we can use language to direct a bot to create an image and use a different bot to describe an existing image. I’ll leave thinking about that as an exercise for the reader, noting only that I’ve got a good number of posts where Claude describes images, and a few where ChatGPT modifies existing images according to simple verbal instructions.
We’re going to see both operations in this post. First I’m going to ask Claude 3.7 to describe an image. Then I’m going to post three copies I had ChatGPT make, each in a different style. Then I’m going to post the conversation I had with Claude about those copies. As always, my comments are in bold face while Claude’s are plain face.
Claude describes a simple country scene
Describe this picture.
This is a colorful drawing of a whimsical village scene with vibrant houses nestled in rolling green hills with mountains in the background. The houses have distinctive architectural features - some with tall spires, others with multiple windows, in bright colors including red, orange, yellow, and blue.
In the foreground is a flower garden full of colorful tulips in red, yellow, orange, and purple, along with blue forget-me-nots or similar small blue flowers. The drawing has a folk art or naive art style with bold colors and simplified forms.
The scene has a storybook quality with its cheerful colors and quaint buildings arranged on the hillside. There appears to be a small signature or date in the bottom right corner of the image, though it's not entirely clear what it says.
The drawing appears to be done with markers or colored pencils on paper, with a charming handmade quality to the artwork.
Right. I belong to an art club where we start with a line drawing and then color it as we wish. Some club members use colored pencils, others use markers. I used a marker.
That's wonderful! Art clubs are such a great way to express creativity in a supportive community. I like how everyone starts with the same line drawing but produces unique interpretations through their color choices and techniques. Your marker work gives the piece a particularly vibrant and bold quality - the colors really pop against each other, especially in the flower garden with those bright tulips against the green background.
Does your art club meet regularly to work on these projects together? The village scene has such a cheerful, storybook quality that feels uplifting.
We meet twice a week.
ChatGPT’s three copies
Note: I did not have ChatGPT make these copies in independent trials. It made all of them in the same session. It did the illuminated manuscript first, then the Mughal miniature, and the Japanese print last. The last two copies have a feature that's missing from the first.
Count the number of flowers and compare with the original. Do the same for the buildings. Notice the small structure at the lower right. That's not in the original image, but if you look closely at the original you'll see portions of some kind of wall peaking out between the leaves of the flowers. This is the third copy that ChatGPT made, but it's the first one Claude commented on.















