Are deep neural networks really black-boxes whose inner workings are opaque?
— Chomba Bupe (@ChombaBupe) September 24, 2023
Not quite:
"... for any feed-forward ReLU network, there exists a global linear approximation to a 2-hidden-layer shallow network with a fixed number of units." https://t.co/dDkcVGRyWV pic.twitter.com/belqidsox9
Sunday, September 24, 2023
Chipping away at the "opacity" of ANNs
Neel Nanda on that paper: [LLMs] have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs.
This paper's been doing the rounds, so I thought I'd give a mechanistic interpretability take on what's going on here!
— Neel Nanda (@NeelNanda5) September 24, 2023
The core intuition is that "When you see 'A is', output B" is implemented as an asymmetric look-up table, with an entry for A->B.
B->A would be a separate entry https://t.co/XROT8D5MrS
Here's the rest of Nanda's tweet stream:
The key question to ask with a mystery like this about models is what algorithms are needed to get the correct answer, and how these can be implemented in transformer weights. These are what get reinforced when fine-tuning.
The two hard parts of "A is B" are recognising the input tokens A (out of all possible input tokens) and connecting this to the action to output tokens B (out of all possible output tokens). These are both hard!
Further, the A -> B look-up must happen on a single token position
Intuitively, the algorithm here has early attention heads attend to the prev token to create a previous token subspace on the Cruise token. Then an MLP neuron activates on "Current==Cruise & Prev==Tom" and outputs "Output=Mary", "Next Output=Lee" and "Next Next Output=Pfeiffer"
"Output=Mary" directly connects to the unembed, and "Next Output=Lee" etc gets moved by late attention heads to subsequent tokens once Mary is output.
Crucially, there's an asymmetry between "input A" and "output A". Inputs are around at early layers, come from input embeddings, and touch the input weights of MLP neurons. Outputs are around more at late layers, compose with the unembedding, and come from output weights of MLPs
This is especially true with multi-token A and B. Detecting "Tom Cruise" is saying "the current token embedding is Cruise, and the prev token space says Tom", while output "Tom Cruise" means to output the token Tom, and then a late attn head move "output Cruise" to the next token
Thus, when given a gradient signal to output B given "A is" it reinforces/creates a lookup "A -> B", but doesn't create "B->A", these are different lookups, in different parameters, and there's no gradient signal from one to the other.
How can you fix this? Honestly, I can't think of anything. I broadly think of this as LLMs working as intended. They have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs. It's wild to me to expect symmetry/flow reversing to be possible
Why is this surprising at all then? My guess is that symmetry is intuitive to us, and we're used to LLMs being capable of surprising and impressive things, so it's weird to see something seemingly basic missing.
LLMs are not human! Certain things are easy for us and not for them, and vice versa. My guess is that the key difference here is that when detecting/outputting specific tokens, the LLM has no notion of a variable that can take on arbitrary values - a direction has fixed meaning Neel Nanda
A better analogy might be in-context learning, where LLMs CAN use "variables". The text "Tom Cruise is the son of Mary Lee Pfeiffer. Mary Lee Pfeiffer is the mother of..." has the algorithmic solution "Attend to the subject of sentence 1 (Tom Cruise), and copy to the output"
Unsurprisingly, the model has no issue with reversing facts in context! Intuitively, when I remember a fact A is B, it's closer to a mix of retrieving it into my "context window" and then doing in-context learning, rather than pure memorised recall.
St. George Variations [ChatGPT]
This post is a companion piece to my earlier piece, Can an LLM identify ring-composition in a literary text? [ChatGPT], where I used the story of St. George and the dragon as an example. After establishing that, yes, ChatGPT did seem able to identify ring-composition, at least in this case, I decided to put it through variations on the St. George story as I had done earlier with the Princess Aurora stories. Thus I repeated the procedure I had used for those stories:
I gave ChatGPT a prompt with two components: 1) a story, and 2) a request to create a new story based on the first, where I specify some changes the new story must incorporate. ChatGPT could make any other changes it wished to make.
The St. George story is more complex than the Aurora stories. ChatGPT, of course, was happy to comply. As far as I can tell, though I haven’t looked closely, the new story retains the ring-composition of the original story.
I did this on Septermber 2, 2023, running against the August 3 Version of ChatGPT. For the purposes of brevity (!) I have omitted the St. George story after the first example, where the dragon is to be replaced by a dog. I did each story in a separate session. I am presenting the stories without commentary beyond noting what ChatGPT did for the last story. I asked it to replace the dragon with a colorless green idea. Chatster placed the idea in people’s heads, “slowly poisoning the harmony of their community.”
Rabid Dog
I’m going to present you with a story. I want you to tell a new story based on it but I want you to change in the following way: Replace “venom-spewing dragon dwelling in a nearby pond” with “large rabid dog on the outskirts of town.” Make any other changes you think are necessary.
Silene in Libya was plagued by a venom-spewing dragon dwelling in a nearby pond, poisoning the countryside. To prevent it from affecting the city itself, the people offered it two sheep daily, then a man and a sheep, and finally their children and youths, chosen by lottery. One time the lot fell on the king's daughter. The king offered all his gold and silver to have his daughter spared, but the people refused. The daughter was sent out to the lake, dressed as a bride, to be fed to the dragon.
Saint George by chance arrived at the spot. The princess tried to send him away, but he vowed to remain. The dragon emerged from the pond while they were conversing. Saint George made the Sign of the Cross and charged it on horseback, seriously wounding it with his lance. He then called to the princess to throw him her girdle (zona), and he put it around the dragon's neck. Wherever she walked, the dragon followed the girl like a "meek beast" on a leash.
The princess and Saint George led the dragon back to the city of Silene, where it terrified the populace. Saint George offered to kill the dragon if they consented to become Christians and be baptized. Fifteen thousand men including the king of Silene converted to Christianity. George then killed the dragon, beheading it with his sword, and the body was carted out of the city on four ox-carts. The king built a church to the Blessed Virgin Mary and Saint George on the site where the dragon died and a spring flowed from its altar with water that cured all disease.
In the quiet town of Altona, nestled on the outskirts of a sprawling forest, a sinister menace had long tormented its people. It was not a venom-spewing dragon dwelling in a nearby pond, but rather a large rabid dog. This ferocious creature had terrorized the countryside, spreading fear and chaos throughout the land. In a desperate bid to protect their beloved town, the citizens resorted to offering sacrifices.
At first, they offered the rabid dog two sheep daily to keep it at bay. But as the dog's appetite for destruction grew, so did the demands. Soon, it required a man and a sheep, and eventually, their children and youths, chosen by a grim lottery. The town's once-peaceful existence had been shattered, and it seemed there was no end in sight.
One fateful day, the ill-fated lottery chose the daughter of Altona's mayor. Desperate to save his beloved child, the mayor offered all the gold and silver he possessed, but the heartless townsfolk remained unmoved. With heavy hearts, they sent the mayor's daughter out to the forest, dressed in bridal attire, to be offered to the rabid dog.
As fate would have it, a wandering traveler named Samuel happened upon this grim scene. The mayor's daughter tried to shoo him away, fearing for his safety, but Samuel vowed to stay and help. Just as the rabid dog emerged from the shadows, Samuel made a solemn gesture, drawing a protective symbol in the air, and then charged at the beast on horseback, his lance striking a critical blow.
Breathing heavily, Samuel called out to the mayor's daughter, asking her to toss him her girdle. She did so without hesitation, and Samuel carefully fastened it around the rabid dog's neck. A miraculous transformation occurred; the once-ferocious beast now followed the girl obediently, like a tamed pet.
Together, the mayor's daughter and Samuel led the rabid dog back to Altona. Though the creature still struck fear into the hearts of the townsfolk, they were willing to listen to Samuel, who had proven his bravery and compassion. He proposed a solution: he would rid the town of the rabid dog, but in return, the citizens must embrace Christianity and be baptized.
Overwhelmed by the promise of salvation, fifteen thousand men, including the mayor, chose to convert to Christianity. Samuel, now revered as a saint, faced the rabid dog one last time. With a swift and merciful stroke, he beheaded the creature with his sword.
The town of Altona, now free from the tyranny of the rabid dog, rejoiced and celebrated their newfound faith. In honor of this miraculous event, the mayor built a beautiful church dedicated to Saint Samuel and the Blessed Virgin Mary on the very spot where the rabid dog met its end. A pure spring of water flowed from the church's altar, and it was said that this water possessed the power to heal all diseases, a testament to the town's newfound hope and faith.
Poisonous Mushroom
I’m going to present you with a story. I want you to tell a new story based on it but I want you to change in the following way: Replace “venom-spewing dragon” with “poisonous mushroom.” Make any other changes you think are necessary.
[story omitted]
In the ancient city of Silene, located in the heart of a lush forest in Libya, a menacing threat loomed. It wasn't a venom-spewing dragon that plagued the land, but rather a mysterious and deadly poisonous mushroom that had sprouted near a tranquil pond, casting a dark shadow over the countryside. Its toxic spores drifted on the wind, contaminating everything they touched and spreading fear among the people.
To safeguard their beloved city from the poisonous mushroom's deadly influence, the citizens devised a grim tribute. They offered two sheep daily as a sacrifice to the malevolent fungus, hoping to appease its hunger for destruction. As time passed and the mushroom's malevolence grew, the sacrifices became increasingly desperate. First, they sent a man and a sheep, and eventually, they resorted to a gruesome lottery system to select children and youths to be offered up to the toxic menace.
One fateful day, the cruel hand of fate chose the king's beloved daughter as the mushroom's next victim. The desperate king, in his anguish, offered all his riches, pledging his gold and silver to have his daughter spared from this horrifying fate. Yet, the desperate pleas of a father fell on deaf ears, as the people remained resolute in their grim tradition. The princess was led to the sinister pond, adorned in bridal attire, to become a sacrificial lamb to the deadly fungus.
However, just as despair began to consume the princess, a courageous figure named Saint George arrived at the scene. The princess, fearing for his safety, attempted to shoo him away, but Saint George was undeterred by her pleas. As the poisonous mushroom's eerie presence grew stronger, it emerged from the ground, casting a sinister shadow over the land. With unwavering determination, Saint George made the Sign of the Cross and charged toward the menacing mushroom on horseback, striking it with a lance that pierced its toxic cap.
With a mighty effort, Saint George subdued the deadly mushroom's malevolent powers and called out to the princess to toss him her girdle. She complied, and he fastened it securely around the mushroom's stalk. Miraculously, wherever the princess walked, the once-menacing fungus now followed her like a tamed creature on a leash.
Together, the princess and Saint George led the poisonous mushroom back to the city of Silene. Though the people were initially terrified by the sight of the once-deadly threat, Saint George offered them a proposition. He vowed to rid their land of the toxic menace once and for all if they agreed to embrace Christianity and be baptized.
The people of Silene, shaken by the events and touched by Saint George's bravery and faith, consented to his offer. Fifteen thousand souls, including the king himself, converted to Christianity, their hearts filled with newfound hope. Saint George, a symbol of courage and compassion, unsheathed his sword and swiftly beheaded the once-menacing mushroom. Its lifeless body was transported out of the city on four ox-drawn carts, a symbol of their deliverance.
In gratitude for their salvation, the king commissioned the construction of a grand church dedicated to the Blessed Virgin Mary and Saint George on the very spot where the poisonous mushroom had met its end. As a sign of the miraculous transformation, a spring of pure water flowed from the altar, believed to possess the power to cure all diseases, a testament to the enduring triumph of faith over fear in the city of Silene.
AI in Japan (ChatGPT)
AI in Japan. pic.twitter.com/IUYa1pAOXr
— tkasasagi 🐻 (@tkasasagi) September 24, 2023
Saturday, September 23, 2023
Culturally, GPT is US-centric [WEIRD in WEIRD out]
GPT falls between Germany and New Zealand...@mmuthukrishna pic.twitter.com/lVkEhgOTO9
— Joe Henrich (@JoHenrich) September 23, 2023
Computer scientists seem unaware of global psychological variation. WEIRD in WEIRD out. Check out our draft here https://t.co/DEbMdry1Wa @asifa_majid
— Joe Henrich (@JoHenrich) September 23, 2023
On Analytic thinking, GPT is Scandinavian @ThomasTalhelm pic.twitter.com/KMuHlV0Hgj
— Joe Henrich (@JoHenrich) September 23, 2023
What we can learn about the origin of the vertebrate brain from the lamprey
What's the origin of the vertebrate brain?
— Luiz Pessoa (@PessoaBrain) September 23, 2023
No cerebellum for one!https://t.co/D8xoSdvQER pic.twitter.com/i7qVqHGUhy
There's a better way to teach music to kids [who are born to groove]
Sammy Miller, We’re Teaching Music to Kids All Wrong, NYTimes, Sept. 23, 2023.
Opening paragraph:
Each fall, as school starts up again, music educators witness a familiar ritual: Eager first-time students squeak on a clarinet, suppress giggles at the noises coming from the tubas and zealously hit a bass drum a little too hard. It’s a moment characterized by excitement, enthusiasm and the anticipation of new beginnings — which is why it’s so disheartening to know that many of those kids will eventually quit their instruments.
And on about this and that, and then:
Rather than fixating on funding, let’s look at taking a whole new approach. Educators lament that, as with other courses, band can frequently fall prey to “teaching to the test” — in this case, teaching to the holiday concert. A class that by definition is meant to be a creative endeavor winds up emphasizing rigid reading and rote memorization, in service of a single performance. We need to abandon that approach and bring play back into the classroom by instructing students how to hear a melody on the radio and learn to play it back by ear, and encouraging students to write their own simple songs using a few chords. (The dirty secret of pop music, as Ed Sheeran has explained, is that most chart-topping songs can be played by using only four chords: G, C, D and E minor.) So start with just one chord, a funky beat and let it rip — and, voilà, you’re making music.
It’s often been repeated that “music is a language,” yet we’re reluctant to teach it that way. When we learn a language, we don’t simply memorize phrases or spend all day reading — we practice the language together, sharing, speaking, stumbling but ultimately finding ways to connect. This should happen in music class, too. Music should be a common pursuit: Ask any dad rock weekend band or church ensemble how it experiences music, and the performers are likely to tell you it’s not a chore but a way of building community.
Most important, we need to let kids be terrible. In fact, we should encourage it. They’ll be plenty terrible on their own — at first. But too often kids associate music in school with a difficult undertaking they can’t hope to master, which leads them to give up. Music does not have to be, and in fact, shouldn’t be, about the pursuit of perfection. And the great musicians have plenty of lessons to teach students about the usefulness of failure.
All together now:
We also teach language through immersion, so let’s focus on creating an immersive experience in the language of music. Kids learn best when they’re part of communities filled with people of all skill levels for them to play along with, listen to music with, mess up with and just be silly with. Parents, this means you. Don’t let instrument instruction simply be something you nag your kids to endure. Music was never meant to be a lonely vigil. Play together. Make noise together. Find joy together. Take out an instrument and learn a song that you and your child both love.
While you're at it, check out:
Available at:
Amazon: https://tinyurl.com/4p67bahh
Barnes & Noble: https://tinyurl.com/5vjf3kjt
Google Play: https://tinyurl.com/39rz43vp
Kobo eReader: https://tinyurl.com/5bmkjsme
Friday, September 22, 2023
Is an AI chill on the horizon?
Called it. https://t.co/IZLLKFgxhD
— Gary Marcus (@GaryMarcus) September 22, 2023
Yeah, he called it. I just wish he weren't so damned self-righteous about it.
Nor is Marcus the only one who understood clearly that the models produced by generative AI have severe limitations. They give us new territory to explore. That's exciting. But, as such, they constitute a digital wilderness. Wilderness must be 'tamed' or 'domesticated' before it can be put to use. That work as yet to be done. In fact, the rush to exploit generative AI takes resources away from research that would be useful in understanding the wilderness.
Better at chess, still sucks at planning [look at the red box]
The new GPT3.5 completion version and the claims on its chess prowess caught my students' attention--and they tested to see if it got better at, you know, planning..
— Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) (@rao2z) September 22, 2023
tldr; Nope--RLHF didn't decimate the planning sparks of GPT3.5--it apparently had none to begin with..😅 https://t.co/JxtUqH1XCO
* * * * *
In related news, Subbarao Kambhampati, Can LLMs Really Reason and Plan? BLOG@CACM, September 12, 2023.
Second paragraph:
Nothing in the training and use of LLMs would seem to suggest remotely that they can do any type of principled reasoning (which, as we know, often involves computationally hard inference/search). While one can dismiss the claims by hype-fueled social media influencers, startup-founders and VC's, it is hard to ignore when there are also peer-reviewed papers in top conferences making similar claims. The "Large Language Models are Zero-Shot
" is almost becoming a meme paper title! At some level, this trend is understandable as in the era of LLMs, AI has become a form of ersatz natural science–driven by observational studies of capabilities of these behemoth systems.
In conclusion:
The fact that LLMs are often good at extracting planning knowledge can indeed be gainfully leveraged. As we have argued in our recent work, LLMs can thus be a rich source of approximate models of world/domain dynamics and user preferences,, as long as the humans (and any specialized critics) in the loop verify and refine those models, and give them over to model-based solvers. This way of using LLMs has the advantage that the humans need only be present when the dynamics/preference model is being teased out and refined, and the actual planning after that can be left to planning algorithms with correctness guarantees (modulo the input model). Such a framework has striking similarities to knowledge-based AI systems of yore, with LLMs effectively replacing the "knowledge engineer." Given the rather quixotic and dogmatic shift of AI away from approaches that accept domain knowledge from human experts, something I bemoaned in Polanyi's Revenge, this new trend of using LLMs as knowledge sources can be viewed as a form of avenging Polanyi's revenge! Indeed, LLMs make it easy to get problem-specific knowledge as long as we are willing to relax correctness requirements of that knowledge. In contrast to the old knowledge engineering approaches, LLMs offer this without making it look like we are inconveniencing any specific human (we are, instead, just leveraging everything humans told each other!). So the million dollar question for reasoning tasks is: "how would you do planning if you have some doddering know-it-all ready to give you any kind of knowledge?" Traditional approaches to model-based reasoning/planning that focus on the incompleteness and incorrectness of the said models (such as model-lite planning, robust planning) can have fresh relevance.
To summarize, nothing that I have read, verified or done gives me any compelling reason to believe that LLMs do reasoning/planning as it is normally understood. What they do, armed with their web-scale training, is a form of universal approximate retrieval which, as we have argued, can sometimes be mistaken for reasoning capabilities. LLMs do excel in idea generation for any task–including those involving reasoning, and as I pointed out, this can be effectively leveraged to support reasoning/planning. In other words, LLMs already have enough amazing approximate retrieval abilities that we can gainfully leverage, that we don't need to ascribe fake reasoning/planning capabilities to them.
There's more at the link.
Thursday, September 21, 2023
AI systems as infrastructure, infrastructure as open source
AI systems are fast becoming a basic infrastructure.
— Yann LeCun (@ylecun) September 21, 2023
Historically, basic infrastructure always ends up being open source (think of the software infra of the internet, including Linux, Apache, JavaScript and browser engines, etc)
It's the only way to make it reliable, secure, and… https://t.co/p5jBkKQgyq
Subscribe to:
Posts (Atom)