This is a fascinating story about how Samsung set out to build a state of the art chip fab (fabrication facility) and the problems that bedeviled it. Without the chips this factory was designed to build, AI is nowhere.
Timestamps:
00:00 - $50B Chip Nightmare
18:26 - Where Everything Went Wrong
29:58 - The Twist
Henry Farrell, America’s plan to control global AI, Programmable Mutter, Jan. 15. 2025.
The idea is to use export controls to restrict the selling and use of to achieve two U.S. policy goals. The first is its desire to keep the most advanced AI out of the grasp of China, for fear that China will use strong AI to undermine U.S. security. The second is its desire to allow some degree of continued access to semiconductors and AI in most countries, to mitigate the anticipated shrieks of protest from big U.S. firms that don’t want to see their export markets disappear.
Hence, this highly complex plan involves controlling access to the advanced semiconductors that are used to train advanced AI models, as well as the model ‘weights’ themselves. The plan continues to very sharply restrict China’s and some other countries’ access to highly advanced semiconductors [...] It allows a much more liberal regime of exports without much in the way of controls to a small group of ‘Tier 1’ countries - important allies and other friendlies such as Norway and Ireland. Finally, there is a large intermediary zone of other countries, including some traditional U.S. allies, that will be allowed access to U.S. semiconductors, but under complex restrictions.
The whole shebang “is intended to cement U.S. power over information technology over the longer term” and depends on “five distinct bets; two on technology, and three on politics.” The technology bets are on 1) scaling and 2) AGI. The political bets are on the 3) effectiveness of export controls, 4) organizational capacity, and 5) politics. I want to comment on 1 and 2 and give you bit of Farrell on 5.
Scaling
The most straightforward bet behind this policy is that the “scaling hypothesis” is right. That is, (a) the more computer power is applied to training AI, the more powerful it will be, and (b) access to the most advanced parallel processing semiconductors is essential to building cutting edge AI models. If this is so, then the U.S. has a possible trump card. U.S. based and dependent companies like Nvidia and AMD, that design the cutting edge semiconductors that are used for training AI, have a considerable advantage over their competitors. China and other U.S. rivals and adversaries have no equivalent producers, and are obliged to rely on the inferior chips that they can make themselves, or that the U.S. allows them access to.
If this bet is right, then the U.S. indeed potentially possesses a chokehold that might allow it to shape the world’s AI system, selectively providing access to those countries and companies that it favors, while denying access to those it does not. Controlling the chips used for training, while restricting the export of AI weights, will allow it to shape what other countries do.
There is, however, some possible evidence suggesting that the relationship between chips and scaling is more complicated than the US might like.
Farrell goes on to mention DeepSeek, a powerful Chinese LLM “that it has trained a frontier AI model without access to the most advanced semiconductors.” Beyond that, I just don’t think that scaling alone is the key to the kingdom. As Gary Marcus, Yann LeCun (just search on the names) and others have been arguing, we need new architectures.
AGI
As you know, I think the term itself (artificial general intelligence) as all but meaningless. AGI’s about as real as the Holy Grail and likely springs from similar psycho-cultural desires.
Farrell notes:
One other belief, which is quite widespread among people in the U.S. national security debate as well as many in Silicon Valley, is that we are on the verge of real AGI - ‘artificial general intelligence.’ In other words, we are about to witness a moment where there will be a vast leap forward in the ability of AI to do things in the world, creating self reinforcing dynamics where those with strong AI are going to be capable of creating yet stronger AI and so on in a feedback loop. This then implies that short term AI superiority over the next couple of years might lead into a long term strategic advantage.
Farrell is skeptical:
Here, for example, Arvind Narayanan and Sayash Kapoor argue that we should be skeptical about the hype that is bubbling out right now from inside the big AI companies.
Industry leaders don’t have a good track record of predicting AI developments. … There are some reasons why we might want to give more weight to insiders’ claims, but also important reasons to give less weight to them. … there’s a huge and obvious reason why we should probably give less weight to their views, which is that they have an incentive to say things that are in their commercial interests, and have a track record of doing so.
There is a lot more in Narayanan and Kapoor’s article, about the specifics of what is happening right now, as we (perhaps) move from one model of AI development to another. I find their arguments compelling - your own mileage may of course vary.
Yes, great things will one day be possible, but not as long as the techbros keep leading us down the path of scaling up LLMs and forms of deep learning. We need new architectures and that’s going to require some fundamental research, research that won’t happen as long as scaling sucks up all the resources, financial, technological, and intellectual.
Politics
None of this will happen if the Trump administration doesn’t want it to. And there are clearly Republicans who are listening to industry protests, and promising to do what they can to get the plan reversed. A lot of people are speculating that the plan is dead on arrival.
That may be premature. One plausible interpretation is that the Biden people are trying to create facts on the ground that will bolster China hawks in the incoming administration, who want strong technology restrictions, so that they have a greater chance of prevailing over the people who want to let technology rip. And that might perhaps work!
It isn’t just the foreign policy people who want sharp restrictions on China. It is also some important people in the AI debate. Pottinger is probably not going to be coming back in (he demonstrated Insufficient Loyalty to the Beloved Leader in the days surrounding January 6 2021) but his co-author, Amodei reflects a general hawkish turn among many people in Silicon Valley. [...]
I don’t feel particularly confident in making any predictions about what the Trump administration will do. I am not the person you ought turn to for accurate gossip about who has influence among the people who are about to take power. But I don’t see any unambiguous signals (yet) that the one side or the other has the upper hand in the internal arguments.
There’s much more at the link.
John Liu, What Works in Taiwan Doesn’t Always in Arizona, a Chipmaking Giant Learns, NYTimes, Aug. 8, 2024:
Taiwan Semiconductor Manufacturing Company, one of the world’s biggest makers of advanced computer chips, announced plans in May 2020 to build a facility on the outskirts of Phoenix. Four years later, the company has yet to start selling semiconductors made in Arizona. [...]
In Taiwan, TSMC has honed a highly complex manufacturing process: A network of skilled engineers and specialized suppliers, backed by government support, etches microscopic pathways into pieces of silicon known as wafers.
But getting all this to take root in the American desert has been a bigger challenge than the company expected.
“We keep reminding ourselves that just because we are doing quite well in Taiwan doesn’t mean that we can actually bring the Taiwan practice here,” said Richard Liu, the director of employee communications and relations at the site.
In recent interviews, 12 TSMC employees, including executives, said culture clashes between Taiwanese managers and American workers had led to frustration on both sides. TSMC is known for its rigorous working conditions. It’s not uncommon for people to be called into work for emergencies in the middle of the night. In Phoenix, some American employees quit after disagreements over expectations boiled over, according to the employees, some of whom asked not to be named because they were not authorized to speak publicly.
The company, which has pushed back the plant’s start date, now says it expects to begin chip production in Arizona in the first half of 2025.
Cultural expectations about work hours are one thing. But there are other factors involved:
On top of working to address the cultural differences in the workplace, TSMC is gearing up to recruit skilled workers to staff the Arizona plant for years to come. The company faces similar challenges in Japan and Germany, where it is also expanding.
In Taiwan, TSMC is able to draw on thousands of engineers and decades of relationships with suppliers. But in the United States, TSMC must build everything from the ground up.
“Here at this site, a lot of things we actually have to do from scratch,” Mr. Liu said.
The article goes on to talk about worker training and talks about how local colleges and universities are creating programs directed at chip manufacturing.
“We have a generation of students whose parents have never once stepped foot into an advanced manufacturing factory,” said Scott Spurgeon, the center’s superintendent. “Their concept of that is still much like the old mom-and-pop manufacturing where you show up every day and come out with dirty clothes and dirty hands.”
I'm wondering how much culturally transmitted tacit knowledge there is in those relationships that exist in Taiwan, but not Arizona.
There's more at the link.
The two Companies I am thinking of are TSMC (Taiwan Semiconductor Manufacturing Company) in Taiwan, which Nicholas Kristof recently wrote about in The New York Times, and ASML Holding in the Netherlands. TSMC is a foundry, producing semiconductor chips designed by other companies, such as AMD, Apple, ARM, Broadcom, Marvell, MediaTek, Qualcomm and Nvidia. They manufacture the most advanced chips. ASML manufactures the most advanced photolithography machines in the world; those are the machines used to fabricate semiconductor chips.
These two companies are, at the moment, unique. Why? I’m guessing – and that’s all it is, a guess – that it is because the knowledge required in both cases does not travel well. Both companies are profit-making businesses and I assume that they do the various things that manufacturing companies do to protect their businesses. They’ve got proprietary knowledge and they take steps to protect that knowledge.
That’s not what I have in mind by saying that their manufacturing knowledge does not travel well. Even if they took no steps to protect their knowledge, that knowledge would not travel well. I’m saying that the knowledge is carried in cultures local to these two firms. Those cultures exist in the minds, bodies, and working relationships between the employees of the two companies. It’s not a matter of one, or seven, or 20 or 50 trade secrets, though there are no doubt trade secrets involved, patents as well. I’m talking about a whole cultural formation.
We’re dealing with very complex physical processes that must be executed to nanometer precision (billionth of a meter). The nature of the industry demands that the level of precision increases on a yearly basis. You can’t create a process and then run it for 10 or 15 years. Everything must be reconceived and reengineered on a continuing basis. That requires a high level of cooperation and coordination within a large network of people processing a wide range of knowledge and skills. Much of the knowledge will necessarily be of the tacit kind that cannot be committed to instructions, formulas, drawings, and procedures writable on paper. It exists only in the minds and habits of interaction of the workers.
How do you transfer such a complex socio-cultural formation from one organization to another? Short of cloning the whole team, you can’t. And so these two companies are, for now, unique.
As I said, I’m just guessing on this. I don’t really know. But this is the only kind of explanation that makes sense to me.
c. 13:15:
Narrator: Before EUV, chipmakers had three companies they could choose from for their photo lithography tools: ASML, Nikon and Canon. Nikon, in Japan, is still a competitor for DUV, but ASML is the only option for EUV. Experts say it could take decades for any other company to catch up, not only because of ASML's proprietary tech, but because it's built complex, often exclusive, deals with nearly 800 suppliers.
Peter Wennink, ASML CEO: And we're unique to our customers, like some of our suppliers are unique to us. And those almost symbiotic relationships, some people say are worse than being married because you cannot divorce.
c. 15:41:
Wennink: It means that we need to ship our machines sooner, earlier, and at higher volume. So it means we need to hire more people in the U.S. It's talent, it's people. I think that's where the biggest challenge will be.
Photographs and Video by Philip Cheung, text by Don Clark, The Huge Endeavor to Produce a Tiny Microchip, NYTimes, April 8, 2022.
High tech runs on microchips. They are difficult to design and fabricate; the "fabs" are major projects in themselves.
Why chip-making is so difficult:
Chip makers are packing more and more transistors onto each piece of silicon, which is why technology does more each year. It’s also the reason that new chip factories cost billions and fewer companies can afford to build them.
In addition to paying for buildings and machinery, companies must spend heavily to develop the complex processing steps used to fabricate chips from plate-size silicon wafers — which is why the factories are called “fabs.”
Enormous machines project designs for chips across each wafer, and then deposit and etch away layers of materials to create their transistors and connect them. Up to 25 wafers at a time move among those systems in special pods on automated overhead tracks.
Processing a wafer takes thousands of steps and up to two months. TSMC has set the pace for output in recent years, operating “gigafabs,” sites with four or more production lines. Dan Hutcheson, vice chair of the market research firm TechInsights, estimates that each site can process more than 100,000 wafers a month. He puts the capacity of Intel’s two planned $10 billion facilities in Arizona at roughly 40,000 wafers a month each.
Factories must be super-clean:
Intel chips typically sell for hundreds to thousands of dollars each. Intel in March released its fastest microprocessor for desktop computers, for example, at a starting price of $739. A piece of dust invisible to the human eye can ruin one. So fabs have to be cleaner than a hospital operating room and need complex systems to filter air and regulate temperature and humidity.
Fabs must also be impervious to just about any vibration, which can cause costly equipment to malfunction. So fab clean rooms are built on enormous concrete slabs on special shock absorbers.
Also critical is the ability to move vast amounts of liquids and gases. The top level of Intel’s factories, which are about 70 feet tall, have giant fans to help circulate air to the clean room directly below. Below the clean room are thousands of pumps, transformers, power cabinets, utility pipes and chillers that connect to production machines.
Fabs require lakes of water "to clean wafers at many stages of the production process." Constructing fabs is difficult:
Excavating the foundations is expected to remove 890,000 cubic yards of dirt, carted away at a rate of one dump truck per minute, said Dan Doron, Intel’s construction chief.
The company expects to pour more than 445,000 cubic yards of concrete and use 100,000 tons of reinforcement steel for the foundations — more than in constructing the world’s tallest building, the Burj Khalifa in Dubai.
Some cranes for the construction are so large that more than 100 trucks are needed to bring the pieces to assemble them, Mr. Doron said. The cranes will lift, among other things, 55-ton chillers for the new fabs.
There's more at the link.
I have some remarks on chip fabrication in Stagnation 1: The phenomenon and a simple-minded model with some remarks on search (pharmaceuticals) and process re-engineering (semiconductors). In particular, I remark on the fact the the laws of the quantum world are different from those of the macroworld.
Dr. Ian Cutress, An AnandTech Interview with Jim Keller: 'The Laziest Person at Tesla',
6.16.21.
I've spoken about Jim Keller many times on AnandTech. In the world of semiconductor design, his name draws attention, simply by the number of large successful projects he has worked on, or led, that have created billions of dollars of revenue for those respective companies. His career spans DEC, AMD, SiByte, Broadcom, PA Semi, Apple, AMD (again), Tesla, Intel, and now he is at Tenstorrent as CTO, developing the next generation of scalable AI hardware. Jim's work ethic has often been described as 'enjoying a challenge', and over the years when I've spoken to him, he always wants to make sure that what he is doing is both that challenge, but also important for who he is working for. More recently that means working on the most exciting semiconductor direction of the day, either high-performance compute, self-driving, or AI.
Note: This interview is intended for an audience with technical expertise in chip design. If, like me, you lack such expertise, you just have to let if flow and be content with a mere flavor for what's going on.
Matrices, graphs, and vectors
IC: I think you said before that going beyond the sort of matrix, you end up with massive graph structures, especially for AI and ML, and the whole point about Tenstorrent, it’s a graph compiler and a graph compute engine, not just a simple matrix multiply.
JK: From old math, and I'm not a mathematician, so mathematicians are going to cringe a little bit, but there was scalar math, like A = B + C x D. When you had a small number of transistors, that's the math you could do. Now we have more transistors you could say ‘I can do a vector of those’, like an equation properly in a step. Then we got more transistors, we could do a matrix multiply. Then as we got more transistors, you wanted to take those big operations and break them up, because if you make your matrix multiplier too big, the power of just getting across the unit is a waste of energy.
So you find you want to build this optimal size block that’s not too small, like a thread in a GPU, but it's not too big, like covering the whole chip with one matrix multiplier. That would be a really dumb idea from a power perspective. So then you get this array of medium size processors, where medium is something like four TOPs. That is still hilarious to me, because I remember when that was a really big number. Once you break that up, now you have to take the big operations and map them to the array of processors and AI looks like a graph of very big operations. It’s still a graph, and then the big operations are factored down into smaller graphs. Now you have to lay that out on a chip with lots of processors, and have the data flow around it.
This is a very different kind of computing than running a vector or a matrix program. So we sometimes call it a scalar vector matrix. Raja used to call it spatial compute, which would probably be a better word.
IC: Alongside the Tensix cores, Tenstorrent is also adding in vector engines into your cores for the next generation? How does that fit in?
JK: Remember the general-purpose CPUs that have vector engines on them – it turns out that when you're running AI programs, there is some general-purpose computing you just want to have. There are also some times in the graph where you want to run a C program on the result of an AI operation, and so having that compute be tightly coupled is nice. [By keeping] it on the same chip, the latency is super low, and the power to get back and forth is reasonable. So yeah, we're working on an interesting roadmap for that. That's a little computer architectural research area, like, what's the right mix with accelerated computing and total purpose computing and how are people using it. Then how do you build it in a way programmers can actually use it? That's the trick, which we're working on. [...]
CPU Instruction Sets: Arm vs x86 vs RISC-V
IC: You’ve spoken about CPU instruction sets in the past, and one of the biggest requests for this interview I got was around your opinion about CPU instruction sets. Specifically questions came in about how we should deal with fundamental limits on them, how we pivot to better ones, and what your skin in the game is in terms of ARM versus x86 versus RISC V. I think at one point, you said most compute happens on a couple of dozen op-codes. Am I remembering that correctly?
JK: [Arguing about instruction sets] is a very sad story. It's not even a couple of dozen [op-codes] - 80% of core execution is only six instructions - you know, load, store, add, subtract, compare and branch. With those you have pretty much covered it. If you're writing in Perl or something, maybe call and return are more important than compare and branch. But instruction sets only matter a little bit - you can lose 10%, or 20%, [of performance] because you're missing instructions.
For a while we thought variable-length instructions were really hard to decode. But we keep figuring out how to do that. You basically predict where all the instructions are in tables, and once you have good predictors, you can predict that stuff well enough. So fixed-length instructions seem really nice when you're building little baby computers, but if you're building a really big computer, to predict or to figure out where all the instructions are, it isn't dominating the die. So it doesn't matter that much.
When RISC first came out, x86 was half microcode. So if you look at the die, half the chip is a ROM, or maybe a third or something. And the RISC guys could say that there is no ROM on a RISC chip, so we get more performance. But now the ROM is so small, you can't find it. Actually, the adder is so small, you can hardly find it? What limits computer performance today is predictability, and the two big ones are instruction/branch predictability, and data locality.
Now the new predictors are really good at that. They're big - two predictors are way bigger than the adder. That's where you get into the CPU versus GPU (or AI engine) debate. The GPU guys will say ‘look there's no branch predictor because we do everything in parallel’. So the chip has way more adders and subtractors, and that's true if that's the problem you have. But they're crap at running C programs.
GPUs were built to run shader programs on pixels, so if you're given 8 million pixels, and the big GPUs now have 6000 threads, you can cover all the pixels with each one of them running 1000 programs per frame. But it's sort of like an army of ants carrying around grains of sand, whereas big AI computers, they have really big matrix multipliers. They like a much smaller number of threads that do a lot more math because the problem is inherently big. Whereas the shader problem was that the problems were inherently small because there are so many pixels.
There are genuinely three different kinds of computers: CPUs, GPUs, and AI. NVIDIA is kind of doing the ‘inbetweener’ thing where they're using a GPU to run AI, and they're trying to enhance it. Some of that is obviously working pretty well, and some of it is obviously fairly complicated. What's interesting, and this happens a lot, is that general-purpose CPUs when they saw the vector performance of GPUs, added vector units. Sometimes that was great, because you only had a little bit of vector computing to do, but if you had a lot, a GPU might be a better solution. [...]