Make no mistake, every time I read a news story about all the time, effort, and investment being poured into building larger and larger generative AI models, LLMs in particular [like today's post about data hunger], not to mention another technical paper, I wonder whether or not they are right and I am wrong. The fact the no one really knows, and so one must be open, weighs on me, though as far as I can tell, it doesn’t weigh on THEM at all. They’re sure. They have to be if they’re going to put all that money into it. Or is it the other way around, it’s the act of investing that makes them sure?
I’ve got explicit arguments and knowledge on my side. They’re by no means conclusive. But they are not nothing either. Nor am I alone in my skepticism. Yet, the roar of the true believers is a distraction.
If you don’t have any explicit arguments against the power of scaling, if you don’t already know a great deal about perception, cognition, and language, the lure of the crowd must be overwhelming.
They don't have to be sure. The payoffs for Amazon and Microsoft are obvious one you see them.
ReplyDeletePart of their plan is "what if we succeed at this?" Part is "what if my competitor succeeds?" Part is "what if no-one ever suceeds?" The third path is paved with considerations about which company is best able to survive wasting money. You can have wide variation in your estimates of the probability of the three paths and still come up with the actions that we now observe.
Personally I think that Microsoft is disingenuous when it says it will spend trillions on AI data centers. Maybe it won't, but it is helpful to its competitive position to say that it will.
It's complicated. One the one hand they need to build out data centers so they can accommodate more users. If they can make money with more or less the current technology, fine. They've certainly got some people willing to pay for what current tech can do. I don't think anyone's breaking even much less making a profit, but it's still early days. But long term it's a crap shoot. & I think worrying about running out of data, generating synthetic data, that strikes me as the tail wagging the dog. The popularity of ChatGPT surprised everyone, and everyone is just treading water as effectively as they can. But I don't see enough attention being given to alternatives to current tech.
ReplyDelete