3.Overall, there is a “de-democratization of AI research” due to the rise of DL, which requires lot of computing power. There's an uneven access to computing power, which resulted in this current divergence b/n haves & have-nots, “compute divide”(similar to the “digital divide”)
— Abeba Birhane (@Abebab) October 30, 2020
Taken together, this suggests that due to the rise of deep learning, we have even more diversity problem in AI research than ever!
— Abeba Birhane (@Abebab) October 30, 2020
As far as I can tell, there's no particularly good reason to think that deep pockets belong to those with greater imagination. And, I must admit, my biases are rather in the other direction. I suspect that those with deep pockets are likely to seek out intellectually conservative investigators in order to protect their investment. See, for example, Gwern on the scaling hypothesis:
The blessings of scale in turn support a radical theory: an old AI paradigm held by a few pioneers in connectionism (early artificial neural network research) and by more recent deep learning researchers, the scaling hypothesis. The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale. As increasing computational resources permit running such algorithms at the necessary scale, the neural networks will get ever more intelligent.
When? Estimates of Moore’s law-like progress curves decades ago by pioneers like Hans Moravec indicated that it would take until the 2010s for the sufficiently-cheap compute for tiny insect-level prototype systems to be available, and the 2020s for the first sub-human systems to become feasible, and these forecasts are holding up. (Despite this vindication, the scaling hypothesis is so unpopular an idea, and difficult to prove in advance rather than as a fait accompli, that while the GPT-3 results finally drew some public notice after OpenAI enabled limited public access & people could experiment with it live, it is unlikely that many entities will modify their research philosophies, much less kick off an ‘arms race’.)
More concerningly, GPT-3’s scaling curves, unpredicted meta-learning, and success on various anti-AI challenges suggests that in terms of futurology, AI researchers’ forecasts are an emperor sans garments: they have no coherent model of how AI progress happens or why GPT-3 was possible or what specific achievements should cause alarm, where intelligence comes from, and do not learn from any falsified predictions. Their primary concerns appear to be supporting the status quo, placating public concern, and remaining respectable. As such, their comments on AI risk are meaningless: they would make the same public statements if the scaling hypothesis were true or not.
Depending on what investments are made into scaling DL, and how fast compute grows, the 2020s should be quite interesting—sigmoid or singularity?
Gwern has a more favorable view of AI than I do, and greater faith in the efficacy of mere scaling, but I do share his skepticism about motivation. These people, whatever they think of themselves, are not intellectual adventurers setting off for intellectual adventure in parts unknown. They are well paid clerks in search of titillation.
No comments:
Post a Comment