"If I were starting grad school" – One person's view about the state of machine learning (ML)

If I were starting grad school now I would not do a PhD in ML. It's not the competition that would steer me away though, I think it's a bad idea for other reasons. https://t.co/Y551gtaXE9

— Misha Denil (@notmisha) July 4, 2020

The rest of Denil's tweet stream:

The field has no consensus on what the big important challenges are. Today is better than a couple years ago when we really had no idea, but the most promising leads come in the form of bringing ML to bear on problems of adjacent fields (chemistry, robotics, economics, etc).

The new challenges all share a common theme which separates them from the old challenges. They all require a much greater depth of domain knowledge to understand if you are making progress.

(Its easy to tell if your dog classifier can classify dogs, or if your translation model can translate sentences. It is hard to tell if your small molecule VAE can generate plausible structures, or if your simulated economy can offer insights to policymakers.)

In the meantime, the ML community seems intent on
1. Hiding domain details behind standardized benchmarks
2. Commoditizing its own tools and methods

Neither of these things are intrinsically bad. Imagenet was a powerful benchmark that drove years of progress, and the democratization of ML tools has been a huge boon in countless ways.

But when your most promising path to impact is to bring your methods to neighboring fields it does seem like a strategic error to avoid learning about the details of those fields and to simultaneously make it easy for them to adopt your tools without you.

The way the landscape looks today, it's a lot easier to teach a chemist to use tensorflow than it is to teach an ML-er to do chemistry, and that gap is only going to get larger as the tooling gets better.

The next LSTM or resnet or transformer will probably come from the core ML community, but developments like that are few and far between.

So when people ask me if they should do a PhD in ML I say no, they should do a PhD in something else and also learn tensorflow. I think they're much more likely to do meaningful ML work that way.