REALM, a new open-source method for language model pre-training

And that's not just a solution to "long documents." It seems, generally, like a way to integrate explicit symbolic knowledge with neural language models. Which would be, um, nice.

— Ted Underwood (@Ted_Underwood) August 12, 2020