Scrambled sentences work for pretraining large language models

"[...] we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order." https://t.co/tNw7Mghsa7

— David McClure (@clured) April 15, 2021