"[...] we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks -- including on tasks specifically designed to be challenging for models that ignore word order." https://t.co/tNw7Mghsa7
— David McClure (@clured) April 15, 2021
Thursday, April 15, 2021
Scrambled sentences work for pretraining large language models
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment