Rotem Botvinik-Nezer, Felix Holzmeister, Colin Camerer, et al., Variability in the analysis of a single neuroimaging dataset by many teams, bioRXiv, Nov. 15, 2019, doi: https://doi.org/10.1101/843193
Abstract: Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.H/t Tyler Cowen.
The importance for consensual validation of much medical "data" is underestimated. Why error as a subject is generally avoided seems to me to be societal. Much of the best learning occurs by overcoming mistakes through trial-and-error, and by comparing one's skills against experts. Error becomes a flaw to be hidden otherwise--not a good thing generally. I had a college teacher who shouted, "You made a mistake" at a student early in the term, followed by, "Rejoice! Now you can learn something." The lecture which followed was a highlight of college.
ReplyDeleteLove the anecdote.
DeleteIt seems to me that avoiding discussion of error goes hand in hand with not publishing negative results – which, after all, IS useful knowledge and would save others from going down a blind conceptual alley – and with publishing the same idea in slightly different forms time after time.
Yes,indeed!
ReplyDeleteIt is also rare for people to say, "I don't know" or "I don't understand." I have an inspiring clip of a brief interview of Richard Dawkins being asked about Brexit. He effectively responds, why is he being asked since he is an ignoramus on the subject. It is inspiring to hear directly about ignoramusitude.
LOL!
DeleteAnecdote from the poet Louise Gluck: She was young and new to studying with Stanley Kunitz who eventually asked her to meet with him to discuss her work at his home (different times, different culture). She was nervous, of course; meeting at his invitation at his home was something. She wondered if perhaps her writing showed an important development she herself wasn't aware of. Kunitz graciously offered her a martini, spread out her poems on the desk in front of them. She was practically holding her breath. Kunitz uttered the words of the nightmare,"Well, they're awful. . . " Gluck didn't breathe. "But you're a poet." Ohh! From which they read through the poems together to see what was salvageable.
ReplyDeleteI hear a fine book of anecdotes forming deep in the internets-- about messing up repeatedly to get things right.
ReplyDeleteMaybe Malcolm Gladwell is heading there in his recent book, "Talking to Strangers." Maybe poetry is essentially 'about' messing up. Hidden in artfulness. OMG! More things to investigate.