Can reviewer anonymity weaken the value of a reviewer’s comments?

Some more crude thoughts in the course of mulling over my rejection at NLH.

A week ago I’d posted, On Discussion and Debate in the Blogosphere, in which I pointed out that when people comment under pseudonyms it can be difficult to interpret their remarks, and this is especially because such discussion is informal and most remarks are made relatively quickly. One and the same remark has one valence if made by someone who is expert in the topic at hand and a different valence if made by someone who is a neophyte.

The same is true of the anonymous reviews one (sometimes) receives on articles submitted for formal publication. That is certainly the case with my recent rejection at New Literary History. I’d submitted an article, Sharing Experience: Computation, Form, and Meaning in the Work of Literature, where I argued that the concept of computation could help us understand the nature of literary form. The reviewer was quite critical of my treatment of computation. Since, however, I do not know who the reviewer is, I don’t have any sense of what they know about computation. In particular, what do they know of computational linguistics and the computational study of story telling, the computational disciplines most germane to my argument? How can I evaluate their comments if I know nothing of their background in computing?

Here’s the reviewer’s comments about computing. Since I am taking these out of context, that may weaken your ability to evaluate them, but I’m not asking you to make any subtle judgment.

Why is it advantageous to use the conception of computation to think about literary processes? How does such a move speak to current debates in literary studies? How might one respond to those skeptical of the spread of computational metaphors through the culture at large? There is, moreover, a long history of conceiving the mind as akin to a machine/calculator/computer that deserves at least brief acknowledgment. [...] The essay repeatedly declares the importance of computational models, but fails to makes its case through a sustained argument.

If the reviewer has some familiarity with computing, then these are serious criticisms indeed. But if the reviewer is uninformed, as I fear most humanists are, then they are still serious, but with rather a different valence.

If the reviewer is unsophisticated how can they judge whether or not the essay addresses “current debates in literary studies”? I wouldn’t have written the piece if I didn’t think it contributed to current debates and if I didn’t indicate just how in my article – form, intersubjectivity (aka sharing). An unsophisticated reviewer, however, might not see that, as they’d find my discussion rather opaque. If the reviewer is sophisticated, though, they might be saying that my article fails on computational terms; the question of whether or not it speaks to literary issues is thus irrelevant.

I know perfectly well that literary critics are relatively uninformed about computational linguistics – that’s my default assumption; digital criticism aside, I see very little computational sophistication in literary criticism [1]. That’s one reason I wrote the article, to show how computation IS relevant. The fact that this reviewer rejected the article means that, on the face of it, I failed to make my case.

Perhaps though I could not possibly have succeeded. What if the reviewer is not only unsophisticated, but also ideologically opposed to computation in the study of the human mind, which is not necessarily the default value? It is quite possible that there is no argument I could make that would have convinced this reviewer.

I’m caught in an impossible inferential circle. The only evidence I have of the reviewer’s computational sophistication is in their remarks. But I can’t make sense of their remarks unless I know something else about their computational sophistication.

Still, let’s circle back one more time. Here are the two opening sentences of the review (I’ve inserted the numbers myself):

(1) This essay starts off well and it addresses some potentially interesting questions. Nonetheless, in spite of its intellectual ambitions, (2) it comes across as rather self-enclosed and lacking the engagement with either current or historical conversations (3) that would make it a good fit for a journal such as NLH.

How much weight do I assign to (1), (2), and (3)? I think (1) is something of a courtesy throwaway and (2) betrays the reviewers lack of computational sophistication – sure, “self-enclosed”, what choice do I have if the profession ignores computational ideas? But (3) is likely a valid conclusion. If the reviewer didn’t get it, then neither will the readers of NLH.

What I fear is that between ideological commitments and lack of knowledge, the readers of NLH cannot be reached. That is not good.

This brings to mind a remark that Neil Lincoln made at a NASA meeting in the summer of 1981. I was part of a team NASA had assembled to help them evaluate their overall computational capability. Lincoln was from the Control Data Corporation (CDC), a now defunct company that pioneered supercomputing. He was spending a year with NASA, on CDS’s payroll, educating NASA about supercomputers. Why? Because NASA couldn’t make intelligent decisions about supercomputing until it knew more about them.

Well, literary critics cannot make intelligent decisions about the relevance of computational ideas to literary study unless they have a deeper understanding of those ideas. Some humanists have ideological commitments that insulate them from computational ideas. Even where that is not the case, gaining useful sophistication is something of a stretch because it cannot be done simply through reading prose. One has to do something beyond that. You have to learn (something of) a different way of thinking. That’s where the problem lies.

* * * * *

This, of course, is just one case, me and NLH and one article. And my case is special/strange/unusual in that I am writing from a conceptual background that is unusual within the humanities. Most submissions would be expressed in terms more or less familiar with NLH’s pool of reviewers.

But the issue is a general one: Why have anonymous reviews at all? Some journals, but not NLH, ask authors to anonymise their submissions so that reviewers will not be biased by knowing who wrote the article. It’s not clear to me, however, that that logic applies to reviewers, but I don’t want to argue that here. I offer three more remarks.

First, and in this particular case, since I know nothing of my reviewer beyond what I can infer from their remarks, I am free to imagine that they are not computationally sophisticated. I know perfectly well that that belief is self-serving. But what if the reviewer is in fact computationally sophisticated? And what if I knew who that reviewer is? Then it would have been much more difficult for me to believe that they simply didn’t understand what I was attempting.

Second, is reviewer anonymity about anything more than power? At some point power must not and cannot be argued with. Power is power.

Third, as I’ve said before, what bothers me is not so much the fact of rejection, but the blindness of it. I see little evidence that the reviewer knew enough about computation to evaluate my article. Why not admit it? Power?

There’s got to be a better way.

* * * * *

[1] Moreover, not so long ago I criticized David Golumbia’s discussion of Chomksy in The Cultural Logic of Computation. Chomsky’s ideas are fundamentally computational, though Chomsky has never, to my knowledge, contributed to projects in computational linguistics. Golumbia is a literary critic. He also has practical experience as a computer programmer. Such experience is only a bit more relevant to his arguments about Chomsky than experience as an automobile driver is relevant to auto repair. His treatment of Chomsky’s ideas is poor; he doesn’t understand what he’s critiquing. 

See Cultural Logic or Transcendental Interpretation? Golumbia on Chomsky's Computationalism, Working Paper, November 2016, 19 pp., URL: https://www.academia.edu/29570825/Cultural_Logic_or_Transcendental_Interpretation_Golumbia_on_Chomskys_Computationalism