Method in DH: Signal and Concept, Operationalization

Signal and concept: distinction employed by Ryan Hauser and Long Le-Khac: A Quantitative Literary History of 2,958 Nineteenth-Century British Novels: The Semantic Cohort Method. Stanford Literary Lab, Pamphlet 4, May 2012.

Operationalization: concept discussed in Franco Moretti, “Operationalizing”: or, the Function of Measurement in Modern Literary Theory, Stanford Literary Lab, Pamphlet 6, December 2013.

Think like a social scientist: Arthur Stinchcombe, Constructing Social Theories, New York: Harcourt, Brace, and World, 1968.

• • • • • •

Toward the end of their pamphlet Hauser and Le-Khac observe (pp. 46-47):

How do we get from numbers to meaning? The objects being tracked, the evidence collected, the ways they’re analyzed—all of these are quantitative. How to move from this kind of evidence and object to qualitative arguments and insights about humanistic subjects— culture, literature, art, etc.—is not clear. In our research we’ve found it useful to think about this problem through two central terms: signal and concept. We define a signal as the behavior of the feature actually being tracked and analyzed. The signal could be any number of things that are readily tracked computationally: the term frequencies of the 50 most frequent words in a corpus, the average lengths of words in a text, the density of a network of dialogue exchanges, etc. A concept, on the other hand, is the phenomenon that we take a signal to stand for, or the phenomenon we take the signal to reveal. It’s always the concept that really matters to us. When we make arguments, we make arguments about concepts not signals.

I’ve already noted that I believe the equation of computing with quantitative is misleading (From Quantification to Patterns in Digital Criticism). That’s one thing.

The solution Heuser and Le-Khac propose is to distinguish between signals, which they can measure, and concepts, which are the phenomena being indicated by the signals. The distinction is useful and it’s also pretty much standard in social science, though not necessarily in those terms. While the mechanisms responsible for the operation of, for example, an automobile, can be observed directly, those operating in human society or the human mind are generally inaccessible to direct observation.

In order to investigate those mechanisms one must propose a theory or a model and then operationalize that theory. What does that mean? Moretti (p. 1):

Forget programs and visions; the operational approach refers specifically to concepts, and in a very specific way: it describes the process whereby concepts are transformed into a series of operations—which, in their turn, allow to measure all sorts of objects. Operationalizing means building a bridge from concepts to measurement, and then to the world. In our case: from the concepts of literary theory, through some form of quantification, to literary texts.

Just so.

However, that is often easier said than done. The problem is exacerbated when the measuring instruments are new and their capacities not yet fully explored. That’s the situation humanists face with the various techniques for statistical analysis of large bodies of texts – corpus linguistics – that have come to our attention in the past decade or so. Consequently one must devote a certain amount of time to fishing expeditions, poking around to see just what one can see. You see what signals your instruments can detect and then figure out what concepts are needed to explain them.

Thus Heuser and Le-Khac started with a general interest in Raymond Williams’s broad “premise that changes in discourse reveal broader historical and sociocultural changes” (p. 2). They then went fishing for signals that could be taken as evidence of such changes. After considerable work in developing their tools they finally detected a signal they could interpret through Williams.

Bingo!

It goes without saying, of course, that logic now requires us to predict other signals that should be a consequence of the proposed mechanism – a population shift to the cities engendering more contact among relative strangers – and see if our instruments can detect them. That brings us to Arthur Stinchcombe’s book, Constructing Social Theories, which is a high-level methodology manual.

I studied with Arthur Stinchcombe when I was an undergraduate at Johns Hopkins back in the 1960s. I took a course in social theory, one of those courses for upper level undergraduates and beginning graduate students. For our final project we had to 1) pick some social phenomenon; 2) provide a theory or model to account for that phenomenon; and 3) list three consequences of that theory that could be detected by empirical means. The phenomenon we choose could be real or imagined. Stinchcombe didn’t care which; he just wanted to see how we reasoned.

We weren’t to choose some large phenomenon – e.g. the rise of capitalism. It should be something on a modest scale. I believe I choose to write about poor student attendance at student-faculty coffee hours, an activity I’d been involved with. Beyond the general topic I remember almost nothing about my paper. But I enjoyed the course and thought Stinchcombe was very sharp.

When he published Constructing Social Theories I snapped it up and read it from cover to cover. It’s not that I wanted to be a sociologist, or any other kind of social scientist. But I figured it would be useful to be able to think like one.

That’s why I mention the book. It was written by a man who knows that, to be useful, a theory must be testable. It’s a handbook on operationalizing social theories.