What’s in a corpus? Words, words organized into texts. Of course.
But that obvious answer not quite what I’m after. I’m interested in how we think about corpora, their role in our work. “We”, who’s that? I’m not sure it much matters, not exactly. It will emerge.
How are these corpora connected to the world? What do we hope to understand about the world by analyzing these corpora? What are our
intuitions in these matters. To some extent I’m trying to track down something I don’t know how to conceptualize. This post from October 20, 2017 is a good example of that:
But when stalking such an abstract beast, it helps to have specific examples in mind. So I’m thinking about the role of corpora in statistical machine translation (MT) vs. their role in topic analysis. Roughly speaking, in MT statistical analysis of corpora are is a means to an end. In topic analysis statistical analysis of a corpus is the end. That difference entails a somewhat different way of thinking about corpora.
Martin Kay, an “ignorance model”
I’m basing this post on some observations by Martin Kay, one of the grand old men of MT. Kay apprenticed with Margaret Masterman at the Cambridge Language Research Unit in the 1950s. In 1951 David Hays hired him to work with the RAND group in MT. He went on to a distinguished career in computational linguistics at the University of California, Irvine, the Xerox Palo Alto Research Center, and Stanford.
Research in MT was undertaken to achieve a practical end, the translation of texts from one language to another. The United States government was particularly interested in obtaining translation of Russian texts. The researchers who undertook this work had various motivations, but some of them were interested in linguistic science and were happy enough to have their work funded by a government agency, the Department of Defense, with a practical goal.
In 2005 the Association for Computational Linguistics gave Kay a Lifetime Achievement Award. On that occasion he looked back over his career and made some observations about the relative merits of statistical and symbolic approaches to MT [1]. He speaks as a man fundamentally interested in basic knowledge who has, however, at times undertaken work with practical ends.
At the beginning of the following passage Kay distinguishes between computational linguistics and natural language processing (NLP). The distinction is a common one, albeit a bit problematic as well [2]. But the distinction Kay makes is clear enough (p. 5):
Computational linguistics is not natural language processing. Computational linguistics is trying to do what linguists do in a computational manner, not trying to process texts, by whatever methods, for practical purposes. Natural Language Processing, on the other hand, is motivated by engineering concerns. I suspect that nobody would care about building probabilistic models of language unless it was thought that they would serve some practical end. There is nothing unworthy in such an enterprise. But ALPAC’s conclusions are as true today as they were in the 1960’s—good engineering requires good science. If one’s view of language is that it is a probability distribution over strings of letter or sounds, one turns one’s back on the scientific achievements of the ages and foreswears the opportunity that computers offer to carry that enterprise forward.
I agree with Kay’s fundamental point, though I note that humanists using NLP techniques are often pursuing basic knowledge rather than a practical end.
Kay wrote that passage in 2005. I don’t know just when literary critics first began exploring NLP techniques, but I first became aware of digital humanities work in topic modeling sometime in 2012 [3]. That’s well after Kay wrote those words.
Let’s return to his remarks. Nearing the end of his talk, Kay remarks (p. 12):
My professional life almost encompasses the history of computational linguistics. But I was only fourteen when Warren Weaver wrote his celebrated memorandum drawing a parallel between machine translation and code breaking. He said that, when he saw a Russian article, he imagined it to be basically in English, but encrypted in some way. To translate it, what we would have to do is break the code and the statistical techniques that he and others had developed during the second world war would be a major step in that direction. However, neither the computer power nor large bilingual corpora were at hand, and so the suggestions were not taken up vigorously at the time. But the wheel has turned, and now statistical approaches are pursued with great confidence and disdain for what went before. In a recent meeting, I heard a well known researcher claim that the field had finally come to realize that quantity was more important than quality.
The young Turks blame their predecessors, the advocates of so-called symbolic systems, for many things. Here are just four of them. First, symbolic systems are not robust in the sense that there are many inputs for which they are not able to produce any out- put at all. Second, each new language is a new challenge and the work that is done on it can profit little, if at all, from what was done previously on other languages. Third, symbolic systems are driven by the highly idiosyncratic concerns of linguists rather than real needs of the technology. Fourth, linguists delight in uncovering ambiguities but do nothing to resolve them. This is actually a variant of the third point.
Kay mounts a quick defense on the first three points, but says a bit more about the fourth, ambiguity (pp. 12-13):
This, I take it, is where statistics really come into their own. Symbolic language processing is highly nondeterministic and often delivers large numbers of alternative results because it has no means of resolving the ambiguities that characterize ordinary language. This is for the clear and obvious reason that the resolution of ambiguities is not a linguistic matter. After a responsible job has been done of linguistic analysis, what remain are questions about the world. They are questions of what would be a reasonable thing to say under the given circumstances, what it would be reasonable to believe, suspect, fear or desire in the given situation. If these questions are in the purview of any academic discipline, it is presumably artificial intelligence. But artificial intelligence has a lot on its plate and to attempt to fill the void that it leaves open, in whatever way comes to hand, is entirely reasonable and proper. But it is important to understand what we are doing when we do this and to calibrate our expectations accordingly. What we are doing is to allow statistics over words that occur very close to one another in a string to stand in for the world construed widely, so as to include myths, and beliefs, and cultures, and truths and lies and so forth. As a stop-gap for the time being, this may be as good as we can do, but we should clearly have only the most limited expectations of it because, for the purpose it is intended to serve, it is clearly pathetically inadequate. The statistics are standing in for a vast number of things for which we have no computer model. They are therefore what I call an “ignorance model”.
That last point is very important.
By the mid-to-late 1960s computational linguists began to realize that they would have to tackle semantics, which covers the relationship between language and the world, in contrast to syntax, morphology, and phonology, which are all internal to the language system. And so computational linguists did that, along with cognitive psychologists and researchers in artificial intelligence. The work then, and now, was interesting and fruitful, but not terribly useful for practical tasks, such as MT. By the 1990s the conjunction of large amounts of cheap computing power and large bodies of digital texts gave statistical approaches a definitive edge in practical applications, an edge which remains.
