Sampling 2: generalizing from case studies
Daniel Gile
18 July, 2006
In a previous text (Sampling-based studies as case studies), it was pointed out that generally, one or more dimensions of a phenomenon are controlled in a study but other relevant dimensions are not, and in that respect, even studies with relatively large samples can be viewed as case studies. Even when a relevant parameter is controlled, it is generally set at just one, two or three values as opposed to many more values it can take in real life. How legitimate is it to generalize from the findings of such studies?
For example, if differences are found in the quality of a translation produced by experienced translators as opposed to novices, but all translators in the sample have the same language combination and have not received formal translator training and the text used was 300 words long, is it legitimate to generalize the findings to the population of translators at large, which includes translators having different language combinations and/or having received formal translator training? Can one assert that the same differences would have been found with texts of 100, 600, 3000 or 10 000 words? (or with texts having different linguistic features, content features, etc.?)
The answer is yes when previous studies have shown that the unattended parameters or parameter values make no major difference in the relevant dependent variable, for instance that basically, translator behavior is similar whatever the working-language combination, whatever the length of the source text and regardless of whether subjects have had formal translator training or not.
One can also conduct replications of the study with various values of all parameters considered relevant to check whether they generate differences or not. For instance, with texts of different lengths, with different language combinations and with translators with formal training. If findings do not vary, scholars have a more solid ground for generalization.
Clearly, the amount of work involved in such replications is considerable, and requires commitment from many researchers over time and other resources. Such efforts are generally devoted only to topics viewed as particularly important or interesting. Many studies, perhaps all empirical studies in TS (not in medical science, in physics, in psychology or in other established empirical disciplines) are not replicated sufficiently for generalizability to be documented.
In such a case, theoretically, according to strict ESP logic, generalization is only legitimate within the space defined by the parameters attended to in the study (for instance to the population of translators without formal training working in the year in which the study was carried out in the specific language combination tested and translating a text similar to the one used in the study).
Such strict application of ESP logic would make most studies virtually useless. What scientists necessarily do is choose what they believe to be the most relevant parameters and the most relevant range of values these can take and control them in the hope that the non-attended parameters and parameter values will not make much difference. On this basis, they do generalize, though they always remember the tentative status of their findings-based inferences. The traditional statement at the end of reports saying that results need to be confirmed in further studies is more than an empty formula.