The editorial process through the looking glass
Daniel Gile – Université Lumière Lyon 2
Gyde Hansen – Copenhagen Business School
Published in Gyde, Hansen, Kirsten Malkmjær and Daniel Gile (eds). 2004. Claims, Changes and Challenges in Translation Studies. Amsterdam/Philadelphia: John Benjamins. 297-306.
Abstract
Editors of scholarly publications are gatekeepers who decide whether to publish manuscripts or not, but their decisions are constrained by institutional factors, sociological factors, and the availability of referees for peer-reviewing. The quality of the input of such referees depends not only on their thematic and methodological expertise, but also on their motivation, on their personal bias and on sociological factors. Although this suggests that editorial decisions are less objective and reliable than their idealized representation, the referees’ comments are often convergent to an encouraging degree. The main comments made by referees on 42 manuscripts submitted to these proceedings are listed. They share a large common denominator, which suggests that the fundamental norms they are based on are the same.
1. Introduction
Conference proceedings are a well-established tradition, and are often awaited impatiently not only as a reflection of the meeting, but also as a publication vector for participants, and frequently as a justification to their university for their presence and participation in the event. However, publishers do not necessarily allow sufficient space for all papers to be published, and editors must choose. How they do this is not always clear to authors who submit their manuscripts.
Basically, the process is very similar to the editorial process in other types of collective volumes and in journals in that it is based on the peer review system. This paper discusses the concept, as well as recurrent issues, in particular as they crystallise in the field of Translation Studies; it then discusses issues in refereeing, and briefly mentions follow-up possibilities after comments are received from the reviewers. This paper is based on its authors’ experience as editors in the field, and also includes, by way of illustration, a case study with an analysis of the most common types of weaknesses reported in papers submitted for publication in proceedings in the field of TS.
2. The editorial process
2.1 The editor’s role as a gatekeeper
In the scientific community, where a scholar’s publications are his/her visiting cards, passports and stepping stones towards professional promotion, editors of journals and collective volumes have an important three-pronged institutional role to play. Firstly, they create and offer publication space to individual scholars. Secondly, they act as gatekeepers and guarantors of the scientific quality of texts they publish (see Judd et al. 1991:17f). Thirdly, through the peer-review system, they offer guidance and text-improvement opportunities.
As gatekeepers who decide to publish or not to publish, editors seem to be in a position of power. And yet, their action is constrained by several factors, including constraints from the publishers and pressure from the research community, difficulties in finding adequate referees for proper implementation of the peer review system, and “political” and diplomatic considerations that inevitably influence their decision-making process.
2.2 Constraints and policies
Thus, a system which is theoretically designed to provide reliable guidance and assessment to editors in their role as gate-keepers in the publication process has in-built limitations which can be difficult to overcome.
To these limitations, one should add the expectations of the publisher in some cases, and of the researchers’ communities in other cases (for instance, when editing the proceedings of a conference), let alone the expectations and reactions of individual researchers, who may need to publish for institutional or personal reasons and who may find it difficult to accept the editors’ decision to reject their paper, or even the critical comments from the referees.
Several types of policy are observed in the field. The easiest one to implement consists in accepting virtually all papers, and merely performing some cosmetic surgery on the wording, correcting errors in the lists of references, re-formatting papers to applicable standards. While this may not be the most efficient policy in terms of scholarly quality and promotion of solid scientific norms, it can be considered legitimate when the purpose is only to reflect the state of the art in a field, or to provide the written record of a scholarly meeting.
At the other extreme, editors may decide to implement a strict quality policy, accepting only manuscripts which pass severe scrutiny. Such a strategy is possible for editors of prestigious journals in established disciplines where there is much research production and much competition. In TS, due to the small size of the community and the general state of research, it is doubtful whether any editorial board can afford to act in such a strict manner.
Most often, the editorial policy with respect to quality is somewhere in-between, and quality thresholds for acceptance vary as a function of supply and demand: if many good papers are offered and fill the required number of pages, there is no need to accept mediocre submissions. When there is more publication space than material, acceptance standards fall.
In such a situation, it is difficult to set standards in advance. For each issue of a journal or collective volume, submissions may have to be refereed first, and the decision to publish or not to publish is taken when the total picture is clear.
2.3 Referees in the peer review system
As its name indicates, the peer review system is based on the work of scholars with a similar background to the authors’, whose role is to provide useful input, which takes the form of both assessment for the benefit of the editor, and guidance to the authors. It is significant, but not often noticed, that in its very name, the system refers to “peers” rather than to “experts” or some other authority-carrying agent: the idea, which is corroborated by experience very regularly, is that even without a substantial difference in training and experience between referees and authors, the very fact that it is easier for anyone to detect another person’s weaknesses than one’s own, makes it possible for the former to help the latter (“…well-trained scientists can make mistakes and sometimes overlook important problems of design” – McBurney 1990:16).
So much for theory. In practice, the system is a bit more complicated. Firstly, “peers” are not always peers. Sometimes, acknowledged authorities are asked to do the reviewing, and sometimes, referees are much less experienced and qualified than the authors themselves. In the ‘guidance’ part, i.e. specific comments on errors or weaknesses and specific suggestions for improvement, this is generally not too problematic. In fact, excellent criticism is often received from conscientious beginners, who may react with high sensitivity to all segments in a text which contain unclear, ambiguous writing or faulty logic, whereas more experienced readers already familiar with the field, who may also be busier, may focus on more specific points and overlook such segments. With respect to overall assessment of manuscripts by referees as opposed to specific comments, experienced, more highly qualified scholars are clearly in a better position than beginners to provide useful input.
In established disciplines with a large community of scholars, referees may not be too difficult to enlist. In other cases, and in particular in the rather small, but diversified TS community, finding such reviewers may be a real problem. Firstly, in some sub-areas of a discipline, there may simply not be more than a very small number of scholars competent enough for refereeing purposes. The problem in international events (as opposed to national ones) is compounded in papers written in languages other than English, which has more or less become the international language of science, in the field of TS as in other fields. More generally, refereeing effectiveness can be discussed under several components:
2.3.1 The language issue
Though not a scientific or intellectual criterion, the language barrier is formidable, even in a multilingual society such as the TS community (or perhaps especially in this community, where authors may insist on their right to write in their language and count on their fellow-researchers’ comprehension). Not only are language skills essential when assessing and/or correcting the style of a text, but they are also important in terms of comprehension, especially in theory-oriented papers. How many TS scholars can review German texts on court interpreting, Spanish texts on research methodology, French texts on contrastive linguistics and translation? The problem was a very concrete one in this volume as well, when scholars well versed in certain fields turned out not to have the language in which the relevant papers were written. In several instances, the less than ideal choice was between refereeing a paper written in a language one knew well, but in a field not familiar enough, and refereeing a paper addressing a field one knew well, but written in a language in which one did not trust one’s comprehension ability fully.2.3.2 ‘Thematic’ expertise
The requirement for sufficient relevant knowledge on the specific theme in the field is obvious, but in practice, it is not always easy to meet, because editors do not always have access to referees with the required qualifications. When they lack ‘thematic’ knowledge, readers may miss errors in the manuscript, especially in the literature review and references, but also in the interpretation of quotations and in the use of concepts. On epistemological and methodological issues, they may also fail to see the relevance of certain points made by authors, or miss gross errors.When the referee’s lack of familiarity with a field leads to unjustified criticism, the refereeing process can become counter-productive. The most conscientious and careful referees faced with this dilemma tend to focus on the structure and rationale of the paper and to formulate questions rather than assertive criticism when in doubt, thus alerting the authors and editors to the existence of potential problems, without making any damaging claims.
2.3.3 Methodological expertise
In a field as heterogeneous as TS, research paradigms are diverse and quite different from each other. Some scholars may adhere to the methods and norms of cognitive psychology, others to those of literary studies, of philosophical discourse, of sociology, of formal linguistics, etc. These are not necessarily mutually compatible, and if referees adhere to one paradigm and the author of a refereed text to another, this may lead to negative assessments predicated on an opposition between schools of thought, not on actual weaknesses in the paper within the paradigm chosen by the author. Challenging a paradigm is legitimate, and a potentially powerful driver of scientific progress, but criticising an author’s method using the criteria of a paradigm other than the one s/he has chosen to follow can be counter-productive (see Kjørup 1996: 109, 113, 116 f.).2.3.4 Motivation
However ‘objective’ and impersonal the peer review system is supposed to be within the idealistic view of science, personal factors may come in strongly in the refereeing process. Critical reading is serious work, as it involves reading a text carefully,making sure one understands it, assessing it, reporting on the findings, and making suggestions for improvement. If there are serious design flaws, and if weaknesses are numerous, it may take much time and effort to list them comprehensively and to suggest alternative formulations, inferences or actions to help an author understand the problem and do the appropriate remedial work. In such cases, the willingness of referees to devote considerable work to the process can make much difference, and it is not rare to see one reviewer only send back one or a few general paragraphs as input to the editors, and another offer several pages of detailed comments on the same paper.2.3.5 Personal bias and sociological factors
One final component, which is not in line with the ideal of objectivity in science, but which is nevertheless endemic in human society, is personal bias. Scientific society is a competitive one, both in terms of theories and schools of thought and in terms of personal rivalry (see Plastow and Igarashi 1989). Actually, as analysed by Kuhn and many others (see Kuhn 1970, Chalmers 1990), such sociological forces may reasonably be considered useful in that they drive researchers into accumulating evidence to attack existing paradigms (or defend them, depending on which side they are on), thus contributing to scientific development. Nevertheless, in an activity having such a strong assessment component as refereeing, positive or negative attitudinal bias, whether conscious or subconscious, can result in unbalanced judgement. Scholars may be eager to find fault with their opponents’ or rivals’ work, or be reluctant to criticise their allies’ and friends’ writings.The problem is compounded by status issues: when reading a well-known, experienced scholar’s work, referees may be reluctant to criticise, not only out of respect for the author, but also out of fear of ill feelings or ‘retaliation’ if recognised. Indeed, in a community as small as that of Translation scholars, it is often easy to recognise an ‘anonymous’ author and an ‘anonymous’ referee through their distinct style, ideas and references.
3. Common problems in submitted papers: a case study
3.1 Introduction
After the EST Congress in Copenhagen, more than 50 papers were submitted for inclusion in the proceedings. Following the tradition and conventions of scientific editing and publishing, they were submitted to the peer-review process. All papers were reviewed by at least 2 referees, and most of them by three. In a few cases, only general assessments were received (see above), but specific comments were made on 42 of them. Referees mentioned both positive aspects, such as the presence of innovative components, interesting information, interesting ideas, etc., and problematic areas. In line with its focus as explained in the introduction, this paper analyses the latter, where constructive work could be done, but should not be misinterpreted as a pessimistic statement or diagnosis.
The following types of perceived problems were reported by referees. Since this is a single case study, they are given not as a demonstration of any general proposition, but by way of an illustration of what we consider a not atypical pattern.
Also note that for reasons that readers will understand easily, no attempt will be made to present the full set of data, with all the comments made by each referee on each paper. This information will remain confidential, and only a synopsis will be given here, in the hope that it will be sufficient to provide interesting and useful input.
3.2 Frequently mentioned weaknesses in papers for this volume
§ The most numerous comments from referees report missing data: in order to gain better understanding of the author’s claims or conclusions, reviewers felt that they needed data which was not provided in the relevant papers, on experimental conditions, on actual results, on the nature of the texts involved, etc. (also see Strauss/Corbin 1998: 268ff, Judd et al. 460ff.) This problem was reported in 18 out of the 42 papers (43%) for which detailed comments were received from referees.
§ In a related but not identical category, in 11 papers (26%), reviewers reported problems with the bibliographical references, which were missing, incorrect, and/or used inappropriately from secondary sources (also see Gibaldi 1999).
§ The second largest number of comments pertained to weaknesses in the rationale of papers. Reviewers reported that authors made unsubstantiated or excessive claims, speculated, mixed categories, over-generalised, and more generally made inferencingerrors in 17 out of the 42 papers (40%) (see Judd et al. 1991: 27ff, 258 f. on various types of validity and bias).
§ This problem in the rationale is linked with problems in the design of empirical studies. Out of 20 papers reporting such studies, 12 (60% – the highest proportion) were reported as having a design that did not correspond to the stated aim of the study, or as comparing groups that were not comparable, or using samples that were too small, or not representative, or as failing to control relevant variables (also see Judd et al. 1991: 75ff, 215ff.).
§ The third largest set of comments considered that papers (13 out of 42 – 31%) presented no innovation or added value.
§ Other frequent comments had to do with specialised concepts, which were not defined at all or not defined clearly enough, or used in a way different from their established use in other disciplines or in paradigms from which they were taken (9 papers out of 42 – 21%).
§ Referees also commented on the excessive length of papers or segments thereof, in particular the introduction (8 papers out of 42 – 19%).
§ Still other comments reported clumsy or pompous style, repetitions in the structure of the paper, missing units in graphs, abbreviations and acronyms which were not spelled out, lack of indications as to the objectives of the papers or studies (see Judd et al. 1991: 472 about procedure and style).
3.3 Assessing the truthfulness of the referees’ comments
Before analysing these data, a few points must be made: Firstly, no strong claim is made as to the degree of representativeness of the sample of papers, of referees and/or of comments. The numbers are given here as a rough indication. Nevertheless, the following could be pointed out:
§ The set of more than 42 papers reviewed for these proceedings can be considered of reasonable size for a sample of this nature, and we (the authors of this paper and editors of this volume) did not feel it differed markedly in respect of problem areas from other sets of papers received and reviewed for other volumes in the field in recent years.
§ An important point is that while there were a few disagreements (less than a handful) between the referees and/or editors on the general value of papers, no disagreement was expressed on any specific comment. Thus, while it is not claimed that referees are holders of the truth (they are as fallible as any scholar – see the discussion in Section 2.3), in this case, there is good reason to consider that their judgement on specific weaknesses has some reasonable inter-subjective value
§ Referees and editors come from different countries and have different backgrounds. These factors suggest that the picture which emerges is not likely to be a highly biased reflection of the general situation in TS. The only potentially marked difference between this case and other, similar cases, is a possibly stronger will on the part of the editors (there was at least one editor among the referees of each paper) to stress quality in the editorial process, in spite of the other factors mentioned in Section 2.1.
3.4 Analysing the data
Looking at the data, it appears that referees have reacted on the basis of a number of norms, which are made explicit in research textbooks (see for instance, McBurney 1983, Shipman 1988, Babbie 1992, Judd et al. 1991, Gibaldi 1999, Strauss/Corbin 1998), in particular the requirements that science be explicit, in providing all the information required for readers to be able to understand and assess reports on scientific studies (“Great care is taken to specify the exact conditions under which observations are made so that other scientists can repeat the observations if they desire and can try to obtain the same results as the first scientists”- McBurney 1990: 7; also see Strauss/Corbin 1991: 266f about replicability), that it be logical in its inferences, and that scientific endeavourscontribute at least some innovation, rather than just review existing data and information (in the words of the editor of the Journal of Bone and Joint Surgery who gave a plenary at a medical conference in Paris in the early nineties, “the reviewer will ask three questions: Is it true? Is it new? Does it matter?” – also see Beaud 1988).
A strikingly large proportion of papers received, more than one in three, did not observe these principles. This suggests a lack of specific awareness of the importance of these norms in research which, in turn, could be ascribed to the lack of appropriate training in research principles and their concrete implementation.
A third interesting observation is the relatively high number of problems arising from trans-disciplinary ventures. As they explore new avenues with theories and concepts from related disciplines, scholars with a solid track record in TS seem to become susceptible to misperceptions. Beyond the case study of this volume of proceedings, problems have been observed when TS scholars without the corresponding background ventured into statistics, philosophy of science, linguistics, and cognitive psychology. Such weaknesses could probably be avoided by co-operating with researchers from the relevant disciplines, and, of course, by appropriate training in such disciplines.
4. Referees’ comments and beyond
The next and obvious questions are, firstly, what authors can/should do after they receive the referees’ comments. Most authors only view the peer review process as an assessment, and focus on its short-term effects in terms of acceptance or rejection of their manuscript and of requirements for rewriting. However, it could also be an excellent opportunity to improve research and writing skills. Comments can be classified into three levels of
§ Some comments are easy to deal with. Following minor suggestions for grammatical correction and stylistic improvement, correcting factual errors, correcting errors in the bibliographical references, improving the graphical presentation of data are ‘mechanical’ and should cause no problems to authors.
§ At a second level of difficulty, the referees’ suggestions challenge the authors’ ‘local’ assessments (on specific points): does a previous finding really mean what the paper says it means? Does another author really say what the paper reports it is saying? Is a quoted piece of work good enough to be used as a reference? Should another study not be cited to strengthen the case? Is the newly coined terminology appropriate, or should it be replaced by better terms? Should a particular assertion based on the findings not be toned down?
§ At the highest level of difficulty are suggestions that challenge fundamental choices and inferences made by the author. In some cases, if the design of an empirical study is seriously flawed, if a theoretical discourse on a concept, model or theory is predicated on a serious misunderstanding of its nature, the paper cannot be salvaged. In other cases, it is possible to improve it without starting again from scratch. For instance, the data may be re-processed with a different method, and findings can be re-interpreted, which may lead to a new conclusion, which may not be in line with the ideas that authors set out to defend. Again, this may be associated with considerable psychological difficulty for them.
5. Conclusion
The peer-review process and the subsequent decisions and actions discussed here are not the only components of editorial work. Upstream, there are contacts with a or several publisher(s), definition of editorial policy, sometimes calls for papers with specific guidelines as to the length and format of texts. Downstream, when revised papers have been accepted, there is still much reading, checking, writing, copy-editing, etc. However, the most stimulating and constructive part certainly lies in dealing with the issues highlighted in this paper.
The preparation of proceedings involves a chain of actions, decisions and reactions from authors, referees and editors. However clear and straightforward the principles may be, their implementation is therefore lengthy, complex, and sometimes a bit painful. However, if referees and editors have enough time and good environmental conditions for serious work on the manuscripts, and if authors accept their comments as potentially useful input, the editorial process offers excellent skill honing opportunities for all parties in the process.
References
Babbie, Earl. 1992. The practice of social research. (Sixth Edition). Belmont, California: Wadsworth.
Beaud, Michel.1988. L’art de la thèse. Paris: Editions la découverte.
Chalmers, Alan. 1982. What is this thing called Science? (Second Edition). Milton Keynes: Open University Press.
Gibaldi, Joseph. 1999. MLA Handbook for Writers of Research Papers. (Fifth Edition). New York: The Modern Language Association of America.
Judd, Charles M. & Eliot R. Smith & Louise H. Kidder. 1991. Research Methods in Social Relations. (Sixth Edition). Orlando: Harcourt Brace Jovanovich College Publishers.
Kjørup, Søren. 1996. Menneskevidenskaberne. Roskilde: Roskilde Universitetsforlag.
Kuhn, Thomas. 1970. The structure of scientific revolutions. (Second Edition, enlarged). Chicago: The University of Chicago Press.
McBurney, Donald. 1990. Experimental Psychology. Belmont, California: Wadsworth.
Plastow, Michael & Yoshihide Igarashi. 1989. The Mind of Science. Tokyo: Kyoritsushuppan.
Shipman, Martin. 1988. The limitations of social research. (Third Edition). London and New York: Longman.
Strauss, Anselm & Juliet Corbin. 1998. Basics of Qualitative Research. (Second Edition). California: Sage.