What makes empirical research ‘scientific’?
Daniel Gile
March 2009
Gartner Group is a company which provides inter alia ‘briefings’ for corporate executives in matters pertaining to the evolution of Information Technology in the marketplace. Its ‘analysts’, as they are called, often give half-day presentations packed with synopses of market evolution worldwide as well as short-term and medium-term predictions which can only be based on extensive research. I have been interpreting at such briefings for many years and have recently had the opportunity to talk to two of Gartner’s analysts and ask them how they arrived at their analyses and predictions. The answer was that in their case (in their particular field of analysis), most of the information comes from their clients. I asked them whether they conducted questionnaire surveys and then processed them statistically, and they answered that most of the information came from personal talks with the clients and with very little statistical processing, and added that this was not “scientific”.
This was an opportunity to reflect upon traditional classifications of research, including my own as set out in a number of texts in this Research Issues section of this website. The systematic collection of information and its analysis for the purpose of generalizing was clearly an essential part of these analysts’ work, as it is an essential part of naturalistic research in academic research work. What are the core differences, if any, between the research done by these two analysts and empirical research in the academic sense of the term (besides the lack of references to existing theories and previous studies of the same or similar phenomena and besides the fact that their research was not reported in an academic publication or meeting)? To me, the main difference lies perhaps in the fact that the two Gartner analysts did not present the data on which their inferences were based as well as the data collection and processing methods they used and the rationale they followed when making inferences. This would make empirical scientists reject their study as bona fide empirical research. But the same could be said about the type of academic research I have called LAP research in other texts in this Research Issues section over the past few years. Could one therefore say that Gartner research is similar to LAP in academic research? Probably not, because it is not a reflection on phenomena and theories, but rather an exercise focusing on data, with limited aims with respect to generalization. On the other hand, LAP researchers may conduct extensive data collection before producing their analyses. Perhaps within LAP, one could talk about ‘empirical LAP studies’ where much information is collected systematically before reflection is attempted (the information being either data on the object of reflection or information on the existing literature on the object of reflection), and about ‘reflective LAP studies’ where reflection is the main component of the research whereas data collection is more limited and less systematic. Finding a solid categorization system for academic research is a tricky exercise, with many fuzzy boundaries. I find that defining provisional categories is useful as a reference to explain some differences and highlight specific norms, especially when training young researchers, but giving them ‘final’, ‘absolute’ status would be unwise.