An undiscovered country

The scientific literature comprises scholarly publications that report original empirical and theoretical work in the natural and social sciences.

It is the principal medium for communicating the results of scientific research and, as such, represents the permanent record of the collective achievements of the scientific community over time. The scope of the literature is hard to conceive. More than 80% of the scientific research literature is published as articles in subject-specific journals. Estimates suggest that more than 1.5 million such papers are published in over 25,000 peer-reviewed journals annually and that over 50 million have been published since the first scientific journal was published in 1665.

The contribution each of the constituent parts of the literature makes to our collective scientific knowledge is a subject of some debate. Value is often determined by the number of times an article is cited in other scientific articles. Over the last decade a new field of study has exploded onto the scientific landscape – that of bibliometrics (you can download a useful description of the field here). Qualitative analysis of the scientific literature is changing rapidly with the creation of new evaluation tools, parameters and normative data. These parameters can be categorised into author or journal focused metrics. Once you have an understanding of the field of bibliometrics it is immediately obvious that publication metrics fail to give a true assessment of research ‘quality’, whatever that may be; it is not yet possible to derive a simple indication of the value of a piece of work. You still have to determine that for yourself.

It has been known for some time that a proportion of scientific papers are never cited. The extent and value of the uncited literature has been the subject of some speculation and discussion. Previous estimates have suggested that up to half of all papers have yet to receive their first citation 5 years after publication, and even 10% of papers by winners of the Nobel Prize go uncited. One often repeated statistic is that more than half of all academic articles remain uncited at 5 years. Researchers often worry that high numbers of uncited papers indicate a useless or irrelevant direction of research.

In order to get a better idea on the extent of this undiscovered country of published research, Nature recently (here) performed its own analysis of just how many papers go uncited. In their article it is noted how difficult it can be to get a grasp on accurate numbers. In this case, the author reviewed data on a core group of 12,000 journals catalogued in the Web of Science. They found evidence that previous estimates are far too high. Looking at recent papers it appeared that fewer than 10% remain uncited. This may also be an over estimate as data recorded on Web of Science does not include citations in journals that are not included on the database they used or appear in books, patents, etc. As for the work of Nobelist’s, it seems that just 0.3% of their work goes uncited.

A closer look at the data suggests that the proportion of uncited papers has been higher in the past and uncited numbers falling steadily. For example, 20% of papers published in 1980 still don't have a single citation today. This could well reflect the revolution we have experienced in the way we undertake background research following the emergence of electronic literature search engines over the last 30 years in addition to the growth in the number of references cited in each paper over this same period.

The authors of the Nature paper note that we shouldn’t assume that an uncited paper is worthless – it is still possible for the work to effect researchers and practitioners. It can still influence readers who may change the way they do something or think but who may not necessarily make comment about the change within the literature. In itself this is worrisome. We have long known (here) that a high proportion of clinical trials, for example, go unpublished – effectively excluding any findings, positive or negative, from consideration. Here we enter a truly undiscovered country. Although it has been suggested that the pharmaceutical industry, in particular, is hesitant to publish negative results, much research suggests that this is not specific to industry-funded projects. In fact, there are serious concerns that in their pursuit for higher publishing metrics journals themselves are biasing the data available to researchers.

The idea that the literature is awash with uncited research goes back to a pair of articles published in Science in the early 1990s. The present evidence shows that concept these reports introduced is now an anachronism and misleading. The Nature article also points out that the original work might have been flawed in that the authors counted documents such as letters, corrections, meeting abstracts and other editorial material, which most scientists accept wouldn’t usually get cited. When these data are removed, leaving only research papers and review articles, rates of uncitedness are reduced.

The last 30 years have seen the way that we undertake our background research change beyond all recognition. Gone are the days of spending an afternoon leafing through the latest version of Index Medicus in the library, taking notes of relevant papers to review in the dusty journals kept in ‘the stacks’. They have been replaced elegant Boolean search strategies performed on multiple data platforms and generating comprehensive candidate articles to review and rank for relevance and importance. Younger researchers, born into these systems now adopt a learn-as-they-go approach, adapting their searching strategy ‘on the fly’. And this is perhaps where the true value of the whole body of the scientific literature comes in. Progress in science occurs through both evolution and revolution. The two are complementary and not mutually exclusive. The body of our research work as a whole, positive, negative and revolutionary, empowers progress. Although it may go uncited, even the most uninspiring paper can impact on our thinking and empower our next great discovery.