Currently peer reviewed journals are peppered with so-called systematic reviews. However, on closer inspection it becomes obvious that the search strategies used to locate evidence for many of them are anything but systematic. Database searching is the initial investigative stage of any systematic review following the development of the protocol and is analogous to laying the foundations of a building; however magnificent the architectural conception, if the foundations are not sound, the entire structure will be at risk, and although the foundations may be the most unexciting part of the project, they are crucial to the integrity of the building. My favourite article on how not to search is, "A surrealistic mega-analysis of redisorganization theories",1 a wonderful send-up of evidence based medicine by David Sackett and others. Unfortunately, it appears that these or similar methods are often applied in a more serious context. Let's look at some of the most frequently published faux pas in so-called systematic review searches.

The first is commonly found in the list of databases searched. "We searched PubMed, MEDLINE[horizontal ellipsis]" PubMed includes MEDLINE,2 so listing both indicates that neither the authors nor the peer reviewers were aware of this. Not a good start to establishing credibility. MEDLINE is also available on other platforms such as Ovid, however the content is identical to that used in PubMed; it is simply leased from the National Library of Medicine and made available on the new provider's platform. A second common fault is the inclusion of ScienceDirect which often appears in the list of databases searched, but is essentially a searchable collection of journals and books from specific publishers. The content is therefore limited, and the search function lacks sophistication. Databases, on the other hand, are not limited by publisher, and are generally designed to allow highly refined searching. Google Scholar is another frequently cited source, and while it certainly has its uses, it does not allow complex searching, nor can the results obtained from a Google Scholar search be reliably replicated

Next comes the description of the search undertaken. "We searched [list of databases] [horizontal ellipsis]using the following keywords[horizontal ellipsis]" This indicates that neither the searchers nor the peer reviewers understood that different databases have different indexing languages, and may use quite different terms for the same concept. For example PubMed's indexing term for "heart attack" is "myocardial infarction", whereas Embase uses the indexing term "heart infarction". A search suited to one database cannot simply be transferred to another. This example also demonstrates ignorance of the range of potential fields that can be searched in many databases - titles of articles, abstracts of articles, authors' keywords, and in some cases the full text of articles. Failure to specify a search field often means that the database's algorithm generates bizarre results by interpreting the search in a completely unexpected manner.3 "We searched PubMed using the following MeSH (indexing terms)[horizontal ellipsis]" Although this search method sounds more convincing, unless correct syntax is used, the search may not work as anticipated.4 Secondly, it does not allow for the possibility of indexer error, or variation in the way indexers may interpret the content of an article. Restricting searching to MeSH alone fails to allow for searching of the hundreds of thousands of new, unindexed articles supplied to PubMed by the various publishers.

The same principle applies to other databases where new material will be unindexed, and therefore accessible only by searching in titles, abstracts or authors' keywords. In Embase, new material is computer indexed, using thesaurus (Emtree) terms, and later checked for accuracy by indexing staff. Recently, I worked with a higher degree student who was researching oropharyngeal dysphagia. I was dismayed to see she had included in her search the Emtree term "swallow (bird)". She explained that she also thought this rather odd, but had seen the indexing term attached to relevant articles, and therefore felt obliged to include it. On further investigation we discovered that this was in fact a computer error, but one that had remained undetected by indexers for a number of years. The Emtree term has now been changed to "swallow" with "(bird)" no longer appearing, but sadly, despite my correspondence with Elsevier technical staff in Frankfurt, the computer indexing error persists.

Database searching in systematic reviews is a demanding task, and one which requires rigorous testing of each individual component of the search, as well as careful attention to its structural and functional elements. It is an exercise involving both logic and linguistics, and while modern technology has been a boon, researchers would do well to remember that the artificial intelligence which generates database algorithms is rather like artificial flavors. It's a valuable asset, but no substitute for the real thing. Effective critical analysis and decision-making in this context remain very much the province of the human brain and its intelligence. So it's really up to you.

References