Development of a Geographic Filter for PubMed to Identify Studies Performed in Spain

INTRODUCTION

With nearly 15 million references, Medline, the biomedical database created and maintained by the US National Library of Medicine, is one of the most complete resources in the world.¹ To a large extent, its success can be attributed to PubMed, its popular, freely available interface. Its vast size, which doubles every 5 years, makes it necessary to resort to specific information retrieval tools (search strategies). Some strategies, or certain parts of them, can serve general purposes and we call these "filters." Their use enables us to save time in constructing searches and increases their reproducibility, as well as achieving greater effectiveness if we take into account certain limitations in the indexing in PubMed.³

The appearance of the "evidence-based medicine" paradigm^4,5 has stimulated the development of methods to identify references in biomedical databases and encouraged the construction of so-called methodological search filters. This term is derived from their purpose which is to identify studies with high internal validity,⁶ above all randomized clinical trials or systematic reviews, among others.³ Given their usefulness, some have even been incorporated into PubMed thru the "Clinical Queries" option.¹

To date, the development of geographical filters, that is, filters oriented towards the identification of information with a specific geographical location, has not been tackled despite the fact that current PubMed options do not facilitate the identification of the geographical origin of populations studied. The principle alternatives available are to enter "Spain" as a restriction criteria either as the affiliation of the first author or as the place of publication, and to activate "Spanish" as the language option. Each of these has been demonstrated to be insufficient for our purpose. Restrictions by language or place of publication ignore numerous Spanish publications in international journals, whereas searches centered on author affiliation are limited as in PubMed this field only offers the affiliation of the first author. Moreover, the efficiency of this strategy is impaired by a lack of standardization. The first author affiliation field contains a wide range of information; frequently, when the country where the journal is published is the same as that of author affiliation, the latter is eliminated.^7-10

Consequently, we set ourselves the following objectives: a) to develop a geographical filter to identify in PubMed those studies in which a Spanish population has participated, and b) to evaluate filter performance in terms of accepted methodological standards and by comparison with simple geographical search criteria.

METHOD

Development of the Geographical Filter

To develop the geographical filter we followed what is known as a second-generation procedure, in the terminology proposed by Jenkins.³ Filter construction is based on: a) applying expert knowledge of the database structure (PubMed) and of the thesaurus or controlled vocabulary (MeSH), and b) validating the strategy obtained by comparison with a gold standard.³

To identify relevant terms we reviewed search strategies used by other authors to identify studies conducted in Spain. These were mainly bibliometric studies aimed at evaluating Spanish scientific production. We contacted authors to obtain their search strategies.^11-17 Based on components we identified in these strategies, we defined 3 syntactic structures of filters and their corresponding terms: a) the term "Spain" in the major PubMed languages (English, Spanish, German, French, Italian); b) place names of Spain's autonomous regions, provinces, provincial capitals and others locations of special interest, excluding any that are also found in other countries; and c) official acronyms of the Spanish autonomous regions' health services.

We specified the search for terms of the first type as free text or affiliation ([ad]). The PubMed search engine applied to free text an automatic mapping process of terms that localizes equivalences consecutively in tables of MeSH terms, journals and authors, in that order, and in which the field "affiliation" is not included.¹ The remaining strategies were applied only to "affiliation." Terms were truncated to permit retrieval of names referring to inhabitants of the country or region. The different strategies were linked using the connector OR (Annex 1).

Performance Evaluation of the Filter

Performance evaluation of the geographical filter was conducted by comparison with a hand search (the gold standard) in an ad hoc validation sample.³

Identification of the Sample and Hand Search

To obtain the sample we selected all PubMed indexed references under MeSH keyword "myocardial infarction." In Spain, myocardial infarction accounted for 39 400 deaths in 2002 alone and its incidence is increasing.^18,19 We only selected clinical trials (Limit: "Clinical Trial") and the default options (activate "Explode" and deactivate "Restrict to Major Subject Headings"). The format selected was "Medline," which details between 20 and 30 characteristics for each reference based on information presented in the publication (title, abstract, authors, affiliation, and MeSH terms among others).¹ Validation samples were exported and analyzed using Reference Manager 10 software²⁰ to guarantee the procedure was auditable.

Two reviewers (JM, JMV) the classified validation sample references by hand ("relevant" vs "non-relevant") applying the following operational definition: references were considered "relevant" when: a) they specifically indicated the study had been conducted in Spain; b) they mentioned the Spanish population as such, c) the study had been conducted by researchers affiliated to Spanish research centers. References connected with multicenter studies (whether international or not) in which a Spanish population participated were also classified as relevant. References failing to meet any of these criteria were considered "non-relevant." If information about author affiliation, where the intervention took place or the population was insufficient, full text articles were located for clarification. In some cases, reviewers needed to identify studies by the same authors or references outside of the sample to determine relevance.

This classification was compared with that obtained after applying the filter to the sample (using the connector "AND"). We included all references in PubMed from its launch in 1966 thru to August 1, 2005, when the electronic searches were performed.

Simple Strategy

To evaluate the relative usefulness of the filter we compared its performance with that of a much simpler strategy. Intuitively, we supposed this would correspond to the use of the free term "Spain." In fact, this is an inefficient term that could artificially increase the relative value of the geographical filter²¹ and therefore selected the strategy "Spain[ad]" which, in practice, retrieves three times as many references.¹

Analysis

Evaluating PubMed searches is similar in operational terms to studying diagnostic test performance in that we apply the concepts of sensitivity (proportion of relevant articles in the validation sample retrieved by the filter) and specificity (proportion of non-relevant articles in the validation sample not retreived by the filter) (Figure).^22,23 We calculated the sensitivity and specificity of applying the geographical filter and the simple strategy on the validation sample.

Figure 1. Performance evaluation indices for filters and search strategies.

We checked the stability of these calculations thru sensitivity analysis of a second, independent, validation sample. We chose a different clinical condition, but one that is also greatly concerns health services, and selected clinical trials indexed in PubMed under MeSH term "Pulmonary Disease, Chronic Obstructive."

RESULTS

Applying the criterion "Spain[ad]" to the entire database retrieved 109 344 references. Applying the filter produced 59 142 additional references giving a total of 168 486 (an increase of 54.1%). Applying the criterion "Spain" and its translations (free text and field [ad]) retrieved 141 961 (84.3%) references; using other place names retrieved an additional 26 499 references (15.7%); and health service acronyms, retrieved another 26 (0.1%).

Under MeSH term "Myocardial Infarction," we found 7312 clinical trial references. Only 2 of these corresponded to the same document, indexed once as a clinical trial and later as a review. The hand review of these references classified 84 (1.15%) as relevant and 7228 as non-relevant (98.85%). Just over half of these had been published in Spanish and in Spanish journals. Seven relevant references corresponded to international trials that included Spanish populations (Table 1)

The geographical filter recovered 75 references, 74 (98.76%) of which were identified as relevant in the hand search. The only reference that did not correspond to a trial conducted in Spain was a PubMed indexing error. In the abstract, the affiliation was given as "Niguarda Ca'Granda" a hospital in Milan, Italy. This had been incorrectly indexed as "Niguarda Ca'Granada, thus locating it in south-eastern Spain even though it was transcribed correctly in the original.²⁴ The criterion "Spain" and its translations (free text and field [ad]) retrieved 53 relevant references (70.67%); that corresponding to autonomous regions, provinces, and cities retrieved 64 (86.47%); and that corresponding to health services, none. of these references, 31 (41.9%) were retrieved by only 1 of the 3 searches, most of them by using place names (n=21, 67.74%), or "Spain" (n=10, 32.26%). The remaining 43 (58.2%) were retrieved by more than 1 of the search criteria. The geographical filter failed to retrieve 10 relevant references. In 2 cases, this was due to the incomplete definition of filter terms (place names not included in the strategy): in 1 case, "Catalonia" was mentioned in the abstract as the place where the trial had taken place²⁵; in another, author affiliation appeared as a hospital located in Cartagena.²⁶ The other 8 lacked sufficient information: most (n=6) had been published between 1970 and 1990 and only reference information was available (title, authors, publication). All these had been classified as relevant in the hand review only after a secondary search for data on the trial and/or author affiliation thru references outside the validation sample.

Altogether, the filter showed sensitivity as 88.10% and specificity, 99.99% (Table 2). In contrast, the simple strategy "Spain[ad]" recovered only 38 relevant references; sensitivity was 45.8% and specificity 100% (Table 2). The effectiveness of estimating performance of the geographical filter in the second validation sample (478 references) was optimal (sensitivity and specificity 100%).

On completing the study, we reported all indexing errors identified to the US National Library of Medicine.

DISCUSSION

We have constructed an easy-to-use geographical filter to identify studies conducted in Spain on the Medline database (PubMed). Performance evaluation has shown excellent specificity and very high sensitivity, above that of the simple search criterion.

Strengths and Weaknesses of the Evaluation Procedure

Inconsistencies in indexing and the presence of transcription errors affect retrieval of references on Medline.^27-31 Our filter validation procedure has been designed with precision bearing in mind the limitations of indexing and overcoming those inherent in methods employed to date. It is based on conducting a hand search of a single ad hoc validation sample, usually defined from references published in selected journals.³ This practice is controversial because the approach to selection introduces a degree of arbitrariness. Moreover, it uses references outside the database so indexing errors affect the estimation of filter performance.³² To take an extreme case, a reference not included in the database cannot be recovered by the filter under evaluation, or by any other, which makes it unreasonable to attribute this limitation to the search strategy. The evaluation procedure took the validation sample directly from PubMed which, moreover, meant the hand search was easily replicable, and guarantees validation samples can be accessed in Reference Manager file format. The construction of an unrelated supplementary sample based on a different clinical entity to determine the stability of the evaluation of filter performance also increases confidence in results.

We would point out that no standards for selection of comparison filters or of samples to evaluate or validate filters currently exist. The simple strategy was selected because it entailed an intuitive approach that trebles the performance of an even simpler strategy ("Spain" as free text), and was the strategy most often described in the literature. The interpretability of results has guided our selection of the validation samples.

Strengths and Weaknesses of the Proposed Filter

Terms have been included on the basis of a systematic procedure derived from the definition of explicit criteria: geographical, with geographical localizations of decreasing size; linguistic, with languages that cover >99.99% of Spanish scientific production³³; and administrative, with the inclusion of the various health services. This same characteristic meant that 2 relevant references could not be recovered electronically because the criterion of localization did not coincide with either of the predefined criteria. Both terms have subsequently been incorporated into the filter (Annex 1), which will certainly continue to be enriched with the personal contributions and suggestions of users.

The geographical filter proposed includes a search for names of regional health services that, on analysis, neither increased nor reduced sensitivity or specificity in the samples used. Bearing in mind their lack of impact, that there are very few of them, that when used separately this criterion identified 1519 references at August 2005, and the authors' prior experience with these terms, we believe they should continue to be included. In any case, the modular construction of the filter means users can easily eliminate the criterion if they wish to.

Increasingly, databases tend to normalize data and have improved their indexing over time. This may lead to the improved performance of some simpler searches which, in any case, have already been inserted into the current filter.

Alternative search strategies based on criteria of language or place of publication, while useful in bibliometric studies, are barely applicable to achieve our objective of identifying studies conducted in the Spanish population. Limiting searches to entries in Spanish would ignore the vast volume of Spanish production written in English (66 855 documents alone in 1994-2002³³). On the other hand, more than half the references contained in PubMed in Spanish correspond to publication outside of Spain which, on the whole, present studies conducted on the American continent.

More and more, researchers publish their results in international journals (>45% in the present study).²⁷ At the same time, some journals published in Spain have a markedly international character: Methods and Findings in Experimental, and Clinical Pharmacology, or Test, for example. Others, which were originally "local" publicatio ns, increasingly aspire to achieve recognition as international journals. This is the case of Revista Española de Cardiología, which has published more than a third of all studies. It seems clear that language and place of publication are unreliable criteria to identify studies conducted in Spain.

Applications of the Geographical Filter

Without doubt, one of the most useful applications of this geographical filter is in to enable us to identify those studies with greater external validity for the application of the data to the Spanish population, what we would call generalizability or proximal similarity.^6,34-36 It is well known that the generalizability of findings depends on various aspects of the studies, ranging from the clinical, socio-demographic, and even genetic characteristics of the population studied to variables related to the process of attention, such as styles of clinical practice and care, and complementary treatments administered.⁶ If the previously indicated aspects could be of special relevance in studies related to cardiovascular risk (consider, for example, the controversial application of the different tables, and equations obtained in different populations), other aspects, such as the legislative or those related to health service organization, could be even more relevant in studies into giving up smoking³⁷ (legislation and model of social behavior), or evaluating the effectiveness of interventions in the home (accessibility and model of attention).³⁸ In both cases, geographical location can be considered an indirect (proxy) variable of many of these factors and may be of great use in evaluating external validity.

Specifically, the availability of a geographical filter of high sensitivity and specificity to identify in PubMed studies conducted in the Spanish population opens the door to its systematic application in clinical practice. Its application will permit quick, reliable identification of relevant information about the effects of procedures and treatments in conditions closer to our reality. Additionally, this information is especially relevant in defining clinical trajectories that justifiably arouse growing interest and in which considering local peculiarities of attention are decisive for effective implementation.^39-42 From a cognitive point of view, retrieving relevant information facilitates learning, as well as improving clinical practice after applying the new information.^43,44

Perhaps because of this, we are still surprised that clinical practice guidelines lack specific sections summarizing information available about using diagnostic and therapeutic procedures in the very contexts where they are intended to be applied^45,46; information which geographical filters like that presented here could play an important role in providing.³⁸ This contextualization of information, rather than contradicting the postulates of evidence-based medicine, confirms them, since considering the external validity of information identified is no less than one step prior to its application in clinical practice.⁴

Finally, this geographical filter offers a useful tool for bibliometric applications which have, to date, primarily centered on the Institute of Scientific Information of Philadelphia (ISI) database and that could now be complemented via PubMed.^15,33 In fact, for this purpose one recent publication⁴⁷ describes the independent use of a geographical filter similar to that presented in our article.

CONCLUSIONS

An efficient filter to identify studies conducted in the Spanish population in PubMed is available. Its application permits us to find those references with greater external validity for clinical practice in Spain and, if used systematically, it could be of great value in constructing clinical practice guidelines and defining clinical trajectories.

ACKNOWLEDGEMENTS

The authors wish to thank the Centro Cochrane Iberoamericano for advice received on filter structure, Dr Alberto Miguel-Dasit and Dr Luis Martí-Bonmatí for generously providing us with the search strategy they designed, and Olatz Garín for her invaluable comments.

See editorial on pages 1221-4

An earlier version of this study was presented at the 23rd Meeting of the Spanish Society of Epidemiology.

This study was part-financed by the Instituto de Salud Carlos III (Red de Investigación Cooperativa en Investigación de Resultados y Servicios Sanitarios [IRYSS] G03/202, Exp No. 3000118).

---

Correspondence: Dr. J.M. Valderas Martínez.
Unidad de Investigación en Servicios Sanitarios. Institut Municipal d'Investigació Mèdica (IMIM).
Dr. Aiguader, 88, edifici PRBB. 08003 Barcelona. España.
E-mail: [e-mail]

Received August 29, 2005.
Accepted for publication August 2, 2006.