We aimed to determine the frequency of use and accessibility over time of electronic citations in Revista Española de Cardiología and 3 other Spanish biomedical journals, and to identify the factors that influence the accessibility and retrievability of these links.
MethodsA cross-sectional, descriptive study was performed of all the references in articles published in the first issue of each even year from 2002 to 2012 in selected journals. Data were collected on the journal, publication, references, and links. The variables analyzed were the percentage of articles with at least 1 electronic citation, the percentage of electronic citations with respect to the total number of citations in the full sample, and the percentages of link accessibility and retrievability.
ResultsWe included 377 articles, of which 73 (19.4%; 95% confidence interval, 15.2%-23.5%) had at least 1 electronic citation, and a total of 7077 references, of which 164 (2.3%; 95% confidence interval, 2.0%-2.7%) were electronic citations. Revista Española de Cardiología had 17 (15.2%) and 38 (1.3%), respectively. Use of electronic citations significantly increased over time (linear tendency, P < .001); 58 (35.4%; 95% confidence interval, 27.7%-43.0%), links were accessible, and information was retrieved in 55 cases (51.9%; 95% confidence interval, 41.9%-61.9%). Accessibility significantly decreased over time (linear tendency, P < .001); 45 electronic citations (27.5%; 95% confidence interval, 20.3%-34.6%) had complete additional information. Retrievability was significantly associated with the amount of additional information (linear tendency, P < .001).
ConclusionsElectronic citations are increasingly used in some Spanish biomedical journals. Access to electronic citations is lost over time, and the probability of its retrieval is associated with the existence of additional information.
Keywords
Electronic citations (EC) have become more common in biomedical journal publications in the past decade. In 2003, a study of 3 international, high-impact journals reported that 30% of articles contained at least 1 EC, accounting for 2.6% of all references at that time.1 Electronic citations are now used to such an extent that they are ranked third in terms of reference type, after biomedical journal articles and books.2
Electronic resources have major advantages over traditional print publications because information can be updated quickly, constantly, and efficiently and can be distributed to potential readers almost instantly. In addition, electronic resources can supply supplementary material without space restrictions and in alternative formats, such as videos, high resolution images, simulations, audio files, databases, and program source code.1–3 However, they have 2 main drawbacks: identifying who is responsible for them, and ensuring their quality, accessibility, and permanence.4 Unlike print media, information on the Internet may suddenly disappear.5 In fact, it has been observed that a web page has an average life of just under 2 years,6 a period that is reduced to 100 days in some contexts.7 In the case of biomedical publications, 1 study reported that 4.4% of ECs in leading journals were inaccessible just 3 months after publication.8
Some articles that analyze the frequency of use or accessibility of ECs in specific journals and areas of knowledge have been published in non-English language journals,1,2,9,10 but very few have been published in Spain. In the case of Revista Española de Cardiología (REC), there are no studies on the use of ECs in its articles. Therefore, in view of the above, this study aimed to: a) determine the frequency of use and longevity of ECs in articles published in REC and 3 other leading Spanish biomedical journals; b) study the quality of the ECs and the characteristics of the links; c) analyze the factors that may influence link accessibility and retrievability, and d) perform a comparative analysis between REC and the other selected medical journals.
METHODSStudy DesignA cross-sectional, descriptive study was performed using systematic sampling to cover all the references in articles published in the first issue of each even year from 2002 to 2012 in REC, Emergencias, Revista Clínica Española and Medicina Clínica. We selected these Spanish medical journals due to their wide dissemination and scientific impact in 2011 (with respective impact factors for that year of 2.530, 2.486, 2.008, and 1.385, and quartile scores of Q2, Q1, Q2 and Q2 in their subject categories, according to Journal Citation Reports Science Edition 2011, which was the latest version available when we planned this study).11 We excluded supplements, special issues, and monographs sponsored by the pharmaceutical industry.
Study ProtocolWe performed a search of all articles and references in the selected journals using their electronic versions between October 15, 2012 and November 15, 2012.
Independent variables were journal name, publication year, article type, total number of references, and the number of ECs. An EC was defined as a reference to a source document or resource published electronically on the Internet. For this study, we included any references with a hyperlink in the form of a uniform resource locator (URL) to a web page as the main source document. Links to articles in electronic format with a DOI (digital object identifier) and publishers with journals indexed by Journal Citation Reports were not considered ECs. To determine the frequency of EC use, we calculated the percentage of articles with at least 1 EC and the percentage of ECs out of all references as dependent variables.
In addition to accessibility, the presence of additional information recommended by the International Committee of Medical Journal Editors12 was also reported for each EC. The recommended additional information was defined as complete if it included the URL, date of any review, and the citation date. For links, we collected data on the publisher, link type, the presence of a quality seal, and localization.
To analyze EC accessibility, we copied the URL from the reference in the article and pasted it in the Google search engine. If the link did not work on at least 2 different days, we then tried the Bing search engine. Links that opened an active web page but not the page referenced in the EC were classified as “accessible but imprecise”. If links were inaccessible or imprecise, we analyzed the error type and degree of retrievability using the additional information provided in the EC.
Statistical AnalysisQualitative variables were expressed as absolute and relative frequencies and the association between them was analyzed using Pearson's chi-square test or Fisher's exact test, as appropriate. Linear tendency was also studied for ordinal variables. To assess annual changes, the beta coefficient (β) was estimated using a linear regression model. To compare REC with the other medical journals in the study (non-REC), we divided the REC sample by non-REC, according to whether or not citations were referenced in REC articles. The probability of a Alpha error was set at 5%. We used the statistical package SPSS 15.0 for the analysis.
RESULTSFrequency of Electronic CitationsWe included 377 published articles, with a total of 7077 references. There were 73 articles (19.4%; 95% confidence interval [95%CI], 15.2%-23.5%) with at least 1 EC. Among the full sample, 164 (2.3%; 95%CI, 2.0%-2.7%) were ECs. The EC percentage differed significantly among the journals (P = .009) and was lower in REC than in Medicina Clínica and Emergencias (P < .05) (Table 1).
Frequency Distribution by Journal
| Total (377 articles; 7077 references) | REC (112 articles; 2821 references) | Emergencias (87 articles; 1454 references) | RCE (102 articles; 1388 references) | Med Clin (76 articles; 1414 references) | P | |
| ECs | 164/7077 (2.3) | 38 (1.3)a | 57 (3.9) | 28 (2.0) | 41 (2.9) | .009 |
| Articles with ≥ 1 EC | 73/377 (19.4) | 17 (15.2) | 25 (28.7) | 15 (14.7) | 16 (21.1) | .170 |
| Accessible ECs | 58/164 (35.4) | 19 (50.0) | 18 (31.6) | 11 (39.3) | 10 (45.5) | .099 |
| Retrieved ECs among inaccessible ones | 55/106 (51.9) | 11 (64.7) | 15 (46.9) | 12 (75.0) | 17 (56.7) | .268 |
| Full additional information | 45/164 (27.5) | 6 (15.8)b | 15 (27.3) | 5 (17.9) | 19 (46.3) | .012 |
EC, electronic citations; Med Clin, Medicina Clínica; RCE, Revista Clínica Española; REC: Revista Española de Cardiología.
Data are expressed as no./No. (%) or no. (%).
At least 1 EC was present in almost half the special articles (Table 2). The percentage of ECs was highest in special articles and was lowest in images. We found a statistically significant difference between the frequency of EC use and publication type (P < .001). Frequency in REC was statistically significantly associated with article type and this finding applied to articles with at least 1 EC (P = .001) and also to ECs (P < .001). Almost all ECs were found in special articles, original articles, and editorials (94.1%).
Distribution of Frequency of Use by Type of Article
| Type of article | Articles with ≥ 1 ECa | ECb | ||||
| Total | REC | Non-REC | Total | REC | Non-REC | |
| Special article | 8/18 (44.4) | 3/5 (60.0) | 5/13 (38.5) | 36/755 (4.8) | 9/475 (1.9) | 27/280 (9.6) |
| Editorial | 9/43 (20.9) | 3/17 (17.6) | 6/26 (23.1) | 19/691 (2.7) | 6/294 (2.0) | 13/397 (3.3) |
| Original article | 30/100 (30.0) | 10/37 (27) | 20/63 (31.7) | 59/2549 (2.3) | 21/1073 (1.9) | 38/1476 (2.6) |
| Clinical case | 5/33 (15.1) | 0/0 (0.0) | 5/33 (15.2) | 9/199 (1.9) | 0/0 (0.0) | 9/199 (4.5) |
| Letter to the editor | 8/112 (7.1) | 0/26 (0.0) | 8/86 (9.3) | 14/783 (1.8) | 0/143 (0.0) | 14/640 (2.2) |
| Brief report | 2/19 (10.5) | 1/17 (5.9) | 1/2 (50.0) | 3/225 (1.3) | 2/207 (0.9) | 1/18 (5.5) |
| Review | 6/23 (26.1) | 0/5 (0.0) | 6/18 (33.3) | 15/1232 (1.2) | 0/466 (0.0) | 15/766 (1.9) |
| Image | 0/17 | 0/0 | 0/17 | 0/131 | 0/0 | 0/131 |
| Other | 5/12 (41.6) | 0/5 (0.0) | 5/7 (71.4) | 9/512 (4.8) | 0/163 (0.0) | 9/349 (2.6) |
EC, electronic citation; non-REC, other journals apart from Revista Española de Cardiología; REC, Revista Española de Cardiología.
Data are expressed as no./No. (%) or no./No.
The percentage of articles with at least 1 EC and the percentage of ECs out of total references in the sample significantly increased over time (linear tendency, P < .001) (Figure 1). Notably, the number of articles with at least 1 EC increased from 4.5% to 27.3% and the frequency of ECs out of total references increased from 1.8% to 3.5% from 2002 to 2012. When analyzing the slope for the frequency of EC use in REC during the study period, we obtained (β)=7.5 (95%CI, 5.5-9.4; P < .001) for the percentage of articles with at least 1 EC and β = 1.1 (95%CI, 0.5-1.7; P = .009) for the percentage of ECs compared with the non-REC group (β = 3.4; 95%CI, 1.6-5.7; P < .001, and β = 0.3; 95%CI, –0.2 to 0.7; P = .158, respectively).
Timeline of the percentage of articles with ≥ 1 electronic citation and of electronic compared with total citations in the full sample and in Revista Española de Cardiología vs other journals; 95%CI, 95% confidence interval; β, beta coefficient; non-REC, other journals apart from Revista Española de Cardiología; REC, Revista Española de Cardiología. Chi-square. Linear regression.
Of the total 164 ECs, 58 (35.4%; 95%CI, 27.7%-43.0%) were accessible. Of the remaining 106 ECs that were not accessible, we managed to retrieve 55 (51.9%; 95%CI, 41.9%-61.9%). There were no significant differences among the journals (P = .099 and P = .268) (Table 1).
Among the links that were broken or imprecise, we detected the type of error on 78 (73.6%) occasions. The most common error was imprecision, ie, the EC link only led to a homepage and not to the specific document, or it did not coincide with the additional information provided in the reference (Table 3).
Distribution by Error Type in Citation Links*
| Error type | All | REC | Non-REC |
| Incomplete (homepage) or does not match additional information in the reference | 35 (33.0) | 8 (42.1) | 27 (31.0) |
| Homepage error message | 22 (20.8) | 3 (15.8) | 19 (21.8) |
| Not found or does not exist (404 and 502 errors) | 22 (20.8) | 3 (15.8) | 19 (21.8) |
| Typographical | 10 (9.4) | 3 (16.7) | 7 (8.0) |
| Other | 17 (16.0) | 2 (10.5) | 15 (17.2) |
Non-REC, other journals apart from Revista Española de Cardiología; REC, Revista Española de Cardiología.
Data are expressed as No. (%).
Link accessibility percentages significantly decreased in parallel with increased time since publication (linear tendency, P < .001) (Figure 2). At the time of consultation, 64% of links were accessible 1 year after the publication date, and almost none was accessible after 8 years. When analyzing the accessibility slope for REC during the study period, we obtained β = –10.4 (95%CI, –28.1 to 7.2; P = .178) compared with Non-REC, with β = –11.4 (95%CI, –14.4 to –8.5; P < .001).
Longevity of percentages of valid links by time since publication in the full sample and in Revista Española de Cardiología vs other journals; 95%CI, 95% confidence interval; β, beta coefficient; non-REC, other journals apart from Revista Española de Cardiología; REC, Revista Española de Cardiología. Chi-square. Linear regression.
We found that 45 ECs (27.5%; 95%CI, 20.3%-34.6%) had complete additional information in the references, complying with the “Uniform requirements for manuscripts submitted to biomedical journals”, issued by the International Committee of Medical Journal Editors (Table 1). There were significant differences among the different journals (P = .012), and REC had the lowest percentage of ECs with complete additional information (15.8%). The most frequent missing information was the date the referenced content had been updated (69%), followed by the date of last access (44.6%).
Link CharacteristicsWith regard to links, 157 (97.5%) were not available in Internet repositories or archives such as Internet Archive13 or WebCite14; 31 (20.9%) ECs were linked to web pages with a quality seal or certificate, 24 (77.4%) of which had at least 2 seals, and 1 (3.2%) had 5 seals. The most common quality seal was the World Wide Web Consortium (W3C),15 which endorsed 14 (45.2%) links, followed by the “web médica acreditada” [accredited medical web] seal,16 issued by the Colegio Oficial de Médicos de Barcelona [Barcelona Official Association of Physicians] in 12 (38.7%) links, and “web de interés sanitario” [useful health web] in 9 (29.0%). The publisher of the homepage referred to by the link was a national or regional government agency or national official institution (eg, Instituto Nacional de Estadística [National Statistics Institute]) in 55 (33.5%) cases and a scientific society in 31 (18.9%) (Table 4). In addition to the aforementioned publishers, REC had a notable proportion of international bodies (13.2%), and nonprofit foundations and research consortia (10.5%). Government agencies, followed by scientific societies, also had the highest percentage of quality certificates on their web pages (42.1% and 30.3%, respectively).
Web publisher frequency distribution*
| Web publisher | All | REC | Non-REC |
| Government (national, local, and official institutions) | 55 (33.5) | 13 (34.2) | 42 (33.3) |
| Scientific societies | 31 (18.9) | 8 (21.1) | 23 (18.3) |
| International bodies | 12 (7.3) | 5 (13.2) | 7 (5.6) |
| Medical web pages | 7 (4.3) | 1 (2.6) | 6 (4.8) |
| Universities | 6 (3.7) | 1 (2.6) | 5 (4.0) |
| Nonprofit foundations and research consortia | 6 (3.7) | 4 (10.5) | 2 (1.6) |
| Medical publishers | 5 (3.0) | 0 (0.0) | 5 (4.0) |
| Cochrane | 4 (2.4) | 1 (2.6) | 3 (2.4) |
| News broadcasters: TV, press, radio, etc. | 2 (1.2) | 0 (0.0) | 2 (1.6) |
| Wikipedia | 1 (0.6) | 0 (0.0) | 1 (0.8) |
| Hospital | 1 (0.6) | 0 (0.0) | 1 (0.8) |
| Other | 34 (20.7) | 5 (13.2) | 29 (23.0) |
Non-REC, other journals apart from Revista Española de Cardiología; REC, Revista Española de Cardiología.
Data are expressed as No. (%).
We analyzed the degree of accessibility by quality-related variables and found that 16 (35.6%) of the references with complete additional information were accessible, and 42 (35.3%) of those with incomplete or no information were accessible (P = .975). By publisher, we observed statistically significant differences in accessibility (P = .004), with > 80% accessibility for nonprofit foundations and research consortia, > 60% for medical publishers and universities, > 40% for government agency pages and international bodies, and about 25% for scientific societies and medical web pages. Analysis of accessibility by quality seal showed that 11 (35.5%) accessible ECs had a certificate and 47 (39.8%) did not (P = .697). By link type, we found that only 11 (44%) ECs were accessible in the case of links to PDF (portable document format) files compared with 20 (52.6%) links to a web page (P < .001). We were unable to analyze web pages available in repositories because very few ECs had this feature.
Analysis of access retrieval when the link did not work revealed that 23 (79.3%) of the 29 ECs were retrieved when the link was broken but that the reference contained complete additional information as defined by the International Committee of Medical Journal Editors. We retrieved 32 (45.1%) ECs that had some additional information, and none when the link to the web page was the only information provided (linear tendency, P < .001).
DISCUSSIONThe first important finding in this study is that the frequency of EC use has increased markedly over the years. However, although 19.4% of articles had at least 1 EC (increasing to 27.3% in 2012), EC frequency compared with total references is still negligible (2.2% in total, and 3.5% in 2012). The percentage of ECs in REC was notably lower than in the other journals, although it has increased more over time than in the non-REC group. We found that the number of ECs was influenced by the type of publication, and that special articles included ECs more frequently than other types of articles. In fact, REC almost only used ECs in editorials, special articles, and original articles. This is the first time that the characteristics and longevity of ECs have been analyzed in Spanish biomedical journals, and in REC in particular. We are aware of only 1 study on electronic resources in a specific Spanish biomedical journal. That study, conducted in 2003, found that 14.5% of articles had at least 1 EC and that 1.4% of all references were ECs.17 Comparing the frequency of EC use in the 4 Spanish journals in our study with that in other international studies, we found that results fall in the same range (0.6%-4%).1 Analysis of EC use should take into account the position of the selected journals in their category, because leading journals in 1 field use ECs more frequently. It is also important to consider the date of the study.1,2 Our finding of increased EC use over time also matches previous data. Electronic citations increased from 0.7% in 2001 to 3.5% in 2004 in AIDS Patient Care and STDs,9 and from 1% in 2000 to 5.4% in 2005 in Annals of Emergency Medicine.2 Another study that analyzed 6 high-impact oncology journals found that 9% of articles had at least 1 EC in 2001, and that this percentage increased to 16% in 2003.10
The second finding in our study is the low rate of EC accessibility in certain Spanish publications. Only 6 out of 10 ECs were accessible at the time of consultation, and almost none was accessible 8 years after publication. The most common reason for lack of accessibility was imprecision. The figure of 40% inaccessible ECs appears to be much higher than the percentages published in non-Spanish journals such as The New England Journal of Medicine (14.6%) and The Lancet (17.9%).1 However, the interval between publication and analysis of accessibility was 0 years to 3 years in these studies, but was 1 year to 9 in the present study. Other studies found that 11.9% of ECs were inaccessible at the time of publication, and 78% were inaccessible 5 years later.2,18,19 Therefore, 2 aspects should be considered when analyzing accessibility: time since publication together with the date of evaluation, and the definition of accessibility. With regard to the latter aspect, most authors do not consider imprecise links as inaccessible. When we used this criterion in our study, 15% of ECs were not accessible in the first year of evaluation. Furthermore, one of the few studies that have analyzed link imprecision found that 7.4% of ECs were imprecise just 3 months after publication.8 In short, imprecise citations appear to be a very common problem in some biomedical journals. This may influence the validity and transmission of knowledge sources used by authors to base their research hypotheses and scientific reasoning, which in turn could have significant implications in scientific methodology, because it hinders future reproducibility.17
Our third finding is that the web page publisher and type of link are possible determinants of the degree of EC accessibility. Nonprofit foundations and research consortia, as well as medical publishers and universities, appear to achieve higher rates of EC accessibility than scientific societies and medical web sites. This finding should be taken into consideration when interpreting the differences between the lower loss of accessibility over time in REC than in the non-REC group. Furthermore, Internet Archive13 and Webcite14 are 2 Internet archiving systems or repositories that ensure that cited web resources remain accessible to future readers. We were unable to analyze the importance of using repository localization as a method to ensure accessibility, because digital repositories were only used exceptionally for ECs in the Spanish biomedical journals in this study, despite recommendations for their use.8,18 Other authors have advocated modifying link type to maintain accessibility.17 Thus, scientific publications have started to adopt DOIs as an alternative to URLs to link to web pages. A DOI consists of assigning a CrossRef name to the published article so that the latter can be located on the Web. Its main advantage is that it does not change over time, even if the article is moved to a different address.
The fourth finding in our study is that ECs can be retrieved more readily if they comply with the standard recommendations for additional information, which can be used if the link does not open the document in question. This aspect should be taken into consideration and corrected in view of the high rate of imprecise links (less than a quarter of ECs studied had complete additional information, as recommended). In particular, ECs in REC had a significantly lower percentage of complete information, which could hinder their retrieval if they become inaccessible. We propose that ECs should not only have a URL, but should also have the additional information recommended in the “Uniform requirements for manuscripts submitted to biomedical journals” for ECs, issued by the International Committee of Medical Journal Editors.18 Like traditional nonelectronic references that have strict requirements and include seemingly redundant information (that nonetheless facilitates locating incorrect links), ECs should have complete additional information that could be used to locate the citation if the link is inaccessible. Specifically, it is important to add the following information in square brackets: type of medium [Internet], revision and citation dates [updated year-month-day; cited year-month-day], length in pages or most suitable format [x-x] and the Internet address [Available from: URL]. This is probably the simplest, fastest, and most inexpensive way of ensuring that, in most cases, readers will have access to the desired content using search engines, even if there are typographical errors or URL changes. It therefore seems unjustifiable that almost half of all ECs are missing the date of last revision. This information is essential for readers, not only to assess the currency of the reference, but also to compare it with the date of access to the document and thus ensure that the retrieved information corresponds to the author's reference. Likewise, almost a quarter of imprecise links could be resolved if those responsible carefully check that the link URLs actually contain the information referred to by the authors. To this end, we need to be more aware of this problem, and involve editors, reviewers, and the authors themselves in the process of checking EC accessibility before publication and reviewing the completeness of additional information.
Finally, in view of the above results, as well as the importance of references when evaluating journal quality and a researcher's scientific activity, we would like to reflect on the lax compliance of some biomedical journals with criteria for ECs in terms of form and content compared with traditional references. With regard to form, editors should check that ECs are correct and complete, and ensure that authors do likewise. We recommend referring to Citing Medicine20 for this purpose. With regard to the quality of referenced content, it is important to know whether the electronic content referenced is of contrasted quality.21,22 In the field of scientific literature, there is controversy as to whether web seals or certificates suffice or whether new indicators should be created to faithfully and universally reflect information quality. These indicators could involve the extrapolation of some of the quality measures used in printed biomedical publications, such as peer review or other bibliometric markers.23–27 It is therefore necessary to create a web quality indicator for homepages or websites, to provide weighted evaluations of aspects such as the number of visits, the date the page was last updated, identification of editors, the existence of any conflict of interest, any document revision dates, the existence of expert review, and the number of broken links reported in the past year. Indeed, some authors are already broaching the subject of cybermetric studies28 and electronic impact factors.29
LimitationsThis study has some limitations. First, we selected only certain journals, according to their area of knowledge and impact, and we reviewed only a specific number of published issues in these journals. Furthermore, since we used a systematic issue sampling method in Spanish biomedical journals with major diffusion and the highest number of citations in their category, our results give only an approximate idea of tendencies over time in this type of journal. Second, there is no standardized, validated instrument to quantify the degree of link accessibility and retrieval, which would facilitate comparisons among different studies. Third, the study design did not rule out the possibility that the lower number of accessible links further back in time may also be due to other causes apart from changes to URLs and closed websites. For example, the higher number of accessible links nowadays may be due to improved citation strategies in electronic resources over the years or to a growing effort by websites to try to ensure EC permanence.
CONCLUSIONSDespite these limitations, this study reflects the growing presence of ECs in some Spanish biomedical journals, and in REC in particular, as well as reduced link accessibility over time. The EC quality is far from optimal, and errors often make it impossible to access the information in references. Finally, this study shows that if the link does not directly open the document in question, ECs can be retrieved more readily if they contain recommended additional information. For this reason, editors of certain Spanish biomedical journals should make every effort to ensure that authors correctly cite references to documents and other electronic resources. As in the case of traditional references, they should check publication data against the original sources, EC accessibility, and the presence of additional information. The latter should have other potential measures to ensure EC accessibility, such as only using links that will remain accessible, or storing them in digital archiving repositories.
CONFLICTS OF INTERESTNone declared.
This study has been funded in part by a grant awarded by Generalitat de Catalunya SGR 2009/1385 to the Grupo de Investigación Consolidado “Urgencias: procesos y patologías”. (“Emergencies: processes and pathologies” Consolidated Research Group).