Keywords
INTRODUCTION
Echocardiography makes it possible to assess morphological, functional and hemodynamic anomalies of the heart and is one of the most frequently used techniques in cardiology, with applications covering the entire spectrum of cardiovascular diseases. The development of new low-cost, laptop-size ultrasound devices can facilitate the echocardiographic examination of patients at the initial visit, as an adjunct to physical examination. Previous studies that evaluated the usefulness of portable devices have shown them to be superior to physical examination,1-3 although there are few data on their diagnostic accuracy as compared to standard equipment.4,5 The purpose of this study was to determine the diagnostic reliability of the portable echocardiographic unit when used by cardiologists experienced in echocardiography.
PATIENTS AND METHODS
We studied a total of 211 unselected patients, 86 women (40.8%) and 125 men (59.2%), referred for ultrasound examination from various hospital departments and from the outpatient service. The referral request included a brief summary of the clinical history, the physical examination and the description of the electrocardiogram.
The Optigo® (Agilent Technologies, Andover) point-of-care ultrasound device was used for the study. This hand-carried device measures 33x23x9 cm, weighs less than 4 kg including the battery, and has a screen resolution of 640x480 pixels. The unit also includes a control panel and an integrated 2.5-MHz transducer. It allows 2D ultrasound imaging and is equipped with color-flow Doppler, with gain and position adjustable at the control panel. Up to two consecutive measurements can be made after freezing the image from the touch pad.
Unlike commonly used ultrasound devices, this unit is not equipped with M-mode continuous wave or pulsed Doppler ultrasound. It does not allow image synchronization with the patient´s surface electrocardiogram or storage of the studies for subsequent review. Moreover, the device does not have special capabilities such as transesophageal study.
For comparison, we used reference diagnoses obtained with high-end devices (Sonos 5500®, Agilent Technologies, and Sequoia®, Acuson); in all cases the second harmonic was used to enhance the image.
Two complete studies were done on each patient, one with the hand-held device by a cardiologist experienced in echocardiography (Level II of the Spanish Society of Cardiology6) and another with the reference equipment, interpreted by a cardiologist with extensive experience in echocardiography (Level III). Standard views were obtained through the parasternal, apical and subcostal windows. The studies were performed in random order and the investigator performing the second test was unaware of the results obtained with the first device. A form was used to record the following data:
1. Subjective assessment of cardiac imaging according to three levels, good, acceptable and poor window, respectively, corresponding to proper visualization of all myocardial segments, proper visualization of most segments, and poor visualization of most segments.
2. Global and regional ventricular function, according to one of four grades (normal, and mild, moderate or severely depressed), depending on the subjectively estimated ejection fraction. Regional contractility was analyzed only by counting the number of abnormal segments found.
3. Measurements of the ventricular chambers, left atrium, and aortic root, and left ventricle thickening, with the findings classified as normal or abnormal for the final analysis.
4. Presence of pericardial effusion.
5. Valve function, estimated with the color-flow Doppler ultrasound option of the hand-held unit. The Doppler mode allows gain adjustment and image scrolling, although gradient measurements and color area outlining are not possible. Therefore, the severity of valvular regurgitation was diagnosed subjectively on the basis of the intensity and size of the color area, as well as indirect data such as chamber enlargement. The diagnosis obtained with the standard equipment was based on continuous wave and pulsed Doppler, with the proximal isovelocity surface area (PISA) method used to calculate the pressure half-time, regurgitant volume and fraction, and area of the regurgitant orifice. The sensitivity and specificity of the hand-held device in diagnosing valvular regurgitation was determined using a dichotomous variable (valvular regurgitation yes/no) for all grades of regurgitation. Because the device had no continuous wave or pulsed Doppler-essential for hemodynamic measurements-valve stenosis was not assessed.
Version 9.0 of the SPSS software was used for the statistical analysis. The degree of interobserver agreement for each diagnosis was calculated using kappa and weighted kappa in the case of variables with more than two categories. The indexes of diagnostic accuracy and the differences in proportions were calculated using the McNemar test for paired data. Deviations of more than one grade when compared to the reference diagnosis were considered clinically relevant. Significance was set at a P<.05. Interobserver agreement was considered very good at kappa values above 0.8, good at 0.6 to 0.8, moderate at 0.4 to 0.6, weak below 0.4 and very weak below 0.2.
Because each investigator performed studies with only one device, the intraobserver variation could not be calculated. According to several other similar studies,5 this variation is assumed to be less than 15% for the purposes of the analysis.
RESULTS
Assessment of the studies
Subjective assessment of cardiac imaging with the two instruments attributed significantly poorer visualization to the point-of-care device (Figure 1). The study was considered of poor quality in 22.4% of studies done with the portable device and 13.3% of those done with the standard devices, due to the poor ultrasound window.
Fig. 1. Assessment of studies performed with the Optigo® and with standard equipment. The results were divided into three levels, depending on visualization of the myocardial segments.
Ventricular function
We found adequate agreement between ventricular function estimates (P=.135) (Table 1) and moderate interobserver agreement (kappa=0.583). Ventricular function estimates were the same in 78.7% of cases. In 21.3% of discordant diagnoses, 17.1% were one grade and 4.3% more than one grade, with a trend toward higher ventricular function estimates in the hand-held device. In one patient, severe ventricular dysfunction was incorrectly estimated as normal with the portable unit; this study had been classified as poor image quality.
There were no differences between the two studies in the number of segments with contractility abnormalities (Table 2), although the segments were not analyzed according to the site or the grade of contractility abnormality.
Pericardial effusion
Pericardial effusion was found in 14 patients, with no significant differences between the two devices (6.2% Optigo®; 6.6% standard device, P=.656). Interobserver agreement was very good (kappa=0.802). Three of the 14 cases with pericardial effusion were missed with the portable device (Table 3), and two patients were incorrectly diagnosed as having effusion. Although the grade of effusion was not considered in the analysis, these cases were diagnosed as mild. Diagnostic sensitivity was 78.6% and specificity was 99% (Figure 2).
Fig. 2. Differences in cardiac measurements and pericardial effusion findings. Statistical significance, sensitivity and specificity are indicated at the bottom.
Cardiac measurements
The left atrium, aortic root, septum, posterior wall, and end-diastolic diameters of the left and right ventricles were measured and the data were used to classify the patients, according to whether the values were abnormal or within normal limits, without distinguishing between grades of severity. Differences were significant for most of the study parameters, except left or right ventricular enlargement (Figure 2). The most frequent finding was left atrial enlargement (60.2%). Consistent diagnoses with both devices were obtained in only 73.5% of cases, with 8.1% false positives and 18.5% false negatives. The least frequent diagnosis was aortic root enlargement (10 cases), of which nine were missed with the hand-held device. One case was falsely diagnosed. In all other diagnoses, agreement was 81.5% to 92.4%, false positives were 2.4% to 3.8% and false negatives were 4.7% to 14.7%. Figure 2 also indicates the sensitivity and specificity of the portable equipment for each diagnosis evaluated, as well as agreement figures. In most cases, there was moderate agreement between investigators.
Valve function
We found 132 patients with mitral regurgitation (62.5%), 104 with tricuspid regurgitation (49.3%), and 68 with aortic regurgitation (32.3%). Significant differences were found for mitral and tricuspid regurgitation (P<.05), but not aortic regurgitation (P=.207). Table 2 presents the sensitivity and specificity of valvular regurgitation diagnosis for the portable equipment, as well as interobserver agreement. This agreement was moderate for the three diagnoses. Tables 3-5 indicate valvular regurgitation differences between the devices.
Agreement was almost 80% in aortic regurgitation (Table 3), but only 58.8% in valvular regurgitation. There were 7.6% false positives (including one patient diagnosed with moderate aortic regurgitation) and 11.4% false negatives. No acute aortic regurgitation was diagnosed.
Agreement in the grade of mitral regurgitation was 65.4% (Table 4), but decreased to 36.5% when patients without regurgitation were excluded. Of the 73 cases with disagreements (34.5%), the difference was more than one grade in three (1.4%) and corresponded to patients with mild acute regurgitation. All other differences were one grade, with 9.9% overestimation and 23.2% underestimation of the grade of regurgitation. There were 18 false positives (8.5%) and 39 false negatives (18.5%).
The overall diagnostic agreement of 67.8% for tricuspid regurgitation (Table 5) decreased to 24.7% when only regurgitation cases were considered. Of the 32.2% cases with disagreements, differences of more than one grade account for 1.9%, specifically, three patients with moderate tricuspid regurgitation and one with acute regurgitation that were not diagnosed. The regurgitation grade was overestimated in 14.2% of the remaining patients and underestimated in 16%. Of these, 7.6% were false positives and 15.6% were false negatives.
DISCUSSION
The introduction of compact, low-cost ultrasound devices will expand the use of echocardiography in initial patient assessment as an adjunct to physical examination. The device may also encourage greater use in other specialties. Therefore, the diagnostic accuracy of this equipment should be determined so the findings can be properly interpreted. The reliability of these studies is highly user-dependent; thus, in keeping with the recommendations of the American Society of Echocardiography7 the hand-held device was used by a cardiologist with Level 2 training in echocardiography, the minimum training level required to perform and interpret a complete echocardiographic study independently.
Our study was not intended to directly compare two devices based on different technological principals and components, but to compare diagnoses made with a point-of-care device to those made with the most reliable reference at hand, a high-end device interpreted by a cardiologist with Level 3 echocardiography experience. Furthermore, these new hand-carried echocardiography devices have not been developed to replace the current equipment, but merely as a new diagnostic tool for initial screening.
The purpose of this study, therefore, was to determine the diagnostic reliability of these portable devices in diseases that can be detected with this technology, when used by qualified personnel (Level 2 training in echocardiography), simulating the probable conditions of use.
Very little data are available at present, although these devices have been shown to be diagnostically superior to physical examination.1-3 However, the results are not entirely consistent with those obtained with better equipment.4,5
Unlike previous studies,5 our study found significant differences in imaging quality. This is expected when working with equipment with differing characteristics, whether screen size or imaging enhancement methods (e.g., second harmonic) shown to be superior to the basic mode.8 However, inferior imaging quality does not prevent good agreement in estimates of ventricular function, which are normally subjective. This agreement was around 80%, with clinically relevant deviations in only 4.3% of patients.
We found discrepancies in heart chamber measurements, although they were not significant in most cases. Apart from adequate visualization, the main problems encountered in measuring a heart chamber with the point-of-care device concern the impossibility of synchronizing the image with the surface electrocardiogram, which makes it impossible to freeze the image at the right time in the heart cycle. This leads to significant variation in the measurements.
Pericardial effusion is an extremely important finding. Although we caution that the number of cases detected in our series (10) was not high, we found no significant differences in detection.
In contrast, valvular regurgitations were not adequately assessed with the hand-held devices (2D imaging and color Doppler ultrasound). We found significant differences in the diagnosis of mitral and tricuspid regurgitation, but not of aortic regurgitation. Diagnostic accuracy with the portable equipment was less than 60% in all cases. Around 30%-35% of diagnoses were inaccurate, although the differences were clinically relevant in less than 2% of cases.
The specificity of the portable device is good for most diagnoses. However, the number of false negatives indicates that the test is not sensitive, and therefore does not meet the requirements for adequate diagnosis during screening for a serious but curable disease.9 Because the portable device has good diagnostic specificity, an abnormal finding will allow the necessary decisions to be made. If the findings are normal, but disease is suspected, a complete echocardiographic study with a high-end unit should be performed.
The diagnostic value of the test is much lower for valve disease, since valve stenoses cannot be detected. In the diagnosis of valvular regurgitation, the sensitivity is lower than 70% and the specificity is lowe r than 85%. Therefore, valve function cannot be adequately analyzed with this device.
LIMITATIONS
Because the reference diagnosis was obtained with high-end equipment and interpreted by a cardiologist with Level 3 training, there is some potential for error by this investigator. This would invalidate the comparisons, as the diagnosis made with the portable device would be compared with a diagnosis that did not describe the patient´s actual condition. In daily clinical practice, however, these are the results used for decision-making with the patient. Another important limitation for the study was that intraobserver variation was not analyzed and some of the differences might be due to this factor. Similar studies that looked at this variation have obtained the same final results, however.
CONCLUSIONS
The availability of new, low-cost, small echocardiography devices will expand the use of echocardiography in initial patient assessments by cardiologists, as well as by other specialists. An understanding of the limitations and the degree of diagnostic accuracy of these devices in clinical assessment is needed, so that the findings can be used properly. Because the diagnostic value of the unit is highly user-dependent, echocardiography training equivalent to Level 2 of the Spanish Society of Cardiology is required, according to current recommendations. In addition, the user must assume responsibility for how the information is obtained, as well as how it is interpreted and used. When these requirements are met, the point-of-care device is superior to physical examination for the detection of heart disease, and accurately assesses ventricular function and pericardial effusion. However, its usefulness in assessing valve function is limited.
In the near future, technological advances in these apparatus will lead to devices equipped with all the components needed for complete echocardiography assessment.
ACKNOWLEDGEMENTS
We would like to thank the Research Department of our hospital for its assistance in reviewing and interpreting the results of the study.
Correspondence: Dr. M.A. García-Fernández.
Servicio de Cardiología, Hospital General Universitario Gregorio Marañón,
Dr. Esquerdo, 46, 28007 Madrid, Spain.
E-mail: magf@seker.es