Keywords
INTRODUCTION
Low cost and speed of data extraction are 2 advantages of using administrative databases to evaluate medical interventions; nevertheless, they have important limitations. First, there is the difficulty of adjusting for confounding factors which cannot be coded. These factors can include severity indicators such as functional class or indicators associated with the care process, such as whether the intervention is scheduled and its urgency. A further difficulty is whether diagnoses correspond to comorbidities or to complications arising during the hospital stay, and whether they occurred before or after the procedure being evaluated. Finally, there are the problems of inaccurate or missing data in the hospital discharge reports.
The hospital discharge Minimum Basic Data Set (MBDS)1 is a population level registry which records information on pathologies treated in Spanish acute care hospitals using the international classification of diseases codes (ICD-9).2 Specific codes are used in the hospital discharge report for each diagnosis and procedure and for each admission. All public and private health care centers in Spain have to provide these data. Because of this, it has been possible to compile an exhaustive and valid database on health care activity and morbidity. The database is intended to be useful for health care planning, as well as in assessing and purchasing health care services. The MBDS has proven useful in epidemiological studies where it has been used to estimate population incidence and procedure performance rates, and to assess raw outcomes such as mortality.3-6 It has also been used to study variations in clinical practice.4,7 Using it to evaluate the outcomes of therapeutic procedures is, however, more controversial.8 Although it has been used to predict hospital mortality in cardiac surgery,9 the results were not compared with data from prospective studies to determine their reliability. Though several studies have analyzed agreement between administrative and clinical databases or specific registries, most such comparisons were carried out in the USA10-12 and Canada13 where the systems for recording and coding data may differ from that used in Spain. A study performed in the UK14 recently showed that administrative databases can be used to predict the risk associated with 3 important interventions (coronary artery bypass graft, repair of abdominal aortic aneurysm, and colorectal excision for cancer). The administrative databases used showed similar properties of calibration and discrimination as a series of clinical databases.
The objectives of the present study were: a) to evaluate the capacity of a MBDS to predict in-hospital mortality after first coronary bypass surgery without associated interventions. Reference data for the analysis came from the same group of patients who were included in a prospective study designed specifically with this aim in mind and which employed several quality controls (the ARCA study15); and b) to analyze the agreement between the 2 sources of information in terms of the most relevant variables in predicting surgical risk.
METHODS
The Prospective, Observational Study Database
Between November 2001 and November 2003, data were collected prospectively for all patients with an indication for first coronary bypass surgery without associated interventions who were treated in 5 hospitals in Catalonia. Three of these hospitals were publicly managed and 2 were privately managed. The private centers accept patients referred from public health centers. In each center, a member of the operating team or someone from the cardiology department reviewed the schedules for heart surgery and completed the data collection form. Following procedures set out in the study manual, data was collected through patient interview and a review of medical records and hospital discharge reports.
Data collection quality was tested through an external review of a random sample of 10% of medical records and through comparison with the hospital discharge MBDS.
The Administrative Database
The MBDS includes clinical data collected at discharge from hospital and coded using the International Classification of Disease, 9th Revision, Clinical Modification (ICD-9-CM).2 Until 2002, the diagnosis considered as the primary motive for admission and up to 3 secondary diagnoses could be recorded, together with the primary procedure and up to 3 secondary procedures. From 2003 on, it is possible to record the main diagnosis and up to 9 secondary diagnoses as well as the primary procedure and up to 7 secondary procedures.
The hospital discharge database (MBDS) for the 5 participating centers was requested from the Catalan Health Service for patients included in the prospective study and in which the code 36.1 appeared (bypass anastomosis for heart revascularization). In order to make record selection comparable with inclusion criteria in the prospective study, records were excluded if they mentioned procedures with codes v45.81 (postsurgical aortocoronary bypass status) and 35 (operations on valves and septa of heart). To be included in the prospective study, patients had to receive a first isolated coronary bypass without associated interventions.
The MBDS for the 5 participating hospitals was also requested for the period 2004-2006 to provide external validation data for the predictive model derived from the earlier cohort. The same selection criteria for procedure codes were applied for the second period.
Linking the Databases
The 2001-2003 MBDS was linked with the ARCA database using the key variables of center, medical record number, and date of discharge. A difference of 1 (1) month was allowed for the date of discharge. Due to errors in entering the medical record number, a second key was defined based on the center, the date of birth and date of discharge.
Analysis of Agreement
In order to analyze agreement between information in the 2 databases, diagnoses, and procedures with equivalent definitions to those used in the ARCA study were selected (Table 1). The way in which variables were defined was not always identical between the 2 studies and the definitions were often more precise in the prospective study. For example, renal failure (codes 585: chronic renal failure, and 586: unspecified renal failure) was defined in the prospective study as creatinine levels ≥1.5 mg/dL. In the prospective study, an urgent intervention was defined as one which was performed during the same period of admission in which surgery was initially indicated. This variable does not exist in the MBDS, although the circumstances of admission are recorded (scheduled or urgent).
The sensitivity and specificity of each diagnosis was calculated using the presence of each in the ARCA database as a reference. The kappa index was used to assess agreement between the 2. Cases of non-agreement were classified as: errors of omission (information collected in the ARCA study but not in the MBDS) and coding errors (diagnosis coded in the MBDS but not in the ARCA study).
At the same time, the overall percentage of agreement between the data in each register was calculated and the mean percentage of agreement was calculated by year of discharge and center.
Predictive Models
Logistic regression methods were used to construct 2 predictive models for hospital mortality, one based onm, ARCA study data (ARCA model) and one based on the 2001-2003 MBDS data (administrative model). Discrimination and calibration parameters were calculated for each model.
Discrimination refers to the model's capacity to distinguish between patients with and without a specific outcome. It was evaluated by calculating the C statistic, which is equivalent to area under the ROC curve; c=1 indicates perfect discrimination, while c=0.5 represents a zero capacity for discrimination; c≥0.7 can be considered acceptable.
Calibration refers to the degree of agreement between predicted mortality (expected cases) and true incidence (observed cases) in all risk strata. This was evaluated using the Hosmer-Lemeshow goodness of fit statistic: the data were grouped in deciles according to the probability estimated by the model and observed and predicted mortality were compared in each decile using χ2. The greater the difference between the expected and observed results, the higher the c2 value and the lower the P value. Low P values indicate poorer calibration.
The administrative model was validated using the ARCA data as the gold standard by means of 2 procedures. First, parameters were recalculated using the equivalent variables from the ARCA study and validity was assessed by calculating discrimination and calibration. Secondly, the administrative model (predictive variables and coefficients obtained from the administrative data) was applied to the ARCA data.
At the same time, the administrative model was validated using data from the 2004-2006 MBDS cohort and by applying similar procedures.
RESULTS
The analysis was performed on a total of 1508 cases which were recorded in both databases (95.9% of the cases in the ARCA database). There were 65 (4.1%) cases in the ARCA study which were not recorded in the MBDS. Hospital mortality was significantly higher in these "non-coincident" cases (12.3% compared to 4.5%).
The proportion of cases with a recorded diagnosis was higher in the ARCA database (Table 2), with notable differences in variables which are important to the evaluation of surgical risk, such as renal failure, previous infarct, or peripheral vascular disease.
Table 3 shows the agreement and validity of the diagnoses and procedures in the MBDS taking the ARCA study for reference. For most of the diagnoses, specificity was high (0.84 to 0.99) and sensitivity was very variable (0.14 to 0.99), although it was generally very low (<0.60 in 77% of the diagnoses). The kappa results were very variable, with almost perfect agreement in the case of non-coded variables such as sex and mortality, moderate for some diagnoses (arterial hypertension, diabetes, acute myocardial infarct, chronic obstructive pulmonary disease), procedures (surgery with extracorporeal circulation), and complications (postoperative infection) but <0.30 for other diagnoses important to the evaluation of surgical risk such as unstable angina, heart failure, renal failure, and peripheral vascular disease. In the majority of cases, most of the disagreement with the MBDS was due to omission of the diagnosis, but for some variables there was a high proportion of codification errors (smoking, prior infarct, use of extracorporeal circulation).
The percentage of agreement for all diagnoses and procedures was significantly higher in the data from 2003 than in the data from 2001 and 2002. It was also higher in data from publicly managed centers than in data from privately managed centers (Table 4).
Table 5 shows the odds ratios (OR) and 95% confidence intervals (CI) for hospital mortality, as well as the discrimination and calibration parameters for the ARCA model and the 2001-2003 administrative model. Table 5 also shows the OR estimates for the same administrative model applied to the ARCA data and to the MBDS data from 2004-2006, as well as the discrimination parameters for both validations.
The index of discrimination for the administrative model was acceptable (area under the ROC curve, 0.77; 95% CI, 0.71-0.83) when applied to the ARCA data, but was lower when applied to data from the 2004-2006 MBDS cohort (area under the ROC curve, 0.71; 95% CI, 0.62-0.81). Likewise, the administrative model showed better external validity when applied to the ARCA data (area under the ROC curve, 0.76; 95% CI, 0.70-0.82) than when applied to MBDS data from the later cohort (area under the ROC curve, 0.65; 95% CI, 0.56-0.74).
Figure 1 shows the number of expected deaths compared to observed deaths for different risk deciles (calibration) for the 2 models (administrative and ARCA) as well as for the administrative model when applied to the ARCA data. When the administrative model is used with the reference data, it tends to considerably overestimate risk, particularly in higher risk strata.
Figure 1. Calibration of the ARCA model, of the administrative model (MBDS), and of the administrative model applied to the prospective data: MBDS (ARCA).
Figure 2 shows the calibration of the administrative model when applied to 2004-2006 MBDS data. It indicates that the administrative model overestimates risk in the second cohort, particularly in the higher risk strata.
Figure 2. Calibration of the administrative model (MBDS 2001-2003) and of the administrative model applied to the MBDS 2004-2006 cohort.
DISCUSSION
Several sources of data are available to evaluate medical procedures, including administrative registries, systematically completed clinical registries, and registries designed for specific studies. Although the latter are much more reliable (particularly if quality controls are enforced), they are also more costly, which prohibits their being permanently or exhaustively applied within a given region or health care system. For that reason, it has been noted on several occasions that administrative databases may have great potential in assessing health care technologies.8 Nevertheless, they also have serious limitations, particularly the fact that they are not designed with health care technology assessment in mind.
The best way of assessing the usefulness of the data is to have valid and reliable parallel information, as in the study by Aylin et al.14 This type of assessment has not been carried out in our setting. However, the existence of an earlier study15 which examined the outcomes of coronary bypass surgery within the same setting provided a reference to assess the reliability of available administrative data and its potential usefulness in estimating risk in coronary surgery. We also evaluated the predictive model derived from 2001-2003 administrative data by applying it to a later cohort (2004-2006) for which administrative data was also available.
The results of the present study show that data quality in the MBDS was generally low. Approximately 4% of cases identified prospectively were not registered in the MBDS and hospital mortality in these unregistered cases was significantly higher than in cases included in both databases. This indicates that mortality in the MBDS is likely to be underestimated. Agreement was only really good for "hard" variables which were not coded, such as gender and hospital mortality. As in other similar studies,16 the specificity of the diagnoses coded using the ICD-9-CM was high, but sensitivity was very variable. Overall, agreement between the 2 databases was low, with errors mainly attributable to the omission of information in the MBDS. In other cases, however, there was a high proportion of codification errors, ie, cases in which the diagnosis in the MBDS was not confirmed by the prospective study. It is striking that this was particularly true in the case of relevant and easily identifiable clinical variables, such as myocardial infarct, renal failure, chronic obstructive pulmonary disease, or the use of extracorporeal circulation.
Paradoxically, the generally low quality of the data did not prevent the model from achieving a reasonable performance in predicting hospital mortality. In fact, the predictive capacity of the model derived from the administrative data was as good, in terms of discrimination and calibration, as that of the model generated from the prospective data (area under the ROC curve, 0.80; with good calibration) and similar to that observed in other studies.14,17 A possible explanation for this apparent paradox might be that, in certain cases, coding is "exaggerated" for some clinical variables which are important in predicting risk. This would also explain why the OR are higher in the administrative model (particularly for heart and renal failure).
This very rigorous coding may stem from different causes: a) co-morbid conditions may be more likely to be recorded in the discharge reports of patients who die in hospital; b) as has been indicated in earlier studies,17 the MBDS may not always distinguish between post-intervention pathology and antecedent complications, even when it is possible to do so, as in the case of heart and renal failure (Table 1). Such differentiation is not possible for other factors typically related to prognosis, such as recent myocardial infarct, which cannot be distinguished from peri-operative infarct; and c) it is likely that in the hospital discharge reports, and consequently in the MBDS, co-morbidities are only recorded when they are quite serious or of considerable importance. An example would be renal failure with very high creatinine values (ie, substantially above 1.5 mg/dL, which is the criterion used in conventional risk scales and the one used in the ARCA study). For that reason, the MBDS tends to overestimate mortality associated with these variables.
However, diagnoses that are theoretically not related to the seriousness of the patient's condition (hypercholesterolemia, high blood pressure, or smoking status, for example) tend to be coded more often in patients with lower risk (the code for hypercholesterolemia was recorded for over 46% of patients in the EuroSCORE low risk group compared to 19% of patients in the high risk group; results not shown). This could be explained by the greater space available to record codes for less serious diagnoses in patients with lower risk who therefore have a lower number of relevant or serious diagnoses.
It should also be borne in mind that, in contrast to specific studies, variables in the MBDS are often imprecisely defined and probably have greater variability. In general, a relationship between surgical risk and the pattern of coding can be deduced, whereby the most serious diagnoses are recorded in higher risk patients and the less serious diagnoses are more frequently recorded in lower risk patients.
This relationship between surgical risk and the pattern of coding would explain the loss of predictive capacity and lack of calibration (over-estimation) when the administrative model was validated with prospective data. The loss of validity was even more marked when the administrative model was applied to MBDS data from 2004-2006, with an area under the ROC curve of 0.65 (a value which is not very different from those observed in previous studies9,10). Two facts might explain, at least partially, this loss of validity. First, predictive models lose validity when they are applied to cohorts other than that from which they were developed and, secondly, they lose discriminative capacity over time18 due to changes in techniques and improvements in outcomes. The format of the MBDS also changed in 2003, which might further limit external validity.
In spite of these limitations, the quality of registries tends to improve over time. As an example, in Catalonia, a further 6 fields to record diagnoses and 6 to record procedures were added in 2003 which had a significant impact on the quality of the information available (Table 4). Furthermore, in general, registries seem to perform better in publicly managed centers, where there is likely to be more pressure to enter information because of the usefulness of registries in monitoring budgetary needs and resource assignation, and in assessing the adequacy of resources in relation to the services provided.
In summary, some limitations of the MBDS stem from design problems such as a lack of severity criteria, or a lack of precision in the definitions used, but others arise from bad management or poor data entry (errors in codification, omission of codes for post-surgical complications). As the quality of the coding is apparently gradually improving, administrative databases may become more useful, not only in estimating raw rates, but in providing adjusted mortality rates. Fomenting the use of administrative databases in research or outcomes assessment should also help to improve their quality.8
For certain interventions, such as heart surgery, which have a significant impact on health and are associated with non-negligible levels of risk and considerable cost, systems of continuous assessment would be recommendable. These could provide periodically adjusted results and act as a benchmark for outcomes from individual suppliers. Such systems could be based, in part, on administrative data,8 perhaps in combination with data collected in systematic clinical registries or specific studies, in a similar fashion to the system used in Ontario, Canada.13 In this case, the administrative data, which are based on ICD-9 diagnostic codes, are periodically crossed with severity variables (ejection fraction and urgent or emergency indication) from a clinical data base designed for systematic completion. Other authors10,12,19 have proposed an alternative, which is used in administrative or clinical databases in a stand alone fashion for risk adjustment. Hannan et al,12 for example, added 3 clinical variables (ejection fraction, reoperation, and narrowing of the left main trunk) to the administrative data, to construct a model that accurately predicted mortality with almost identical precision to that obtained from prospective data. Aylin et al14 also recently demonstrated that creative use of administrative databases can efficiently complement clinical registries.
CONCLUSIONS
This study has shown that administrative databases are currently insufficient to adequately assess the risk of hospital mortality associated with coronary surgery. Nevertheless, some of our findings indicate that, if information is entered correctly, they could play a role in this endeavor. Even in that case, however, it is likely that specific objectives would have to be stipulated, that some clinical variables would need to be added, and that variables would need to be more precisely defined.
Funding: Instituto de Salud Carlos III (FIS: 05/2108), CIBER de Epidemiología y Salud Pública.
ABBREVIATIONS
MBDS: Minimum Basic Data Set
Correspondence:
Dra. A. Ribera.
Unidad de Epidemiología. Servicio de Cardiología. Hospital Universitario Vall d'Hebron. Pg. de la Vall d'Hebron, 119-129. 08035 Barcelona. España.
E-mail: aribera@vhebron.net
Received January 9, 2008.
Accepted for publication April 9, 2008.