Validation of the EuroSCORE Probabilistic Model in Patients Undergoing Coronary Bypass Grafting

INTRODUCTION

Health-care providers need to be able to reliably assess their activities in terms of outcomes, quality, and cost-effectiveness. This is, in part, due to the constant rise in health care costs and the fact that resources are limited, though other drivers include the increased demand for health services and the need to compare clinical outcomes between centers. As all institutions require information on quality of care they need to be able to summarize their activities in terms of outcomes adjusted by the center's specific characteristics.¹

In the area of heart surgery, both in Europe² and the United States,³ the periodic publication of outcomes reports is now widespread and adequately regulated.^4,5 These reports incorporate information from mathematical models which are used to predict the likelihood of a certain event, such as death, occurring in a given individual based on a group of risk factors attributed to that particular patient.⁶

The European System for Cardiac Operative Risk Evaluation (EuroSCORE) is a logistic model which is used to predict hospital mortality in patients undergoing a cardiac intervention. Using 18 risk variables and a beta coefficient associated with each variable (Table 1), the model provides the likelihood of death for any individual. The model was created and initially validated in a cross-sectional study^7,8 of 19 030 European patients in 1999. Since then, it has become the most widely used model worldwide in this type of patient.

A much simpler variant of the logistic model is the additive EuroSCORE, which assigns a weight to each risk factor presented by the patient. The sum of the weights provides the likelihood of dying for that patient. The widespread and uniform use of a single probabilistic model allows for internal and external comparisons over time and can help to minimize risk adverse behavior which might be fomented if comparisons are made using unadjusted outcomes.^9,10

The primary objective of the present study was to validate this predictive model of mortality in a large teaching hospital in Barcelona, Spain. The model was assessed in terms of fit and discriminatory capacity.¹¹A second study objective was to compare the additive and logistic versions of the model and to determine which was the most appropriate for use in different groups of patients defined by their level of risk. The additive version has been the most widely used of the model variants because, although it is less precise, it is much easier to calculate and it can be calculated at the bedside. Nevertheless, the logistic equation has been shown to better predict mortality, particularly in high risk patients, and is recommended for use in those patients.^12-14

METHODS

Patients

This was a validation study performed in the Hospital Clínic of Barcelona (HCB) using a retrospective design. The hospital is a 720-740 bed center which deals with 40 000 admissions annually. Patients with cardiovascular disease are attended in the Clinical Institute for Diseases of the Thorax which includes, among others, the Cardiovascular Surgery and Cardiology services. The study was approved by the Institutional Review Board of the HCB.

The center's computerized database (SAP^®) was used to obtain data on all patients who underwent procedures defined in the CIE-9-CM¹⁵ and who were assigned codes 36.10 to 36.17 and 36.19. The SAP is used for both clinical and administrative purposes and has high reliability. All discharges are coded. The study population consisted of patients derived for coronary surgery and who underwent surgery between July 1, 2004 and July 1, 2006.

Variables and Statistical Analysis

The 18 variables included in the EuroSCORE predictive model were identified for all patients included in the study (Table 1) together with administrative variables (date of admission and discharge) and data on deaths occurring while patients were hospitalized in relation to the intervention. These data were obtained from computerized clinical records which included discharge history as well as surgery, pre-anesthesia, and laboratory reports. If no value was recorded for a specific risk factors, it was assumed that the risk factor was absent.

Both the logistic and additive versions of the Euro-SCORE¹² were used to predict mortality. In the case of the additive EuroSCORE, the probability of dying was calculated by summing the relative weights for each risk factor for all individuals.

In order to calculate predicted mortality using the logistic model, the following equation was used

where b₀ takes a value of -4.789594 (logistic regression constant) and b_i is the regression coefficient for the variable c_i in Table 1. For the age variable, in the logistic method b was multiplied by the number of years that the patient exceeded 60 years of age. In the additive method, a weight of 1 was assigned for every 5 years (or part of 5 years) over 60.

The validity of the logistic regression model was analyzed by examining its goodness of fit and discriminatory capacity. Goodness of fit was assessed using the Hosmer-Lemeshow test^16-19 which estimates a C statistic from the difference between observed and expected values for mortality in different risk groups. The lower the C statistic the better the model's fit. A value of P>.05 indicates that the model fits the data well and that it therefore accurately predicts mortality. The test is most frequently used to validate recently created models but it is equally useful in validating an existing model which has been applied in a new set of data, as in the present study.

The model's ability to discriminate is assessed in terms of its capacity to distinguish between patients who died during hospitalization from those who did not. Discriminatory capacity was analyzed by calculating the area under the ROC curve. A value of 0.5 indicates that the model is equivalent to pure chance and a value of 1 indicates perfect discrimination.¹⁷

The logistic and additive models were compared by calculating the mortality predicted by each in both the overall sample and in 2 sub-groups defined by level of risk. The high and low risk groups were defined using a cut-off point on the additive EuroSCORE of 6 points,⁶ a cut point which had previously been used for this purpose.^8,12 ROC curves were calculated for both models.

The validation analysis was carried out using the STATA^® v.8 statistical package and the comparative analysis using SPSS^® v.12.0.

RESULTS

A total of 498 patients underwent heart surgery in HCB between July 1, 2004 and June 30, 2006. The distribution of EuroSCORE risk variables in these patients is summarized in Table 2.

For the logistic model, a C statistic of 11.51 (P=.32) was obtained on the Hosmer-Lemeshow test and the area under the ROC curve was 0.83 (Figure 1).

Figure 1. ROC curve for EuroSCORE logistic model.

Among the patients who underwent surgery during the study period, there were 29 hospital deaths, giving an overall mortality rate of 5.8%. Total predicted mortality was 3.9% using the additive model and 4.2% using the logistic model (Table 3).

In the low risk group (n=412), observed mortality was very similar to that predicted by both models; however, in the high risk group (n=86) the values predicted by the logistic model were closer to observed values (Table 3). Both models showed good discriminatory power, with an area under the ROC curve of 0.84 for the additive model and 0.83 for the logistic model (Figures 1 and 2).

Figure 2. ROC curve for EuroSCORE additive model.

DISCUSSION

In order to evaluate the quality of health care services and to be able to adequately inform patients on the likely outcomes of the health care process, crude values for overall observed or expected outcomes are often not sufficient. Prognostic models which take into account patients' specific characteristics and which provide risk-adjusted outcomes for interventions are required and more useful.⁶

There are many risk-adjusted models available to predict mortality in cardiac surgery interventions, though in recent years the EuroSCORE⁸ has become one of the most widely used in western countries.

The HCB is a reference center for this type of heart surgery and performs a large number of coronary interventions. In this type of center, reliable and comparable data are required in order to assess the quality of care.

Before using a probabilistic model in a context other than that for which it was created,²⁰ the model should be validated to ensure that it does not generate erroneous probabilities. The aim of the present study was to validate²¹ the EuroSCORE in the HCB.

The C statistic obtained with the Hosmer-Lemeshow was P=.32, which indicates satisfactory model fit for patients undergoing surgery in our center. The model's discriminatory power was also adequate, as indicated by an area under the ROC curve of 0.83. Given these results, we can conclude that the EuroSCORE model has been validated for use in this center and that it has proven to be a reliable instrument. This signifies that the model's predictions of the probability of dying are valid and appropriately risk-adjusted for surgery patients in HCB. It should be pointed out that mortality during hospitalization is a very favorable measure of mortality as it does not incorporate mortality after discharge. Nevertheless, we believe that intervention-related mortality after discharge was practically null in this series.

The results of this study are in line with those reported in the earlier article on model validation in six countries in the European Union,²² to which Spain contributed 2422 patients. In the earlier study, the Hosmer-Lemeshow C statistic was P =.38 and the area under the ROC curve was 0.87. Nevertheless, the earlier validation in Spain took place in very diverse settings and conditions, which meant there was a need for further validation in specific contexts before the model could be used with confidence.

The additive, or standard, model has been the most widely used because it is easy to use. It is a simplified version of the logistic model and the weights it uses are derived from that model. A previous study¹² indicated that the additive model tended to underestimate the probability of death in high risk patients.

In the comparison of the 2 models, only low and high risk groups were studied because of the relatively low mortality in the study population. Creating a larger number of risk groups would have led to very broad confidence intervals for the predicted mortality rates and would have hindered comparisons. The cut point for defining the 2 groups was a EuroSCORE value on the additive model of >6. In this study, we found that the logistic model more accurately predicted the probability of death in the high risk group, a result in line with previous studies which have compared the 2 models.¹²

These results indicate that the logistic model is more accurate and more appropriate for use in daily practice in heart surgery units. The fact that most such units now have the technical resources to rapidly calculate a score using the logistic model further supports its use.

A study in a larger sample would generate a larger number of events (deaths) and would provide more solid results as the Hosmer-Lemeshow test is based on contrasting predicted events with observed events.

A direct comparison of real and expected events was not carried out as they are two distinct variables which provide information about events. True mortality describes the event death, whether observed or otherwise, for each patient (dichotomous variable). Predicted mortality, on the other hand, indicates the likelihood of dying for each patient based on specific characteristics included in the model (quantitative continuous variable). The analysis of the relationship between these 2 variables is precisely what constitutes the validation of the model. The validated model is useful because it allows us to perform risk assessments for patients which can then be compared with observed outcomes, while taking into account the level of risk.

The publication of the outcomes of health care has been a reality in the United States and the United Kingdom for over 15 years. The impact of such publications has been extensively analyzed, with the results showing a clear improvement in the quality of health care.²³ The instrument validated in the present study could be useful in providing systematic information on the outcome of interventions in centers providing the relevant services. The information could be published and made available both to citizens and the purchasers of health care.

CONCLUSIONS

The results of this study allow us to conclude that the EuroSCORE is a useful probabilistic model in this public, teaching hospital. It can be used to estimate the probability of death in patients scheduled for heart surgery and to assess the outcomes of health care. The logistic model is the most reliable of the 2 versions, particularly in high risk patients.

ABBREVIATIONS
HCB: Hospital Clínic of Barcelona
EU-A: additive EuroSCORE
EU-L: logistic EuroSCORE
EuroSCORE: European System for Cardiac Operative Risk Evaluation
ROC: receiver operating characteristic

SEE EDITORIAL ON PAGES 567-71

---

Correspondence:
Dr. A. Trilla.
Servicio de Medicina Preventiva y Epidemiología. Hospital Clínic. Villarroel, 170. 08036 Barcelona. España.
E-mail: [e-mail]

Received August 11, 2007.
Accepted for publication December 19, 2007.