Effectiveness of a Multifactorial Strategy for Implementing Clinical Guidelines on Unstable Angina: Cluster Randomized Trial

INTRODUCTION

Clinical practice in unstable angina (UA) is not always based on best evidence. High rates of inappropriate care for this disease have been described worldwide.^1,2 In Spain, one recent study that used data from an UA national register showed that there is a minimal use of well established efficacious drugs and invasive procedures (coronary angiography [CA], 32%).³ Evidence-based clinical practice guidelines (CPG) have been advocated as a useful method to improve quality of health care,⁴ although evidence shows that passive dissemination of guidelines is not always enough.⁵ Based on this, during the last 10 years a big effort has been made internationally to develop guidelines for a whole range of diseases and conditions.⁶

Guideline implementation strategies have been shown to be moderately effective at increasing the use of appropriate care.⁷ Inappropriate care for UA in Spain could be improved, but unfortunately at the present there is a lack of a national coordinated strategy of CPG development, implementation, and assessment, based on homogeneous and rigorous methodology.⁸ Decision makers for a national or regional implementation strategy need the best evidence that comes from rigorous evaluations, but currently, there are no cluster randomised controlled trials assessing guideline implementation strategies conducted in the public healthcare system of Spain, although there are some prospective studies.^9-11 Moreover, the majority of the existing evidence comes from completely different settings (USA, UK, Canada).

The aim of our study was to test the effectiveness of 2 different UA guideline implementation strategies delivered to wards of cardiologists and internists. We chose interventions that could be feasible in the real world setting of public hospitals in Spain, and also that had some empirical evidence of success in previously published studies.⁵ A multifaceted strategy, consisting of interactive educational meetings plus local consensus process, was compared with a passive dissemination, a single lecture where the guideline was introduced to the control group. The rationale for the interventions in the control group was that passive dissemination of guidelines implicitly recognised that when people are taught, they change; and also represented current way of introducing new information in a ward. On the other hand, the multifaceted strategy implied that for people to change their clinical practice they must first recognise "the needs," the "what," the "when," and "how" (interactive educational meetings), and feel comfortable with the new practice (local adaptation and guideline ownership).

Our hypothesis was that the multifaceted strategy was more effective than the control one. Groups of physicians (clusters), instead of single physicians, were randomized to avoid managing both study and control patients in the same way, diluting the effects of the intervention.

This study was part of the CAMBIE project that tried an overall approach to improve quality of health care, by development and implementation of evidence based CPGs, and monitoring the results.¹²

METHODS

Participants

The study was based in the departments of cardiology, and internal medicine of three university hospitals in Seville (Spain). Groups of physicians clustered by a common place of work (wards) within departments, were randomised to be allocated to the study or control group. All the members of the medical staff in each ward were considered eligible. Chiefs of cardiology, and internal medicine departments at each hospital were asked for consent. The study obtained the approval of the ethical committees. Episodes of UA attended in those wards were identified from databases using several main diagnosis discharge codes (unstable angina, progressive angina, rest angina, new onset angina, post-infarction angina; that were the criteria accepted in Braunwald's definition at the time the intervention was delivered). We excluded those episodes with unknown main health care provider, episodes of silent ischemia, significant aortic valvulopathy, Prinzmetal's angina, typical chest pain with raised cardiac enzymes, home address outside from the area of the study, or death during admission period.

Interventions

We implemented an evidence based CPG for UA, developed by the Andalusian Agency for Health Technology Assessment (AETSA) that used systematic review of the literature, and expert panel opinion (RAND method).^13,14 We adapted the recommendations to local conditions (for example the availability of CA).¹⁵ Intervention group: 3, 45 minutes meetings were hold separately with each 1 of the study wards, the meetings had mixed components (local consensus process, and interactive educational meetings).^16,17 In the first meeting, a draft of the guideline was presented, we invited physicians to discuss the recommendations made in the guideline and asked them for feedback; relevant comments about barriers to implement the guideline were taken into account to modify some recommendations of the final version of the guideline. Doing this we tried to achieve a feeling of guideline ownership in the intervened wards. The second and the third meetings were after the publication of the guidelines, several representative cases of UA inappropriately attended in each intervened ward were reviewed and oriented according the recommendations in the guideline. The emphasis during these meetings was put on teaching the use of the guideline and clarifying doubtful interpretations of recommendations. We mainly focused on recommendations of use of CA and stress testing. Control group: a single didactic session of 30 minutes presenting the guidelines was given to the control wards. All the study wards were presented the same material by the same researcher (IM). The guidelines were delivered by hand to all participants at their workplace. The guidelines included a pocket card with decision aids algorithms on CA and stress testing use. The study took place from January to April 1998.

Outcomes

The primary outcome was the proportion of UA episodes attended at each ward with appropriate use of stress test and coronary angiography. Each episode of UA was rated appropriate when the physician decision adhered to guideline recommendations for both components. To assess adherence to the guideline we developed the criteria taking into account relevant variables for decision-making mentioned in the recommendations. Further information on clinical scenarios considered and how we rated the appropriateness of physician's performance for the primary outcome is available at http://www.redeguias.org/public_articulos/anexoREC.pdf

Secondary outcomes: we measured the proportion of UA episodes that assessed the left ventricular ejection fraction (EF). The assessment was considered valid if the EF reported in the clinical record was assessed during admission or at any moment in the previous 6 months.

Study Design and Sample Size Calculation

The study was a before and after, pragmatic cluster randomised controlled trial. This design is considered the optimal for implementation research.¹⁸ Randomisation of wards, stratified by hospital and specialty department was done by the main researcher (IM) using a random numbers table. Study participants were blinded to the study objectives and design, that we tried to achieve by 3 steps: delaying the time between intervention and recruitment of patients with a wash-out of 3 months; avoiding to tell when we started the recruitment of the sample; and by collecting data from clinical records at the end of the recruitment period, to avoid over-observation upon the clinical process. As the research team was practising at the study setting, those wards that included a researcher were allocated to the study group (compelling allocation). For sample size calculation we estimated, from a previous study, an appropriateness rate around 60% for CA and for stress testing (Briones E, personal communication). We considered relevant to detect a 15% change in the appropriateness of use of CA. For a fixed number of 10 clusters eligible, a statistician (JRL) estimated a total number of 872 UA episodes necessary to detect that difference with an 80% power, assuming α=.05. The intraclass correlation coefficient used was 0.05. The sample size was recruited proportionally according to the ward size. To calculate the power for the cluster level analysis we estimated that the variance of the primary outcome at cluster level, using the baseline data from the 10 clusters, was 101.4, we used a significance level of .05 (2 tailed), so the power to detect a difference of 20 in the means of the primary outcome between groups was 81.7%, and for a difference of 15 the power was 57.5%.¹⁹

Data Collection

Recruitment of UA episodes for the post-intervention period started three months after the intervention was delivered, from September 1998 to June 1999. Recruitment of pre-intervention episodes was from January to October 1997.

Detailed demographic and clinical data from each episode were obtained from retrospective clinical record review in July 1999. A team of abstractors blinded to the study objectives and design were trained to collect the data. Outcomes from each episode were assessed using the same criteria by one researcher (CA, AR) that was blinded to the intervention status of the physician. Doubtful episodes were resolved by consensus between assessors. Physicians' attitudes towards guidelines were surveyed in a physician sample, before the intervention using a 13-items validated translated version of the CPP-18 questionnaire.²⁰

Statistical Methods

We studied the same physicians per each cluster before and after the intervention. A t test was used to compare the mean change (before-after) in appropriateness between intervention and control wards for the primary and secondary outcomes. We did a pragmatic intention to treat analysis on all the physicians, whether or not they attended all the meetings. Analyses were conducted using SPSS (version 9.0) and SAS (version 8.2).

RESULTS

All eligible wards were enrolled; one ward (a coronary unit) withdrew from the study because there was a change in the organization of the inhospital flow of patients during the study (patients were not discharged directly to home but to a cardiology or internal medicine ward) that caused a low recruitment rate for that ward. More than 85% of doctors attended the sessions. Data abstraction method was considered reliable.²¹ Figure 1 shows the flow of participants through the study and the randomization process. Differences in the number of episodes recruited between the control and intervention group were explained due to changes introduced during the trial in the inhospital flow of patients between specialties. This imbalance was maintained in order to avoid an extended recruitment period, and to observe both study arms concurrently. There were no significant differences in the clinical characteristics of UA episodes between study groups (Table 1). The majority of the physicians had a good attitude towards guidelines (Table 2).

Figure. Randomization process and flow diagram of the progress of wards and physicians in the trial (post-intervention phase). ep indicates episode; phy, physician.

Primary Outcome

A total of 1734 episodes were analysed. Overall, the appropriateness of use of CA and stress testing in each ward before the intervention was very low (mean appropriateness, 31%; SD, 10), the lowest performance was observed in 2 intervened wards (around 24%). The multifaceted implementation strategy compared with the passive one, was associated with an absolute improvement in the appropriateness of use of CA and stress testing of 11% (95% CI, 0.85-21.1; P=.03), this represents a relative improvement of about 130% (Table 3).

Secondary Outcomes

Baseline data showed that the mean rate of EF assessment in UA episodes was 57% (SD, 17.1). Both implementation strategies were equally effective at improving the number of episodes with an assessment of the EF performed (mean change in each at improving group around 12%). For the difference between the 2 strategies, the study group did not do better than the control group, with an absolute improvement of 1.1% (95% CI, ­-15.9 to 18.1; P=.88) (Table 4).

DISCUSSION

These results have shown that interactive educational meetings and local consensus process delivered to wards of physicians may improve the appropriateness of use of CA and stress testing by 11%. The results are in concordance with those from a systematic review on educational meetings¹⁷ where interactive seminars and workshops were considered to cause moderately large changes in professional practice. Generally, no benefit was reported for didactic sessions in that systematic review.

Local consensus processes can help people involved in the decision-making consider issues that will directly affect them, increasing their sense of ownership and commitment to adhering the decision reached.²² Currently, there is conflicting evidence on the effectiveness of local consensus processes.²³ We didn't plan to study which component of the multifaceted intervention that we used in our study was more active.

In our study a relevant change in both groups was found for the assessment of the EF; this was probably due to contamination of the intervention effects between groups. Although physicians of each ward didn't know that they were part of a study, communication between members of different wards in the same hospital department is common. This communication could have lead to an increased interest in the adoption of guideline recommendations by the control wards members. Contamination of intervention effects is a well known problem associated with some cluster randomised trials.²⁴

For the primary outcome, the decision to include or exclude the ward 0 had implications on the statistical significance found. We decided to report the results without that ward not considering it a major threat to an intention to treat analysis, because only 3 UA episodes registered are not enough to precisely estimate the ward performance.

Another interesting finding was the overall physicians' positive attitudes towards guidelines. This means that good attitude towards guidelines is not enough to change practice. The finding agrees with those of a systematic review and stress the necessity of doing educational, organisational, and structural changes in the healthcare system to change physicians' behaviour.²⁵

Strengths

Our study, like few others in Spain (CAM¹¹), measured changes in appropriateness of use of treatments and diagnostic procedures and not only rate of utilization, which is a better way to assess changes in quality of care delivered.

To our knowledge this is the first study on quality improvement research done in Spain with the rigorous methodology of a cluster randomised trial. Although before and after studies (uncontrolled or controlled) are appropriate for local quality improvement projects, generalisable knowledge can only be produced using randomised controlled trial designs.²⁶ The CAM study with a before and after design, measured the effect of a simple educational intervention that yielded an absolute change of 8% in the appropiateness of use of i schaemia testing, a result like ours.¹¹ However the absolute change obtained in the CAM study could represent an overestimation, mainly due to a design that did not allowcontrolling sudden changes or secular trends, so making it difficult to attribute the observed effect to the intervention.²⁶ In this sense, the absolute effect obtained in our study is more realistic; randomised designs allow that all the relevant factors (known or unknown) for the outcome are evenly distributed in both control and intervened groups. In that way differences in the effect obtained after the intervention can be better attributed to the intervention rather to other factors (like the self-selection of participants in the CAM study).²⁶

Our study was conducted in public teaching hospitals and, in our view, the findings are generalisable at least to similar centers in Spain. The preparation of the meetings weren't resource intensive and any cardiology or internal medicine department could deliver that intervention when a reliable guideline is available.

There are 2 other implementation prospective studies in Spain, with a different approach that our study. The first, is the implementation of a quality improvement program in acute myocardial infarction (using audit and feedback, and computer generated reminders in the discharge report).⁹ They reported a range of absolute improvement from 5% to 46% depending on the indicator, achieving for all the indicators the surprising success of 100% patients correctly treated. These results could be considered as an outlier if we take into account the data reported in a recent systematic review of the effectiveness of guideline implementation strategies where audit and feedback showed a median effect of +7% (range, +1.3%±16%), reminder systems a median effect of +14% (range, from -­1% to +34%), and multifaceted (mixed strategy) a median effect of +6% (range, ­-4% to +17.4%).²⁷

Another study that focused on an education program directed to patients and families, suggested that it could be effective to improve the rate of adoption of secondary prevention measures after a myocardial infarction. This strategy promises a great future but is not comparable with our study because the target group studied were patients and we focused on professionals. Besides, the lack of a control group and the high rate of lost to follow-up limit the generalisability of the results.¹⁰

Limitations

Our study has some limitations. The results should be interpreted in the context of the flaw of using a local guideline not endorsed by the national cardiology scientific society. This could have reduced the credibility of the guidelines between the professionals in the study, explaining part of the small effect shown in some wards. Furthermore during the intervention period, relevant advances on the classification and clinical approach to acute coronary syndromes were taking place. Some participants aware of this new knowledge could have considered the guideline not updated and for that reason not useful.

Study groups weren't similar at baseline for the primary outcome, two wards in the intervention group had the lowest performance, and the bigger change in the appropriateness after the intervention was observed in them. This effect is common in quality improvement research, where low performers usually benefit more from the intervention.²⁶ Another problem was that some researchers were involved in the study as participants; as they were unblinded to the study objectives and methods, we had to allocate their wards (3) to the intervention group. Two out of 3 of these wards obtained the highest benefits from the intervention, this could be due to the fact that those wards had the lowest performance at baseline, but also to the fact that researchers' practice was more adapted to the guideline, effects of these researchers' new practice on their ward's fellows, or both. Covariate adjustment at cluster level (specialty, baseline performance, co-morbidity) was not possible due to the small number of clusters.

These common problems and methodological difficulties are threats to the internal validity of our study, although the circumstances that caused these limitations represents the real world of quality improvement research in Spain and in that way the results are valuable to extract some lessons and conclusions.

CONCLUSIONS

Our results provided evidence that interactive educational meetings and local consensus process delivered to wards of physicians may improve the appropriateness of use of coronary angiography and stress testing. The assessment of the ejection fraction was improved in a relevant way with the implementation strategies in both groups but without difference between them. Even with a low performance at baseline, physicians' attitudes towards guidelines were not negative.

Implications for Clinicians, Health Care Managers, and Policymakers

In order to improve clinical practice, active implementation of best evidence is needed. This is a complex process, which is difficult to achieve, and can not always be done using an intuitive approach based on personal beliefs of what interventions can improve clinical practice²⁸; besides there is good evidence available that can help to guide these decisions.^7,27 This evidence together with better global standards allows us to foresee an optimistic future for quality improvement research. Fortunately, scientific societies are committed to produce global guidelines sharing resources to increase credibility and quality of the guidelines. Today it is not justifiable to produce local guidelines, but to adapt global guidelines.²⁹

This study has tried to raise awareness about the necessity of well designed and conducted quality improvement research in Spain in order to make relevant informed decisions on health care at regional and national level.

ACKNOWLEDGEMENTS

To Drs Burgos, Carneado, Cruz, Grilo, Pastor, Pérez Cano, and Rivera, chiefs of their respective intervened wards. To Jacques Lemelin for his unconditional help that made possible the writing of this paper. To Lorenzo Moja for his comments to a previous version of this paper.

Others members of the CAMBIE study group are: Drs Cotos, Gómez-Gras, and Reyes Álvarez who performed the guidelines attitudes study.

Funding: Fondo de Investigaciones Sanitarias. PI: 96/1226; PI: 00/0667; PI: SAS, Grant, 96/105, European Community FEDER FUNDS. Redes Temáticas Program, FIS-G03/90, Spain.

Authors disclose any conflict of interest. This work was awarded the 2005 Dr López Laguna prize by the SADEMI.

---

Correspondence: Dr. A. Romero.
Servicio de Medicina Interna. Hospital Universitario Valme.
Ctra. Cádiz-Bellavista, km 548,9. 41014 Sevilla. España.
E-mail: [e-mail]