Qualitative Evaluation to Explain Success of Multifaceted Technology-Driven Hypertension Intervention

Published Online: December 16, 2011
Mari Millery, PhD; Donna Shelley, MD, MPH; Daren Wu, MD; Pamela Ferrari, RN; Tuo-Yen Tseng, MA; and Helene Kopal, MPA, MPH

Objectives: This study sought to examine the implementation of an electronic health record– based intervention to improve quality of hypertension care in community health centers. The primary goal was to use qualitative analysis to explain how different components of the intervention contributed to positive patient-level outcomes.


Study Design: Qualitative process evaluation.


Methods: The intervention included alerts, order sets, templates, clinical reminder algorithms, and provider performance feedback. Semi-structured interviews were conducted with primary care providers before (n = 16) and after (n = 16) intervention, and with key staff and leadership involved in the implementation (n = 6). The research team applied an iterative systematic qualitative coding process to identify salient themes. Several constructs from IT implementation theories guided the analysis.


Results: The analysis focused on: (1) satisfaction and perceived usefulness of intervention components, (2) perceived proximal changes resulting from intervention, and (3) perceived facilitators of change. Different participants found different components useful. Proximal impact manifested in multiple ways (eg, more aggressive follow-up
appointments and prescribing) and in increased overall attention to hypertension. Facilitators of success included leadership, organizational culture, provider engagement, rigorous implementation process, framing of intervention as quality improvement (QI), and health center capacity to process data.


Conclusions: We attribute the success of the intervention to a multifaceted approach where the combination of multiple intervention components resulted in across the-board change in hypertension care practices. In contrast with research that attempts to isolate the impact of circumscribed health information technology (HIT) tools, our experience suggests that HIT can achieve success in patient outcomes when rigorously implemented as a multifaceted intervention and framed as QI activity.


(Am J Manag Care. 2011;17(12 Spec No.):SP95-SP102)

Qualitative process evaluation of a multifaceted technology-driven hypertension intervention in community health centers resulted in the following key findings and conclusions:


  •  Multiple intervention components contributed to changes in hypertension care practices, supporting the multifaceted approach to health information technology implementation.


  •  Framing the technology-driven intervention as quality improvement was a facilitator of success.


  •  Other facilitators included leadership, organizational culture, provider engagement, rigorous implementation process, and health center capacity to process electronic health record data.


  •  Qualitative process evaluation was found to be a feasible and useful method for investigating reasons for the success of a multifaceted technology-based intervention.
Hypertension affects one-third of the American adult population.1 Appropriate management of hypertension presents a significant opportunity to improve cardiovascular health outcomes through interventions in primary care settings. The costs of healthcare, medication, and missed days of work from hypertensionrelated symptoms and complications were $76.6 billion in 2010.1 Improved control of blood pressure among those diagnosed with hypertension would pay off in lowered incidence of stroke and heart disease.

Health information technologies (HITs) are a promising tool for improving quality of care in primary care settings, including underserved settings such as community health centers (CHCs).2 Much of HIT research tests the efficacy of isolated technology components, such as a decision support tool or computerized order entry. Less is known about the effectiveness of HIT-based interventions in community-based settings that combine multiple intervention components in a comprehensive effort to improve quality-of-care outcomes.3 Evidence of effectiveness of HIT to improve hypertension outcomes is mixed.4-6

Effective interventions to improve community health outcomes call for complex multi-level and multi-component approaches.7 However, it is challenging to design studies that can attribute quantitative health outcomes to particular intervention components.8 Instead, the outcomes indicate success of the entire “package.” Process evaluation is an avenue to explore questions about the effective “ingredients” of such interventions. One important purpose of process evaluation is to explain the results of an outcome evaluation,9 also called “interpretive evaluation.”10 Process evaluation using qualitative methods may be particularly wellsuited for understanding why a complex intervention led to its outcomes.

In the domain of information technology (IT) implementation, several models posit factors that explain success of IT. The Technology Acceptance Model (TAM)11-13 builds on general social-behavioral theories and proposes constructs such as perceived usefulness and perceived ease of use as predictors of end-user acceptance and use of IT. The DeLone & McLean Information System Success Model14 predicts organizational and individual impact of IT systems from system use, user satisfaction, system quality, and information quality. A third model proposed by Ovretveit et al15 identifies factors that help or hinder HIT implementation, including characteristics of the HIT system, the implementation process, leadership, resources, and organization culture and climate. These models are helpful in categorizing and labeling factors of importance in HIT implementation research.

This qualitative study examined a multi-component quality improvement (QI) intervention in a CHC with several practice sites in New York that included clinical decision support within an electronic health record (EHR) and provider feedback. The intervention had a positive impact on provider adherence to hypertension guidelines16 and hypertension control among patients.17 The goal of the qualitative analysis was to explain how different facets of the multi-component intervention contributed to the positive impact of improved hypertension control.


Setting and Patient Population

The study setting was Open Door Family Medical Centers, a federally qualifying CHC with 4 primary care sites located in suburban communities in New York. The CHC provides primary care to approximately 40,000 patients annually. It is a safety-net provider with a patient population approximately 74% Hispanic, 15% non-Hispanic white, and 9% non-Hispanic black; 35% of the patients have Medicaid, 4% Medicare, 4% private insurance, and 57% no insurance. In May 2007, the CHC started using an EHR system (eClinicalWorks). The leadership of the CHC has a strong interest in systematic QI activities and was eager to use the new EHR as a QI tool. At baseline of the study, 14% of all CHC patients were diagnosed with hypertension and an additional 5% were found to have undiagnosed hypertension. About half of hypertensives had controlled blood pressure at their last visit.18


The CHC leadership had identified hypertension as a QI target because it was the most prevalent adult chronic condition, a significant contributor to leading causes of mortality, and appeared feasible to improve through appropriate management. The CHC had recently engaged in a successful QI diabetes program and wanted to now turn their attention to hypertension. The intervention was designed by a collaborative team that included the administrative and medical staff from the CHC and public health researchers. Components of the intervention were informed by pre-intervention interviews with all providers practicing at the 4 study sites. Several key components of the intervention were implemented in the CHC’s EHR system, including alerts of high blood pressure readings, and templates, order sets, and clinical reminder algorithms for hypertension management. The templates and order sets offered standard sets of items to be included in hypertension visits while the clinical reminder algorithm prompted specific actions based on patient characteristics. Providers also received trainings and quarterly report cards (feedback) indicating levels of hypertension control among their panel of patients. The report cards were discussed in group meetings, followed up by one-on-one meetings with the CHC Medical Director for providers who performed below average. Additional details of all components of the intervention are described elsewhere.17

The outcome evaluation design compared repeated measures of hypertension control before and after the intervention. The proportion of visits in which hypertension was controlled was 51% at baseline and increased significantly to 61% in the post-intervention period.17 The process evaluation included quantitative surveys of providers and qualitative interviews of providers and key informants. The quantitative surveys were primarily used to measure provider attitudes preintervention and to inform intervention design. This paper reports results based on the qualitative interviews.

Provider and Key Informant Interviews

All providers practicing at the 4 health centers (n = 16) were interviewed about 6 months prior to and at 3 to 4 months following the launch of the intervention. The interview questions focused on perceptions and experiences regarding hypertension care, hypertension guidelines, EHRs, organizational environment, intervention components, and intervention implementation. Key informants (n = 6) representing leadership and staff actively involved in implementation of the intervention were interviewed about the implementation experience about 5 to 6 months after the intervention began. The semistructured interviews were conducted by the evaluators and lasted about 30 to 45 minutes. They were audio-recorded and transcribed for analysis. Human subject participation was approved by the Institutional Review Boards at New York University and Columbia University Medical Center. Informed consent was obtained from all interview participants.

Data Analysis

A team of 4 researchers conducted an iterative process of identifying a set of thematic codes and then applying them systematically to all interview transcripts. The team first reviewed key concepts in the 3 theoretical models (TAM, De- Lone & McLean’s model, and Øvretveit’s model). They then read all interview transcripts, identifying salient themes and notable quotations, and collaboratively drafted a hierarchically organized list of thematic codes. The final coding manual included a total of 78 themes and sub-themes. Using Atlas.ti qualitative analysis software, the researchers applied the codes systematically to all transcripts. To verify agreement, 10% of transcripts were coded by 2 researchers. For this paper, the codes were further reorganized under key process evaluation questions, as described below and shown in the Table.


The findings were organized into 3 main domains that emerged as we examined the data in light of process evaluation research questions and implementation theories:

1. Satisfaction and perceived usefulness of intervention components

2. Perceived proximal changes resulting from the intervention

3. Perceived facilitators of change Each of the 3 domains contributes answers to why the intervention had a successful outcome. The Table shows how the 3 domains relate to key functions of process evaluation and to major constructs of the 3 theoretical models used to guide the interpretation of the data.

1. Satisfaction and Perceived Usefulness of Intervention Components

PDF is available on the last page.


Recommended Reading