 Those 2 words conjure various images to many people. Some clinicians still think of "managed care" as a reductionist, mindless management technique designed to reduce their clinical freedom and impose an accountant's will on their decision making. However, over time, most practitioners have recognized that the proper application of managed care leads to better healthcare for their patients with variable constraints imposed on the clinician-patient interface. 

 Two letters that also convey a variety of images to those who hear them or read them. For those who practiced medicine prior to the mid-1990s, the Veterans Health Administration (VA) system usually meant older red-brick buildings, populated by caring physicians fighting an autocratic, nonresponsive administration to provide care for their patients. Patients often endured months of waiting for appointments, endless hours of sitting in clinic reception areas before being seen, a lack of continuity of care, and bureaucratic hurdles for both patients and physicians to overcome. However, like managed care, the VA is now seen as a leader in providing high-quality care to its patients. 

, we highlight health services research performed in the VA system. In particular, we focus on issues and areas that may resonate with readers who work in managed care and confront problems similar to those of their VA colleagues. Yet, this research and the quality of care that VA provides today were not achieved through happenstance, but rather represent the VA's active transformation of itself. As with the televised "extreme makeovers" of individuals or their homes, the VA's "extreme makeover" resulted from an awareness of the need for change. The VA's managers recognized that the organization had to change and they set the stage to support those efforts. In the first article in this issue, Jonathan Perlin, the VA Acting Undersecretary for Health, and his co-authors review this period of change.

 We learn how the VA modified its structure to reflect a more regional approach to healthcare compared to its previous hospital-centric focus. The VA developed advanced electronic records, allowing for automated integration of clinical records, electronic order entry, laboratory test results, imaging studies, and clinical reminders. Leadership at all levels of the organization was held to higher objective standards than before. This transformation has led to the VA's current position as a leader in the quality of healthcare. Along the way, the VA encountered occasional problems. Its electronic record systems have failed from time to time, leaving healthcare providers with little or no access to patient records and an inability to get laboratory test results or order medications for their patients. In addition, change mandated from above has not always been well received below, leading to occasional adversarial relationships within the organization. Despite these intermittent obstacles, the VA has much to be proud of. 

Some of that pride can be centered on its research programs. The remaining articles in this issue present a variety of Health Services Research investigations designed to showcase both the breadth of VA research and its applicability to non-VA settings. 

For example, can one develop an accurate single-item question for depression screening? Corson et al. took up this challenge and found that, although a 1-item screening question was relatively specific, it was not sensitive enough to replace more accurate questionnaires.

 However, the authors also report that a 2-item depression screen from the Patient Health Questionnaire (PHQ-2) had similar sensitivity and specificity for depression as the standard 8- and 9-item scales. For patients who are asked to complete large numbers of forms or answer a series of questions, a reduction from 8 to 2 questions is a significant achievement and one that might be worthwhile for managed care organizations searching for a screening instrument for depression. 

Of course, screening for depression is only useful if an organization can then assure that it can provide quality depression care. Continuous Quality Improvement (CQI) is one mechanism organizations often use in an effort to assure quality healthcare. If CQI is truly related to quality of care, one would expect to find a strong association. Charbonneau et al. examined whether the quality of depression care provided at 14 VA hospitals was associated with each hospital's CQI and organizational culture.

 The authors combined data from 2 separate studies, a novel way to save time and expense to answer relevant policy questions, and found no such correlation. However, their study was underpowered to truly examine this question. They were also not able to examine whether CQI was utilized specifically for depression care nor could they evaluate the organizational culture in the units providing depression care. But this study may lead other organizations to consider whether this is the approach to take to assess such relationships. Few healthcare systems in the United States can match the size of VA; if a study with 14 hospitals and more than 12,000 patients was unable to demonstrate a relationship between CQI and quality of care, how large a study is necessary? 

The VA is also interested in what its stakeholders think about the care they receive. One of its prime stakeholders is veterans of the US Armed Services. Unfortunately, some veterans have low literacy levels. Weiner et al. report on the iterative design process they used to develop a patient satisfaction questionnaire for low-literacy populations.

 Such questionnaires may be useful in the future for organizations with many low-literacy or limited-English-speaking members. 

Another issue confronting VA and non-VA patients is rising pharmaceutical costs. Each year, drug prices continue to rise faster than overall healthcare costs. Since 1996, the number of veterans seeking care from the VA has increased by 75% from 2.8 million to 4.9 million. Many of these patients come to the VA because they can no longer afford the cost of their medications under other healthcare systems. Piette and Heisler report on the impact of escalating pharmaceutical costs on VA patients compared to those with Medicare, Medicaid, private health insurance, and no insurance coverage.

 The researchers found that both cost-related underuse of medications and having to forego basic needs to pay for medications were more common among patients with Medicare or no health insurance than among those receiving VA care. As Medicare promotes alternative systems to help senior citizens pay for their medications, the government may consider comparing the cost effectiveness of the VA program to the newer programs being promoted to senior citizens. 

The final 3 articles in this issue address different aspects of the electronic medical record and utilization of administrative data to assess healthcare quality. While the VA is justifiably proud of its electronic medical record, Kaboli et al. find that databases can only reflect the information entered into them.

 The researchers examined the level of agreement between patients' reports of the medications they were taking and information in the VA electronic system and found complete agreement for only 5% of patients. Among these patients, who were on a minimum of 5 medications, patients reported taking approximately 3 medications that were missing from the VA list, while a mean of 1.3 medications were on the list, despite the patient no longer taking them. The VA and other healthcare systems will need to remain vigilant and not naively assume that an electronic record is always accurate. 

Similarly, the development and incorporation of a clinical reminder system into an electronic medical record does not guarantee its deployment or utilization. Fung et al. surveyed participants at a national VA electronic health record meeting about the number and type of clinical reminders in use at their facilities.

 While reminders for conditions measured nationally by the VA were commonly used, others that were not nationally measured were less consistently used. Some of this variation may be attributable to provider attitudes toward electronic reminders, with greater numbers of reminders at sites where providers reported finding them more useful. 

The final article, by Tseng et al., is more pertinent to the fee-for-service sector than to managed care.

 In contrast to managed care, where patients generally receive all their care, many veterans are dual system users who may have private health insurance or Medicare, but also seek care to varying degrees at the VA. Like the fee-for-service sector, the VA often finds it impossible to have complete information on the care patients receive from other healthcare providers. In the VA the importance of this dual care is clearly demonstrated when examining regional variation in lower extremity amputation rates among veterans. The addition of Medicare data increased reported amputation rates among patients with diabetes by 34% to 151% and changed outlier status for 8 of the 22 regions studies. Similar problems may confront other healthcare organizations as they move toward measuring and improving the quality of care for all the patients they care for and not merely those in managed care. 

Few Americans would dispute the need to provide high-quality care to those who have served in the Armed Services. As one of the largest healthcare systems in the country, the VA is in a unique position to analyze all aspects of its care and report the results to guide other healthcare organizations. Thus, patients throughout the world will potentially benefit from the recent investments in the VA system. 

Articles

INTRODUCTION

VA Health Services Research: Lessons for the World's Healthcare Organizations

The Veterans Health Administration is the United States' largestintegrated health system. Once disparaged as a bureaucracy providingmediocre care, the Department of Veterans Affairs (VA) reinventeditself during the past decade through a policy shiftmandating structural and organizational change, rationalization ofresource allocation, explicit measurement and accountability forquality and value, and development of an information infrastructuresupporting the needs of patients, clinicians, and administrators.Today, the VA is recognized for leadership in clinicalinformatics and performance improvement, cares for more patientswith proportionally fewer resources, and sets national benchmarksin patient satisfaction and for 18 indicators of quality in diseaseprevention and treatment.

(Am J Manag Care. 2004;10(part 2):828-836)

The Veterans Health Administration (VHA), oneof three administrations within the Departmentof Veterans Affairs (VA), is the largest integratedhealth system in the United States. Suffering deservedlyor not during the 1980s and early 1990s from a tarnishedreputation of bureaucracy, inefficiency, andmediocre care, the VA sought to reinvent itself beginningin 1995 as a model system characterized bypatient-centered, high-quality, high-value healthcare.This reinvention mandated structural and organizationalchanges, rationalization of resource allocation,measurement and active management of quality andvalue (and clear accountability for quality and value),and an information infrastructure that would increasinglysupport the needs of patients, clinicians, andadministrators.

Although predating the US Institute of Medicine'srecent recommendations for a more ideal health system,

 the VA's improvement using strategies remarkablysimilar to those enunciated in the report providesincreasing evidence for the utility of the recommendationsin closing the "quality chasm." Through adoptionof evidence-based practices, proactive approaches topatient safety, and use of advanced technologies (eg, afully deployed electronic health record, bar-coded medicationadministration), the VA's success in improvingquality, safety, and value have allowed it to emerge asan increasingly recognized leader in healthcare.

HISTORY OF THE VETERANS HEALTHADMINISTRATION

Origins of the Veterans Health Administration

Although health and social support for aged or disabledsoldiers has existed in the United States sinceColonial times, the spectrum of national programs forAmerican veterans was consolidated with the establishmentof the Veterans Administration in 1930. Resourcesfor social services expanded rapidly followingWorld War II with the Servicemen's Readjustment Actof 1944 (better known as the GI Bill of Rights), and ahospital system that specialized in meeting the rehabilitativeneeds of more than 1 million returning troopswho had experienced physical and emotional traumaexpanded and evolved. The Veterans Administrationwas elevated to Cabinet status and became theDepartment of Veterans Affairs in 1989, with financialsupport programs such as pensions administered underthe aegis of the Veterans Benefits Administration andhealth services consolidated in the Veterans HealthAdministration (VHA). The Secretary of Veterans Affairsdirects the activities of the department, and the UnderSecretary for Health serves as the chief executive officerof VHA.

Structural and Organizational TransformationSince 1995

Until the mid-1990s, the VA operated largely as a hospitalsystem providing general medical and surgical services,specialized care in mental health and spinal cordinjury, and long-term care throughdirectly operated or indirectlysupported facilities. Medical centersand other facilities operatedrelatively independently of eachother, even competitively duplicatingservices. Anachronisticlaws required virtually all healthcareservices to be provided inhospitals, counter to the movementof care into the ambulatoryenvironment. In 1996, theVeterans Health Care EligibilityReform Act enabled the system tobe restructured "from a hospitalsystem to a health care system,"as directed by then UnderSecretary for Health, Kenneth W.Kizer, MD. The structural changeswere predicated on the assumptionthat providing the most effective,efficient care required coordination among facilitiesand synergy of resources, including that care be providedin the most appropriate environments.

The structural transformation was characterized bycreation of 22 geographically defined Veterans IntegratedService Networks (VISNs) in 1995. In additionto redirecting resources allocations to follow the geographicallyshifting veteran population, resources wereallocated to each network rather than to each facility.Within VISNs, this created financial incentives for coordinationof care and resources among previously competingfacilities. Although the portfolio of medicalcenters still exists today, medical centers now belong to1 of 21 VISNs (2 VISNs were recently merged), as docommunity-based outpatient clinics, which increasedfrom fewer than 200 in 1996 to more than 850 today,and more than 300 other long-term care facilities, domiciliaries,veterans' counseling centers, and home-careprograms. This structural transformation facilitatedshifting care from the hospital to ambulatory-care facilitiesand the home environment, allowing a reductionof authorized hospital and long-term care beds fromapproximately 92 000 to 53 000, with a concomitantdecrease in hospitalizations and an increase in ambulatory-care visits and home care services (

It should be noted that from 1996 to 2003, the numberof veterans treated annually increased by 75% fromapproximately 2.8 to 4.9 million. The appropriatedbudget to care for those increasing numbers of patientsremained flat at $19 billion from 1995 to 1999, and hasincreased to approximately $25 billion for fiscal year2003, or about 32% cumulatively over 6 years.

INTRODUCTION OF ACCOUNTABILITYFOR PERFORMANCE

Quality and Value as Organizing Strategies

Because of its public nature, the VA is perhaps themost scrutinized health system in the United States. Inthe late 1980s and early 1990s, the VA was beset byincreasing public anxiety about the quality of care. A1992 movie titled 

, made in Hollywood byOrion Pictures, parodied the VA as a hapless and dangerousbureaucracy, and the challenging US economy atthe close of the 1980s and opening of the next decaderaised concern about the economic viability of the system.The broader American healthcare context saw theincreasing emergence of managed care, offering the hopeof improved quality and the promise of a mechanism forcontrolling healthcare cost inflation. At the extremes, atension emerged between the desire to maintain a systemdedicated to veterans' health needs and voucheringout (contracting for) care for presumably greater qualityand efficiency. It was increasingly apparent that if theVA were to survive, it would need to prove its value toCongress and its quality to veterans themselves.

Vision for Change andPrescription for Change

, published in 1995 and 1996,respectively, outlined the challenges facing the VA andserved as the strategic outline for organizationalrestructuring and a new strategy for systematizing qualityand value.

The VA sought to operationalize value in terms of therelationship of outputs to inputs, in contrast to the moresimplistic, prevalent, and less meaningful concept ofunit cost. Expanding on the definition of "value" as therelationship of quality to cost,

 the VA objectified qualityas a constellation of outcomes of interest to veteransand stakeholders that were known as the valuedomains. The value domains now include 6 dimensionsof effectiveness that the VA holds itself accountable forthrough performance measurement. The first 5 can beconstrued as the outputs of the system, and includetechnical quality of care, access to services, patientfunctional status, patient satisfaction, and communityhealth. The inputs are the resources, ultimately financial,that the VA works with. The sixth value domain,cost-effectiveness, emerges as the ratio of outputs toinputs, a relationship sometimes referred to as the"value equation."

The objectification of quality and value serves as thebasis for internal performance improvement efforts, andboth internal and external accountability. Measures aredetermined in each of the value domains. In the arenaof quality, performance measures largely are derivedfrom rates of providing evidence-based healthcare services(processes and intermediate outcomes) in theareas of preventive health, disease treatment, and palliation.Novel composite measures, known as the preventionindex (see 

), chronic disease index,and palliative care index, serve to focus providerattention on these areas and summarize performance.Examples of measure topics in each domain aredescribed in 

Accountability Through a NationalPerformance Contract

The VA operates with both formal external and internalaccountability for performance. As part of theGovernment Performance and Results Act, major federalagencies now engage in a performanceagreement with the White House,administered through the Office ofManagement and Budget. Internally,since 1995, an annual performance contracthas been in place between theUnder Secretary for Health and seniornetwork (VISN) leaders. The content ofthis performance contract has beenconstructed around the value domains,now known as the "strategic goal areas."Measures are developed by using anevidence-based approach that extendsthe principles of evidence-based medicineto the administrative arena, a conceptthat might be termed"evidence-based quality management."Thus, the VA's accountability and improvement systemis both rigorous and data intensive. Operating in parallelwith the Performance Measurement Program is theNational Advisory Council for Clinical PracticeGuidelines. In the clinical arena, the VA has the strategicadvantage of affiliation with 107 academic healthsystems and the Department of Defense Military HealthSystem; and in conjunction with its own directlyemployed professional work force, expertise in specificclinical disciplines and evidence synthesis is robust.Many professionals are involved in VA Health ServicesResearch and Development Service as well as the VA's 8Quality Enhancement Research Initiatives (or QUERIprograms), each of which focus on either highly prevalentdiseases such as diabetes or heart failure, or onconditions conferring unique vulnerability such as mentalillness and spinal cord injury. The collective effortsserve to systematically translate the best evidence intorecommendations for best practice.

 Although moreanalysis is required to determine what aspects of thetranslational process may contribute to performanceimprovement, it has been suggested that the process ofengaging health systems in this critical analysis of theevidence and outcomes creates awareness of performancegaps and defensible approaches to improvement.

The VA's clinical performance measures are generallyconstructed to determine compliance with evidence-basedclinical guidelines or other recommendations inthe areas of preventive medicine, disease treatment,and palliative care. In the remaining domains of satisfaction,access, function, community health, and cost-effectiveness,experts similarly reconcile data toidentify and support areas for improvement. The guidingprinciple for determining which measures areselected for inclusion in the performance contract is tochoose measures which are ambitious and "transformative,"helping the VA and itscare of veterans to meaningfullymove forward.

The performance contract iscreated as a collaborativeprocess involving central managementand field leaders. ThePerformance MeasurementWork Group is both co-chairedby and comprised of central andfield leaders, and it includesboth clinicians and administrators.The group serves as amechanism for vetting and prioritizingmeasures for inclusionin a performance contract recommendedto the UnderSecretary for Health. Thus, theultimate contract establishedbetween the Under Secretaryand VISN leaders, and then cascadedto clinicians and managersthroughout the system, isa collaborative product, whichis thought to reduce the traditional"us-them" tensionbetween central and field leadershipor between administratorsand clinicians.

Results of the performancecontract form the basis for quarterly managementreviews. Although extremely modest managementincentives exist, the performance results are broadlydistributed within the VA and are known to key stakeholderssuch as Congress, veteran advocacy groups, andthe Office of Management and Budget. The performancedata that result are published in hard copy quarterlyand annually and, since 2002, are increasingly availableas they accrue in real time on Intranet sites.

INFORMATION SYSTEMS TO MONITORAND SUPPORT PERFORMANCE

Effective information systems are the prerequisitefor the effective delivery of services that maximize valuein each of the domains, as well as the source of data foroperation of the Performance Management Program.The VA's clinical information system is remarkably welldesigned to support patient care; however, the system'scapacity for national "roll-up" of all discrete data elementsdesired is currently limited. So, although mostclinical data and patient records are fully electronic, theVA has invested in an audit program to assess clinicalperformance. Using VA performance criteria, audits areperformed by an independent external contractor underthe External Peer Review Program (EPRP). This programprovides data to support measurement primarilyin the more clinical domains of quality and function, asdescribed elsewhere.

 The VA's new health data repositorywill markedly expand the capacity for automatedaggregation of national performance data.

In the domain of satisfaction, traditional event-drivensurveys of satisfaction with ambulatory-care visits,hospitalizations, or other services (eg, prosthetics,spinal cord injury, pharmacy) have been used. However,recognizing that satisfaction is only 1 componentof the patient experience, a new omnibus Survey ofHealth Experiences of Patients has been introduced toacquire data about general healthcare experiences (eg,waiting times) and satisfaction, patient functional status(the veterans SF-12

), and health risk behaviors (eg,nutrition, exercise, tobacco) that link with clinicalinformation acquired through external peer review.These pooled data more richly support improvement,program planning, policy development, and (with allidentifying information redacted) health servicesresearch. Corporate data from scheduling and fiscal systems(and some survey information) are used to supportmeasurement and improvement in the domains ofaccess, cost, and community health.

The VA's approach to both improvement of healthcaredelivery and improvement of information systemsis reflected well in models identifying the convergenceof patients, providers, and the health system for optimaloutcomes, as articulated by Glasgow and colleagues.

These models suggest that the most productive interactionsoccur when prepared, proactive providers andinformed, activated patients interact in the context of asupportive, informed health system.

 The VA's clinicalinformation system provides support for improvementfor the system, for providers, and for patients.Standardized data elements can be aggregated to assessperformance on a clinical measure at the team, clinic,facility, network, or system level. These same data elementsserve as the basis for implementing clinicalreminders used to support immediate feedback andimprovement for care providers. Finally, these dataincreasingly will serve as the basis for online healthassessment and education for patients and caregivers,who ideally will use that knowledge for more effectivemanagement of their health needs.

The Electronic Health Record forClinical Data Management

The VA has had automated information systemsproviding extensive clinical and administrative capabilitiesin all of its medical facilities since 1985, when itsdecentralized hospital computer program began operating.The veterans health information systems andtechnology architecture (VistA), which supports ambulatory,inpatient, and long-term care, provided significantenhancements to the original system with therelease of the computerized patient record system forclinicians in 1997. The computerized patient recordsystem (CPRS) was developed to provide a single, highlygraphical interface for healthcare providers to reviewand update a patient's medical record and to placeorders for various items including medications, procedures,x-rays and imaging, patient care nursing orders,diets, and laboratory tests. The computerized patientrecord system is flexible enough to be implemented ina wide variety of settings, both inpatient and outpatient,ranging from home and long-term care to operatingrooms and intensive-care units. It serves a broadrange of healthcare workers, and provides a consistent,event-driven, Windows-style interface across functionsand locations.

The computerized patient record system organizesand presents all relevant patient data in a way thatdirectly supports clinical decision making. Its comprehensivecover sheet displays timely, patient-centricinformation including active problems, allergies, currentmedications, recent laboratory results, vital signs,hospitalization, and outpatient clinic history. Thisinformation is displayed immediately when a patient isselected and provides an accurate overview of thepatient's current status before any clinical interventionsare ordered.

Today, the CPRS is fully operational at all medicalcenters and most other VA sites of care. VistA Imaging,which provides a multimedia, online patient record thatintegrates traditional medical chart information withmedical images of all kinds (eg, x-rays, pathology slides,video views, scanned documents, cardiology examresults, dental images, endoscopies) is also now operationalat VA medical centers (

Electronic Health Information to SupportPerformance Improvement

Beyond serving as a complete electronic healthrecord, other capabilities in the CPRS support performanceimprovement, including computerized providerorder entry, critical alerts, remote data view toaccess health information from other VA facilities, anda clinical reminder system to provide real-time decisionsupport.

Computerized provider order entry has been shownto decrease rates of adverse drug events.

 The VA'scomputerized provider order entry, with real-timeorder-checking system, alerts clinicians during theordering session that a possible problem could exist ifthe order is processed (eg, drug—drug interactions,duplicate laboratory values). Since implementation,order checking has required some reengineering toensure that attention to important alerts is not diminishedby frequent, trivial messages. Currently, 94% of allpharmacy orders throughout the VA are electronicallyentered directly by the prescriber.

Other features of CPRS include a notification systemthat immediately alerts clinicians about clinically significantevents such as abnormal test results, a strategythat helps prevent errors by requiring an activeresponse for critical information.

 A patient postingsystem, displayed on every CPRS screen, alerts cliniciansto issues related to the patient, including crisisnotes, special warnings, adverse reactions, and advancedirectives. The remote data view functionality allowsclinicians to view a veteran's medical information fromanother VA facility or from Department of Defense medicaltreatment facilities to ensure the clinician hasaccess to all clinically relevantdata.

The clinical reminder systemallows caregivers to track andimprove preventive healthcareand disease treatment forpatients and to ensure that timelyclinical interventions are initiated.The clinical decisionsupport it provides is contextsensitive (eg, it recognizes thatthe patient has a particular diagnosissuch as diabetes), and timesensitive (eg, 12 months haveelapsed since the service, such asan influenza vaccination, was lastprovided). The clinical remindersystem is now the VA's preferredmechanism for implementingclinical practice guidelines, andfacilitates linking the evidencewith the real-time clinicalreminder, with the action (egpneumococcal vaccination inelderly or chronically ill patients),and with the automaticallygenerated documentation aswell as with a trail of standardized performance data(

A more recent addition to CPRS provides a multipatientview for follow-up on clinical interventions. A listof patients can be generated based on abnormal testresults, or based on a clinic schedule, inpatient wardcensus, or team roster. Using this new care managementsoftware, clinicians can manage a group ofpatients—seeing and taking action on test results, signingnotes, or generating new tasks.

NEW DEVELOPMENTS TO SUPPORTCONTINUING TRANSFORMATION

The VA is currently transforming the architectureunderlying its health information systems to more effectivelyserve the needs of patients, providers, and thehealth system. The new architectural strategy, knownas Health

Vet, fully integrates a health data repositorywith registration systems, provider systems, managementand financial systems, and information and educationsystems.

The health data repository creates a true longitudinalhealthcare record including data from VA and non-VAsources, supporting research and population analyses,improving data quality and security, and facilitatingpatient access to data and health information. With anemphasis on "eHealth," a secure patient portal known asMy Health

Vet provides patients access to their personalhealth record, online health assessment tools, mechanismsfor prescription refills and making appointments,and access to high-quality consumer health information.The consumer information is evidence based, consistentwith clinical practice guideline recommendations (madeproactive through clinical reminders), and ideally,inspires the patient to act. Although deployed nationally,a major barrier to the complete penetration of CPRSand Health

Vet extensions at every VA site is the challengeof an inadequate high-speed telecommunicationsinfrastructure in more remote and rural parts of thecountry. Otherwise, My Health

Vet is available to veteranswherever Internet access is possible.

Safety and effectiveness are fundamental expectationsfor healthcare services, but do not independentlyconstitute patient-centered care. VHA aspires to providehealthcare that is safe, effective, and meaningfullypatient centered. Such care is organized so that thelocus of control is the patient and the experience of careis seamless across environments. Furthermore, theenvironment of care now extends beyond the provider-centricdomains of the hospital and clinic to thepatient's home, work place, and community.

Patient-centered care coordination extends the focusof disease management to better and more efficientlyintegrate every patient's disease-specific and generalhealth needs with the resources of the health system. Apatient with diabetes and heart failure is no longer managedwith separate but overlapping services for each disease;instead, care coordination seeks to rationalize andunify the care approach. In an environment of constrainedresources, care coordination also seeks toensure that healthcare is provided when the patientneeds it, and is not determined by arbitrary, provider-basedrules.

COMMENTARY

The Veterans Health Administration: Quality, Value, Accountability, and Information as Transforming Strategies for Patient-Centered Care

Up to 25% of adults in the United States have difficulty witheveryday reading tasks. As patients, adults with low literacy maynot be able to complete many self-administered written questionnaires,which often are used to obtain information from patientsand to gauge their satisfaction with care. We developed an illustratedversion of a patient satisfaction instrument used by theVeterans Health Administration. This paper describes the extensivedesign process used to develop, pilot-test, and revise this 63-itemillustrated instrument. A total of 438 patients were interviewed overa 1-year period to obtain feedback on illustrations, with at least 15people viewing and commenting on each picture and revision. Allpictures were revised, with the majority revised at least 4 times. Wereport on this iterative design process as well as on lessons welearned in illustrating questions for low-literacy populations.

(Am J Manag Care. 2004;10(part 2):853-860)

Many adults in the United States have difficultywith everyday reading tasks. In 1992, theNational Adult Literacy Survey found that 40-44 million people (nearly one quarter of adults) scoredin the lowest of 5 levels in reading, writing, and numericalskills.

 Most people who score at this level cannotread and write well enough to meet the needs of everydayliving and working. Another 50 million demonstratedskills in the next higher level—meaning that nearlyhalf of the US population lacks the reading skills necessaryto function well in an increasingly complex society.

Recently, researchers have documented low levels of"health literacy," defined as the degree to which individualshave the capacity to obtain, process, and understandbasic information and services needed to makeappropriate decisions regarding their health.

 Contributingto poor health literacy is the consistent findingthat the literacy demands of most printed health materialsexceed the reading abilities of the averageAmerican adult.

 These materials include consentforms, drug package inserts, emergency department andhospital discharge instructions, and patient educationbrochures.

Healthcare providers and organizations rely onprinted material to convey information 

 patients, aswell as to gather information 

 them. The latter taskusually involves completing written forms or questionnairesthat provide, for example, critical details aboutmedical history or satisfaction with care, a recognizedindicator of quality.

 In addition to facing barriers tousing and navigating the healthcare system, patientswith low literacy also may have difficulty responding tothe system and being full participants in their care. Arecent study documented a link between low literacyand participants' inability to accurately complete a writtenhealth questionnaire.

Simplifying the language of written materials canimprove their comprehensibility,

 although thisstrategy mostly benefits higher-level readers.

 Evenmaterials scored at a fifth-grade reading level may notbe understood by about one quarter to one half of manypatient populations.

 To improve readability even further,literacy experts recommend visual strategies suchas limiting the number of concepts per page, using headersto break up text, using typefaces of 12 points orlarger, and illustrating the text.

Health educators have investigated the use of pictorialrepresentations such as photo essays,

 and illustrations to improve the readability ofwritten material. Some studies have found that illustrationsand graphics improve the comprehensibility ofhealth materials,

These findings are consistent with educational researchand theory, which indicates that the effectiveness ofillustrations seems to vary with the ability of the readers,the type of pictures, and the difficulty of the text.

Little research exists on whether illustrating writtenquestionnaires improves response rates and accessibilityfor low-literacy populations. In this project, wesought to develop and test an illustrated version of apatient satisfaction instrument. This task addresses akey element of the national healthcare quality agenda,which defines quality care as effective, safe, timely, andpatient centered.

 The new National HealthcareQuality Report emphasizes the need to measure thepatient-centeredness of care using instruments thatelicit patient perceptions of care.

 Patients with low literacymay be unreachable through conventional text-basedinstruments and also may have differentexperiences and perceptions of their healthcarebecause of their low literacy. This article describes theextensive design process used to produce an illustratedform of the Veterans Health Administration AmbulatoryCare Customer Satisfaction Survey.

A multidisciplinary team (composed of experts inpsychometrics, health services research, medicine, andliteracy) and several trained patient interviewers used acombination of qualitative and quantitative methods todevelop an illustrated version of the Veterans HealthAdministration's Ambulatory Care Customer SatisfactionSurvey. The Department of Veterans Affairs (VA)Performance Analysis Center for Excellence conductsthis survey using an instrument developed by the PickerInstitute in Boston.

 The questionnaire has 62 itemsand is designed for written self-administration. It assessespatient satisfaction with recent ambulatory-careencounters (visits in the last 2 months and specificallythe most recent visit) along 9 dimensions: access, continuityof care, courtesy, emotional support, patient education,patient preferences, pharmacy, specialist care,and visit coordination of care. There also are scores foroverall satisfaction and overall coordination of care. Theitems require different types of response formats such asrating scales (eg, poor, fair, good, very good, excellent),agreement (eg, yes, completely; yes, somewhat; no); andreporting experience (eg, same day, 1 to 14 days, 15 to30 days, 61 to 120 days, and more than 120 days).Because measurement properties are sample dependent,there is no single set of summary performance measures.However, in a sample of veterans, the internal consistencyreliabilities among the subscales ranged from .59to .85 (JAS, unpublished data, 2002). We added 1 itemto address adequacy of parking, as focus group participantshad identified parking as particularly troublesome.The reading level of the VA instrument has not been formallyassessed, though item responses are reviewedeach year, and items are edited if nonresponse patternsindicate a problem with the item. We found that thequestions on the VA instrument had a Flesch ReadingEase score of 76.6 (indicating "easy" or grade-schoollevel) and a Flesch-Kincaid Grade level of score of 6when analyzed by a computerized program.

The team began by auditioning graphic artists toillustrate the questions. Two artists were chosen toillustrate a sample of questions. The team worked withthe artists to brainstorm ideas for depicting the conceptsin the survey. Ideas were shared between the 2artists, who used different styles and often developeddifferent visual concepts to illustrate items.

To begin pretesting, the team conducted 4 focusgroups, 2 at the VA Medical Center in Philadelphia and 2at a nearby academic medical center hospital. A total ofapproximately 200 patients scheduled for visits on designateddays were sent letters inviting them to participatein a focus group during the lunch hour, either before orafter their clinic visit. About a week after the letters wentout, a research assistant called patients to ask them ifthey would like to participate. Once the target numberper group was reached, no more patients were called.Twelve patients were scheduled per group, and a total of31 participated. Participants ranged in age from 21 to 76years of age (mean age of 57 years). Education levels werehigh school diploma or less (39%), some college (39%),and college degree or more (19%). Fifty-two percent ofthe participants were African American, and 32% werewomen. At the VA sites, 20 of 21 were men. At the academicmedical center, 9 of 10 were women. Literacy levelwas not assessed in this phase of the study.

After the moderator explained the nature of the surveyand the task of creating an instrument that was easierto read, each group was shown illustrations withoutand then with the written questions. The moderatorasked participants a series of semistructured questionsabout how they would interpret the drawings and whichstyle of drawings they preferred.

After these focus groups, the team conducted theremainder of the pilot testing through one-on-one interviewswith patients drawn from the same 4 clinic sites.These interviews were preferable to focus groupsbecause they were more appropriate for the task(assessing individual understanding) and more efficient(many more patients could be interviewed). Patientswere recruited in an area that served as a waiting roomfor both the primary care clinics and the outpatientpharmacy. Interviewers approached waiting patients,introduced the study, and collected demographic information(sex, age, education, and race). Approximately85% of the patients approached agreed to participate inthe study. There were no differences in demographicsbetween those who agreed to participate and those whodid not. Interviews were conducted in a corner of thewaiting area where clinic personnel provided a tableand chairs, slightly removed from the main waiting areabut still allowing the patient to hear his/her name ifcalled for the appointment. Patients were eligible if theywere at least 18 years old and were current patients atthe outpatient clinics. All aspects of the study (focusgroups and interviews) were approved by the institutionalreview boards at the Philadelphia Veterans AffairsMedical Center and the University of Pennsylvania. Oralconsent was obtained. Both institutional review boardswaived written consent.

In the interview, patients were asked to review a setof 4-6 pictures. Interviewers worked in pairs. One conductedthe interview, while another took notes. Theinterview started with a sample picture (held constantacross all interviews). The interviewer showed thepatient a picture without any text and asked, "What doyou think is going on in this picture?" Then the interviewershowed the patient the picture with the text andasked, "What question do you think we are trying to askhere?" Lastly, the interviewer asked, "How would youanswer this question?" The interviewer rated thepatients on their understanding of the picture (withouttext) and their understanding of the question (picturesand text) using a 3-point scale (1 = yes, 2 = partially, 3 =no). After each interview, the interviewer and note takerreviewed the standardized rating form that documentedthe patient's responses and the note taker's preliminaryjudgments, and came to consensus on the ratings for thepatients' understanding of the picture and the question.The team reviewed free-text comments about aspects ofthe pictures that patients did not understand, and usedthis feedback to revise many of the pictures.

A total of 438 patients were recruited on-site andinterviewed over a 1-year period. Their mean age was51 years, and 72% were men. Approximately 51% of patientsinterviewed had a highschool diploma or less. Participantsidentified themselves asAfrican American (59%), white(23%), or Hispanic or other (18%).

Modifications to illustrationswere made iteratively, based primarilyon patients' feedback.However, the study team itselfbecame more active in critiquingillustrations as it gained expertiseabout potential patient comprehensionproblems. All 63 pictureswere revised, many of them multipletimes. At least 15 peopleviewed and commented on each ofthe 63 illustrated items. The revised pictures also werereviewed by at least 15 people. The final illustratedbooklet contained 22 pictures that had a rating of"understood," 39 pictures that had a rating of "partiallyunderstood," and 2 pictures that had a rating of "notunderstood." The 

 summarizes the number oftimes items were revised. For most illustrations, thefinal revision was to make text fonts and details such ascollars on shirts consistent with all other illustrations.As shown, 2 items could not be illustrated effectively;however, these illustrations were retained to keep thebooklet consistently illustrated, and to remain true tothe original questionnaire.

The multiple rounds of pretesting of the items generatedvaluable information on the comprehension andacceptability of the draft illustrations. We report belowon some of the lessons learned.

Artistic style matters—people prefer realistic illustrationsand familiar circumstances.

 The two artistshad different artistic and communicative styles. 

 show their initial attempts to illustrate question17: "Did you have trouble understanding theprovider because of a language problem?" Pretestingrevealed that patients preferred the artistic style ofFigure 1B, because it was more realistic and lessabstract. However, more patients understood the questionin Figure 1A, misinterpreting the various symbolsin the bubble in Figure 1B as curses. 

 showsthe final version, which blends successful elements ofthe previous drawings such as saying hello, using thelight bulb as a symbol of understanding, and using thequestion mark as a symbol of confusion. In general, thedrawings worked better when they portrayed everydaysituations. It might be noted, however, that preferencefor a style was not necessarily related to the comprehensibility of the message conveyed with that style, justas someone might find typewritten text very readable,but calligraphy more attractive.

Using multiple illustrationsfor each question-and-answeroption creates visual clutterand obscures meaning.

 Theteam initially faced decisionsabout what to illustrate in thequestionnaire—the questions(stems), the answers (responseoptions), or both. The strategyselected was to pick the mainconcepts in the stem and usethem to support answer options.For example, 

 showsmultiple concepts from the stemfor question 47: "Did you knowwhom to ask when you hadquestions about your healthcare?" The concepts chosen forillustration were "questions,""healthcare (eg, medications,immunizations, electrocardiograms),"and "whom to ask." Inaddition, we tried to portray theprocess of having a question(top frames) and knowingwhom to ask (bottom frames).

 shows the final version.Two concepts wereretained ("questions" and"whom to ask") in a greatly simplifiedformat. In the end, mostillustrations focused on just 1concept.

The group also needed todecide whether to illustrate allanswer options, especiallywhen the answers were scaled.

 shows initial drawingsillustrating the 3 optionsfor question 33: "How wellorganized was the clinic?"Pretesting revealed that illustratingjust the end points(rather than every option) ledto better patient understandingand also reduced patient confusionby eliminating visual clutter.We also experimented witha single illustration conveyingthe general theme, rather than2 illustrations conveying the scale end points. 

 shows the final drawing illustrating the end points,with an arrow connoting the scale.

The group initiallytried to eliminateany reliance onreading ability byillustrating as manyaspects of the questionsand answersas possible, and notusing any words.However, thisapproach resultedin too many picturesand too muchdetail for thepatients to follow.As the pictures progressed,the artistsremoved extraneousdetails and simplifiedtheirdrawings. Forexample, 

shows the initialdrawings for question27: "Did youspend as much timewith your provideras you wanted?"Patients did notunderstand the useof split frames to communicate the beginning and end ofa visit; further, the details of the provider's office (eg,door, diploma, clock on the wall, examination table)obscured the message. 

 shows the drawingsafter multiple revisions. The final version features simple,uncluttered cues for "provider" (a stethoscope),"time" (looking at a watch), and meeting expectations(same simple text in thought bubble, with patient andprovider smiling). In addition to simplicity, we foundthat patients' interpretations relied heavily on the characters'facial expressions. They quickly keyed in on theexpressions and made appropriate interpretations of the"good" and "bad" situations.

 Initially, the team usedno text in the illustrations, in an attempt to develop aquestionnaire that could be understood by patientswith practically no reading ability. However, we soonfound that use of simple words and repeating key wordsfrom questions actually supported understanding. Thefinal illustrations shown in Figures 1C and 4B use textsuccessfully.

Mathematical symbols and graphics with numbersmay not work.

 Many of the initial drawings sought tocommunicate aspects of time through clocks and calendars,and used mathematical symbols for "same as" and"less than." Figure 4A shows initial attempts to useclocks to illustrate whether time expectations were met.Patient interviews revealed that clocks, as symbolic representationsof time, did not work for our audience. Thegroup experimented with the use of arrows and shadingon clocks, but understanding was not improved. AsFigure 4B shows, in the final version, the clocks wereeliminated, and words, gestures, and facial expressionswere used to communicate satisfaction or dissatisfactionwith the time spent.

In interpreting illustrations, patients bring theirstereotypes with them.

 The team made a conscious effortto use both women and men and people of different racesand ethnicities in the roles of patients, staff, andproviders. Early on we learned that some gender-basedassumptions hindered understanding of the pictures. Forexample, patients tended to identify women as nurses,even with lab coats. We debated butrejected the idea of accepting thisstereotype to improve understanding(by portraying all physicians as men).Instead, we chose other visual clues(eg, lengthening the coats of all physicians,adding stethoscopes for allproviders) to distinguish providersfrom clinic staff, and to delineatepatients and providers. Unlike gender,race- and ethnicity-basedassumptions did not affect understandingof the illustrations in thisethnically diverse patient population.Patients rarely mentioned race whencommenting on the illustrations, andsaid it was not an issue when askeddirectly about racial portrayals.

Consistent visual cues help getthe point across.

 Using the same pictureor symbol in several illustrationshelps give patients a sense of themeaning of a picture, even if they didnot understand its use initially.Examples of standard visual cuesinclude a light bulb (for understanding),question marks (for confusion),scribbles (for incomprehensible language),test tubes (for tests), long labcoats (for physicians), and a stethoscope(for a healthcare provider).Once pilot testing provided evidencethat these standard cues were understandable,the artists revised illustrationsso that these cues were usedconsistently.

Some concepts remain difficult totranslate well into pictures andgraphics.

 Despite applying all of thelessons learned regarding simplificationand judicious use of words, someconcepts remained difficult to convey.It became apparent throughpretesting that "specialist" could notbe successfully communicated withoutsome words. 

 shows initialillustrations of the question 41:"Overall, how would you rate thequality of your most recent specialistvisit?" Using the symbol of a heartand a caduceus did not allow patientsto identify the provider as a specialist.

 shows the final versionof the illustration, in which thespecialist is labeled.

One of the most challengingconcepts to portray was theanswer option "not applicable,"which was available for 8 questions.Many attempts weremade to illustrate that optionfor each question, or to developa symbol for "not applicable"that could be used each time.Figure 5A shows an attempt toillustrate this option by usingthe universal circle and slashsymbol. Patients did not understandthis symbol in this context,nor did they understandother illustrations for the option.In the end, as indicated in Figure5B, the option was "illustrated"by repeating the words in awhite box with a larger font.

Increasingly, patient satisfactionis viewed as a criterion bywhich the quality of healthcareservices can be measured.However, most evaluations ofpatient satisfaction rely on self-administered writtenquestionnaires, which may lie beyond the patient's abilityto complete. Because patients with low health literacyreport poorer health status and less use of preventive services

and may face greater barriers to accessing and navigatingthe healthcare system, it is especially important todevelop instruments that can reach this population.

The value of illustrations, especially in questionnaires,remains unknown. (The instrument currently beingevaluated is a 24-page booklet stapled in the middle, witha finished size of 8&#189; by 11, a large (14-point) serif font,2 questions per page, and generous use of white space,printed in black and white to make it easy to reproduce.)A recent study found that an illustrated version of a 10-item adult Dermatology Life Quality Index was superiorto a text-only version in terms of patient preference andease of use, but equivalence between the versions couldnot be demonstrated.

 Other investigators have developedillustrated instruments for young children, mostnotably to assess psychiatric symptoms, but they areadministered by the interviewer.

 For a target audienceof adults, however, others have noted the importanceof a rigorous design process for the success ofpictures as a communication aid for patients and families.

 To produce clear, culturally acceptable pictures,it is particularly important to collaborate with the targetpopulation.

Many of these findings confirm the advice of literacyexperts in terms of using simple drawings, familiarimages, and realistic illustrations, rather than cartoon-likeor stick-figure sketches.

 However, this patientpopulation did not voice strong preferences for illustrationsof people of their own race or sex. In general,patients appreciated racial and ethnic diversity in thepictures, but did not require a specific racial compositionacross the pictures.

Some questions remained difficult to illustrate,even after applying all lessons that were learned alongthe way. Our inability to produce illustrations that wereuniformly understood could be due to the complexityof the questions in their written form, or to a more fundamentaldifficulty in graphically representing abstractconcepts. In either case, these issues should be consideredwhen deciding whether to enhance an existinginstrument with illustrations, or to develop a new illustratedinstrument for low-literacy populations.

We translated an existing instrument into pictures andmade the decision to stay true to the original text. Wesuspect our task would have been easier had the originalinstrument been simpler or, better still, had it been developedfrom the start with the idea of producing paralleltext and illustrated versions. Under that circumstance,item development might have been based in part onpotential ease of illustration.

Finally, the failure of clocks, calendars, and mathematicalsymbols in pretesting may reflect the strongcorrelation between prose literacy and numeracy(quantitative communication skills).

 Patients with lowliteracy (as well as some with high literacy) often havedifficulty decoding quantitative symbols and representationssuch as charts and graphs. Our findings pointout the importance of pretesting to ascertain the abilityof the target audience to interpret these conventionsthat, at least on the surface, appear to communicateconcepts without words. In some cases, a picture (orsymbol) is worth far less than 1000 words.

From the Center for Health Equity Research and Promotion, Philadelphia VeteransAffairs Medical Center, Philadelphia, Pa (JW, AA, KR, KK, LM, JM, DAA, JAS); the Divisionof General Internal Medicine, University of Pennsylvania, Philadelphia (JW, AA, KR, JM,DAA, JAS); and the Leonard Davis Institute of Health Economics, University of Pennsylvania(JW, DAA, JAS).

This work was supported by grant PCC-98-0871-1 from the Health Research andDevelopment Service, Department of Veterans Affairs.

Address correspondence to: Janet Weiner, MPH, Leonard Davis Institute ofHealth Economics, 3641 Locust Walk, Philadelphia, PA 19104-6218. E-mail: weinerja@mail.med.upenn.edu.

Adult Literacy in America: A FirstLook at the Findings of the National Adult Literacy Survey.

1. Kirsch IS, Jungeblut A, Jenkins L, Kolstad A.

 US Department ofEducation, National Center for Educational Statistics, National Adult LiteracySurvey. Princeton, NJ: Educational Testing Service; 1993. 

National Library of Medicine Current Bibliographies in Medicine: HealthLiteracy.

METHODS

Designing an Illustrated Patient Satisfaction Instrument for Low-literacy Populations

 To identify patterns of use of computerized clinicalreminders (CCRs) across an integrated healthcare system anddescribe institutional factors associated with their implementation.

 At a national electronic health record (EHR) meeting,we surveyed 261 participants from 104 Veterans Health Administration(VHA) healthcare facilities regarding the number and typesof CCRs available at each facility. Potential explanatory measuresincluded perceived utility and ease of use of CCRs, training andpersonnel support for computer use, EHR functionalities, and performancedata feedback to providers at each facility.

 The number of conditions with CCRs in use at a facilityranged from 1 to 15; most reported implementation of remindersfor 10 of the 15 conditions surveyed. The most commonly implementedCCRs, used in more than 85% of facilities, were for conditionswith VHA national performance measures (eg, tobaccocessation, immunizations, diabetes mellitus). The least commonlyimplemented CCRs were for post-deployment health evaluationand management, medically unexplained symptoms, and erectiledysfunction. Facilities that had implemented greater numbers ofclinical reminders had providers who reported greater ease of useand utility of the reminders (

 VHA facilities vary markedly in their implementationof CCRs. This effect may be partly explained by greater incorporationof clinical reminders for conditions with performancemeasures. Further study is needed to determine how to best implementclinical reminders and the institutional factors important intheir use.

(Am J Manag Care. 2004;10(part 2):878-885)

Computerized clinical reminders (CCRs) havebeen widely publicized as potential tools forchanging behavior

 They have been particularly effective in improvingadherence with preventive care and screeningguidelines,

 However, factors such as workload, time, andperceived reduction of the quality of the provider-patientinteraction may be barriers to effective use ofcomputerized reminders.

The Veterans Health Administration (VHA), theUnited States' largest integrated healthcare delivery system,has invested heavily in the informatics infrastructurenecessary to support CCRs. It has been on theforefront of developing these tools and incorporatingthem into the computerized patient record system(CPRS), which is the VHA's electronic health record(EHR). The VHA developed CPRS, an application usedthroughout VHA facilities that enables clinicians toreview and analyze patient data and supports clinicaldecision making.

Computerized clinical reminders may be developedand distributed at the national, regional, or local level.Although VHA mandates use of a few clinical reminderssuch as assessing hepatitis C risk and possible sexualtrauma during military service, most CCRs have beenlocally initiated.

 After reminders are created, they aretested and activated. Some reminders are applicable toall patients, while others apply to a particular group ofpatients. VHA facilities can generate clinical reminderreports, which provide information, for example, aboutthe number of patients in a clinic with completed clinicalreminders and the number of patients eligible for thereminder.

Although CCR technology is in widespread use inVHA, data regarding CCR patterns of use in VHA andinstitutional factors associated with their use have notbeen available. A description of the different types ofreminders implemented in VHA may help other largeorganizations interested in promoting this technology.Identifying factors influencing widespread implementationof CCRs is a necessary step to promote their dissemination.Hence, our primary objective was to surveyVHA facilities for variations in the numbers and typesof CCRs used across VHA facilities nationally. Oursecondary objective was to identify institutional factors associated with increased implementation of CCRtechnology.

We assessed CCR use across the VHA by distributinga survey instrument to participants at the nationalCamp CPRS meeting that took place in Georgia in May2003. This meeting focused on the VHA's EHR and in2003, included 1304 representatives from 136 of the142 VHA medical facilities participating in the VHA'sExternal Peer Review Program (EPRP). The representatives,who are nominated by their facilities, may be clinicalstaff (eg, physician, nurses, others), administrativepersonnel (eg, chief of staff), or informatics experts.Many attendees are opinion leaders at their facility andhave extensive experience with local CPRS capabilities,either as users or developers of clinical applications forthe EHR and its decision support tools.

We received responses from 261 participants (20%)representing 104 VHA facilities (76%). Nonrespondentswere not tracked because only those who volunteeredwere given the survey instrument. Sixty-five of the 104facilities that responded to our survey had more than1 respondent.

The American Hospital Association database providedadditional background information about VHAfacilities. It provided information about geographicdistribution of participating facilities, teaching statusaffiliation, and size of the facilities.

After discussion with the VHA staff that oversee,develop, or perform research on clinical reminders,including members of the National Clinical PracticeGuideline Council and members of its ad hoc ClinicalReminders Committee, we chose to evaluate facilities'use of 15 different types of CCRs. We selected remindersthat represent a broad range of conditions cliniciansmight encounter in various clinic settings. Somereminders were for conditions that have VHA nationalperformance measures (eg, addressing tobacco cessation),whereas others (eg, low back pain) were for conditionswithout VHA national performance measures.

The remainder of the 77 items in the survey instrumentfocused on institutional factors hypothesized fromour ongoing studies to be important in implementationof clinical guidelines

 or CCRs. The instrumentassessed different forms of computer use (provider education,performance feedback, and clinical support suchas the ability to retrieve radiological images), perceivedutility and ease of use of CCRs, adequacy of computertraining and CCR training, organizational support, hospitalculture/climate, and availability of feedback mechanismsfor modifying CCRs. Responses for this portionof the survey instrument were either dichotomous (yesor no) or measured on a 5-point Likert-type scale.Respondents who answered "don't know" were recodedas "missing" or "no." The questionnaire is available,upon request, from Dr. Doebbeling.

The University of Iowa/Iowa City VA institutionalreview boards approved the study protocol. The datamanagement plan underwent subsequent review andapproval from the institutional review board of the VAGreater Los Angeles Healthcare System.

Outcome measures were obtained from questionnaireresponses. For these measures, we aggregatedindividual responses at the facility level. Although 39facilities (37.5%) had only a single respondent, the otherfacilities had multiple respondents, including 29 facilitieswith 2 respondents, 23 with 3 respondents, 6 with4 respondents, 1 with 5 respondents, 3 with 6 respondents,and 3 with 10 or more respondents. When morethan 1 response per facility was available, we used meanscores for questions that had Likert-like responsescales. For dichotomous variables, we assumed thatrespondents from any given facility, while knowledgeableabout their home facility, would be unlikely toreport seeing or using a clinical reminder that they havenever seen or used. In contrast, it is possible that theymay not have seen or used a clinical reminder that wasavailable at their facility, but a colleague at their facilitymay have reported seeing or using the clinicalreminder. Because our goal was to determine whether aclinical reminder exists or not at a given facility, weweighted "yes" responses more than "no" responses.We reasoned that the union rather than intersection ofpositive responses would more accurately reflect, forexample, the actual number of conditions with clinicalreminders in place at a particular facility. Therefore, wetreated any positive answer among respondents at thefacility as a true-positive facility response. For example,if there were 3 responses and 2 stated that CPRS wasused for requesting consults and 1 did not (or viceversa), the facility score was based on a "yes" answer.In a sensitivity analysis, we weighted negative responsesmore heavily than positive responses.

The first outcome measure represents which CCRswere available at the facility. The second outcome measure,a "facility clinical reminder score" (minimum possiblescore = 0, maximum possible score = 15), wascreated by summing "yes" responses to questions surveyingwhether a facility had at least 1 clinical reminderfor a particular health condition. This variable representedthe level of implementation of CCR technology inVHA facilities. Although some conditions, such as diabetes,may have more than 1 clinical reminder (eg, glycosylatedhemoglobin, microalbumin/creatinine ratio),we counted each condition only once in our facility clinicalreminder score. We used 

 tests to determinewhether responses to each question differed significantlyfor facilities with complete versus incomplete responses.There were 72 facilities with complete data and 32facilities with incomplete data (missing data or "don'tknow" response) for the questions used to construct thefacility clinical reminder score. The mean facility clinicalreminder score was 9.1 (95% confidence interval[CI]: 8.6, 9.7) for facilities with complete responses versus7.4 (95% CI: 6.3, 8.5) for those with incompleteresponses (a difference of 1.7; 95% CI: 0.7, 2.8). 

presents comparison data for VHA facilities that participatedin Camp CPRS and had respondents to the surveyversus VHA facilities that did not participate in CampCPRS or did not have respondents to the survey. Facilitieswith complete and incomplete responses were similarwith respect to number of acute-care beds, trainee andresident full-time equivalents, academic affiliation, andurban versus rural location (

To identify potential explanatory variables, we developedscales from questionnaire items. The process ofdeveloping scales also served as a method of data reduction.Existing literature and conceptual models, as well asfactor analysis, guided development of the scales.

 Werelied on a priori hypotheses developed from literaturereview, our conceptual framework, and clinical experiencein the identification and labeling of a 5-factor solution.The resulting domains included:

, measuringperceived adequacy of training and personnelsupport (8 items; alpha = .842).

, measuring the number of featuresavailable online to clinicians at the point ofcare and at the facility in general (12 items; alpha = .86).

Clinical reminders utility and ease of use

, measuring availabilityof graphical display of individual and clinic performance(2 items; alpha = .95).

For each scale, higher scores indicated greater perceivedsupport, ease of use/utility, functionality, oravailability (

To meet our primary objective, we used simpledescriptive statistics to present data on the type and distributionof CCRs across VHA. For our secondary objective,bivariate relationships were tested and amultivariate linear regressionmodel was estimated to determinewhich American HospitalAssociation facility characteristicsand domains from our factoranalysis were independentpredictors of having a higherfacility clinical reminder score.The number of facilities withcomplete responses for all variablesincluded in the analysis (n= 62) limited the number ofpotential variables to no morethan 6. Because there weremore than 6 potential variables,only variables with a 

 value of&#8804;.25 in our bivariate analyseswere considered for the multivariateanalysis. Then, we usedstepwise regression to build ourmultivariate model. We used analpha of <.10 for inclusion inthe final model (the level of significancethat was used todefine the minimum acceptable F limit for adding orremoving a variable in the model). All statistical testingwas performed with SAS, version 8 (SAS Institute, Inc,Cary, NC).

Table 1 demonstrates that participating facilitieswere geographically distributed across the UnitedStates. Approximately one half (48%) of the hospitalswere members of the Council of Teaching Hospitals.One third of the hospitals had fewer than 200 acute-carebeds, one third had 200-399 acute-care beds, andone third had 400 or more acute-care beds.Participating facilities were similar to facilities that didnot participate in the conference or respond to the surveyin terms of geographic region (

 summarizes the percent of respondents whowere clinicians, information technologists, and clinicalapplication coordinators. Clinical application coordinators,who are liaisons between informatics and clinicalservices, provide services such as implementing softwarepackages and clinical support systems for VHA'sEHR and customizing national-level VHA packages forlocal VHA healthcare facilities.

 summarizes the number of conditions withCCRs across 104 facilities (the facility clinical reminderscore). Of the 15 conditions surveyed, the number ofconditions with reminders implemented across facilitiesranged from 1 to 15, with a median of 9 (interquartilerange 7, 10). In a sensitivity analysis, weighting negativeresponses more than positive responses, themedian number of conditions with remindersimplemented was 6 (interquartile range 4, 8),but variation across facilities remainedunchanged. 

 summarizes the types ofclinical reminders reported at each facility. Themost commonly implemented reminders acrossfacilities were for conditions with VHA nationalperformance measures, including tobacco cessation(n = 99), immunizations (n = 97), diabetesmellitus (n = 95), hypertension (n = 91), anddyslipidemia (n = 88). In contrast, conditions forwhich reminders were infrequently implementedincluded low back pain (n = 14), erectile dysfunction(n = 8), medically unexplainedsymptoms (n = 7), and post-deployment healthevaluation and management (n = 5).

Most facilities had implemented basic componentsof an EHR: entering patient notes (n =100), requesting consultations (n = 101), orderingradiological procedures (n = 101), ordering laboratorytests (n = 102), and prescribing medications (n =104). Many facilities also provided customization of thereminders (n = 96) or formats tailored to specificpatient populations (n = 97). Respondents consideredclinical reminders generally useful, with an overallmean response of 4.0 on a scale 1 to 5 (1 = not at alland 5 = very great).

 summarizes the institutional structure andprocess variables associated at a bivariate level with thefacility clinical reminder score. Multivariable regression(

) found 2 hospital-level variables associatedwith implementation of clinical reminders; these were"perceived utility and ease of use" and "number of full-timeequivalent trainees and residents" (F statistic =5.78; 

 = .005). Each 5-point increase on the "ease ofuse and utility" scale was associated with a 1-unitincrease in the number of CCRs at the facility (parameterestimate of 0.21; 

November 2004 - Part 2

VA Health Services Research: Lessons for the World's Healthcare Organizations

The Veterans Health Administration: Quality, Value, Accountability, and Information as Transforming Strategies for Patient-Centered Care

Designing an Illustrated Patient Satisfaction Instrument for Low-literacy Populations

Variation in Implementation and Use of Computerized Clinical Reminders in an Integrated Healthcare System

Dual-system Utilization Affects Regional Variation in Prevention Quality Indicators: The Case of Amputations Among Veterans With Diabetes

Problems Due to Medication Costs Among VA and Non-VA Patients With Chronic Illnesses

Assessing the Accuracy of Computerized Medication Histories

Screening for Depression and Suicidality in a VA Primary Care Setting: 2 Items Are Better Than 1 Item

The Relationship of System-Level Quality Improvement With Quality of Depression Care