To gain early experience with a networked system designed to assess a patient’s adherence to oral medication and physiologic metrics in an ambulatory, at-home setting.

This networked system for patient self-management consists of ingestible markers and a wearable, personal monitor. When a marker is ingested, it communicates to a monitor that time-stamps the ingestion and identifies the marker as unique. The monitor also records heart rate and activity. Data from third-party monitoring equipment (eg, sphygmomanometer, weight scale) can be integrated into the system. Collected data are summarized for patient and physician review. Directly observed ingestion (DOI) of placebo tablet markers was used to assess the system’s technical performance. Markers were also coencapsulated with drugs to capture at-home adherence. A performance criterion of ≥95% was set as the objective for system performance.

 A total of 111 subjects ingested 7144 ingestible markers; 3298 were DOIs. The system’s positive detection accuracy and negative detection accuracy in detecting ingested markers were 97.1% and 97.7%, respectively. It differentiated 100% of multiple drugs and doses taken simultaneously by type and by dose. Medication adherence was >85%. The most common adverse effect was mild skin rash from the monitor’s electrodes. No definitive marker-related adverse effects were reported.

 The system appears to be safe and effective in capturing and integrating adherence and physiologic data. Efforts are under way to enhance system functionalities and refine user interfaces. By providing context-rich information, this system may enhance patient-provider collaboration.

	Early clinical experience with a networked system consisting of ingestible markers (integrated with oral medications to capture at-home adherence patterns) and a wearable, personal monitor indicates that this system appears to be safe and effective in capturing, integrating, and presenting medication adherence and physiologic information.

		Compared with directly observed ingestion, the system’s ability to detect ingested markers was 97.1% and 97.7% for positive and negative detection accuracy, respectively.

		Adherence was juxtaposed with physiologic data for a comprehensive report.

		Sharing of context-rich information between patient, providers, and caregivers may enhance collaboration to achieve and sustain patient wellness.

	The World Health Organization has characterized poor adherence to treatment as a worldwide problem affecting developing as well as developed countries. In the United States, it is estimated that one-third to one-half of all patients do not take their medications properly.

 Patients who do not take their medications as prescribed cost the US healthcare system an estimated $290 billion in avoidable medical spending annually.

 The World Health Organization considers adherence to be the single modifiable factor that compromises treatment outcome across diseases. It has called for an evolution of the health systems to provide patient-tailored interventions to address poor adherence and to consider the patient’s family, community, and affiliated organizations to be key contributors to success in improving medication adherence.

	Barriers that patients face in taking their medications have been studied extensively.

 Typical reasons cited by patients for not taking their medications include forgetfulness (30%), other priorities (16%), deliberate decision to omit doses (11%), lack of information (9%), and emotional factors (7%).

 These barriers to adherence oftentimes can be improved upon or overcome by patient efforts independent of providers or other members of the healthcare system. Thus, a necessary and important first step in improving adherence is to provide attention and aid to patients.

	Barriers to adherence are also related to poor interactions and/or communication between the patient, the healthcare providers, and the healthcare system. Traditionally, physicians and designated healthcare workers have played a key role in achieving adherence by providing the rationale for treatment to their patients, and involving family and caregivers whenever possible. However, it is time and resource intensive for providers to assess and influence adherence alone on a timely and ongoing basis. There remains an unmet need for a reliable and broadly utilizable means to accurately assess and manage a patient’s medication adherence.

	An ideal system for improving adherence would not only engage the patients themselves, but also leverage their selected health providers, caregivers, and communal support network. The system would also allow a secure exchange of reliable and robust information among patients and clinical decision and support systems.

 By utilizing the latest medical technology, patients could be empowered in a novel way to promote self-management of their chronic conditions, beginning with better management of their adherence.

	The purpose of this report is to discuss the early clinical experience with a networked system for promoting patient self-management. This system is designed to utilize unique ingestible markers that confirm the ingestion of individual oral medications and doses, to integrate this adherence data with physiologic parameters and wellness metrics, to offer patientdirected sharing of health information with caregivers and providers, and to incorporate individualized behavioral support tools. Some of the above mentioned functionalities have been implemented and evaluated in 4 separate clinical studies of 4 populations; early safety and technical performance results are presented.

	A networked system has been developed to electronically confirm adherence to oral medication, to acquire physiologic metrics periodically and remotely, and to gather and present this information in a comprehensive, centralized manner

). Briefly, upon ingestion, the ingestible marker dissociates from the medication or carrier within the stomach and is activated by stomach fluids. The marker then begins communicating its unique signature to the wearable, personal monitor. This data communication scheme is similar to what a heartbeat does in transmitting an electrocardiogram to the surface of the body. The communication process is unnoticed by, and not detected beyond, the system user. The personal monitor interprets the information from the marker, identifies it as unique, and stores the marker ingestion event to the memory of the monitor. Periodically, the personal monitor transfers the collected data to a mobile phone, which relays the data to a secure server. Once validated at the server level, adherence and physiologic data can be viewed via a Web browser on a computer or a mobile phone. The system components are described in detail below.

 The system utilizes ingestible markers that are incorporated into oral medication to electronically confirm medication adherence. The ingestible marker is considered biocompatible and toxicologically safe. It is 1 mm x 1 mm in size, and it is coated with a copper salt on one side and magnesium on the other. The levels of silicon, copper, magnesium, minerals, and cellulose present in the marker are far below the levels commonly present in the diet. The ingestible marker is also designed to be compatible with the formulation of oral pharmaceuticals. Upon ingestion, the marker is activated by gastric fluids and communicates a unique identifying signature to the personal monitor. The marker is currently designed to communicate for approximately 7 minutes. After that, the communication ceases and the marker becomes completely inactive; it goes through normal gastric transit and is eliminated from the body via fecal elimination.

 The personal monitor serves 2 purposes. First, it records the identifying signatures from the ingestible marker-enabled medications and time-stamps the drug taken, its dose, and its time of ingestion. Second, it has internal markers that are capable of reporting metrics such as heart rate, respirations, activity, body position, and monitor-wearing compliance. The current iteration of the personal monitor is a miniaturized, soft foam, skin-patch device measuring 5 cm x 11 cm x 1 cm. It is capable of recording a variety of physiologic parameters and periodically transmitting stored data to a computing device via Bluetooth telemetry. It adheres to the torso of a subject in a similar fashion to an adhesive bandage.

 In addition to adherence and physiologic data collected by the ingestible markers and the personal monitor, other metrics can be collected and integrated into the networked system using peripheral wireless devices. A wireless sphygmomanometer, weight scale, and base station can be set up at a patient’s home. When a blood pressure or weight measurement is made, it is sent to the base station and subsequently to a centralized, secure data server via a phone or Internet connection. At the server level, data from these peripheral wireless devices are integrated with those from the ingestible markers and the personal monitor, to provide a comprehensive view of a patient’s physiologic and wellness profile. Data from other telemetric devices (eg, glucometer) may also be integrated with the system in this manner.

Mobile Phone, Web-Based Patient Communication.

 Once data are aggregated, they can be disseminated to appropriate stakeholders. Data can be securely viewed using a Web-based platform. Summary reports can be generated and shared with the patients. In the next-generation system, data could be sent to the patients directly, discreetly, and securely via a Web-based or mobile phone platform. With the patient’s permission, the same set or a subset of the gathered data could be shared with the persons they designate, including family members, caregivers, and/or healthcare providers.

	To gather comprehensive information regarding system use, 4 separate small-scale, prospective, observational studies were conducted in 4 populations over a course of 19 months. Each of the 4 studies targeted a specific patient subject population including healthy volunteers and patients with tuberculosis (TBC), heart failure (HF), and hypertension (HTN). Studies were conducted at 5 clinical sites in the United States: 2 were clinical research organizations and 3 were outpatient clinics. Studies were approved by institutional review boards at Chesapeake Research Review, Inc, Western Institutional Review Board, University of North Texas Health Science Center, and Independent Review Consulting, Inc. Studies were routinely monitored by the sponsor according to Good Clinical Practice guidelines.

	Study candidates were recruited through advertisement and referrals. In general, candidates identified for inclusion in the study were those who were willing to adhere to study procedure, had the capacity to provide informed consent, and for women of childbearing age, had a negative urine pregnancy test. Study candidates were excluded if they had acute or chronic gastrointestinal symptoms, terminal illness, known allergies that could preclude safe participation in the study, or current presence of an electronically active implanted medical device (except in the HF population study), or if they were participating or had participated in any clinical trial (drug or device) <30 days prior to study start.

	Since each study targeted a particular subject population with specific procedural requirements, additional inclusion and exclusion criteria were implemented. 

 summarizes these specific subject selection criteria. Study subjects were enrolled after providing written, informed consent per the International Conference on Harmonisation Guideline for Good Clinical Practice.

	Two main objectives were common to all 4 studies. First, the networked system safety, including ingestible marker ingestion and personal monitor wearing, was evaluated. Secpropriate ondly, the technical performance of the networked system was characterized. Namely, the reliability of the system in detecting ingestible marker ingestions was compared with a gold standard: directly observed ingestion.

	In addition to the main objectives described above, a number of corollary aims were unique to each study and subject population. The purpose of including these study aims was to expand the scope of the technical assessment of the networked system. The study aims and the population(s) that these aims were evaluated in were (1) to assess the effect of body mass index (BMI) on ingestible marker detection in all studies, (2) to assess the effect of meal type on ingestible marker detection in healthy users, (3) to characterize the effect of the ingestible marker carrier form factor on marker detection in all studies, (4) to characterize the unsupervised, at-home adherence pattern using marker-enabled medications in HF and HTN patients, and (5) to demonstrate the use of peripheral wireless devices for additional physiologic monitoring in HF and HTN patients.

	To achieve the main objectives described above, several procedures were implemented and repeated in each study. First, a safety assessment was conducted in each protocol by evaluating the frequency and nature of the adverse events (AEs) occurring during the active and the follow-up phases of the study. An AE was defined as “any undesirable medical event occurring in a subject, whether or not the event is considered related to the investigational device.” Study investigators were asked to document and classify all AEs based on severity and relatedness to the study device and procedure.

	Second, investigators periodically conducted direct observation of marker ingestions to assess the system’s accuracy in detecting marker ingestion. While at the clinic, subjects were instructed to ingest 2 to 4 markers in the presence of research personnel, who would observe and log each ingestion. Both functional and some nonfunctional markers were administered, to evaluate the positive detection accuracy and negative detection accuracy of the system, respectively. The system’s identification accuracy—the ability to correctly identify the unique identifier of the ingested device&mdash;was also assessed.

	To address the corollary study aims, specific procedures were conducted. Subjects’ height and weight were recorded at enrollment to assess the effect of BMI on marker detection accuracy. Healthy subjects were given meals and beverages designed by a dietician prior to marker ingestion, to evaluate the effects of meal composition on system performance. Markers were delivered on a placebo tablet ingested alone, on a placebo tablet coingested with medication, or attached to a capsule body containing medication in these studies, to assess the impact of the marker carrier form factor on detection. In the HF and HTN studies, subjects’ prescribed furosemide and valsartan were marker enabled to capture subjects’ unsupervised, at-home medication adherence pattern using the networked system. Wireless devices including blood pressure cuff and weight scale were distributed to the subjects for use, to evaluate the system’s ability to integrate peripheral health monitoring equipment.

	In these 4 studies, the active study participation duration varied from 2 to 42 days with periodic visits at the clinic with the study coordinators. The safety follow-up period ranged from 5 to 21 days, depending on the duration of study participation.

	To characterize the technical performance of the networked system, a number of outcome measures were defined and assessed. The key outcome measures are summarized in 

	Moreover, to focus primarily on ingestible marker in vivo activation and detectability in these early feasibility studies, the technical evaluation of positive detection accuracy (PDA) and negative detection accuracy (NDA) excluded instances where (1) data were collected from subjects during training prior to the observation phase, (2) there were user errors in properly activating monitor prototypes prior to marker ingestion, (3) there were malfunctions in prototype software or hardware, or (4) placement or location of the monitor prototype was unsatisfactory, as indicated by a measured electrode impedance of 1000 ohms or greater.

Articles

WEB EXCLUSIVE

A networked system designed to promote patient self-management appears to be safe and effective in capturing, integrating, and presenting medication adherence and physiologic information.

Early Clinical Experience With Networked System for Promoting Patient Self-Management

 With the introduction of Part D drug benefits, Medicare began to collect information on diagnoses, treatments, and clinical events for millions of beneficiaries. These data are a promising resource for comparative effectiveness research (CER) on treatments, benefit designs, and delivery systems.

 To explore the data available for researchers and approaches that could be used to enhance the value of Medicare data for CER.

 Using currently available Medicare data for CER is challenging; as with all administrative data, it is not possible to capture every factor that contributes to prescribing decisions and patients are not randomly assigned to treatments. In addition, Part D plan selection and switching may influence treatment decisions and contribute to selection bias. Exploiting certain program aspects could address these limitations. For example, ongoing changes in Medicare or plan policies and the random assignment of beneficiaries with Part D low-income subsidies into plans with different formularies could yield natural experiments.

Refining policies for time to data release, provision of additional data elements, and linkage with more beneficiary level information would improve the value and usability of these data. Improving the transparency and reproducibility of findings, and potential open access for qualified stakeholders are also important policy considerations. Data needs must be reconciled with current policies and goals.

 Medicare data provide a rich resource for CER. Leveraging existing program elements, combined with some administrative changes in data availability, could create large data sets for evaluating treatment patterns, spending, and coverage decisions.

	Medicare now collects information on diagnoses, treatments, prescription drug use, and clinical events for millions of beneficiaries. These data are a promising resource for comparative effectiveness research (CER) on treatments, benefit designs, and delivery systems; however, there are a number of challenges to using these data for CER.

		 We explore the data available for researchers and approaches that could be used to enhance the value of Medicare data for CER.

		 Leveraging existing program elements, combined with some administrative changes in data availability, could create large data sets for evaluating treatment patterns, spending, and coverage decisions.

	Information on the safety, effectiveness, and value of medical care requires detailed clinical data from large numbers of patients receiving care in real-world settings. With the introduction of Medicare Part D prescription drug benefits in 2006, Medicare began to collect information on the use of prescription drugs for more than 27 million beneficiaries.

 Previously, this information was not widely available for Medicare beneficiaries, although drug use data were available for dual-eligible Medicare beneficiaries via Medicaid claims. The more comprehensive collection of drug use data allows for linkage with previously available information from Parts A and B including

	inpatient and outpatient diagnoses, and major clinical events for millions of beneficiaries, including many persons over the age of 65 years. These data provide a promising resource for assessing the comparative effectiveness of many types of care across a range of settings and geographic areas in the United States.

	Because Medicare collects data for payment and administrative purposes and not for research, there are several limitations to using these observational data. The care beneficiaries receive will vary depending on where they live, the types of Medicare plans they choose, and their physicians and hospitals. In other words, there may be factors associated with both the care beneficiaries receive and the outcomes of care; these factors confound assessments of care effectiveness and limit the validity of simple comparisons. However, exploiting certain program aspects and ongoing natural experiments within the Medicare program can mitigate some biases associated with purely observational data.

	In this review article, we discuss these strengths and limitations of using Medicare data for comparative effectiveness research (CER) and propose policy recommendations for improving the usefulness of these data for patients, providers, and policy makers.

	There is a profound need for more evidence to guide clinical and policy decisions on drug treatments, devices, interventions, care delivery, payment models, and delivery systems. For example, while there is arguably substantial trial evidence supporting the use of many prescription drugs, this evidence often provides limited guidance for actual clinical decisions.

	There are 2 main approaches for developing comparative effectiveness evidence: (1) clinical trials, including randomized clinical trials (RCTs) and pragmatic trials, and (2) studies using observational data from actual practice. While doubleblinded RCTs represent the gold standard for generating clinical evidence, they have a number of practical limitations. Specifically, trials have historically compared single drugs with placebo rather than with existing alternative drugs, or examined them in combination with commonly used drug regimens. Trials are done under rigorous experimental conditions (efficacy) rather than real-world situations (effectiveness), and are not designed to evaluate costs or rare adverse events. Randomized clinical trials also tend to be expensive and examine relatively short-term effects. Moreover, trials may have limited generalizability to specific subgroups of patients (eg, the elderly, racial/ethnic minorities, those with severe diseases) because they tend to target relatively homogeneous patient groups rather than the broader mix seen in actual practice.

Pragmatic trials attempt to overcome some of these limitations by focusing on more heterogeneous groups of patients and by evaluating effectiveness under routine care; however, existing evidence from pragmatic trials is in short supply and funding for these types of studies is limited.

	Studies using observational, longitudinal data sets could provide complementary information that addresses many of the limitations associated with RCTs. For example, Medicare collects information on millions of beneficiaries and allows for linkages across a range of claims data, including inpatient, outpatient, and prescription drug data. Having a large sample of individuals is critical for ensuring adequate statistical power for studying rare conditions or specific patient subgroups. Use of observational data (including Medicare data), however, requires consideration of numerous factors such as the types of Medicare plans for inpatient, outpatient, and drug services; coverage/cost sharing for treatments; availability of physicians and hospitals; and clustering of patients by physician. In addition, there can be variations in practice patterns across geographic areas. Examinations of drug use within Part D should consider these various levels of analysis and account for factors

	that could affect drug use, adherence, and ultimately outcomes, including a range of patient, provider, and plan-level characteristics. In short, the strengths and limitations of both clinical trials and observational data analyses should be considered carefully when evaluating the value of these approaches for addressing specific questions.

	The following sections describe relevant structural aspects of the Medicare Part D program, the Medicare data available for researchers, and potential approaches that could be used to create quasi-experiments and enhance the value of historical Medicare data for CER.

	Medicare currently collects diagnostic and treatment information through 4 programs: Part A (inpatient), Part B (outpatient), Part C (Medicare Advantage, which includes medical information for beneficiaries enrolled in managed care organizations), and Part D (prescription drugs). Part D is administered by private plans either as stand-alone Prescription Drug Plans (PDPs) that supplement traditional Medicare or Medicare Advantage Prescription Drug (MAPD) plans that bundle Part A, B, and D benefits. Part D is a voluntary benefit; in 2009 about 27 of 45 million Medicare beneficiaries were enrolled in a Part D plan, including 9.6 million lowincome beneficiaries who received additional premium and cost-sharing subsidies from Medicare. The Centers for Medicare & Medicaid Services (CMS) randomly assigns low-income subsidy beneficiaries who have not chosen a Part D plan to qualified stand-alone drug plans.

	Beneficiaries choose their Part D plans; these plans have some autonomy in determining their benefit structures and formulary drug lists provided they meet basic Medicare requirements. For example, all plans must offer benefits at least as generous as the defined standard. Medicare also requires that plans cover at least 2 drugs within a therapeutic class. However, plans can determine coverage for specific drugs within a class, as well as tier placement and utilization management requirements.

	The use of utilization management tools such as prior authorization has grown since Part D’s introduction; these tools are most often used for drugs that are newer, more expensive, or more risky, with greater potential for adverse effects or with less available evidence on the possible benefits or harms.

MEDICARE DATA CURRENTLY AVAILABLE FOR RESEARCH

	Since the introduction of the drug benefits, Part D plans have submitted detailed information on prescription drug events for all Part D beneficiaries to CMS. Researchers can apply for access to Research Identifiable Files that include beneficiarylevel information on Part A, B, and D claims. Proposals are reviewed by the CMS privacy board; if approved, researchers must sign a Data Use Agreement specifying the terms of use, including the destruction or return of data at the study’s end.

 The final rule permitting release of the newly available Part D data to researchers was issued in May 2008. In addition to usual protections for beneficiary privacy, this rule includes additional protections for commercially sensitive plan information. Since the initial release of Part D data, CMS has rolled out an increasing number of data elements and linkages, and is continuing to assemble supplemental data files.

 outlines the available Part D research data files.

 The primary data source for Part D drug utilization is the Part D Event (PDE) files, which are currently available for

	2006-2008. These files contain detailed information on each drug event for PDP and MAPD plan beneficiaries, and encrypted beneficiary, pharmacy, prescriber, and plan identifiers that allow linkage with other files such as inpatient and outpatient claims data, and Part D plan characteristics files. The PDE contains information on each drug dispensed including the National Drug Code, the quantity dispensed, and days of supply, allowing for the examination of therapy adherence and persistence based on dispensing data, which have been previously validated.

	The PDE data also capture cost information such as total drug costs and patient payments. These data allow for examination of variation in spending patterns and cost-of-care analyses. The PDE data also specify the benefit phase during which each prescription was filled (eg, deductible or initial coverage phase) based on the benefit structure implemented by each beneficiary’s plan, which affects costs for both patients and payers. This file also includes plan-specific information on the formulary coverage for each drug dispensed, including the tier and utilization management requirements. Plan-level information on formulary and benefit structures can be valuable for identifying quasi-experiments or instrumental variables for statistical analyses.

	The PDE data can be linked with information on individual beneficiary characteristics. The Beneficiary Summary File contains beneficiary-level information on basic demographics, including age, sex, race, and geographic location. This file also describes beneficiaries’ months of enrollment in Parts A, B, C, and D, including the type of coverage (eg, retiree, Part D stand-alone plan, Part D Medicare Advantage plan), dual-eligibility status, and whether they are receiving the Part D low-income subsidy. The Beneficiary Annual Summary File contains additional information on patients’ inpatient diagnosis-related groups, as well as 2 sets of chronic condition flags.

	Since the Part D data rule was originally issued, CMS has rolled out an increasing number of data elements that supplement the PDE and beneficiary data (Table 1). The Drug Characteristics File includes drug names (generic and brand) and strength and dosage form information by the National Drug Code. Plan characteristics files can be linked to the PDE data using the encrypted plan identifier to examine detailed information on plan type, cost-sharing levels, premium information, and service area. The Pharmacy Characteristics File includes information on the type of pharmacy where beneficiaries filled their prescriptions. Lastly, the Prescriber Characteristics File contains information on the prescriber’s specialty, credentials, and geographic location.

	Part D data can also be linked to traditional Medicare data on beneficiaries’ other medical claims (eg, hospital, skilled nursing facility, hospice, physician). For example, the Inpatient Standard Analytic File includes claims for inpatient stays, including diagnosis and procedure codes (

International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM]

), Diagnosis Related Groups (a classification system used for prospective Medicare payments), date, facility, and cost information. The Carrier File contains claims for noninstitutional providers, largely physicians. It also contains diagnosis and procedure codes (

 and CMS Healthcare Common Procedure Coding System codes), reimbursement amounts, and provider identifiers.

	Information at the beneficiary level on inpatient and outpatient service use and diagnoses within the Medicare Advantage program is not currently available to researchers.

	Monthly enrollment information is available in the Beneficiary Summary File so beneficiaries’ transitions between traditional fee-for-service Medicare and Medicare Advantage can be tracked. Prior to the introduction of Part D, beneficiaries could switch between traditional Medicare and Medicare Advantage plans monthly. In 2006, beneficiaries were restricted to switching only during the first 6 months of the year, and from 2007 onward, the first 3 months of the year.

	Medicare claims data also can be linked with Medicaid data for dual-eligible beneficiaries, as well as data sets focused on specific subpopulations or collected via surveys, such as the Surveillance Epidemiology and End Results Cancer Registry, the Long Term Care Minimum Data Set, the Home Health Outcome and Assessment Information Set, the Medicare Current Beneficiary Survey, and the Health and Retirement Survey. While the available sample size is more limited when linking to these data sources, they can provide a richer set of sociodemographic, health, and clinical characteristics than is available in the claims data alone.

	A substantial body of literature describes the limitations of using observational data for CER, as well as potential strategies for addressing these limitations.

 Data and measurement quality, the formation and stability of comparison groups, and methods to deal with crossover effects or switching between therapies are particularly relevant for observational data studies and have been described elsewhere.

	Despite the range of information available on Medicare beneficiaries’ drug treatments and diagnoses, simple analyses using currently available Medicare data are likely to yield biased results (

). Because patients are not randomly assigned to treatments, there may be important differences between the comparison groups with respect to the factors that contributed to patients receiving different treatments or therapies, such as disease severity, risk factors, and comorbidities (eg, confounding by indication).

 The lack of randomization creates analytic challenges because it is not possible using administrative data to capture all factors that contribute to physicians’ prescribing decisions.

	Plan selection and switching within the Part D program beneficiaries also create challenges. Beneficiaries choose to enroll in plans with varying characteristics that can influence treatment decisions, such as different delivery systems (eg, Medicare Advantage vs PDP), benefit designs, and formularies. Beneficiaries are encouraged to choose plans basedon these characteristics to minimize treatment disruptions and lower out-of-pocket drug costs, so there is likely to be subst antial self-selection into Part D plans.

 For example, there was evidence of substantial adverse selection into plans that offered full coverage during the standard coverage gap in 2006, prompting these plans to leave the market the following year.

 In addition, the availability of employer-sponsored drug coverage for current or retired employees also influences the characteristics of beneficiaries choosing to enroll in PartD plans; an estimated 8.3 million beneficiaries received retiree drug coverage in 2010. Beneficiaries can also switch to and from plans during the open enrollment phase of each year (eg, based on changes in clinical need). Beneficiaries’ plan choice and geographic location may have important implications for their access to different treatment alternatives.

 While a number of patient, provider, and plan characteristics are available in Medicare data sets, simply adjusting for these factors in a multivariate model is unlikely to sufficiently address selection differences between patient groups that contribute to treatment decisions, plan choice, or health outcomes.

Structural Aspects of the Medicare Program

REVIEW ARTICLE

This review article explores the Medicare data available for researchers and approaches that could be used to enhance the data%u2019s value for comparative effectiveness research.

Using Medicare Data for Comparative Effectiveness Research: Opportunities and Challenges

To compare the incidence of serious gastrointestinal (GI) complications and associated medical costs in a population with either osteoarthritis (OA) or rheumatoid arthritis (RA) enrolled in Medicare plans with celecoxib formulary restrictions versus plans without such restrictions.

 This study was a retrospective cohort analysis of Medicare members in plans with and without celecoxib restrictions. Members diagnosed with OA or RA were identified and followed for 1 year.

The restricted group had higher levels of nonselective nonsteroidal anti-inflammatory drug use (51% vs 40%; 

 <.001), and celecoxib use was double in the unrestricted group (16% vs 8%; 

 <.001). The incidence of a serious GI complication was slightly higher in the restricted group (5.4% vs 4.6%; 

 <.001). The adjusted mean serious GI complication—related cost for the restricted group was more than 15 times higher than that for the nonrestricted group ($1559 [95% confidence interval (CI) $1341-$1811] vs $101 [95% CI $87-$117]); adjusted mean arthritis-related medical costs were $5733 per year (95% CI $5097-$6448) for the restricted group and $3170 (95% CI $2816-$3569) for the unrestricted group.

 The restricted group had significantly less use of celecoxib, indicating that restriction was effective at reducing celecoxib utilization. Although limitations exist when comparing populations from different health plans, and the underlying causes of serious GI complications are multifactorial, the restricted group had a higher

	incidence of serious GI complications and higher costs related to serious GI complications and arthritis.

	This study compared the incidence of serious gastrointestinal (GI) complications and associated GI- and arthritis-related medical costs between Medicare plans with and without formulary restriction policies for celecoxib.

		 Plans with a formulary restriction policy had a significantly higher incidence of serious GI complications.

		 Plans with a formulary restriction policy had significantly higher serious GI-related medical costs.

		Plans with a formulary restriction policy had significantly higher arthritis-related medical costs.

	Nonsteroidal anti-inflammatory drugs (NSAIDs) are widely used in the treatment of osteoarthritis (OA) and rheumatoid arthritis (RA).

 The most common adverse events experienced by NSAID users are gastrointestinal (GI) related. Adverse GI events can range from very minor dyspepsia to life-threatening GI bleeding or perforation.

 The major risk factors for GI complications related to NSAID use include history of GI bleed, concomitant use of anticoagulants, use of 2 or more NSAIDs, history of peptic ulcer, high-dose NSAID, age >60 years, severe illness, Helicobacter pylori infection, and concomitant use of corticosteroids.

 Lanas et al indicated that individuals over the age of 75 years are considered to be at high risk and their risk equals that of persons with a history of peptic ulcer disease.

 It has been estimated that more than 90% of all NSAIDs prescribed are for individuals over the age of 65 years.

 It is well documented that the elderly experience significant and serious GI complications with NSAIDs.

	Celecoxib, a selective cyclo-oxygenase (COX-2) inhibitor NSAID, has been shown to be as effective as traditional nonselective NSAIDs (nsNSAIDs), 

 although the product labeling for all NSAIDs carries the same GI and cardiovascular warning. The majority of traditional nsNSAIDs have generic equivalents available, while there are no COX-2 generic equivalents. Many health plans, in an effort to control costs, have implemented utilization management techniques such as tier placement, prior authorization, or step therapy to decrease utilization of COX-2 agents.

 In 2004, Briesacher et al found that members in a 3-tier pharmacy benefit design with arthritis and serious GI comorbidity were less likely to use a COX-2 agent compared with those with a 1-tier pharmacy benefit.

 Johnson et al evaluated the impact of NSAID formulary restriction on medical costs.

 They showed that the more restrictive formularies led to higher hospitalizations for OA and RA members and higher total healthcare costs for RA members.

 Although these approaches have the ability to lower pharmacy costs by decreasing utilization, very little has been published to show the impact of the restrictions on medical outcomes. Since 2007, some Medicare Advantage and Prescription Drug Plans have used various formulary management strategies to restrict the use of celecoxib. Humana health plans have used a step-therapy policy with prior authorization restriction. That is, in order to get celecoxib covered through their prescription benefit, Humana Medicare members must have tried and failed with 2 prescription-strength nsNSAIDs on the preferred drug list in the previous 6 months or must satisfy at least 1 of the following secondary criteria: currently on a prescription-strength GI medication, anticoagulant or antiplatelet therapy, bisphosphonate, chronic oral corticosteroid, or antineoplastic agent. If neither of these criteria are satisfied, the member’s provider could file a prior authorization request.

	The purpose of this study was to compare the incidence of serious GI complications and associated medical costs in an elderly Medicare population with OA and/or RA enrolled in plans with restrictions on celecoxib versus plans without restrictions on celecoxib.

	This retrospective study used a cohort design to identify and track members with arthritis who were covered under a Medicare Advantage or Medicare Supplemental plan. Two groups were identified for comparison: (1) restricted celecoxib coverage and (2) unrestricted celecoxib coverage. The study design was approved by an institutional review board.

	All members with coverage through a Medicare Advantage or Medicare Supplemental health benefit plan who were continuously enrolled during an observation period 365 days prior to and after the index date and who were between 65 and 90 years of age on the index date were included in the study. Members who met either of the following 2 criteria were excluded from the study: (1) diagnosis of familial adenomatous polyposis during the preindex or follow-up period (defined as International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] code 211.3 in any diagnosis field on a medical claim) or (2) diagnosis of ankylosing spondylitis during the preindex or follow-up period (defined as ICD-9- CM code 720.0 in any diagnosis field on a medical claim).

	The restricted celecoxib group was identified from Humana’s Medicare population using medical, pharmacy, and eligibility data. The unrestricted celecoxib group was identified using the MarketScan Medicare Supplemental and Coordination of Benefits Database from Thomson Reuters. Unrestricted status was based on participating in a plan that did not have a prior authorization/step therapy requirement for celecoxib. Thomson Medstat associates assigned the unrestricted status specific to each member for each calendar year. To assign this indication, members were first grouped by drug plan and year. Details on celecoxib coverage by plan and year were determined using Fingertip Formulary (Fingertip Formulary LLC, Glen Rock, New Jersey), Epocrates (Epocrates Inc, San Mateo, California), or the plans’ Web sites. When information was not available, the proportion of celecoxib fills relative to all drug fills was used as a proxy for restriction. Members enrolled in plans with <1.5% fills for celecoxib were tagged as restricted. Only members enrolled in a plan with an unrestricted status during their entire study period were considered for inclusion.

	The study period was January 1, 2006, to December 31, 2008. Members who met all of the following criteria were included in the study: medical claim with an ICD-9-CM diagnosis code in any of the first 3 diagnosis fields for OA (715.x) or RA (714.x or v82.1) between January 1, 2007, and December 31, 2007. The service date of the first claim was defined as the index date. Those members who met the inclusion criterion were observed for 12 months prior to their index date and 12 months after their index date.

	Pharmacy and medical claims with service dates that occurred during the individual subjects’ 24-month observation period were used to identify the outcomes of interest for the study. The composite primary end point of a serious GI complication was defined as a GI bleed or perforation (

). Serious GI complications were further identified as upper or lower in nature. The relevant codes were selected based on published literature and expert opinion. The time until a serious GI complication was calculated as the number of days between the index date and service date on the medical claim.

	Serious GI complication—related total medical per member per year (PMPY) costs during the follow-up period included the total outpatient and inpatient medical costs where the primary or secondary diagnosis code was for a serious GI complication. Arthritis-related total medical PMPY costs in the follow-up period included the total outpatient and inpatient medical costs where the primary or secondary diagnosis code was for RA or OA. Both plan-paid and member out-of-pocket costs were included in the calculation of serious GI- and arthritis-related medical costs. Gastroprotective and gastrotoxic medication utilization was evaluated using the pharmacy claims for the member. Arthritis treatment utilization was also evaluated (

). An RxRisk-V score, which is an enhancement of the Chronic Disease Score,

 was calculated using pharmacy utilization during the 12 months prior to the index date. The unweighted version used in this analysis was calculated using a simple count of drugs in unique drug categories. The score can range from 0 to 27.

	Bivariate descriptive analyses were conducted on all study variables. The 

 test or Wilcoxon rank-sum methods were utilized for comparing continuous variables, and X

tests were utilized for the categorical variables. The baseline characteristics (demographic, clinical, and drug utilization patterns) were compared between the celecoxib-restricted and celecoxibunrestricted groups. Statistical significance was set at 

	Cox proportional hazard models were used to compare the restricted and unrestricted groups on the risk of serious GI complications. Time until a serious GI complication was the dependent (outcome) variable used in the model. The independent (predictor) variables used in the model were age, sex, region, OA/RA, use of medications or clinical conditions that increase risk of GI complications, and use of gastroprotective medications. Members were censored if they did not have a serious GI complication by the end of their follow-up period. Collinearity among the covariates was assessed, and based on the results there were no significant correlations between the covariates. Time-dependent covariates were included in the models to assess the proportional hazards assumption.

	Generalized linear models (GLMs) using a gamma distribution as their probability distribution and log-link as their link function were used to compare the medical costs of serious GI complications and arthritis for the groups. The usefulness of the GLMs for modeling the medical costs is their ability to handle nonconstant variance (heteroscedasticity) and avoid having to retransform log-transformed costs (a common strategy for handling highly skewed medical cost data).

 The covariates used in the models were the same as those used in the proportional hazard model. Members with zero costs (related to serious GI complications or arthritis) during the follow-up period had $1 added to their costs. Models provided the adjusted estimates of mean cost for each of the groups. SAS version 9.2 (SAS Institute Inc, Cary, North Carolina) was used for all statistical analyses.

	A total of 172,118 members met the inclusion criteria, with 86,011 members in the restricted group and 86,107 in the unrestricted group. 

 shows the results of the application of the inclusion/exclusion criteria for the groups.

 shows the baseline characteristics for the groups. The majority of members in both groups were female, with approximately 33% of both groups being male. Large differences were seen in the geographic regions between the restricted and unrestricted groups. The majority of members in the restricted group were located in the South and Midwest regions (70% and 23%, respectively). A more even distribution was seen in the unrestricted group between the Midwest, South, and West regions (29%, 37%, and 28%, respectively). The mean age for each of the groups was similar (75 years), although found to be statistically significantly different (

 <0.01) due to the large sample sizes of the groups. The restricted and unrestricted groups showed an equal distribution of members with a diagnosis of OA (88%), RA (5.3%), and OA with RA (6%). The unrestricted group had more members with a prior history of a GI bleed (4.4% vs 2.1%). There were slightly more members with a history of H pylori infection in the restricted group (0.6% vs 0.4%). The unrestricted group had a slightly higher RxRisk-V score (5.9 vs 3.8).

	With respect to the utilization of gastroprotective medications, more members in the unrestricted group had a claim(s) for a proton pump inhibitor (PPI) (37% vs 32%; 

 <.01), whereas more members in the restricted group had a claim(s) for a histamine H2-receptor antagonist (H2RA) (13% vs 8%; 

 <.01). Compared with the unrestricted group, the restricted group had more claims for drugs recognized as gastrotoxic: nsNSAIDs (51% vs 40%) and oral glucocorticosteroids (33% vs 29%). More utilization of bisphosphonates, antiplatelets, anticoagulants, and heparin was seen in the unrestricted group. As expected, fewer members used celecoxib in the restricted group compared with the unrestricted group (8% vs 16%). The unrestricted group also utilized more nonbiologic and biologic disease-modifying antirheumatic drugs.

	The members in the restricted group had a slightly higher risk for a serious GI complication compared with the unrestricted group (5.4% vs 4.6%; 

]). This translates into a higher relative risk (1.17 [95% confidence interval (CI) 1.12-1.21]) in the restricted group, although the absolute difference in risk was small (0.8%). While we agree that the clinical relevance of this finding is debatable due to the small difference, the difference was statistically significant, and given the prevalence of arthritis in the Medicare population, it could translate into significant outcome and cost differences between the 2 groups, as examined in subsequent sections. While no differences existed in lower GI complications between the groups, the incidence of upper GI complications was higher in the restricted group (1.0% vs 0.7%; 

 <.01). Both groups were similar with respect to time until a serious GI complication occurred. The average number of days until a serious GI event occurred was 167 days from the index date.

	Cox proportional hazard models were also used to predict the risk of serious GI complications. After controlling for demographic and clinical characteristics and medications that impact the risk for a serious GI complication, the restricted group still showed a higher risk for a serious GI complication (hazard ratio 1.69; 95% CI 1.60-1.78), confirming the results of the unadjusted results. Covariates associated with an increased risk were age, prior GI bleed, 

 diagnosis, RxRisk score, and sucralfate use. Those associated with a decreased risk were OA (vs OA and RA combined), and PPI, H2RA, glucocorticosteroid, oral bisphosphonate, antineoplastic, antiplatelet, anticoagulant, heparin, and celecoxib use (

). A sensitivity analysis was performed by excluding members with a prior GI bleed. The results from the proportional hazards were similar (hazard ratio 1.63; 95% CI 1.54-1.73).

	A second Cox proportional hazards model evaluating any gastroprotective therapy use and level of exposure was applied. Gastroprotective therapy use was defined as a member using a PPI, H2RA, sucralfate, or misoprostol between the index date and the end of their follow-up period. Gastroprotective therapy use was higher in the unrestricted group than the restricted group (36% vs 28%; 

 <.0001). The level of exposure was the number of claims made by the member during the period. The results of the Cox regression model indicated that users of any versus no gastroprotective therapy (dichotomous predictor) had more than 2.2 times the risk for a serious GI complication of nonusers (

). However, the increase in the number of gastroprotective therapy refills was associated with a 20% decrease in the risk for a serious GI complication.

	The unadjusted mean serious GI complication medical costs PMPY were $146 and $29 for the restricted and unrestricted groups, respectively. The unadjusted arthritisrelated medical PMPY costs were higher for the restricted group ($1436 vs $764; 

 presents the results from the generalized linear regression models for serious GI complication—related and arthritis-related medical costs after controlling for age, sex, geographic region, underlying clinical conditions, and drug utilization. The adjusted mean PMPY costs for serious GI complications were $1559 vs $101 (

 <.0001) for the restricted and unrestricted groups, respectively, across the entire arthritis patient population. A second GLM model was used to compare the serious GI complication costs between the restricted and unrestricted groups for the subset of arthritis members with a serious GI complication only. This model showed that among those with a serious GI complication, the adjusted serious GI complication PMPY costs were 4 times higher in the restricted group compared with the unrestricted group ($4025 vs $1028, 

 <.0001). The model for arthritis-related medical PMPY costs showed that the adjusted costs for the restricted group were 80% higher than those for the unrestricted group.

	This study compared the incidence of serious GI complications and disease-related expenditures in 2 groups of members covered under a Medicare Advantage or Medicare Supplemental plan who were diagnosed with OA or RA. The groups differed in terms of whether or not the health plan that administered the drug coverage benefit had a restriction (ie, step therapy edit) in place for celecoxib. Overall the groups were similar in terms of age and sex. However, large differences were seen in the geographic regions represented in each of the groups. The analyses indicated that region did not have an impact on serious GI event rates, but there was an impact on the costs associated with treating serious GI complications and arthritis. This could be due to variations in treatment patterns and reimbursement rates between regions. Both groups were also similar with respect to the prevalence of OA, RA, or both conditions. As would be expected, the level of celecoxib utilization was much lower in the restricted group.

	Several other studies in Medicaid populations have demonstrated that prior authorization programs have a significant impact on reducing the utilization of COX-2 inhibitors with an expected increase in nsNSAID utilization.

 However, to our knowledge, no study has examined the association between restrictions and serious GI complications in a Medicare population. This association is important because for most health plans, age greater than 65 years is one of the risk factors indicating that the restriction should not be enforced (ie, members should be able to receive celecoxib). In spite of having this risk factor, only 8% of the restricted group received celecoxib, which is cause for concern and raises questions about factors influencing prescriber behavior. Future studies could focus on the outcomes observed in those individuals who are directly impacted by a formulary restriction policy to determine what alternative therapies, if any, are used if the decision is made to avoid or bypass the step therapy or prior authorization process.

	Another key finding that may have an impact on the incidence of GI complications was the difference in utilization of PPIs. The unrestricted group had higher levels of PPI use. It was confirmed that the restricted plan also has some formulary restrictions on PPIs, which might explain some of this difference. Other studies have shown that PPIs are used more often with COX-2 inhibitors than nsNSAIDs.

 Although this relationship was not directly analyzed in this study, it could be an underlying reason why PPI use was higher in the unrestricted group. Interestingly, the unrestricted plan members also utilized more of nonbiologic and biologic disease-modifying antirheumatic drugs, even though the incidence of RA in both groups was similar. The increased use and resulting effectiveness of these agents could be an underlying factor for the  lower arthritis-related medical costs in the unrestricted group.

	First, the study was observational, so it was unable to identify a causal relationship between formulary restriction policy and serious GI complications and treatment

	costs. Second, cumulative medication exposure was not accounted for when evaluating gastrotoxic/gastroprotective medication exposure. There could be significant differences between the therapeutic categories included in the analysis in terms of the cumulative amount of exposure required to impact the risk for a GI complication. For example, 30 days of continuous NSAID therapy may put a member at higher risk than 30 days of continuous antiplatelet therapy; however, based on presence of claims they are given equal weighting. Third, another clinical limitation is that other GI bleed risk factors were unaccounted for (eg, residing in a skilled nursing facility, where the likelihood of being sedentary may be higher and thus put the member at a higher risk for a serious GI complication). Fourth, the study was unable to account for confounding by health plan benefit structure. The restricted group consisted of members enrolled in a Medicare Advantage plan, whereas the nonrestricted group consisted of members enrolled in a Medicare Supplemental plan. The types of individuals enrolled in these types of plans could differ in terms of benefit structure. Lastly, members enrolled in a capitated (at-risk) benefit were not excluded. Providers commonly submit encounter claims only for their capitated members. The lack of reliable payment information for these members may have affected the cost analysis.

MANAGERIAL

A formulary restriction policy in a Medicare population was associated with lower celecoxib utilization; however, higher gastrointestinal- and arthritis-related medical costs were observed.

Impact of Celecoxib Restrictions in Medicare Beneficiaries With Arthritis

 Engaging patients in their healthcare is a goal of healthcare reform. Obtaining sufficient, reliable patient feedback about their experiences in an office encounter has been a challenge.

 To determine the feasibility of collecting feedback from patients regarding their office encounter at the point of care using touch screen kiosk technology in an urban primary care clinic.

 We analyzed response rate, ease of use, provider data, and condition-specific data. The study was conducted over a 45-day period at 1 internal medicine academic teaching practice. Providers, staff, and a sponsor-supported monitor directed patients to use the kiosk after an office visit.

 A total of 1923 surveys were completed from 3850 office visits (50%). There was no appreciable impact on office flow in terms of wait time, checkout procedures, or visit with provider. Characteristics of patients completing the surveys were similar to practice demographics of patients with an office visit during the study period in terms of sex, but differed by age and race. Small but statistically significant differences were seen among patient ratings of resident versus attending physicians. Patients with depression were less likely than patients with diabetes, chronic low back pain, or asthma to report that they had set personal goals to manage their condition.

 This technology represents an important advance in our ability to capture the patient’s opinion regarding quality and practice improvement initiatives, and has the potential for directly engaging patients in their care.

	Electronic kiosks, which enable data to be collected at the point of care, allow patients to report on specific elements of their care visit with the provider.

		This method of surveying patients provides a much larger data set than traditional paper-and-pencil or mailed surveys, allowing for provider-level analysis.

		Electronic kiosk surveys are simple to implement into clinic flow and require little additional staff resources.

	Goals of healthcare reform include improving the quality of patient care and reducing costs. How improvement is measured must include consideration of the actual process and outcome of care from the patient’s perspective.

 For effective chronic disease management, patients should leave a visit with their care provider having a clear understanding of their diagnosis, goals, and treatment plan.

Traditional methods of evaluating a visit from the patient’s side that focused on satisfaction with care through postcare surveys and interviews have been limited by a number of factors, including low response rate, recall bias related to delay in obtaining information, selection bias related to which patients actually complete a survey, and the halo effect, which reflects patients’ desires to speak highly of their providers.

 In addition, the logistics of collecting this information may be complex, requiring staff time to distribute surveys and collect and analyze data.

	New information technology greatly expands the ability of all stakeholders (patients, providers, practices, systems, and payers) to obtain patient feedback at the point of care, with the potential to link to the patient’s electronic health record to track the relationship between provision of care (what medications were prescribed, what tests were ordered), outcomes, and patient understanding of what happened during the visit and their role after the visit. These data can be used to evaluate, manage, and potentially reward systems, practices, patients, and providers. Electronic kiosks offer instant data access, feedback capabilities, and generation of patient-specific health information,

 and are increasingly being used within ambulatory practices to collect data at the point of care delivery in a time-, cost-, and space-efficient manner.

	We hypothesized that structuring and standardizing a process that gives patients the opportunity to provide immediate, anonymous electronic feedback using health information technologies integrated into the daily practice of medicine would lead to higher response rates than the traditional paper and pencil surveys being utilized by the practice, while allowing the clinician and practice to receive more detailed, provider-level information about the patient’s experience. We also sought to determine whether this point-of-care system can be used to assess patients’ understanding of their care plan.

	The study was conducted over a 45-day period at 1 inter-nal medicine academic teaching practice of the University of Pennsylvania. The practice consists of 9 faculty, 2 nurse practitioners, and 48 categorical residents, and conducts an average of 25,000 patient visits per year. The patient mix of the practice includes a range of socioeconomic status and insurance types.

	Clinicians and staff were instructed to direct all patients to the kiosk immediately after the office visit to voluntarily complete an anonymous survey. Clinicians were also asked to review the primary reason for visit (acute visit, checkup or screening, a specific condition such as diabetes) with patients and were given a set of “visit confirmation” notepads to assist with this process.

	We used the CarePartners Plus® touch screen Well™ Patient-Interactive Healthcare Management System (P-IHMS) to generate surveys and collect data. Two kiosks were installed in close proximity to the checkout desks where patients routinely stop after their visit.

	The study consisted of 2 phases. During the first phase a sponsor-supported monitor directed patients to use the kiosk and was available to answer questions and respond to technical issues. The monitor kept an observation log of all patients seen exiting the practice through the checkout hallway (an additional exit exists in the clinic, and unobserved patients could have exited the clinic via that route). Each observation was individually indexed and included details regarding whether the patient had been directed to the kiosk by the practitioners, staff, or monitor, and whether the patient successfully completed the survey. During the second phase, the monitor did not actively direct patients toward the kiosks but continued to record observations of use. Providers and practice staff actively directed patients to use the kiosk during both phases of the study. Phase 1 lasted for 37 days and phase 2 lasted for 8 days.

	Patients were initially presented with a welcome and consent screen informing them that participation was voluntary, that responses to the survey were anonymous and for research purposes only, and that their responses would not have any impact on the care provided. Permission to continue was requested. Those who did not wish to continue could select "no” and end the survey.

	Participants were then asked to provide demographic information (sex, race, age) and to select the primary reason for the visit from the options in the medical conditions list. On the following screen, they were asked to select the clinician who treated them by photograph and name.

	All participants were then asked 5 questions (the Patient Experience Module) based on the National Committee for Quality Assurance’s Healthcare Effectiveness Data and Information Set & Quality Measurement standards. These questions concerned their experience with their provider and the visit. The next set of questions was based on the reason for visit selection (Disease Management Module). If the patient selected checkup, screening, acute care, or other as the reason for the visit, they were asked about the preventive medicine and patient education services provided. For those that selected a specific disease state, questions were asked about whether the care provider treated the patient using best practice guidelines for disease management of the condition, such as performing a foot exam for a diabetic patient.

	The next set of questions, presented to all participants, was related to medication safety (Medication Module). Patients were asked to indicate whether the provider had prescribed written educational materials and whether they had received any medication prescriptions. They were also asked whether the medication they received was accompanied by an explanation from their care provider on proper use and what to do in the case of an adverse reaction.

	We compared the demographics of patients completing surveys with the demographics of the overall patient population to assess representativeness using X

 tests. Response rates were compared for directed versus undirected kiosk use and for phase 1 (monitor and practice directed) and phase 2 (only practice directed) using z tests of proportions.

	We assessed feasibility using observational data on office flow, ease of kiosk use for participants, and effort required of staff and providers. Additionally, survey content analysis included provider-level ratings of experience of care, patient understanding of provider explanations of medications prescribed, and goal setting for self-management. Patient experience data were used to create a score for the provider (attending physicians and resident physicians). Each “yes” response to the Patient Perception Module questions was assigned a value of 1, and all “no” and “not sure” responses were assigned a null value. The average survey score for each provider type was then determined based on a scale of 1 to 5, and the average scores for resident and attending provider type were compared. Descriptive statistics for patient-level feedback on the Medication Module questions and personal goal setting by condition from the Disease State Module questions are presented as well. This study was granted approval by the University of Pennsylvania Institutional Review Board.

	The total number of completed surveys used in analysis was 1923. Four completed surveys where the respondent indicated an age of 0 to 17 years were removed from analysis because the practice does not provide care to patients younger than 18 years. Among patients who approached the kiosk and began the survey, 16.5% declined to go past the consent screen.

 compares respondents with all unique patients who had office visits during the period of kiosk activity. There were small but statistically significant variations in the racial/ethnic and age mixes, with fewer people older than 65 years completing the survey. More Asian and Hispanic patients engaged in the survey than expected, while fewer African American and Caucasian patients participated. Sex of survey participants mirrored that of the patient mix that presented to the practice for office visits during the study period.

 shows combined data for both phases of the study. Overall, a total of 1923 surveys were completed (50% response rate). For phase 1 the overall response rate was 55% (1766/3219), and for phase 2 it was 26% (161/631). There were 3850 office visits during the study period, and 3141 patients were observed exiting the clinic and passing by the kiosks (709 patients exited without passing by the kiosks or without being observed). Of the patients who were directed to use the system, there was no difference in response rate between phase 1 and phase 2 (64% vs 67%, z = 0.768, 

 = .22, data not shown on the Figure). Of the 370 patients who were not directed to use the kiosk by a provider, staff, or monitor, 140 completed surveys (38%). Overall, patients were more likely to use the kiosks when directed, and direction from practice staff and providers was more likely to result in survey completion than when direction came from the sponsor monitor (z = 8.612, 

	Data obtained by the monitor indicated that the average time to complete a survey was 3 minutes per participant. Informal interviews with checkout staff and providers revealed there was no appreciable impact on office flow in terms of wait time to check in or check out. Provider and staff effort in directing patients was minimal. No complaints were received from patients, providers, or staff regarding impact of kiosk project on office flow. During phase 2, when no technical support was available, staff received few questions about use.

Patient Experience Ratings by Provider Type. 

One potential advantage of the kiosk system is the ability to obtain sufficient data to distinguish between performance of individual providers or subgroups of providers. Data were therefore analyzed to determine whether any differences in patient experience could be detected between resident and attending physicians. Results are displayed in 

 and demonstrate that patients were more satisfied with care delivered by attending physicians compared with less experienced resident physicians (

 <.001), though the effect size for this difference was small (Cohen’s 

 = 0.21). The average number of completed patient surveys was 93.7 per attending physician and 15.2 per resident.

 Questions regarding medication prescribing were asked of all participants who completed a survey. Less than half (43.9%) of patients reported receiving educational material from their provider. We found high levels of patient-reported understanding of explanations from providers on how to use medications (99.9%).

Patients were asked to answer specific questions relating to the reason for their visit. More than three-fourths of patients (77%) chose checkup/screening or acute care visit as their reason for visit, and 23% chose a specific condition. Patients who chose a specific condition were asked whether they set personal goals with their provider to manage that condition (

). Although the number of respondents was low, patients being treated for diabetes (

Electronic kiosks were used to survey patients on their experience with care at the time of care delivery in an urban primary care practice.

July 2011

Early Clinical Experience With Networked System for Promoting Patient Self-Management

Using Medicare Data for Comparative Effectiveness Research: Opportunities and Challenges

Impact of Celecoxib Restrictions in Medicare Beneficiaries With Arthritis

Obtaining Patient Feedback at Point of Service Using Electronic Kiosks

Addressing Healthcare Inequities in Israel by Eliminating Prescription Drug Copayments

Adherence to Medication Under Mandatory and Voluntary Mail Benefit Designs

Systolic Blood Pressure Control After Participation in a Hypertension Intervention Study

Colorectal Cancer Screening Use Among Insured Primary Care Patients

Effects of Standardized Outreach for Patients Refusing Preventive Services: A Quasiexperimental Quality Improvement Study

Cost-Effectiveness of 21-Gene Assay in Node-Positive, Early-Stage Breast Cancer