Some patients lack regular computer access and experience a digital divide that causes them to miss internet-based health innovations. The diffusion of smartphones has increased internet access across the socioeconomic spectrum, and increasing the channels through which patients can access their personal health records (PHRs) could help bridge the divide in PHR use. We examined PHR use through a computer-based Web browser or mobile device.

Cross-sectional historical cohort analysis.

 Among adult patients in the diabetes registry of an integrated healthcare delivery system, we studied the devices used to access their PHR during 2016.

 Among 267,208 patients with diabetes, 68.1% used the PHR in 2016; 60.6% of all log-ins were via computer and 39.4% were via mobile device. Overall, 63.9% used it from both a computer and mobile device, 29.6% used only a computer, and 6.5% used only a mobile device. After adjustment, patients who were black, Hispanic, or Asian; lived in lower socioeconomic status (SES) neighborhoods; or had lower engagement were all significantly more likely to use the PHR only from a mobile device (

 <.05). Patients using the PHR only via mobile device used it less frequently.

 Mobile-ready PHRs may increase access among patients facing a digital divide in computer use, disproportionately reaching racial/ethnic minorities and lower SES patients. Nonetheless, even with a mobile-optimized and app-accessible PHR, differences in PHR use by race/ethnicity and SES remain. Continued efforts are needed to increase equitable access to PHRs among patients with chronic conditions.

In an integrated delivery system, mobile access to personal health records (PHRs) may increase their use among patients with limited computer access, but differences in PHR use by race/ethnicity and neighborhood socioeconomic status (SES) remain. 

Seventy percent of PHR users accessed it with a mobile device at least once in 2016, with nearly 40% of all log-ins done using a mobile device. 

Patients who were black, Hispanic, or Asian or who lived in lower SES neighborhoods were significantly more likely to use the PHR exclusively via a mobile device. Still, these groups of patients were less likely to use the PHR at all in 2016.

Although more than 200,000 health-promoting mobile apps are available for patient download, with 1.7 billion users worldwide, research is needed to identify the clinical usefulness of mobile tools in self-management and care quality for patients.

 Importantly, the majority of these apps lack any integration with patients’ ongoing healthcare services and providers.

 Apps that are integrated with a clinical electronic health record (EHR) and that make patient-reported data available to clinicians may hold the most promise to improve well-coordinated, high-quality healthcare delivery. Within healthcare, this timely expansion to mobile-connected devices complements the growing availability of personal health records (PHRs). PHRs could be particularly relevant for patients with chronic conditions, such as diabetes, who require ongoing self-management that can be facilitated via PHRs.

However, the long-standing digital divide, defined as the gulf between individuals with and without ready access to the internet, is well documented.

 EHR requirements from CMS (Stage 3 Meaningful Use objectives) include that physicians provide patients with electronic access to their health records and tailored patient education via a Web-based tethered PHR that is linked to the patient’s EHR, also known as a patient portal.

 Although Meaningful Use financial incentives continue to promote the widespread adoption of PHRs among eligible providers, they do not require that PHRs be easily accessible via mobile devices.

 Therefore, with only computer-based access, many patients might be left out.

Previous research shows that use of computer-based PHRs has been consistently lower among racial and ethnic minorities and patients with lower education and health literacy levels.

 Recently, the diffusion of smartphone technology has increased mobile access to the internet and apps among individuals most likely to be affected by the digital divide, including racial/ethnic minorities and those with lower socioeconomic status (SES).

 Mobile-accessible PHRs can help engage patients in managing their health through convenient and timely access to personal health data, provider messaging, refilling prescriptions, or scheduling appointments.

 With many healthcare innovations, specifically advancements in PHRs, changes have favored those who have social advantages, such as higher education, greater income or wealth, more knowledge of how to navigate the healthcare system, and nonminority race/ethnicity. In this manuscript, we focus specifically on a different type of technological innovation: the introduction of PHRs that are more easily accessible using mobile devices.

Diabetes is more prevalent among individuals of lower SES and racial/ethnic minorities.

 Patients with diabetes often have other chronic conditions with complex clinical needs that require ongoing self-management.

 Diabetes self-management is crucial and requires extensive self-monitoring, adherence to medications, proper diet, and adequate exercise.

 Any practical realization of a model for coordinated safe care must rely on timely availability and use of comprehensive electronic clinical information that is not only available to providers through an EHR, but also to patients through a PHR.

 Previous studies have found that PHR use was associated with improved diabetes quality measures.

 Thus, mobile-accessible PHRs could be particularly relevant for patients with diabetes. Yet, in the absence of mobile-accessible PHRs, we found that lack of computer access accounted for most of the variation in PHR use by race and income.

 Consequently, we expect that mobile access to PHRs may facilitate PHR use among individuals who are mobile dependent. Little is known about the use of PHRs that are easily accessible and optimized for use via mobile devices. 

Within an integrated delivery system that provided all members with multiple channels to access their PHR, we examined the channel through which an adult population of patients with diabetes used their PHR (ie, through a computer-based Web browser, smartphone-accessible website, or mobile apps). We also assessed the association between patient characteristics and PHR use via mobile device.

Kaiser Permanente Northern California is an integrated delivery system that provides comprehensive care, including inpatient, outpatient, and pharmacy services, to more than 3 million members via employer-sponsored, individual, or publicly sponsored insurance. Members who register to use the password-protected patient PHR can access it free of charge via computer browser, mobile-optimized website, or mobile app. The computer-based Web portal has been available to members for more than 10 years, and the mobile-optimized website and Android or iOS apps have been available to members since 2013. The PHR offers patients a number of services, including the ability to exchange secure messages with providers on their healthcare team, view lab results, request medication refills, view portions of their health records, schedule office visits, and pay bills. The mobile-accessible and computer browser versions of the PHR offer comparable functions, although features changed slightly over time.

Our study population included all adult (≥18 years) members of an integrated delivery system, Kaiser Permanente Northern California, who were in the health plan diabetes clinical registry as of the last quarter of 2015. We chose to focus on patients with diabetes in order to examine patients with a chronic condition who would likely have ongoing need for healthcare services and self-management functions available in PHRs. This analysis was part of a larger study focused on how patients with diabetes use technology to manage their care. We included all patients who maintained continuous health plan coverage in 2016. Because our study focused on patient characteristics, we excluded PHR use via designated proxies. 

We used automated datasets to capture PHR use by channel (mobile app, mobile website, or computer browser) in 2016 among the full study population. To calculate PHR use counts, we identified use episodes by counting the number of days during the year with any PHR use (eg, multiple log-ins in 1 day counted as a single episode). In addition, we measured if patients used 3 key PHR functions (order prescription refill, send secure message, or view lab result) at any time in 2016. We also used EHR data to capture patient characteristics (age, gender, race/ethnicity) and linked patients’ residential addresses to 2010 US Census measures of education and income to define neighborhood SES at the Census block group level. Census block groups are defined as neighborhoods of lower SES if at least 20% of residents have household incomes below the federal poverty level or at least 25% of residents 25 years or older have less than a high school education.

 We also identified patients’ additional chronic conditions, other than diabetes, during the last quarter of 2015 using the health plan’s clinical chronic condition registries for asthma, coronary artery disease, heart failure, and hypertension. As an indicator of patient engagement in 2015, we used clinical quality registries to create an overall measure of patients’ histories of adherence to chronic condition medications (with 80% or more days covered by medications) and to preventive care recommendations (up-to-date flu shot, blood pressure measure, low-density lipoprotein [LDL] cholesterol measure, and glycated hemoglobin [A1C] measure for those with diabetes). We categorized patients as being highly engaged in their care if they were adherent to their chronic condition oral medications for diabetes, hypertension, or cholesterol (≥80% of days covered) and recommended preventive care services (flu vaccine and A1C, LDL cholesterol, and blood pressure screening) in 2015.

We studied patient characteristics associated with the channel(s) used to access the PHR during 2016 (computer and mobile, computer only, and mobile only). We used multivariable logistic regression to measure the association between PHR use (any use in 2016 vs no use) and patient characteristics, as well as multinomial logistic regression to measure the association between device used (mobile only, computer only, or mobile and computer) among PHR users and patient characteristics (age, gender, race/ethnicity, neighborhood SES, number of chronic conditions, and health engagement). For both models, we calculated the adjusted percentage of patients using the PHR and device type by patient characteristics, assuming patients in the subgroup had the same other characteristics as the full study population (margins command in Stata). We included the main effects of each covariate and first order interaction of all covariates (except for age group and number of other chronic conditions due to an empty cell problem). All analyses were conducted using Stata 14 (StataCorp LP; College Station, Texas).

The Kaiser Foundation Research Institute Institutional Review Board reviewed and approved the study protocol.

 shows characteristics of the 267,208 patients with diabetes included in the study and their adjusted rates of PHR use in 2016. In that year, 49.1% were 65 years or older, 47.7% were female, 43.7% were white, 22.6% were Asian, 21.6% were Hispanic, 10.2% were black, 24.0% lived in low SES neighborhoods, and 75.1% had multiple chronic conditions. Nearly one-third of all study patients (31.9%) did not use the PHR in 2016. More than half (58.0%) of patients were categorized as highly engaged, meaning that they were adherent to their oral medications for chronic conditions and received the flu vaccine and recommended screenings for A1C, LDL cholesterol, and blood pressure in 2015. 

During 2016, 181,981 patients (68.1% of the total number) accessed their PHR 8.9 million times: 60.6% of log-ins were via a computer, 19.9% via mobile device browser, and 19.5% via smartphone apps. Of all PHR users, 6.5% used it only with a mobile device, 29.6% only with a computer, and 63.9% used both a computer and mobile device (

). Patients who accessed the PHR only via a mobile device used it less frequently (median of 11 days with PHR use) than those using only a computer (15 days) or both a computer and mobile device (30 days). Most PHR users used it to view lab results (81.0%), send a secure message (75.8%), and order a prescription refill (66.0%). For the 3 key functions, use was highest among patients who accessed the PHR via both a computer and mobile device and lowest among those who used only a mobile device (

Adjusted percentages of patients who used the PHR, calculated using results from the multivariable logistic regressions, are shown in Table 1. After adjustment, a larger percentage of patients who were younger (84.7% for ages 18-29 years), white (78.4%), living in higher SES neighborhoods (71.1%), and highly engaged (71.0%) used the PHR in 2016 compared with those who were older (60.9% for ≥65 years), black (57.8%), Hispanic (54.3%), Asian (67.7%), living in lower SES neighborhoods (59.5%), and not highly engaged in their care (64.2%) (all 

 shows the adjusted percentages of PHR users who accessed their PHR only from a mobile device by patient characteristics. After adjustment, a higher percentage of patients who were black (8.8%), Hispanic (9.6%), Asian (6.7%), living in lower SES neighborhoods (7.9%), younger (11.9% for ages 18-30 years), or not highly engaged in their health (7.2%) used the PHR only via a mobile device in 2016 relative to those who were white (4.7%), from higher SES neighborhoods (6.0%), older (4.3% for ≥65 years), or highly engaged (5.8%) (all 

In 2015, a Pew survey found that 64% of US adults owned a smartphone and 19% relied exclusively on smartphones for internet access.

 In our study of use of a mobile-accessible PHR among patients with diabetes, most users (70%) accessed their PHR with a mobile device at least once and nearly 40% of all PHR log-ins were done via a mobile device. Among PHR users, nearly 7% accessed the PHR exclusively via a mobile device. These users accessed the PHR less frequently than those who used computers. In addition, nearly one-third of patients did not use the PHR at all in 2016. Patients who were black, Hispanic, or Asian; who lived in lower SES neighborhoods; or who had lower engagement in their care were significantly more likely to access the PHR exclusively using a mobile device in 2016, which is consistent with findings from other reports of mobile-dependent individuals.

 It is possible that many of the patients who accessed the PHR only by mobile device were dependent on that device for internet access and would not have had convenient computer access to use the PHR if a mobile option had not been available. 

In our previous analysis conducted in 2010, before the PHR was mobile-accessible, we found that computer internet access explained most of the variation in use of secure messaging via the PHR by race/ethnicity and income.

 Thus, the current finding that mobile access is disproportionately reaching members living in lower SES neighborhoods and racial/ethnic minorities is encouraging and suggests that mobile PHR access may, in fact, be helping to bridge the digital divide by reducing disparities in PHR use. 

Of note, we found that patients with diabetes who were not considered to be highly engaged in their health (ie, not previously adherent to chronic condition medications and recommended preventive care services) were significantly more likely to use the PHR exclusively via a mobile device. For patients coping with diabetes and other chronic conditions, which require substantial self-management practices, health engagement is critical to maintaining their health.

 Recent studies of PHR use among patients with diabetes found that it is associated with improved self-management practices and glycemic control.

 Mobile PHR use could be an important gateway for potentially reaching those patients who previously had limited engagement with preventive care and medication adherence. 

The federal government has invested more than $30 billion to promote the widespread adoption of EHRs and tethered PHRs as a way to improve access and quality of care.

 However, these financial incentives do not require that PHRs be easily accessible via mobile devices. In fact, eligible providers caring for patients in regions with low broadband internet access are exempted from Meaningful Use objectives related to PHRs.

 Although it is possible for patients with smartphones to access any available computer-based PHR using their mobile devices, websites that are not optimized for mobile use can be exceedingly difficult to navigate using the relatively small-sized smartphone screens. To the extent that PHR use can improve chronic care management and clinical outcomes,

 limiting PHR access to patients with easy access to internet-connected computers could contribute to existing disparities in healthcare access and outcomes. Policy makers should consider extending Meaningful Use PHR objectives to require easy access via mobile devices.

Nonetheless, even with a mobile-accessible PHR, we found that differences in use remained, with black, Hispanic, and Asian patients and those living in lower SES neighborhoods still significantly less likely to have used the PHR at all in 2016. This finding is similar to those of previous studies of PHRs that were not mobile-accessible.

 Mobile-dependent internet users often face important constraints to accessing the internet, such as data usage limits and small screens. It is possible that these constraints limited any PHR use or frequency of use among mobile-dependent individuals. Although smartphones have increased mobile access to the internet among underserved groups, they may be insufficient to fully bridge the digital divide.

Policy makers and healthcare administrators should continue to increase the accessibility of PHRs by making them easy to navigate and available on multiple platforms in order to reach patients with limited computer access or literacy. Increasing promotion of and education about mobile-accessible PHR availability and its salience to health management, particularly for patients with chronic conditions, could help to further bridge the divide in PHR use. Still, it is likely that PHR use may not be preferred by all patients, so it is equally important to make alternative methods for easily accessing care available.

This study has some limitations. It was conducted with members of a single health plan in northern California. Results may differ in other healthcare settings, such as those with higher poverty rates or that provide services to the uninsured. Also, because our results were based on cross-sectional data, they should be interpreted as associations only. We were not able to confirm causality. Our study did not have any direct measure of the types of internet-connected devices patients have easy access to and relied only on the type of device used to access the PHR itself. Thus, we made the assumption that patients who use a device to access the PHR have access to such a device. Although we picked a patient population with diabetes in order to examine patients with a chronic condition who would likely have ongoing need for the healthcare services and self-management functions available in PHRs and we adjusted for number of chronic conditions, individual clinical need varied by patients and over time. 

Our findings suggest that offering mobile access to PHRs may increase engagement with healthcare among vulnerable patients facing a digital divide in computer technology access. Patients who were of nonwhite race/ethnicity, lived in lower SES neighborhoods, and had lower prior health engagement were more likely to rely exclusively on mobile devices to access the PHR. Nonetheless, even with access to a mobile-optimized and app-accessible PHR, differences in PHR use by race and neighborhood SES remain. Continued efforts are needed to increase equitable access to PHRs and electronic patient self-management technologies among patients with chronic conditions who may not have convenient computer access.

Department of Preventive Medicine, University of Tennessee Health Science Center (IG), Memphis, TN; Kaiser Permanente Division of Research (JHua, CKY, MER), Oakland, CA; Department of Epidemiology and Biostatistics, University of California at San Francisco (RB), San Francisco, CA; Mongan Institute, Massachusetts General Hospital (JHsu), Boston, MA; Department of Health Care Policy, Harvard Medical School (JHsu), Boston,

 This research was supported by the National Institute of Diabetes and Digestive and Kidney Diseases (R01DK085070).

 Dr Brand is a consultant for the Kaiser Permanente Division of Research, where the research for this paper was completed with the support of a federal grant. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

 Concept and design (IG, JHua, RJB, JHsu, MER); acquisition of data (IG, JHua, MER); analysis and interpretation of data (IG, JHua, RJB, JHsu, CKY, MER); drafting of the manuscript (IG, CKY); critical revision of the manuscript for important intellectual content (IG, JHsu, CKY, MER); statistical analysis (JHua, RJB); obtaining funding (IG, RJB, MER); administrative, technical, or logistic support (MER); and supervision (JHsu, MER). 

 Ilana Graetz, PhD, University of Tennessee Health Science Center, 66 N Pauline St, Ste 620, Memphis, TN 38163. Email: igraetz@uthsc.edu.

1. Cortez NG, Cohen IG, Kesselheim AS. FDA regulation of mobile health technologies. 

2014;371(4):372-379. doi: 10.1056/NEJMhle1403384.

2. Kuehn BM. Is there an app to solve app overload? 

2015;313(14):1405-1407. doi: 10.1001/jama.2015.2381.

3. Uscher-Pines L, Mehrotra A. Analysis of Teladoc use seems to indicate expanded access to care for patients without prior connection to a provider. 

2014;33(2):258-264. doi: 10.1377/hlthaff.2013.0989.

4. Burke BL Jr, Hall RW. Telemedicine: pediatric applications

2015;136(1):e293-e308. doi: 10.1542/peds.2015-1517.

Digital Divide: Civic Engagement, Information Poverty, and the Internet Worldwide.

 Cambridge, UK: Cambridge University Press; 2001.

6. CMS, HHS. Medicare and Medicaid programs; Electronic Health Record Incentive program—Stage 3 and modifications to Meaningful Use in 2015 through 2017. final rules with comment period. 

Articles

MANAGERIAL

Racial/ethnic minorities and patients living in poorer neighborhoods were more likely to access their personal health record exclusively with a mobile device.

Bridging the Digital Divide: Mobile Access to Personal Health Records Among Patients With Diabetes

 To measure overuse of low-value care using electronic health record (EHR) data and manual chart review and to evaluate whether certain low-value services are better captured using EHR data.

We implemented algorithms to extract performance on 13 Choosing Wisely—identified healthcare services using EHR data at a large physician practice group between 2011 and 2013.

 We calculated rates of overuse using automated EHR extracts. We manually reviewed the charts for 200 cases of overuse for each measure to determine if they had clinical risk factors that could explain use of the low-value service and then calculated adjusted rates of overuse. We explored trends in overuse for each low-value service in the 3-year duration using logistic regression.

Unadjusted rates of overuse ranged from 0.2% to 92%. Automated EHR extracts and manual chart review identified explanatory risk factors for most measures, although the magnitude varied: for some measures (eg, bone densitometry exam for women younger than 65 years), manual chart review did not identify many additional risks (3.0%). In contrast, in patients who had sinus computed tomography or an antibiotic prescription for uncomplicated acute rhinosinusitis, manual chart review identified more explanatory risk factors (22.5%) than the automated EHR extract (9.5%). Adjusted rates of overuse ranged from 0.2% to 61.9%. Eight services demonstrated a statistically significant decrease in overuse over 3 years, while 1 increased significantly.

The use of EHR data, both extracted and manually abstracted, provides an opportunity to more accurately and reliably identify overuse of low-value healthcare services.

In 13 low-value tests and screenings, we found varying levels of overuse using both automatically extracted electronic health record (EHR) data and manual chart review. 

Although several studies reporting overuse of low-value health services rely on administrative claims data, we found that EHR data may accurately and reliably measure overuse. 

EHR data can provide important insights on the presence of clinical risk factors that may trigger or explain the use of low-value health services. 

EHR data extracts and manual chart review should be considered alongside other methodologies of measuring overuse to develop higher-value local treatment norms for clinicians.

In recent years, various stakeholders have sought strategies to reduce healthcare costs while improving the quality of care.

 One such strategy is to define, identify, and reduce wasteful spending on the delivery of unnecessary healthcare services. In 2012, the American Board of Internal Medicine launched the Choosing Wisely campaign, an initiative aimed at engaging patients and physicians in discussions regarding how to avoid unnecessary healthcare.

 More than 80 specialty societies have developed a list of recommendations to help physicians avoid unnecessary treatments, which include imaging studies, surgical procedures, medications, and laboratory testing.

Findings of some early studies suggest that the Choosing Wisely initiative is showing modest results in its first years

; however, the success of this initiative, and others like it, is critically dependent on our ability to accurately measure healthcare overuse.

 Most studies in the literature have relied on administrative claims data,

 although these data are better suited for measuring simple processes of care and are not suitable to capture the nuances of appropriateness of care. Studies estimating overuse with administrative claims report that claims data can lack clinical context and that the use of such data risks misclassification of potentially appropriate testing or imaging.

 Electronic health records (EHRs) may capture more detailed clinical information and context behind inappropriate use that are not available in claims data.

The goal of our study was to measure overuse defined by the Choosing Wisely initiative using a combination of structured data extracts and manual chart review from EHRs. Using this combination of rich clinical information, we sought to determine whether certain types of low-value healthcare services are better captured than others using EHR data.

We conducted this study at Atrius Health, a large physician practice group with 900 physicians and more than 400 advanced practice clinicians that provides primary and specialty care for more than 740,000 patients in eastern Massachusetts. Atrius Health physicians have utilized an integrated EHR (Epic Systems) since 1995 to support computerized outpatient ordering of medications, laboratory tests, and radiologic studies. All outpatient encounters are entered into the medical record, including vital signs, clinical notes, diagnostic and procedure codes, and all laboratory and radiology results.

A team of 2 physician health services researchers and 2 health economists reviewed available Choosing Wisely recommendations for inclusion in the study. We selected 13 Choosing Wisely recommendations (

) for analysis based on their relevance to both primary and specialty care; their inclusion of medications, procedures, and laboratory testing; and their focus on ambulatory care, given that Atrius Health does not provide hospital care and therefore does not capture these data reliably. 

We implemented algorithms to electronically capture performance on the selected Choosing Wisely recommendations between 2011 and 2013 using automated data extracts from coded data in the EHR. These data included the problem and medication lists and all diagnostic, lab, and procedure codes entered by clinicians during clinical encounters (

]). The extracts implemented numerator and denominator inclusion definitions and captured any potential denominator exclusion criteria for each measure. We also used manual chart reviews to collect information on “explanatory risk factors” from the EHR, which were conditions that indicated why the Choosing Wisely test or treatment would have been clinically appropriate. Two measures (repeat bone densitometry [DEXA] exams and repeat endoscopy for Barrett’s esophagus) were assessed in years prior to 2011, as they required historic data to measure whether the repeat exams occurred between 2011 and 2013.

We measured overuse via a denominator at either the exam or patient level, based on whether we could identify a patient population accurately. Measures that defined the patient denominator based only on demographic criteria and required little clinical evaluation (eg, cancer screening measures for women) were measured using an exam-level denominator. In contrast, measures that defined the patient denominator based on presentation to the office with a symptom (eg, headache or syncope) were measured using a patient-level denominator. This approach allowed us to avoid making assumptions about whether patients were receiving their care at the physician group practice during the entire study period and should thus be included in the denominator. 

From the electronic data extracts, we selected a random sample of 200 cases of overuse for each measure based on its numerator definition (eAppendix I) from each study year for manual chart review. Manual chart review data were considered the gold standard for our analyses, as such data are generally the most robust clinical information source available. The chart review team consisted of 2 board-certified internal medicine physicians and a research assistant. The manual chart reviews focused on the index clinical note associated with ordering of the targeted overuse service, as well as any other information referenced in this index clinical note, such as prior clinical notes, study results, or medication lists. 

The review team collected information on whether the EHR data extracts were working as intended in capturing the numerator, denominator, and exclusion definitions of each measure; developed a list of clinical explanatory risk factors for each measure based on guidelines; and examined whether these were present in the clinical notes or test ordering documentation. 

We fit multivariable logistic regression models to assess trends of overuse using EHR data for each Choosing Wisely recommendation, incorporating definitions of overuse (eAppendix I) and controlling for each study year between 2009 and 2013.

The chart review team collected data on the number of explanatory factors found from EHR extracts and manual chart review. In cases where there was confusion about whether a test or procedure might have been clinically indicated, the 3 reviewers used a consensus approach to determine whether a legitimate explanatory risk factor was present.

We reported unadjusted and adjusted performance rates after excluding patients with explanatory risk factors. All analyses were conducted using Stata version 14 (StataCorp LP; College Station, Texas). The study protocol was approved by the Partners Healthcare System Human Studies Review Committee. 

The prevalence of overuse varied widely, both among those measures with exam-level denominators and those with patient-level denominators, after accounting for explanatory risk factors found in the EHR (

). In 2013, among exam-level measures, the prevalence of overuse ranged from a low of 0.2% (proportion of Pap smears performed on women aged 18-21 years) to a high of 57% (proportion of DEXA exams performed on women aged 18-65 years). Among patient-level measures in 2013, the prevalence of overuse ranged from a low of 8% (use of head imaging for syncope) to a high of 92% (use of sinus computed tomography [CT] or antibiotics for acute rhinosinusitis).

Among the 12 measures with 3 years of data, the prevalence of overuse demonstrated a statistically significant decrease over time for 8 (67%) measures. There was a significant increase in prevalence for only 1 (8%) measure, the use of head imaging for patients with uncomplicated headache.

We found wide variation in the number of risk factors identified by the automated EHR extract across measures (

). In Vitamin D deficiency screening, imaging for low-back pain, and both DEXA measures, over half of the sample was shown to have risk factors that could explain the test or screening using the automated EHR extract; however, other measures, like Pap smears performed on women younger than 21 years, found no explanatory risk factors when using the EHR extract alone. 

Across nearly all measures, the manual chart review identified additional explanatory risk factors, although the magnitude varied by measure (Table 3). Manual chart review identified few additional explanatory risk factors in DEXA exams performed on women younger than 65 years (3.0%) and prescription of opioids or butalbital treatment for patients with migraine (5.0%), but more in patients who had a sinus CT or antibiotic prescription for uncomplicated acute rhinosinusitis (22.5%) and Pap smears performed on women with total hysterectomy for noncancer disease (29.0%). 

 presents the explanatory risk factors for each measure. 

We calculated an adjusted overuse rate in 2013 based on the proportion of EHR-identified overuse that manual chart review determined had risk factors that could explain the test or screening (Table 3). The prevalence of overuse after adjustment ranged between 0.2% for Pap smears on women younger than 21 years and 61.9% for patients with uncomplicated acute rhinosinusitis who had a sinus CT or were prescribed antibiotics. 

There is widespread agreement that low-value services are a substantial problem leading to enormous waste in healthcare, and Choosing Wisely recommendations are often suggested as a tool to begin to define and identify waste. We examined how easily EHR data from a large ambulatory care center could be used to identify overuse on a selection of Choosing Wisely measures, and we found mixed results. 

Previous study findings indicate that administrative claims are a useful, albeit limited, source of data for identifying overuse.

 Primarily used for documenting diagnoses and procedures provided for the purpose of payment, administrative claims data do not provide detailed clinical context, which can ultimately misclassify, underestimate, or overestimate indicated tests or screenings.

 Reliance on claims alone may misclassify a clinically appropriate test or screening as overuse of low-value care, as patient history is an integral factor in clinical decision making.

 Several studies, including our own, cite reliance solely on administrative claims data for measuring overuse as a limitation for accurately reporting overuse.

Our findings suggest that EHR data can be an important, although variable, source of information in identifying overuse of clinical services. For some measures of overuse, such as Pap smears in women younger than 21 years, structured EHR extracts were sufficient for identifying rates of overuse and relevant risk factors. In these cases, manual chart review added little insight into the potential clinical justification for a test or screening.

For most other measures, the combination of EHR data and manual chart review provided valuable information and elucidated some of the inherent complexities in overuse measures. For instance, cases of DEXAs in women younger than 65 years were easy to identify in the EHR, although clinical risk factors, such as previous osteopenia, were often identified in manual chart review. The initial diagnosis of osteopenia was often obtained from a DEXA that was not clearly indicated, thereby suggesting that an initial low-value test or screening may lead to subsequent low-value services. Further, there is debate on what the correct duration of follow-up should be after identifying mild or moderate osteopenia.

 Understanding these clinical nuances that may explain imaging, testing, or procedures is important, given the potential implication on costs (eg, the estimated cost of a single DEXA exam is approximately $125).

 The measure regarding overuse of antibiotics for sinusitis also proved challenging. Most cases of sinusitis identified using the EHR received antibiotics, which might represent a coding bias of clinicians in which the diagnosis is only listed when treated. 

Although there is variation in magnitude across measures, our study results suggest that EHR data provide important insights on overuse and presence of risk factors for several Choosing Wisely recommendations. However, both claims data and EHR data have limitations that can overestimate overuse, as the presence of risk factors is often only captured in chart review. We used manual chart review as the gold standard in our study, but recognize the labor-intensive nature of this methodology. Development of more automated text-based extracts (eg, natural language processing) could provide a less resource-intensive means to identify legitimate explanatory clinical risk factors of overuse. Further, as practices incorporate clinical decision support to identify low-value testing in real time and query providers to specify a clinical justification, the utility of EHR data extracts should improve. 

Our analysis has a number of limitations. We examined the use of EHR data in a large ambulatory care system that uses Epic software. The data warehouse structure and information available for procedures, medications, and laboratory tests likely have large variations compared with other EHRs. Second, we examined only a selection of Choosing Wisely recommendations, although the sample had variety in measures pertaining to medications, imaging, and procedures. Third, our study relied on manual chart review as the gold standard for determining overuse. Although manual chart review provides more clinical information than administrative claims data, we relied on information documented in the patient chart and therefore may be missing data that were not documented. Fourth, our chart reviews only examined clinical information from the encounter associated with the test order of each Choosing Wisely measure. A more thorough chart review looking back at previous notes and outside notes would likely yield more explanatory information, although this type of review requires more resources to perform. Finally, our EHR extracts and chart reviews examining explanatory factors and risk factors for each measure are open to clinical interpretation, and the clinical opinion of reviewers would impact the reproducibility of our results.

As clinicians and policy makers continue to gather data on overuse of low-value services, the methodologies and data sources utilized to measure overuse have become increasingly important. Developing more accurate and reliable calculations of overuse would be instrumental for policy makers and providers to identify opportunities for changing care delivery. Our work suggests that EHRs are an important source of data to quantify overuse and that EHRs can capture clinical information that often explains why a test or treatment is clinically indicated. Further, manual chart review, although more resource-intensive, may identify the presence of important risk factors that automated EHR data extracts cannot, and it should be considered alongside other methodologies of measuring overuse. The data from such manual chart reviews might be particularly important when engaging clinicians in the development and implementation of care delivery practices that reduce overuse of low-value services.

 Atrius Health (TI), Newton, MA; Department of Health Policy and Management, Harvard T.H. Chan School of Public Health (MBR, ZL, KHN), Boston, MA; The Dartmouth Institute for Health Policy and Clinical Practice (CHC, NEM, AJM), Lebanon, NH; Norris Cotton Cancer Center, Dartmouth-Hitchcock Medical Center (CHC), Lebanon, NH; Department of Community and Family Medicine, Geisel School of Medicine at Dartmouth (NEM), Lebanon, NH; Division of General Internal Medicine, Brigham and Women’s Hospital (EAK, TDS), Boston, MA; Partners HealthCare (EAK, TDS), Boston, MA; Department of Health Care Policy, Harvard Medical School (TI, TDS), Boston, MA.

 The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article. 

 Concept and design (TI, MBR, CHC, NEM, EAK, TDS); acquisition of data (TI, MBR, ZL, EAK, TDS); analysis and interpretation of data (TI, MBR, CHC, NEM, AJM, ZL, KHN, EAK, TDS); drafting of the manuscript (TI, MBR, CHC, NEM, AJM, KHN); critical revision of the manuscript for important intellectual content (TI, MBR, CHC, NEM, AJM, TDS); statistical analysis (MBR, CHC, ZL, KHN, TDS); obtaining funding (MBR, CHC, TDS); administrative, technical, or logistic support (CHC, AJM, KHN, EAK); and supervision (MBR). 

 Thomas Isaac, MD, Atrius Health, 275 Grove St, Ste 3-100, Auburndale, MA 02466. Email: thomas_isaac@atriushealth.org.

1. Schwartz AL, Chernew ME, Landon BE, McWilliams JM. Changes in low-value services in year 1 of the Medicare Pioneer Accountable Care Organization program. 

2015;175(11):1815-1825. doi: 10.1001/jamainternmed.2015.4525.

2. About. Choosing Wisely website. choosingwisely.org/about-us. Accessed December 11, 2015. 

3. Lists. Choosing Wisely website. choosingwisely.org/doctor-patient-lists/. Accessed December 11, 2015.

4. Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the Choosing Wisely campaign. 

2015;175(12):1913-1920. doi: 10.1001/jamainternmed.2015.5441.

5. Grover M, McLemore R, Tilburt J. Clinicians report difficulty limiting low-value services in daily practice. 

2016;7(2):135-138. doi: 10.1177/2150131915624112.

CLINICAL

Electronic health records data can accurately quantify overuse of clinical services and the risk factors that may trigger low-value testing and screening.

Measuring Overuse With Electronic Health Records Data

 This study explored variation in the extent of use of electronic health record (EHR)-based health information technology (IT) functionalities across US ambulatory care practices. Use of health IT functionalities in ambulatory care is important for delivering high-quality care, including that provided in coordination with multiple practitioners.

We used data from the 2014 Healthcare Information and Management Systems Society Analytics survey. The responses of 30,123 ambulatory practices with an operational EHR were analyzed to examine the extent of use of EHR-based health IT functionalities for each practice.

 We created a novel framework for classifying ambulatory care practices employing 7 domains of health IT functionality. Drawing from the survey responses, we created a composite “use” variable indicating the extent of health IT functionality use across these domains. “Super-user” practices were defined as having near-full employment of the 7 domains of health IT functionalities and “under-users” as those with minimal or no use of health IT functionalities. We used multivariable logistic regression to investigate how the odds of super-use and under-use varied by practice size, type, urban or rural location, and geographic region.

 Seventy-three percent of practices were not using EHR technologies to their full capability, and nearly 40% were classified as under-users. Under-user practices were more likely to be of smaller size, situated in the West, and located outside a metropolitan area.

 To achieve the broader benefits of the EHR and health IT, health systems and policy makers need to identify and address barriers to full use of health IT functionalities.

As of 2014, 73% of ambulatory practices were not using electronic health record (EHR)-based functionalities to their full capability, and nearly 40% were classified as health information technology (IT) “under-users.”

Under-use of health IT in ambulatory care has implications for the ability of the health system as a whole to provide coordinated and efficient care. 

Facilitating the full use of a range of health IT tools in the ambulatory setting may help the broader health system gain the full benefit of investments in EHR-based technologies. 

Efforts to increase the use of health IT functionalities should focus on practices that are small, are located in nonmetropolitan areas, and provide specialty care.

Healthcare organizations across the United States have invested substantially in electronic health record (EHR) systems, incentivized by federal investment and legislation.

 Ambulatory care practices have steadily improved their EHR adoption over the last decade; 2014 estimates indicated that approximately 78% of ambulatory care practices had a certified EHR platform.

 There is substantial heterogeneity within this group, however. The EHR acts as a backbone for a range of health information technology (IT) functionalities with multiple potential applications to care delivery; practices vary in their adoption of these functionalities and in the extent of their use of these tools in routine practice.

Empirical data show benefit to processes of care from an array of health IT functionalities, including data repository,

 electronic messaging and health information exchange,

 In addition, quality improvements from the EHR and associated functionalities likely transcend the individual provider organization, with some tools (such as health information exchange) designed to work in synergy for coordination of care among multiple practitioners.

 Practices restricting themselves to the more basic features of this technology may limit the potential impact of the EHR on their own performance

; it is also possible that slow or elementary adopters may have a negative impact on the quality of the health system as a whole.

In this study, we explored variation in the extent of use of EHR-based health IT functionalities in the ambulatory care setting. We used data from the Healthcare Information and Management Systems Society (HIMSS) Analytics ambulatory practice surveys to create a new framework of EHR use across 7 domains of health IT functionality, and we identified practices that were high users of a range of functionalities (“super-users”) and those that used these EHR tools only minimally (“under-users”). Noting that studies on hospital EHR adoption suggest that small and rural hospitals may experience greater barriers in implementing this technology,

 we investigated how the rates of super-use and under-use vary according to practice size, type, urban or rural location, and geographic region.

HIMSS conducts annual surveys of US health systems and organizations, with a particular focus on structural characteristics of their EHR and health IT functionalities in use, generating a comprehensive database that has been frequently used in empirical research.

 To date, published studies that have employed these data utilized only the data regarding hospitals.

 However, HIMSS also obtains data on ambulatory care practices, defined as facilities providing “preventative, diagnostic, therapeutic, surgical, and/or rehabilitative outpatient care where the duration of treatment is less than 24 hours—and is generally referred to as outpatient care.” We used data from the 2014 ambulatory practice survey, which contains information on more than 75% of US health system—associated ambulatory care practices. HIMSS defines a health system as an organization composed of at least 1 hospital and its associated nonacute facilities, and “associated” as a governance relationship (ie, they are owned, leased, or managed by a health system). Eligible practices for our study were those that indicated they had a “live and operational” EHR and had completed at least 1 health IT functionality survey question. We linked the practice site zip code with a publicly available dataset providing a geographic taxonomy to develop a measure of rurality.

Existing EHR classifications applicable to the ambulatory care setting have limitations; many are defined by only short lists of Meaningful Use criteria,

 and categorizations of “basic” or “comprehensive” systems are largely hospital-focused. We created a novel framework for classifying ambulatory care practices using 7 domains of health IT functionality, referencing the structure of the HIMSS survey and historical taxonomies (such as that by Des Roches et al

). The 7 domains were data repository, clinical decision support, order entry management, electronic messaging, results management, health information exchange, and patient use. The HIMSS survey asks respondents to indicate if they use any of more than 50 EHR-based health IT functionalities and, in some cases, assesses the intensity of this use (eg, “What proportion of orders are completed using the EHR?”). We matched all of these items to 1 of the 7 domains of functionality (details are given in the 

We used a 3-step process to define a practice as a super-user or under-user of health IT functionalities. First, we classified practices into 3 categories based on the number of functionalities employed within each domain. Practices in the lower quartile for their sum total of functionality within a domain were categorized as “low” (score of 0), those in the upper quartile were defined as “high” (score of 2), and practices in the interquartile range were categorized as “moderate” (score of 1). Second, we created a composite “use” variable by summing the domain scores for each practice (composite scores ranged from a minimum of 0 to a maximum of 14). Third, we ranked practices according to this composite variable. We explored the natural distribution of the data in order to identify practices that were low and high outliers on the composite score. We defined practices as super-users if they had a composite score of 12 to 14 and under-users if their composite score was 0 to 2. We performed sensitivity analyses to explore the impact of alternative criteria; our findings were robust to alternate specification of the cut points.

We examined characteristics of practices according to their classification as a super-user or under-user, using Pearson’s χ

 test for the categorical variables and a 2-sided 

 test for the continuous variable. Variables of interest included the size of the practice (defined as number of affiliated physicians, in 4 categories), location (metropolitan, midsize, small town, or rural), geographical region (Northeast, Midwest, South, or West), and type of practice (primary/family care; single-specialty, multispecialty, and allied health; or urgent care and specialist services). Allied health practices included those practicing podiatry, occupational health, weight management, and holistic medicine, among others. Practices providing “specialist services” were those giving specialty-circumscribed care to a defined population (eg, patients undergoing dialysis or cardiac rehabilitation). Using multivariable logistic regression models, we estimated odds ratios associated with super-user and under-user status, according to practice characteristics. Analyses were performed using Stata version 14.2 (StataCorp LLC; College Station, Texas). We used Quantum Geographic Information Software to create maps showing the distribution of use categories across the United States.

There were 38,638 health system—affiliated practices in the HIMSS data; 32,236 (83.4%) indicated they had a live and operational EHR, and of these, 30,123 (93.5%) provided survey responses. The majority (77.4%) of responding practices in the sample had fewer than 7 associated physicians; however, the distribution of this variable was skewed by some practices with large numbers of physicians (maximum, 2300) such that the median number of physicians per practice was 2 and the mean was 5.6. The dominant practice type was single or multiple specialty and allied health practitioners (62.5%), whereas 30.8% were primary/family medicine. Nearly 75% of practices were located in metropolitan areas; only 4.7% were rurally located.

shows the proportions of practices with low, moderate, and high use by domain of health IT functionality. The eAppendix

provides the full table of functionalities and frequency of responses and the descriptive characteristics of the total sample and super-user and under-user practices. Among practices indicating any use of computerized physician order entry, only 35.6% used this capacity for more than 75% of orders. Additionally, although the majority of practices were adept at using their EHR for more elementary functions, such as data storage (100% of practices stored transcribed reports electronically and 61.1% used the EHR for nursing documentation), some of the more advanced functionalities (such as the ability to find and modify orders for all patients on a specific medication) were used at much lower rates (29.3%).

 gives the findings of the multivariable analyses, in which 8003 practices were classified as health IT super-users (26.6%). The odds of super-user status were lower for single-specialty, multispecialty, and allied health practices than for primary/family care clinics, and lower still for practices providing specialist services or acute care. The likelihood of super-use increased as the number of affiliated physicians increased, and super-users were more than twice as likely to be located in metropolitan areas than rural. Overall, the odds of being a super-user were highest for practices in the Midwest.

In contrast, 11,706 practices (38.9%) were classified as health IT under-users. Under-user practices were more likely to be situated in the West, have fewer affiliated physicians, and be located outside of metropolitan centers. Compared with primary/family care practices, single-specialty, multispecialty, and allied health practices were more likely to be under-users, as were those that provided specialist or acute care services. 

 give the geographical location of super- and under-users and the proportion of these practices by county.

We examined variation in the extent of use of EHR-based health IT functionalities in a national sample of US ambulatory care practices. Among 30,123 practices that were affiliated with a health system and had an operational EHR, only 27% were super-users, meaning they were maximally using EHR functionalities designed to improve patient care and facilitate high-quality performance across the broader health system. Of concern was that nearly 40% of ambulatory practices were categorized as under-users, indicating minimal use of the EHR and associated health IT functionalities. Under-use was more likely in smaller practices, those located outside of metropolitan centers, non—primary care practices, and those situated in the West.

There are likely multiple factors involved in EHR and health IT under-use by ambulatory care practices. Cost has been cited as the primary barrier to adopting an EHR system

; similarly, upgrading a basic EHR to one more comprehensive may not be financially possible for practices with limited resources. Cost may also be a factor for these practices because of the health IT support resources required to train users and maintain the systems. Smaller and rural practices were more likely to be under-users, a pattern also seen in the adoption of EHR and health IT in hospitals.

 These types of practices may face financial, human resource, or structural barriers that impede their ability to use their EHR to full capacity. We found that super-use was least prevalent in practices providing specialist-level care and services; it is possible that these practices are less well served by existing health IT functionalities and require specific tools developed for more specialized clinical scenarios.

Policy makers should consider how to address the barriers of the small, nonmetropolitan, and specialist practices to using their health IT functionalities more fully, as the relative under-use of these tools has far-reaching implications. First, suboptimal use of critical health IT functionalities may have direct relevance for the quality of care provided by an individual practice as part of routine patient care. Second, under-use of these technologies (such as health information exchange) may have consequences for the quality of care provided across the ambulatory care sector. Current policies, such as shared risk programs, encourage community-based strategies to avoid costly hospitalizations; similarly, value-based purchasing holds providers accountable for care delivered by multiple practitioners. It is also notable that the diversity of ambulatory care providers has expanded in recent years

; our sample included more than 50 types of ambulatory specialty services. This trend and the aforementioned policies suggest that communication and care coordination among ambulatory care providers is more crucial than ever.

Third, the interoperability of the broader digital health system is essential for the network benefits of health IT and EHR systems, yet differences in EHR capacity between the US hospital sector and ambulatory care are substantial. For example, transitions in care are a crucial task for both primary and tertiary health providers. The Office of the National Coordinator for Health Information Technology reported that approximately 49% of hospitals could generate a care summary document in 2014 (a low estimate that shows room for improvement)

; however, only 39% of ambulatory practices in our sample were able to create and transmit an equivalent report. Especially telling is the 2013 estimate that 77% of hospitals had the capacity to send laboratory results to ambulatory providers

 compared with the 49% of ambulatory practices that were able to communicate with hospitals for clinical information. Hospitals with advanced EHR systems are fundamentally limited if there are functional restrictions on their ability to interact with caregivers and organizations in the community setting.

This study has some limitations. First, we used 2014 reported data, and practices may have since expanded their health IT functionality. However, changes since this time are likely to be incremental only; given that we have focused on the “outliers” of EHR use, it is unlikely that there would be substantial alterations in the proportions of either super- or under-users. Second, this is the first publication using HIMSS ambulatory care data, and their validity has not been examined by the research community. However, many published studies have used the HIMSS hospital dataset,

 which utilizes the same sampling and survey methodology as the ambulatory practice survey that provided the data in our study. One such study describes this source as the “industry standard for information on EMR [electronic medical record] adoption.”

 Accordingly, a strength of our study is its presentation of the first-ever analysis of the corresponding data from HIMSS about ambulatory care health IT use. The HIMSS survey represents one of the most comprehensive assessments of use of health IT that currently exists; our study extended the current taxonomy of EHR systems well beyond that of “basic” and “comprehensive.” Finally, the survey includes only ambulatory practices that are affiliated with a health system. Given a presumed desire for system interoperability, we might expect greater use of some health IT functionalities (such as health information exchange) by the ambulatory practices in our sample compared with independent practices. The use of this subset, in conjunction with our focus on those practices with a preexisting operational EHR, suggests that our results may overestimate the true proportion of super-users in the broader ambulatory care setting and that the national rate is even lower; the reverse is also likely true for the estimates of under-users.

Although it is critical for ambulatory care practices to have the building block of the EHR, a substantial proportion of these practices use this technology only minimally, indicating there is capacity for significant improvement. It is important that policy makers and healthcare providers understand the limits of health IT functionality in ambulatory care practices, as strategies aimed at improving the coordination of care or those relying on the EHR as a vehicle for intervention may be hindered by the technological capacity of ambulatory care partners. We also suggest that policy makers identify the barriers limiting the use of these tools in ambulatory care (in particular those related to small, rural, and specialty practices) and consider how best to facilitate the full use of a range of EHR-based health IT functionalities. Investment in EHR-based health IT capacity of individual ambulatory practices will likely have benefits to providers across the ambulatory setting and to the performance of the broader health system.

The authors wish to thank Julie Lai and Julie Newell of RAND Health for their assistance in preparing the data and performing the GIS mapping.

 RAND Health (JRS, PS, CLD), Santa Monica, CA; Northland District Health Board (JRS), Whangarei, New Zealand; General Internal Medicine, Greater Los Angeles VA Healthcare System (PS), Los Angeles, CA.

 Support for this research was made possible by a Harkness Fellowship in Healthcare Policy and Practice (Dr Rumball-Smith), awarded by The Commonwealth Fund. This work was also supported, in whole or in part, through a cooperative agreement (1U19HS024067-01) between the RAND Corporation and the Agency for Healthcare Research and Quality. The content and opinions expressed in this publication are solely the responsibility of the authors and do not reflect the official position of the Agency, HHS, or the Commonwealth Fund or its directors, officers, or staff. 

 Concept and design (JRS); acquisition of data (CLD); analysis and interpretation of data (JRS, PS, CLD); drafting of the manuscript (JRS, PS, CLD); critical revision of the manuscript for important intellectual content (JRS, PS, CLD); statistical analysis (JRS); obtaining funding (JRS, PS, CLD); and supervision (PS, CLD).

 Juliet Rumball-Smith, MBChB, PhD, Tohora House, Northland District Health Board, Whangarei 0110, New Zealand. Email: jrs@health.rumballsmith.nz. 

1. American Recovery and Reinvestment Act of 2009, S 1, 111th Cong (2009).

2. Jamoom EW, Yang N, Hing E. Adoption of certified electronic health record systems and electronic information sharing in physician offices: United States, 2013 and 2014. NCHS data brief, no. 236. National Center for Health Statistics website. cdc.gov/nchs/products/databriefs/db236.htm. Published January 2016. Accessed May 1, 2017.

3. Office-based physician electronic health record adoption. Office of the National Coordinator for Health Information Technology website. dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php. Published 2015. Accessed May 1, 2017.

4. Ancker JS, Kern LM, Edwards A, et al; HITEC Investigators. Associations between healthcare quality and use of electronic health record functions in ambulatory care. 

. 2015;22(4):864-871. doi: 10.1093/jamia/ocv030.

5. Prgomet M, Li L, Niazkhani Z, Georgiou A, Westbrook JI. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. 

. 2017;24(2):413-422. doi: 10.1093/jamia/ocw145.

6. Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. 

Nearly 40% of US ambulatory care practices are “under-users” of health information technology functionalities, which impacts the ability of the health system as a whole to provide coordinated, efficient care.

January 2018

Bridging the Digital Divide: Mobile Access to Personal Health Records Among Patients With Diabetes

Measuring Overuse With Electronic Health Records Data

Electronic Health Record "Super-Users" and "Under-Users" in Ambulatory Care Practices

Electronic Sharing of Diagnostic Information and Patient Outcomes

Hospital Participation in Meaningful Use and Racial Disparities in Readmissions

A Cost-Effectiveness Analysis of Cardiology eConsults for Medicaid Patients

The Health Information Technology Special Issue: Has IT Become a Mandatory Part of Health and Healthcare?

Electronic Health Record Problem Lists: Accurate Enough for Risk Adjustment?

Racial/Ethnic Variation in Devices Used to Access Patient Portals

Hospitalized Patients' and Family Members' Preferences for Real-Time, Transparent Access to Their Hospital Records